From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 500D9C433E2 for ; Sun, 13 Sep 2020 00:39:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A340321531 for ; Sun, 13 Sep 2020 00:38:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="C6aY/h6a" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A340321531 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A653B6B0002; Sat, 12 Sep 2020 20:38:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A17AD6B0037; Sat, 12 Sep 2020 20:38:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92B696B0055; Sat, 12 Sep 2020 20:38:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 793846B0002 for ; Sat, 12 Sep 2020 20:38:57 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3B3AF2031D for ; Sun, 13 Sep 2020 00:38:57 +0000 (UTC) X-FDA: 77256178314.29.dog01_0b0e0c1270fb Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id 18E8918116E24 for ; Sun, 13 Sep 2020 00:38:57 +0000 (UTC) X-HE-Tag: dog01_0b0e0c1270fb X-Filterd-Recvd-Size: 8713 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Sun, 13 Sep 2020 00:38:56 +0000 (UTC) Received: from X1 (unknown [209.33.215.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2CADC2087C; Sun, 13 Sep 2020 00:38:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1599957535; bh=Qrk6RDTgFet/9FovmpYoVRdKZmBgNObTMBNLu7D+HSE=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=C6aY/h6a0sjbC+rKHE7zhe9UPGhCN9fBqoREF1af+4VKdxLEfdy6fswxk84+1PCGu R4G5HkKzyOhmwjFBIYuhx4jxUbgh0a88GH0XgVrSNAB9zU6aivamtXso09eAxUfsIg 7IBmjoLdcSiVQh62kXeDg2Yt3RW+5Bv4CEaE/SvM= Date: Sat, 12 Sep 2020 17:38:54 -0700 From: Andrew Morton To: Christoph Hellwig , Al Viro Cc: bugzilla-daemon@bugzilla.kernel.org, gaetan@nexylan.com, linux-mm@kvack.org, Alexey Dobriyan Subject: Re: [Bug 209245] New: 5.8.x Cannot allocate memory requesting sysctl Message-Id: <20200912173854.8ec0189616e541691644c4d0@linux-foundation.org> In-Reply-To: References: X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.32; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 18E8918116E24 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: (switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface). On Sat, 12 Sep 2020 09:12:18 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=209245 > > Bug ID: 209245 > Summary: 5.8.x Cannot allocate memory requesting sysctl > Product: Memory Management > Version: 2.5 > Kernel Version: 5.8.x > Hardware: All > OS: Linux > Tree: Mainline > Status: NEW > Severity: high > Priority: P1 > Component: Page Allocator > Assignee: akpm@linux-foundation.org > Reporter: gaetan@nexylan.com > Regression: No Actually, "Regression: yes". > Created attachment 292483 > --> https://bugzilla.kernel.org/attachment.cgi?id=292483&action=edit > dmesg > > Since we upgraded a few nodes to 5.8.x (Gentoo Linux), we are having random > "Cannot allocate memory" when requesting infos from sysctl despite system has > free memory available. > > Steps to reproduce : > > 1/ Have system with heavy workload under 5.8.2 (reproduced with 5.8.8) > 2/ At some point, accessing sysctl is not possible anymore : > > > ~# cat /proc/sys/net/netfilter/nf_conntrack_count > cat: /proc/sys/net/netfilter/nf_conntrack_count: Cannot allocate memory > > > 3/ Dmesg drops memory allocation errors (see attached) > > > Memory usage : > > total used free shared buff/cache available > Mem: 32009 22029 1547 0 8431 9506 > Swap: 7811 621 7190 > > Node is running a single mongodb process. > The out-of-memory dump is below. proc_sys_call_handler() is attempting an order-5 kmalloc for a read of a simple int. Presumably because cat(1) is starting out with a 128kb read(). AFAICT this is due to changes made in 32927393dc1cc ("sysctl: pass kernel pointers to ->proc_handler")? [Sat Sep 12 11:07:55 2020] cat: page allocation failure: order:5, mode:0x40dc0(GFP_KERNEL|__GFP_COMP|__GFP_ZERO), nodemask=(null),cpuset=/,mems_allowed=0 [Sat Sep 12 11:07:55 2020] CPU: 10 PID: 32410 Comm: cat Not tainted 5.8.8-gentoo #1 [Sat Sep 12 11:07:55 2020] Hardware name: Supermicro MBI-6119M-T2N/B2SC1-CPU, BIOS 1.1 06/25/2019 [Sat Sep 12 11:07:55 2020] Call Trace: [Sat Sep 12 11:07:55 2020] dump_stack+0x6d/0x8b [Sat Sep 12 11:07:55 2020] warn_alloc+0xf6/0x160 [Sat Sep 12 11:07:55 2020] __alloc_pages_slowpath.constprop.123+0xb28/0xb70 [Sat Sep 12 11:07:55 2020] ? __alloc_pages_nodemask+0x144/0x2e0 [Sat Sep 12 11:07:55 2020] __alloc_pages_nodemask+0x2af/0x2e0 [Sat Sep 12 11:07:55 2020] kmalloc_order+0x13/0x70 [Sat Sep 12 11:07:55 2020] kmalloc_order_trace+0x19/0xa0 [Sat Sep 12 11:07:55 2020] proc_sys_call_handler+0xf9/0x170 [Sat Sep 12 11:07:55 2020] vfs_read+0x96/0x170 [Sat Sep 12 11:07:55 2020] ksys_read+0x9c/0xd0 [Sat Sep 12 11:07:55 2020] do_syscall_64+0x44/0x70 [Sat Sep 12 11:07:55 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Sat Sep 12 11:07:55 2020] RIP: 0033:0x7f2c7a19ac7b [Sat Sep 12 11:07:55 2020] Code: 00 3d 00 00 41 00 74 07 31 c0 e9 e9 fe ff ff 50 48 8d 3d 60 63 09 00 e8 3b c4 01 00 64 8b 04 25 18 00 00 00 85 c0 75 0b 0f 05 <48> 3d 00 f0 ff ff 77 4c c3 48 83 ec 28 48 89 54 24 18 48 89 74 24 [Sat Sep 12 11:07:55 2020] RSP: 002b:00007ffec0af8ec8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [Sat Sep 12 11:07:55 2020] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f2c7a19ac7b [Sat Sep 12 11:07:55 2020] RDX: 0000000000020000 RSI: 00007f2c7a097000 RDI: 0000000000000003 [Sat Sep 12 11:07:55 2020] RBP: 00007f2c7a097000 R08: 00007f2c7a096010 R09: 0000000000000000 [Sat Sep 12 11:07:55 2020] R10: 0000559b45d2aa00 R11: 0000000000000246 R12: 00007f2c7a097000 [Sat Sep 12 11:07:55 2020] R13: 0000000000000003 R14: 0000000000000fff R15: 0000000000020000 [Sat Sep 12 11:07:55 2020] Mem-Info: [Sat Sep 12 11:07:55 2020] active_anon:5210006 inactive_anon:373749 isolated_anon:0 active_file:972354 inactive_file:990026 isolated_file:0 unevictable:1958 dirty:70 writeback:0 slab_reclaimable:198141 slab_unreclaimable:19557 mapped:8172 shmem:29 pagetables:11756 bounce:0 free:391832 free_pcp:0 free_cma:0 [Sat Sep 12 11:07:55 2020] Node 0 active_anon:20840024kB inactive_anon:1494996kB active_file:3889416kB inactive_file:3960104kB unevictable:7832kB isolated(anon):0kB isolated(file):0kB mapped:32688kB dirty:280kB writeback:0kB shmem:116kB writeback_tmp:0kB all_unreclaimable? no [Sat Sep 12 11:07:55 2020] Node 0 DMA free:11800kB min:8kB low:20kB high:32kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15980kB managed:15896kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 2108 31993 31993 [Sat Sep 12 11:07:55 2020] Node 0 DMA32 free:153020kB min:1504kB low:3656kB high:5808kB reserved_highatomic:0KB active_anon:1169844kB inactive_anon:206804kB active_file:180128kB inactive_file:209060kB unevictable:0kB writepending:0kB present:2257504kB managed:2159144kB mlocked:0kB kernel_stack:16kB pagetables:1848kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 29884 29884 [Sat Sep 12 11:07:55 2020] Node 0 Normal free:1402508kB min:21380kB low:51980kB high:82580kB reserved_highatomic:0KB active_anon:19670180kB inactive_anon:1288192kB active_file:3709496kB inactive_file:3751000kB unevictable:7832kB writepending:280kB present:31178752kB managed:30602236kB mlocked:7832kB kernel_stack:4496kB pagetables:45176kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [Sat Sep 12 11:07:55 2020] lowmem_reserve[]: 0 0 0 0 [Sat Sep 12 11:07:55 2020] Node 0 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11800kB [Sat Sep 12 11:07:55 2020] Node 0 DMA32: 2120*4kB (UME) 9907*8kB (UME) 3974*16kB (UM) 56*32kB (UM) 3*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 153304kB [Sat Sep 12 11:07:55 2020] Node 0 Normal: 8036*4kB (UME) 62654*8kB (UME) 54156*16kB (UM) 155*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1404832kB [Sat Sep 12 11:07:55 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [Sat Sep 12 11:07:55 2020] 1994278 total pagecache pages [Sat Sep 12 11:07:55 2020] 33986 pages in swap cache [Sat Sep 12 11:07:55 2020] Swap cache stats: add 205324, delete 171339, find 1160015/1183131 [Sat Sep 12 11:07:55 2020] Free swap = 7363576kB [Sat Sep 12 11:07:55 2020] Total swap = 7999480kB [Sat Sep 12 11:07:55 2020] 8363059 pages RAM [Sat Sep 12 11:07:55 2020] 0 pages HighMem/MovableOnly [Sat Sep 12 11:07:55 2020] 168740 pages reserved [Sat Sep 12 11:07:55 2020] 0 pages hwpoisoned