From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) by kanga.kvack.org (Postfix) with ESMTP id AE8496B0266 for ; Thu, 26 Jul 2018 04:03:03 -0400 (EDT) Received: by mail-ed1-f71.google.com with SMTP id r9-v6so481992edh.14 for ; Thu, 26 Jul 2018 01:03:03 -0700 (PDT) Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id r12-v6si885227eda.307.2018.07.26.01.03.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Jul 2018 01:03:02 -0700 (PDT) Date: Thu, 26 Jul 2018 10:03:01 +0200 From: Michal Hocko Subject: Re: [Bug 200651] New: cgroups iptables-restor: vmalloc: allocation failure Message-ID: <20180726080301.GW28386@dhcp22.suse.cz> References: <20180725125239.b591e4df270145f9064fe2c5@linux-foundation.org> <20180726072622.GS28386@dhcp22.suse.cz> <67d5e4ef-c040-6852-ad93-6f2528df0982@suse.cz> <20180726074219.GU28386@dhcp22.suse.cz> <36043c6b-4960-8001-4039-99525dcc3e05@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <36043c6b-4960-8001-4039-99525dcc3e05@suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: Andrew Morton , gnikolov@icdsoft.com, bugzilla-daemon@bugzilla.kernel.org, linux-mm@kvack.org, netfilter-devel@vger.kernel.org On Thu 26-07-18 09:50:45, Vlastimil Babka wrote: > On 07/26/2018 09:42 AM, Michal Hocko wrote: > > On Thu 26-07-18 09:34:58, Vlastimil Babka wrote: > >> On 07/26/2018 09:26 AM, Michal Hocko wrote: > >>> On Thu 26-07-18 09:18:57, Vlastimil Babka wrote: > >>>> On 07/25/2018 09:52 PM, Andrew Morton wrote: > >>>>> (switched to email. Please respond via emailed reply-to-all, not via the > >>>>> bugzilla web interface). > >>>>> > >>>>> On Wed, 25 Jul 2018 11:42:57 +0000 bugzilla-daemon@bugzilla.kernel.org wrote: > >>>>> > >>>>>> https://bugzilla.kernel.org/show_bug.cgi?id=200651 > >>>>>> > >>>>>> Bug ID: 200651 > >>>>>> Summary: cgroups iptables-restor: vmalloc: allocation failure > >>>>> > >>>>> Thanks. Please do note the above request. > >>>>> > >>>>>> Product: Memory Management > >>>>>> Version: 2.5 > >>>>>> Kernel Version: 4.14 > >>>>>> Hardware: All > >>>>>> OS: Linux > >>>>>> Tree: Mainline > >>>>>> Status: NEW > >>>>>> Severity: normal > >>>>>> Priority: P1 > >>>>>> Component: Other > >>>>>> Assignee: akpm@linux-foundation.org > >>>>>> Reporter: gnikolov@icdsoft.com > >>>>>> Regression: No > >>>>>> > >>>>>> Created attachment 277505 > >>>>>> --> https://bugzilla.kernel.org/attachment.cgi?id=277505&action=edit > >>>>>> iptables save > >>>>>> > >>>>>> After creating large number of cgroups and under memory pressure, iptables > >>>>>> command fails with following error: > >>>>>> > >>>>>> "iptables-restor: vmalloc: allocation failure, allocated 3047424 of 3465216 > >>>>>> bytes, mode:0x14010c0(GFP_KERNEL|__GFP_NORETRY), nodemask=(null)" > >>>> > >>>> This is likely the kvmalloc() in xt_alloc_table_info(). Between 4.13 and > >>>> 4.17 it shouldn't use __GFP_NORETRY, but looks like commit 0537250fdc6c > >>>> ("netfilter: x_tables: make allocation less aggressive") was backported > >>>> to 4.14. Removing __GFP_NORETRY might help here, but bring back other > >>>> issues. Less than 4MB is not that much though, maybe find some "sane" > >>>> limit and use __GFP_NORETRY only above that? > >>> > >>> I have seen the same report via http://lkml.kernel.org/r/df6f501c-8546-1f55-40b1-7e3a8f54d872@icdsoft.com > >>> and the reported confirmed that kvmalloc is not a real culprit > >>> http://lkml.kernel.org/r/d99a9598-808a-6968-4131-c3949b752004@icdsoft.com > >> > >> Hmm but that was revert of eacd86ca3b03 ("net/netfilter/x_tables.c: use > >> kvmalloc() in xt_alloc_table_info()") which was the 4.13 commit that > >> removed __GFP_NORETRY (there's no __GFP_NORETRY under net/netfilter in > >> v4.14). I assume it was reverted on top of vanilla v4.14 as there would > >> be conflict on the stable with 0537250fdc6c backport. So what should be > >> tested to be sure is either vanilla v4.14 without stable backports, or > >> latest v4.14.y with revert of 0537250fdc6c. > > > > But 0537250fdc6c simply restored the previous NORETRY behavior from > > before eacd86ca3b03. So whatever causes these issues doesn't seem to be > > directly related to the kvmalloc change. Or do I miss what you are > > saying? > > I'm saying that although it's not a regression, as you say (the > vmalloc() there was only for a few kernel versions called without > __GFP_NORETRY), it's still possible that removing __GFP_NORETRY will fix > the issue and thus we will rule out other possibilities. http://lkml.kernel.org/r/d99a9598-808a-6968-4131-c3949b752004@icdsoft.com claims that reverting eacd86ca3b03 didn't really help. -- Michal Hocko SUSE Labs