Re: [Bug 200651] New: cgroups iptables-restor: vmalloc: allocation failure

[Michal Hocko <mhocko@kernel.org>]

On Thu 26-07-18 09:18:57, Vlastimil Babka wrote:
> On 07/25/2018 09:52 PM, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> > 
> > On Wed, 25 Jul 2018 11:42:57 +0000 bugzilla-daemon@bugzilla.kernel.org wrote:
> > 
> >> https://bugzilla.kernel.org/show_bug.cgi?id=200651
> >>
> >>             Bug ID: 200651
> >>            Summary: cgroups iptables-restor: vmalloc: allocation failure
> > 
> > Thanks.  Please do note the above request.
> > 
> >>            Product: Memory Management
> >>            Version: 2.5
> >>     Kernel Version: 4.14
> >>           Hardware: All
> >>                 OS: Linux
> >>               Tree: Mainline
> >>             Status: NEW
> >>           Severity: normal
> >>           Priority: P1
> >>          Component: Other
> >>           Assignee: akpm@linux-foundation.org
> >>           Reporter: gnikolov@icdsoft.com
> >>         Regression: No
> >>
> >> Created attachment 277505
> >>   --> https://bugzilla.kernel.org/attachment.cgi?id=277505&action=edit
> >> iptables save
> >>
> >> After creating large number of cgroups and under memory pressure, iptables
> >> command fails with following error:
> >>
> >> "iptables-restor: vmalloc: allocation failure, allocated 3047424 of 3465216
> >> bytes, mode:0x14010c0(GFP_KERNEL|__GFP_NORETRY), nodemask=(null)"
> 
> This is likely the kvmalloc() in xt_alloc_table_info(). Between 4.13 and
> 4.17 it shouldn't use __GFP_NORETRY, but looks like commit 0537250fdc6c
> ("netfilter: x_tables: make allocation less aggressive") was backported
> to 4.14. Removing __GFP_NORETRY might help here, but bring back other
> issues. Less than 4MB is not that much though, maybe find some "sane"
> limit and use __GFP_NORETRY only above that?

I have seen the same report via http://lkml.kernel.org/r/df6f501c-8546-1f55-40b1-7e3a8f54d872@icdsoft.com
and the reported confirmed that kvmalloc is not a real culprit
http://lkml.kernel.org/r/d99a9598-808a-6968-4131-c3949b752004@icdsoft.com
 
> > I'm not sure what the problem is here, apart from iptables being
> > over-optimistic about vmalloc()'s abilities.
> > 
> > Are cgroups having any impact on this, or is it simply vmalloc arena
> > fragmentation, and the iptables code should use some data structure
> > more sophisticated than a massive array?
> > 
> > Maybe all that ccgroup metadata is contributing to the arena
> > fragmentation, but that allocations will be small and the two systems
> > should be able to live alongside, by being realistic about vmalloc.
> > 
> >> System which is used to reproduce the bug is with 2 vcpus and 2GB of ram, but
> >> it happens on more powerfull systems.
> >>
> >> Steps to reproduce:
> >>
> >> mkdir /cgroup
> >> mount cgroup -t cgroup -omemory,pids,blkio,cpuacct /cgroup
> >> for a in `seq 1 1000`; do for b in `seq 1 4` ; do mkdir -p
> >> "/cgroup/user/$a/$b"; done; done
> >>
> >> Then in separate consoles
> >>
> >> cat /dev/vda > /dev/null
> >> ./test
> >> ./test
> >> i=0;while sleep 0 ; do iptables-restore < iptables.save ; i=$(($i+1)); echo $i;
> >> done
> >>
> >> Here is the source of "test" program and attached iptables.save. It happens
> >> also with smaller iptables.save file.
> >>
> >> #include <stdio.h>
> >> #include <stdlib.h>
> >>
> >> int main(void) {
> >>
> >>     srand(time(NULL));
> >>     int i = 0, j = 0, randnum=0;
> >>     int arr[6] = { 3072, 7168, 15360 , 31744, 64512, 130048}; 
> >>     while(1) {
> >>
> >>         for (i = 0; i < 6 ; i++) {
> >>
> >>             int *ptr = (int*) malloc(arr[i] * 93);  
> >>
> >>             for(j = 0 ; j < arr[i] * 93 / sizeof(int); j++) {
> >>                 *(ptr+j) = j+1;
> >>             }
> >>
> >>             free(ptr);
> >>         }
> >>     }       
> >> }
> >>
> > 

-- 
Michal Hocko
SUSE Labs