Re: [Bug 200651] New: cgroups iptables-restor: vmalloc: allocation failure

[Georgi Nikolov <gnikolov@icdsoft.com>]

On 07/30/2018 04:57 PM, Michal Hocko wrote:

> On Mon 30-07-18 16:37:07, Georgi Nikolov wrote:
>> On 07/26/2018 12:02 PM, Georgi Nikolov wrote:
> [...]
>>> Here is the patch applied to this version which masks errors:
>>>
>>> --- net/netfilter/x_tables.c    2018-06-18 14:18:21.138347416 +0300
>>> +++ net/netfilter/x_tables.c    2018-07-26 11:58:01.721932962 +0300
>>> @@ -1059,9 +1059,19 @@
>>>       * than shoot all processes down before realizing there is nothing
>>>       * more to reclaim.
>>>       */
>>> -    info = kvmalloc(sz, GFP_KERNEL | __GFP_NORETRY);
>>> +/*    info = kvmalloc(sz, GFP_KERNEL | __GFP_NORETRY);
>>>      if (!info)
>>>          return NULL;
>>> +*/
>>> +
>>> +    if (sz <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER))
>>> +        info = kmalloc(sz, GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY);
>>> +    if (!info) {
>>> +        info = __vmalloc(sz, GFP_KERNEL | __GFP_NOWARN | __GFP_NORETRY,
>>> +         PAGE_KERNEL);
>>> +        if (!info)
>>> +        return NULL;
>>> +    }
>>>  
>>>      memset(info, 0, sizeof(*info));
>>>      info->size = size;
>>>
>>>
>>> I will try to reproduce it with only
>>>
>>> info = kvmalloc(sz, GFP_KERNEL);
>>>
>>> Regards,
>>>
>>> --
>>> Georgi Nikolov
>>>
>> Hello,
>>
>> Without GFP_NORETRY problem disappears.
> Hmm, there are two allocation paths which have __GFP_NORETRY here.
> I expect you have removed both of them, right?
>
> kvmalloc implicitly performs __GFP_NORETRY on kmalloc path but it
> doesn't have it for the vmalloc fallback. This would match
> kvmalloc(GFP_KERNEL). I thought you were testing this code path
> previously but there is some confusion flying around because you have
> claimed that the regressions started with eacd86ca3b036. If the
> regression is really with __GFP_NORETRY being used for the vmalloc
> fallback which would be kvmalloc(GFP_KERNEL | __GFP_NORETRY) then
> I am still confused because that would match the original code.

No i was wrong. The regression starts actually with 0537250fdc6c8.
- old code, which opencodes kvmalloc, is masking error but error is there
- kvmalloc without GFP_NORETRY works fine, but probably can consume a
lot of memory - commit: eacd86ca3b036
- kvmalloc with GFP_NORETRY shows error - commit: 0537250fdc6c8

>> What is correct way to fix it.
>> - inside xt_alloc_table_info remove GFP_NORETRY from kvmalloc or add
>> this flag only for sizes bigger than some threshold
> This would reintroduce issue fixed by 0537250fdc6c8. Note that
> kvmalloc(GFP_KERNEL | __GFP_NORETRY) is more or less equivalent to the
> original code (well, except for __GFP_NOWARN).

So probably we should pass GFP_NORETRY only for large requests (above
some threshold).

>
>> - inside kvmalloc_node remove GFP_NORETRY from
>> __vmalloc_node_flags_caller (i don't know if it honors this flag, or
>> the problem is elsewhere)
> No, not really. This is basically equivalent to kvmalloc(GFP_KERNEL).
>
> I strongly suspect that this is not a regression in this code but rather
> a side effect of larger memory fragmentation caused by something else.
> In any case do you see this failure also without artificial test case
> with a standard workload?

Yes i can see failures with standard workload, in fact it was hard to
reproduce it.
Here is the error from production servers where allocation is smaller:
iptables: vmalloc: allocation failure, allocated 131072 of 225280 bytes,
mode:0x14010c0(GFP_KERNEL|__GFP_NORETRY), nodemask=(null)

I didn't understand if vmalloc honors GFP_NORETRY.

Regards,

--
Georgi Nikolov