Re: Cannot allocate memory despite buff/cache non-zero

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear David,


Thank you for your reply.

Am 09.06.21 um 13:17 schrieb David Hildenbrand:
> On 04.06.21 13:36, Paul Menzel wrote:

>> On a 1 TB RAM compute server with Linux 5.10.24 and memory
>> overcommitting disabled, we ran into a situation where processes like
>> SSH couldn’t allocate memory anymore.
>>
>>       $ more /proc/cmdline
>>       BOOT_IMAGE=/boot/bzImage-5.10.24.mx64.375 root=LABEL=root ro crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on
>>
>>       2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]: pam_systemd(sshd:session): Failed to create session: Unit session-25654.scope not found.
>>       2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error: do_exec_no_pty: fork: Cannot allocate memory
>>       2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: pam_unix(sshd:session): session closed for user root
>>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork: Cannot allocate memory
>>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: ssh_msg_send: write: Broken pipe
>>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: send_rexec_state: ssh_msg_send failed
>>
>>       $ free -h
>>                     total        used        free      shared  buff/cache available
>>       Mem:           1.0T        606G        2.6G        2.2M        395G      391G
>>       Swap:            0B          0B          0B
>>
>> Looking at this, I would have expected, that the pages(?) in buff/cache
>> would be moved/deleted to make memory available.
>>
>> Looking at `/proc/meminfo` (attached):
>>
>>       MemTotal:       1052411824 kB
>>       MemFree:         2709976 kB
>>       MemAvailable:   410847908 kB
>>       […]
>>       CommitLimit:    1052411824 kB
>>       Committed_AS:   1052455260 kB
>>       […]
> 
> With memory overcommit disabled, each accountable mapping 
> (mm/mmap.c:accountable_mapping()) will count towards Committed_AS. So 
> you might still have plenty of free memory in the system reserved for 
> these mappings, yet Linux won't allow for more accountable mappings. 
> That's why you see the "Cannot allocate memory" messages. mmap() failed.

     Buffers:            3212 kB
     Cached:         411083788 kB
     SwapCached:            0 kB
     Active:         303175824 kB
     Inactive:       740080100 kB
     Active(anon):       1448 kB
     Inactive(anon): 632169724 kB
     Active(file):   303174376 kB
     Inactive(file): 107910376 kB

The documentation [1] describes *MemAvailable*, *Buffers*, and *Cached*:

> MemAvailable
>               An estimate of how much memory is available for starting new
>               applications, without swapping. Calculated from MemFree,
>               SReclaimable, the size of the file LRU lists, and the low
>               watermarks in each zone.
>               The estimate takes into account that the system needs some
>               page cache to function well, and that not all reclaimable
>               slab will be reclaimable, due to items being in use. The
>               impact of those factors will vary from system to system.
> Buffers
>               Relatively temporary storage for raw disk blocks
>               shouldn't get tremendously large (20MB or so)
> Cached
>               in-memory cache for files read from the disk (the
>               pagecache).  Doesn't include SwapCached

So I would have assumed, the kernel removes files from the in-memory 
cache for files.

>> Committed_AS is greater than the commit limit (total memory).
>>
>> Is such behavior expected?
> 
> We're talking about 43436 kB that exceed the CommitLimit.
> 
> The CommitLimit might change (grow/shrink) when
> a) The number of hugetlb pages changes
> b) Swap space is resized
> 
> If CommitLimit did not change, Committed_AS should actually not exceed 
> it. IIUC, it can only happen temporarily while trying creation of a new 
> mapping. We increase Committed_AS unconditionally and decrease it again 
> if we reject it.

I can’t say for sure, as the system was rebooted, but I thought the 
value stayed the same.


Kind regards,

Paul


[1]: https://www.kernel.org/doc/html/latest/filesystems/proc.html#meminfo