Re: Cannot allocate memory despite buff/cache non-zero

[David Hildenbrand <david@redhat.com>]

On 04.06.21 13:36, Paul Menzel wrote:
> Dear Linux folks,
> 
> 
> On a 1 TB RAM compute server with Linux 5.10.24 and memory
> overcommitting disabled, we ran into a situation where processes like
> SSH couldn’t allocate memory anymore.
> 
>       $ more /proc/cmdline
>       BOOT_IMAGE=/boot/bzImage-5.10.24.mx64.375 root=LABEL=root ro
> crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd
> audit=0 random.trust_cpu=on
> 
>       2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]:
> pam_systemd(sshd:session): Failed to create session: Unit
> session-25654.scope not found.
>       2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error:
> do_exec_no_pty: fork: Cannot allocate memory
>       2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]:
> pam_unix(sshd:session): session closed for user root
>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork:
> Cannot allocate memory
>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:
> ssh_msg_send: write: Broken pipe
>       2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:
> send_rexec_state: ssh_msg_send failed
> 
>       $ free -h
>                     total        used        free      shared  buff/cache
> available
>       Mem:           1.0T        606G        2.6G        2.2M        395G
>      391G
>       Swap:            0B          0B          0B
> 
> Looking at this, I would have expected, that the pages(?) in buff/cache
> would be moved/deleted to make memory available.
> 
> Looking at `/proc/meminfo` (attached):
> 
>       MemTotal:       1052411824 kB
>       MemFree:         2709976 kB
>       MemAvailable:   410847908 kB
>       […]
>       CommitLimit:    1052411824 kB
>       Committed_AS:   1052455260 kB
>       […]
> 

With memory overcommit disabled, each accountable mapping 
(mm/mmap.c:accountable_mapping()) will count towards Committed_AS. So 
you might still have plenty of free memory in the system reserved for 
these mappings, yet Linux won't allow for more accountable mappings. 
That's why you see the "Cannot allocate memory" messages. mmap() failed.

> Committed_AS is greater than the commit limit (total memory).
> 
> Is such behavior expected?

We're talking about 43436 kB that exceed the CommitLimit.

The CommitLimit might change (grow/shrink) when
a) The number of hugetlb pages changes
b) Swap space is resized

If CommitLimit did not change, Committed_AS should actually not exceed 
it. IIUC, it can only happen temporarily while trying creation of a new 
mapping. We increase Committed_AS unconditionally and decrease it again 
if we reject it.

-- 
Thanks,

David / dhildenb