Cannot allocate memory despite buff/cache non-zero

[Paul Menzel <pmenzel@molgen.mpg.de>]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

Dear Linux folks,


On a 1 TB RAM compute server with Linux 5.10.24 and memory 
overcommitting disabled, we ran into a situation where processes like 
SSH couldn’t allocate memory anymore.

     $ more /proc/cmdline
     BOOT_IMAGE=/boot/bzImage-5.10.24.mx64.375 root=LABEL=root ro 
crashkernel=256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd 
audit=0 random.trust_cpu=on

     2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]: 
pam_systemd(sshd:session): Failed to create session: Unit 
session-25654.scope not found.
     2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error: 
do_exec_no_pty: fork: Cannot allocate memory
     2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: 
pam_unix(sshd:session): session closed for user root
     2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork: 
Cannot allocate memory
     2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: 
ssh_msg_send: write: Broken pipe
     2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: 
send_rexec_state: ssh_msg_send failed

     $ free -h
                   total        used        free      shared  buff/cache 
available
     Mem:           1.0T        606G        2.6G        2.2M        395G 
    391G
     Swap:            0B          0B          0B

Looking at this, I would have expected, that the pages(?) in buff/cache 
would be moved/deleted to make memory available.

Looking at `/proc/meminfo` (attached):

     MemTotal:       1052411824 kB
     MemFree:         2709976 kB
     MemAvailable:   410847908 kB
     […]
     CommitLimit:    1052411824 kB
     Committed_AS:   1052455260 kB
     […]

Committed_AS is greater than the commit limit (total memory).

Is such behavior expected?


Kind regards,

Paul

[-- Attachment #2: 20210603-godsavethequeen-meminfo.txt --]
[-- Type: text/plain, Size: 1408 bytes --]

MemTotal:       1052411824 kB
MemFree:         2709976 kB
MemAvailable:   410847908 kB
Buffers:            3212 kB
Cached:         411083788 kB
SwapCached:            0 kB
Active:         303175824 kB
Inactive:       740080100 kB
Active(anon):       1448 kB
Inactive(anon): 632169724 kB
Active(file):   303174376 kB
Inactive(file): 107910376 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:      632161948 kB
Mapped:           508880 kB
Shmem:              2248 kB
KReclaimable:    2357416 kB
Slab:            3119240 kB
SReclaimable:    2357416 kB
SUnreclaim:       761824 kB
KernelStack:       50512 kB
PageTables:      1715372 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1052411824 kB
Committed_AS:   1052455260 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      480508 kB
VmallocChunk:          0 kB
Percpu:           460800 kB
AnonHugePages:  509384704 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     5995344 kB
DirectMap2M:    483590144 kB
DirectMap1G:    579862528 kB