Re: debug linux kernel memory management / pressure

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Hi Vlastimil,

sorry for the late reply i was on holiday.

Am 05.04.19 um 12:37 schrieb Vlastimil Babka:
> On 3/29/19 10:41 AM, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> nobody an idea? I had another system today:
> 
> Well, isn't it still the same thing as we discussed in last autumn?
> You did report success with the ill-fated patch "mm: thp:  relax __GFP_THISNODE
> for MADV_HUGEPAGE mappings", or not?

No it's not. Last year those were KVM Host machines. These time it's a
LAMP machine. But i think i'll upgrade to 4.19.36 LTS and see if that
fixes the problem.

Thanks!

> 
>> # cat /proc/meminfo
>> MemTotal:       131911684 kB
>> MemFree:        25734836 kB
>> MemAvailable:   78158816 kB
>> Buffers:            2916 kB
>> Cached:         20650184 kB
>> SwapCached:       544016 kB
>> Active:         58999352 kB
>> Inactive:       10084060 kB
>> Active(anon):   43412532 kB
>> Inactive(anon):  5583220 kB
>> Active(file):   15586820 kB
>> Inactive(file):  4500840 kB
>> Unevictable:       35032 kB
>> Mlocked:           35032 kB
>> SwapTotal:       3905532 kB
>> SwapFree:              0 kB
>> Dirty:              1048 kB
>> Writeback:         20144 kB
>> AnonPages:      47923392 kB
>> Mapped:           775376 kB
>> Shmem:            561420 kB
>> Slab:           35798052 kB
>> SReclaimable:   34309112 kB
> 
> That's rather significant. Got a /proc/slabinfo from such system state?
> 
>> SUnreclaim:      1488940 kB
>> KernelStack:       42160 kB
>> PageTables:       248008 kB
>> NFS_Unstable:          0 kB
>> Bounce:                0 kB
>> WritebackTmp:          0 kB
>> CommitLimit:    69861372 kB
>> Committed_AS:   100328892 kB
>> VmallocTotal:   34359738367 kB
>> VmallocUsed:           0 kB
>> VmallocChunk:          0 kB
>> HardwareCorrupted:     0 kB
>> AnonHugePages:  19177472 kB
>> ShmemHugePages:        0 kB
>> ShmemPmdMapped:        0 kB
>> HugePages_Total:       0
>> HugePages_Free:        0
>> HugePages_Rsvd:        0
>> HugePages_Surp:        0
>> Hugepagesize:       2048 kB
>> DirectMap4k:      951376 kB
>> DirectMap2M:    87015424 kB
>> DirectMap1G:    48234496 kB
>>
>> # cat /proc/buddyinfo
>> Node 0, zone      DMA      1      0      0      0      2      1      1
>>     0      1      1      3
>> Node 0, zone    DMA32    372    418    403    395    371    322    262
>>   179    114      0      0
>> Node 0, zone   Normal  89147  96397  76496  56407  41671  29289  18142
>> 10278   4075      0      0
>> Node 1, zone   Normal 113266      0      1      1      1      1      1
>>     1      1      0      0
> 
> Node 1 seems quite fragmented. Again from last year I recall somebody (was it
> you?) capturing a larger series of snapshots where we saw a Sreclaimable rise
> due to some overnight 'find /' activity inflating dentry/inode caches which then
> got slowly reclaimed, but memory remained fragmented until enough of slab was
> reclaimed, and compaction couldn't help. drop_caches did help. Looks like this
> might be the same case. Add in something that tries to get large-order
> allocations on node 1 (e.g. with __GFP_THISNODE) and overreclaim will happen.
> 
>> But with high PSI / memory pressure values above 10-30.
>>
>> Greets,
>> Stefan
>> Am 27.03.19 um 11:56 schrieb Stefan Priebe - Profihost AG:
>>> Hello list,
>>>
>>> i hope this is the right place to ask. If not i would be happy to point
>>> me to something else.
>>>
>>> I'm seeing the following behaviour on some of our hosts running a SLES
>>> 15 kernel (kernel v4.12 as it's base) but i don't think it's related to
>>> the kernel.
>>>
>>> At some "random" interval - mostly 3-6 weeks of uptime. Suddenly mem
>>> pressure rises and the linux cache (Cached: /proc/meminfo) drops from
>>> 12G to 3G. After that io pressure rises most probably due to low cache.
>>> But at the same time i've MemFree und MemAvailable at 19-22G.
>>>
>>> Why does this happen? How can i debug this situation? I would expect
>>> that the page / file cache never drops if there is so much free mem.
>>>
>>> Thanks a lot for your help.
>>>
>>> Greets,
>>> Stefan
>>>
>>> Not sure whether needed but these are the vm. kernel settings:
>>> vm.admin_reserve_kbytes = 8192
>>> vm.block_dump = 0
>>> vm.compact_unevictable_allowed = 1
>>> vm.dirty_background_bytes = 0
>>> vm.dirty_background_ratio = 10
>>> vm.dirty_bytes = 0
>>> vm.dirty_expire_centisecs = 3000
>>> vm.dirty_ratio = 20
>>> vm.dirty_writeback_centisecs = 500
>>> vm.dirtytime_expire_seconds = 43200
>>> vm.drop_caches = 0
>>> vm.extfrag_threshold = 500
>>> vm.hugepages_treat_as_movable = 0
>>> vm.hugetlb_shm_group = 0
>>> vm.laptop_mode = 0
>>> vm.legacy_va_layout = 0
>>> vm.lowmem_reserve_ratio = 256   256     32      1
>>> vm.max_map_count = 65530
>>> vm.memory_failure_early_kill = 0
>>> vm.memory_failure_recovery = 1
>>> vm.min_free_kbytes = 393216
>>> vm.min_slab_ratio = 5
>>> vm.min_unmapped_ratio = 1
>>> vm.mmap_min_addr = 65536
>>> vm.mmap_rnd_bits = 28
>>> vm.mmap_rnd_compat_bits = 8
>>> vm.nr_hugepages = 0
>>> vm.nr_hugepages_mempolicy = 0
>>> vm.nr_overcommit_hugepages = 0
>>> vm.nr_pdflush_threads = 0
>>> vm.numa_zonelist_order = default
>>> vm.oom_dump_tasks = 1
>>> vm.oom_kill_allocating_task = 0
>>> vm.overcommit_kbytes = 0
>>> vm.overcommit_memory = 0
>>> vm.overcommit_ratio = 50
>>> vm.page-cluster = 3
>>> vm.panic_on_oom = 0
>>> vm.percpu_pagelist_fraction = 0
>>> vm.stat_interval = 1
>>> vm.swappiness = 50
>>> vm.user_reserve_kbytes = 131072
>>> vm.vfs_cache_pressure = 100
>>> vm.watermark_scale_factor = 10
>>> vm.zone_reclaim_mode = 0
>>>
>>
>