Re: Caching/buffers become useless after some time

[Marinko Catovic <marinko.catovic@gmail.com>]

> I went through the whole thread again as it was spread over months, and
> finally connected some dots. In one mail you said:
>
> > There is one thing I forgot to mention: the hosts perform find and du (I mean the commands, finding files and disk usage)
> > on the HDDs every night, starting from 00:20 AM up until in the morning 07:45 AM, for maintenance and stats.
>
> The timespan above roughly matches the phase where reclaimable slab grow
> (samples 2000-6000 over 5 seconds is roughly 5.5 hours). The find will
> fetch a lots of metadata in dentries, inodes etc. which are part of
> reclaimable slabs. In other mail you posted a slabinfo
> https://pastebin.com/81QAFgke in the phase where it's already being
> slowly reclaimed, but still occupies 6.5GB, and mostly it's
> ext4_inode_cache, and dentry cache (also very much internally fragmented).
> In another mail I suggest that maybe fragmentation happened because the
> slab filled up much more at some point, and I think we now have that
> solidly confirmed from the vmstat plots.
> I think one workaround is for you to perform echo 2 > drop_caches (not
> 3) right after the find/du maintenance finishes. At that point you don't
> have too much page cache anyway, since the slabs have pushed it out.
> It's also overnight so there are not many users yet?
> Alternatively the find/du could run in a memcg limiting its slab use.
> Michal would know the details.
>
> Long term we should do something about these slab objects that are only
> used briefly (once?) so there's no point in caching them and letting the
> cache grow like this.
>

Well caching of any operations with find/du is not necessary imho
anyway, since walking over all these millions of files in that time
period is really not worth caching at all - if there is a way you
mentioned to limit the commands there, that would be great.
Also I want to mention that these operations were in use with 3.x
kernels as well, for years, with absolutely zero issues.

2 > drop_caches right after that is something I considered, I just had
some bad experience with this, since I tried it around 5:00 AM in the
first place to give it enough spare time to finish, since sync; echo 2
> drop_caches can take some time, hence my question about lowering the
limits in mm/vmscan.c, void drop_slab_node(int nid)

I could do this effectively right after find/du at 07:45, just hoping
that this is finished soon enough - in one worst case it took over 2
hours (from 05:00 AM to 07:00 AM), since the host was busy during that
time with find/du, never having freed enough caches to continue, hence
my question to let it stop earlier with the modification of
drop_slab_node ... it was just an idea, nevermind if you believe that
it was a bad one :)