how periodically do you want them? I assumed this some-hours and days snapshots would be sufficient.

any particular command with or without grep perhaps?

I just had to drop caches, right before your response, the performance was simply too bad.

this is for your information, how it was right after dropping and 0+5+25 minutes later

https://pastebin.com/LcjKgQkg .. this is what it looks like just after sync; echo 2 > /proc/sys/vm/drop_caches
https://pastebin.com/ZCeFCKrb .. 5 minutes later, when performance is starting to get better again

https://pastebin.com/8hij8Lid .. 20 minutes after that, you can expect this to consume all the available ram within 1-2 hours

2018-07-16 18:23 GMT+02:00 Michal Hocko <mhocko@kernel.org>:

On Mon 16-07-18 17:53:42, Marinko Catovic wrote:
> I can provide further data now, monitoring vmstat:
>
> https://pastebin.com/j0dMGBe4 .. 1 day later, 600MB/13GB in use, 35GB free
> https://pastebin.com/N011kYyd .. 1 day later, 300MB/10GB in use, 40GB free,
> performance becomes even worse
>
> the issue raises up again, I would have to drop caches by now to restore
> normal usage for another day or two.
>
> Afaik there should be no reason at all to not have the buffers/cache fill
> up the entire memory, isn't that true?
> There is to my knowledge almost no O_DIRECT involved, also as mentioned
> before: when dropping caches
> the buffers/cache usage would eat up all RAM within the hour as usual for
> 1-2 days until it starts to go crazy again.
>
> As mentioned, the usage oscillates up and down instead of up until all RAM
> is consumed.
>
> Please tell me if there is anything else I can do to help investigate this.

Do you have periodic /proc/vmstat snapshots I have asked before?

--
Michal Hocko
SUSE Labs