From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by kanga.kvack.org (Postfix) with ESMTP id 19F3C6B0003 for ; Mon, 30 Jul 2018 10:40:51 -0400 (EDT) Received: by mail-ed1-f70.google.com with SMTP id n4-v6so2504107edr.5 for ; Mon, 30 Jul 2018 07:40:51 -0700 (PDT) Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id 37-v6si3768214edt.319.2018.07.30.07.40.49 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 30 Jul 2018 07:40:49 -0700 (PDT) Date: Mon, 30 Jul 2018 16:40:48 +0200 From: Michal Hocko Subject: Re: Caching/buffers become useless after some time Message-ID: <20180730144048.GW24267@dhcp22.suse.cz> References: <20180712113411.GB328@dhcp22.suse.cz> <20180716162337.GY17280@dhcp22.suse.cz> <20180716164500.GZ17280@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: Marinko Catovic , linux-mm@kvack.org On Fri 27-07-18 13:15:33, Vlastimil Babka wrote: > On 07/21/2018 12:03 AM, Marinko Catovic wrote: > > I let this run for 3 days now, so it is quite a lot, there you go: > > https://nofile.io/f/egGyRjf0NPs/vmstat.tar.gz > > The stats show that compaction has very bad results. Between first and > last snapshot, compact_fail grew by 80k and compact_success by 1300. > High-order allocations will thus cycle between (failing) compaction and > reclaim that removes the buffer/caches from memory. I guess you are right. I've just looked at random large direct reclaim activity $ grep -w pgscan_direct vmstat*| awk '{diff=$2-old; if (old && diff > 100000) printf "%s %d\n", $1, diff; old=$2}' vmstat.1531957422:pgscan_direct 114334 vmstat.1532047588:pgscan_direct 111796 $ paste-with-diff.sh vmstat.1532047578 vmstat.1532047588 | grep "pgscan\|pgsteal\|compact\|pgalloc" | sort # counter value1 value2-value1 compact_daemon_free_scanned 2628160139 0 compact_daemon_migrate_scanned 797948703 0 compact_daemon_wake 23634 0 compact_fail 124806 108 compact_free_scanned 226181616304 295560271 compact_isolated 2881602028 480577 compact_migrate_scanned 147900786550 27834455 compact_stall 146749 108 compact_success 21943 0 pgalloc_dma 0 0 pgalloc_dma32 1577060946 10752 pgalloc_movable 0 0 pgalloc_normal 29389246430 343249 pgscan_direct 737335028 111796 pgscan_direct_throttle 0 0 pgscan_kswapd 1177909394 0 pgsteal_direct 704542843 111784 pgsteal_kswapd 898170720 0 There is zero kswapd activity so this must have been higher order allocation activity and all the direct compaction failed so we keep reclaiming. -- Michal Hocko SUSE Labs