From: Johannes Weiner <hannes@cmpxchg.org>
To: Mel Gorman <mgorman@suse.de>
Cc: Johannes Weiner <jweiner@redhat.com>,
Andrew Morton <akpm@linux-foundation.org>,
Andrea Arcangeli <aarcange@redhat.com>,
Minchan Kim <minchan.kim@gmail.com>,
Dave Jones <davej@redhat.com>, Jan Kara <jack@suse.cz>,
Andy Isaacson <adi@hexapodia.org>, Rik van Riel <riel@redhat.com>,
Nai Xia <nai.xia@gmail.com>, Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 11/11] mm: Isolate pages for immediate reclaim on their own LRU
Date: Mon, 19 Dec 2011 17:14:36 +0100 [thread overview]
Message-ID: <20111219161436.GF1415@cmpxchg.org> (raw)
In-Reply-To: <20111216160728.GI3487@suse.de>
On Fri, Dec 16, 2011 at 04:07:28PM +0000, Mel Gorman wrote:
> On Fri, Dec 16, 2011 at 04:17:31PM +0100, Johannes Weiner wrote:
> > On Wed, Dec 14, 2011 at 03:41:33PM +0000, Mel Gorman wrote:
> > > It was observed that scan rates from direct reclaim during tests
> > > writing to both fast and slow storage were extraordinarily high. The
> > > problem was that while pages were being marked for immediate reclaim
> > > when writeback completed, the same pages were being encountered over
> > > and over again during LRU scanning.
> > >
> > > This patch isolates file-backed pages that are to be reclaimed when
> > > clean on their own LRU list.
> >
> > Excuse me if I sound like a broken record, but have those observations
> > of high scan rates persisted with the per-zone dirty limits patchset?
> >
>
> Unfortunately I wasn't testing that series. The focus of this series
> was primarily on THP-related stalls incurred by compaction which
> did not have a dependency on that series. Even with dirty balancing,
> similar stalls would be observed once dirty pages were in the zone
> at all.
>
> > In my tests with pzd, the scan rates went down considerably together
> > with the immediate reclaim / vmscan writes.
> >
>
> I probably should know but what is pzd?
Oops. Per-Zone Dirty limits.
> > Our dirty limits are pretty low - if reclaim keeps shuffling through
> > dirty pages, where are the 80% reclaimable pages?! To me, this sounds
> > like the unfair distribution of dirty pages among zones again. Is
> > there are a different explanation that I missed?
> >
>
> The alternative explanation is that the 20% dirty pages are all
> long-lived, at the end of the highest zone which is always scanned first
> so we continually have to scan over these dirty pages for prolonged
> periods of time.
That certainly makes sense to me and is consistent with your test case
having a fast producer of clean cache while the dirty cache is against
a slow backing device, so it may survive multiple full inactive cycles
before writeback finishes.
> > PS: It also seems a bit out of place in this series...?
>
> Without the last path, the System CPU time was stupidly high. In part,
> this is because we are no longer calling ->writepage from direct
> reclaim. If we were, the CPU usage would be far lower but it would
> be a lot slower too. It seemed remiss to leave system CPU usage that
> high without some explanation or patch dealing with it.
>
> The following replaces this patch with your series. dirtybalance-v7r1 is
> yours.
>
> 3.1.0-vanilla rc5-vanilla freemore-v6r1 isolate-v6r1 dirtybalance-v7r1
> System Time 1.22 ( 0.00%) 13.89 (-1040.72%) 46.40 (-3709.20%) 4.44 ( -264.37%) 43.05 (-3434.81%)
> +/- 0.06 ( 0.00%) 22.82 (-37635.56%) 3.84 (-6249.44%) 6.48 (-10618.92%) 4.04 (-6581.33%)
> User Time 0.06 ( 0.00%) 0.06 ( -6.90%) 0.05 ( 17.24%) 0.05 ( 13.79%) 0.05 ( 20.69%)
> +/- 0.01 ( 0.00%) 0.01 ( 33.33%) 0.01 ( 33.33%) 0.01 ( 39.14%) 0.01 ( -1.84%)
> Elapsed Time 10445.54 ( 0.00%) 2249.92 ( 78.46%) 70.06 ( 99.33%) 16.59 ( 99.84%) 73.71 ( 99.29%)
> +/- 643.98 ( 0.00%) 811.62 ( -26.03%) 10.02 ( 98.44%) 7.03 ( 98.91%) 17.90 ( 97.22%)
> THP Active 15.60 ( 0.00%) 35.20 ( 225.64%) 65.00 ( 416.67%) 70.80 ( 453.85%) 102.60 ( 657.69%)
> +/- 18.48 ( 0.00%) 51.29 ( 277.59%) 15.99 ( 86.52%) 37.91 ( 205.18%) 26.06 ( 141.02%)
> Fault Alloc 121.80 ( 0.00%) 76.60 ( 62.89%) 155.40 ( 127.59%) 181.20 ( 148.77%) 214.80 ( 176.35%)
> +/- 73.51 ( 0.00%) 61.11 ( 83.12%) 34.89 ( 47.46%) 31.88 ( 43.36%) 53.21 ( 72.39%)
> Fault Fallback 881.20 ( 0.00%) 926.60 ( -5.15%) 847.60 ( 3.81%) 822.00 ( 6.72%) 788.40 ( 10.53%)
> +/- 73.51 ( 0.00%) 61.26 ( 16.67%) 34.89 ( 52.54%) 31.65 ( 56.94%) 53.41 ( 27.35%)
> MMTests Statistics: duration
> User/Sys Time Running Test (seconds) 3540.88 1945.37 716.04 64.97 715.04
> Total Elapsed Time (seconds) 52417.33 11425.90 501.02 230.95 549.64
>
> Your series does help the System CPU time begining it from 46.4 seconds
> to 43.05 seconds. That is within the noise but towards the edge of
> one standard deviation. With such a small reduction, elapsed time was
> not helped. However, it did help THP allocation success rates - still
> within the noise but again at the edge of the noise which indicates
> a solid improvement.
>
> MMTests Statistics: vmstat
> Page Ins 3257266139 1111844061 17263623 10901575 20870385
> Page Outs 81054922 30364312 3626530 3657687 3665499
> Swap Ins 3294 2851 6560 4964 6598
> Swap Outs 390073 528094 620197 790912 604228
> Direct pages scanned 1077581700 3024951463 1764930052 115140570 1796314840
> Kswapd pages scanned 34826043 7112868 2131265 1686942 2093637
> Kswapd pages reclaimed 28950067 4911036 1246044 966475 1319662
> Direct pages reclaimed 805148398 280167837 3623473 2215044 4182274
> Kswapd efficiency 83% 69% 58% 57% 63%
> Kswapd velocity 664.399 622.521 4253.852 7304.360 3809.106
> Direct efficiency 74% 9% 0% 1% 0%
> Direct velocity 20557.737 264745.137 3522673.849 498551.938 3268166.145
> Percentage direct scans 96% 99% 99% 98% 99%
> Page writes by reclaim 722646 529174 620319 791018 604368
> Page writes file 332573 1080 122 106 140
> Page writes anon 390073 528094 620197 790912 604228
> Page reclaim immediate 0 2552514720 1635858848 111281140 1661416934
> Page rescued immediate 0 0 0 87848 0
> Slabs scanned 23552 23552 9216 8192 8192
> Direct inode steals 231 0 0 0 0
> Kswapd inode steals 0 0 0 0 0
> Kswapd skipped wait 28076 786 0 61 1
> THP fault alloc 609 383 753 906 1074
> THP collapse alloc 12 6 0 0 0
> THP splits 536 211 456 593 561
> THP fault fallback 4406 4633 4263 4110 3942
> THP collapse fail 120 127 0 0 0
> Compaction stalls 1810 728 623 779 869
> Compaction success 196 53 60 80 99
> Compaction failures 1614 675 563 699 770
> Compaction pages moved 193158 53545 243185 333457 409585
> Compaction move failure 9952 9396 16424 23676 30668
>
> The direct page scanned figure with your patch is still very high
> unfortunately.
>
> Overall, I would say that your series is not a replacement for the last
> patch in this series.
Agreed, thanks for clearing this up.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2011-12-19 16:14 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-12-14 15:41 [PATCH 0/11] Reduce compaction-related stalls and improve asynchronous migration of dirty pages v6 Mel Gorman
2011-12-14 15:41 ` [PATCH 01/11] mm: compaction: Allow compaction to isolate dirty pages Mel Gorman
2011-12-14 15:41 ` [PATCH 02/11] mm: compaction: Use synchronous compaction for /proc/sys/vm/compact_memory Mel Gorman
2011-12-14 15:41 ` [PATCH 03/11] mm: vmscan: Check if we isolated a compound page during lumpy scan Mel Gorman
2011-12-15 23:21 ` Rik van Riel
2011-12-14 15:41 ` [PATCH 04/11] mm: vmscan: Do not OOM if aborting reclaim to start compaction Mel Gorman
2011-12-15 23:36 ` Rik van Riel
2011-12-14 15:41 ` [PATCH 05/11] mm: compaction: Determine if dirty pages can be migrated without blocking within ->migratepage Mel Gorman
2011-12-16 3:32 ` Rik van Riel
2011-12-16 23:20 ` Andrew Morton
2011-12-17 3:03 ` Nai Xia
2011-12-17 3:26 ` Andrew Morton
2011-12-19 11:05 ` Mel Gorman
2011-12-19 13:12 ` nai.xia
2011-12-14 15:41 ` [PATCH 06/11] mm: compaction: make isolate_lru_page() filter-aware again Mel Gorman
2011-12-16 3:34 ` Rik van Riel
2011-12-18 1:53 ` Minchan Kim
2011-12-14 15:41 ` [PATCH 07/11] mm: page allocator: Do not call direct reclaim for THP allocations while compaction is deferred Mel Gorman
2011-12-16 4:10 ` Rik van Riel
2011-12-14 15:41 ` [PATCH 08/11] mm: compaction: Introduce sync-light migration for use by compaction Mel Gorman
2011-12-16 4:31 ` Rik van Riel
2011-12-18 2:05 ` Minchan Kim
2011-12-19 11:45 ` Mel Gorman
2011-12-20 7:18 ` Minchan Kim
2012-01-13 21:25 ` Andrew Morton
2012-01-16 11:33 ` Mel Gorman
2011-12-14 15:41 ` [PATCH 09/11] mm: vmscan: When reclaiming for compaction, ensure there are sufficient free pages available Mel Gorman
2011-12-16 4:35 ` Rik van Riel
2011-12-14 15:41 ` [PATCH 10/11] mm: vmscan: Check if reclaim should really abort even if compaction_ready() is true for one zone Mel Gorman
2011-12-16 4:38 ` Rik van Riel
2011-12-16 11:29 ` Mel Gorman
2011-12-14 15:41 ` [PATCH 11/11] mm: Isolate pages for immediate reclaim on their own LRU Mel Gorman
2011-12-16 4:47 ` Rik van Riel
2011-12-16 12:26 ` Mel Gorman
2011-12-16 15:17 ` Johannes Weiner
2011-12-16 16:07 ` Mel Gorman
2011-12-19 16:14 ` Johannes Weiner [this message]
2011-12-17 16:08 ` Minchan Kim
2011-12-19 13:26 ` Mel Gorman
2011-12-20 7:10 ` Minchan Kim
2011-12-20 9:55 ` Mel Gorman
2011-12-23 19:08 ` Hugh Dickins
2011-12-29 16:59 ` Mel Gorman
2011-12-29 19:31 ` Rik van Riel
2011-12-30 11:27 ` Mel Gorman
2011-12-16 22:56 ` [PATCH 0/11] Reduce compaction-related stalls and improve asynchronous migration of dirty pages v6 Andrew Morton
2011-12-19 14:40 ` Mel Gorman
2011-12-16 23:37 ` Andrew Morton
2011-12-19 14:20 ` Mel Gorman
-- strict thread matches above, loose matches on Subject: below --
2011-12-01 17:36 [PATCH 0/11] Reduce compaction-related stalls and improve asynchronous migration of dirty pages v5 Mel Gorman
2011-12-01 17:36 ` [PATCH 11/11] mm: Isolate pages for immediate reclaim on their own LRU Mel Gorman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20111219161436.GF1415@cmpxchg.org \
--to=hannes@cmpxchg.org \
--cc=aarcange@redhat.com \
--cc=adi@hexapodia.org \
--cc=akpm@linux-foundation.org \
--cc=davej@redhat.com \
--cc=jack@suse.cz \
--cc=jweiner@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@suse.de \
--cc=minchan.kim@gmail.com \
--cc=nai.xia@gmail.com \
--cc=riel@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox