From: Mel Gorman <mgorman@suse.de>
To: Ingo Molnar <mingo@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Alex Shi <alex.shi@linaro.org>,
Thomas Gleixner <tglx@linutronix.de>,
Andrew Morton <akpm@linux-foundation.org>,
Fengguang Wu <fengguang.wu@intel.com>,
H Peter Anvin <hpa@zytor.com>, Linux-X86 <x86@kernel.org>,
Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Subject: Re: [PATCH 0/4] Fix ebizzy performance regression due to X86 TLB range flush v2
Date: Mon, 16 Dec 2013 12:59:23 +0000 [thread overview]
Message-ID: <20131216125923.GS11295@suse.de> (raw)
In-Reply-To: <20131216102439.GA21624@gmail.com>
On Mon, Dec 16, 2013 at 11:24:39AM +0100, Ingo Molnar wrote:
>
> * Mel Gorman <mgorman@suse.de> wrote:
>
> > I had hacked ebizzy to report on the performance of each thread, not
> > just the overall result and worked out the difference in performance
> > of each thread. In a complete fair test you would expect the
> > performance of each thread to be identical and so the spread would
> > be 0
> >
> > ebizzy thread spread
> > 3.13.0-rc3 3.13.0-rc3 3.4.69
> > vanilla nowalk-v2r7 vanilla
> > Mean 1 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%)
> > Mean 2 0.34 ( 0.00%) 0.30 (-11.76%) 0.07 (-79.41%)
> > Mean 3 1.29 ( 0.00%) 0.92 (-28.68%) 0.29 (-77.52%)
> > Mean 4 7.08 ( 0.00%) 42.38 (498.59%) 0.22 (-96.89%)
> > Mean 5 193.54 ( 0.00%) 483.41 (149.77%) 0.41 (-99.79%)
> > Mean 6 151.12 ( 0.00%) 198.22 ( 31.17%) 0.42 (-99.72%)
> > Mean 7 115.38 ( 0.00%) 160.29 ( 38.92%) 0.58 (-99.50%)
> > Mean 8 108.65 ( 0.00%) 138.96 ( 27.90%) 0.44 (-99.60%)
> > Range 1 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%)
> > Range 2 5.00 ( 0.00%) 6.00 ( 20.00%) 2.00 (-60.00%)
> > Range 3 10.00 ( 0.00%) 17.00 ( 70.00%) 9.00 (-10.00%)
> > Range 4 256.00 ( 0.00%) 1001.00 (291.02%) 5.00 (-98.05%)
> > Range 5 456.00 ( 0.00%) 1226.00 (168.86%) 6.00 (-98.68%)
> > Range 6 298.00 ( 0.00%) 294.00 ( -1.34%) 8.00 (-97.32%)
> > Range 7 192.00 ( 0.00%) 220.00 ( 14.58%) 7.00 (-96.35%)
> > Range 8 171.00 ( 0.00%) 163.00 ( -4.68%) 8.00 (-95.32%)
> > Stddev 1 0.00 ( 0.00%) 0.00 ( 0.00%) 0.00 ( 0.00%)
> > Stddev 2 0.72 ( 0.00%) 0.85 (-17.99%) 0.29 ( 59.72%)
> > Stddev 3 1.42 ( 0.00%) 1.90 (-34.22%) 1.12 ( 21.19%)
> > Stddev 4 33.83 ( 0.00%) 127.26 (-276.15%) 0.79 ( 97.65%)
> > Stddev 5 92.08 ( 0.00%) 225.01 (-144.35%) 1.06 ( 98.85%)
> > Stddev 6 64.82 ( 0.00%) 69.43 ( -7.11%) 1.28 ( 98.02%)
> > Stddev 7 36.66 ( 0.00%) 49.19 (-34.20%) 1.18 ( 96.79%)
> > Stddev 8 30.79 ( 0.00%) 36.23 (-17.64%) 1.06 ( 96.55%)
> >
> > For example, this is saying that with 8 threads on 3.13-rc3 that the
> > difference between the slowest and fastest thread was 171
> > records/second.
>
> We aren't blind fairness fetishists, but the noise difference between
> v3.4 and v3.13 appears to be staggering, it's a serious anomaly in
> itself.
>
Agreed.
> Whatever we did right in v3.4 we want to do in v3.13 as well - or at
> least understand it.
>
Also agreed. I started a bisection before answering this mail. It would
be cooler and potentially faster to figure it out from direct analysis
but bisection is reliable and less guesswork.
> I agree that the absolute numbers would probably only be interesting
> once v3.13 is fixed to not spread thread performance that wildly
> again.
>
> > [...] Because of this bug, I'd be wary about drawing too many
> > conclusions about ebizzy performance when the number of threads
> > exceed the number of CPUs.
>
> Yes.
>
> Could it be that the v3.13 workload context switches a lot more than
> v3.4 workload?
The opposite. 3.13 context switches and interrupts less.
> That would magnify any TLB range flushing costs and
> would make it essentially a secondary symptom, not a primary cause of
> the regression. (I'm only guessing blindly here though.)
>
Fortunately, I had collected data on context switches
4 core machine: http://www.csn.ul.ie/~mel/postings/spread-20131216/global-ebizzy/ivor/report.html
8 core machine: http://www.csn.ul.ie/~mel/postings/spread-20131216/global-ebizzy/ivy/report.html
The ebizzy results are at the end. One of the graphs are for context
switches as measured by vmstat running during the test.
In both cases you can see that context switches are higher for 3.4 as
are interrupts. The difference in context switches are why I thought this
might be scheduler related but the difference in interrupts was harder to
explain. I'm guessing they're IPIs but did not record /proc/interrupts
to answer that. I lack familiarity with scheduler changes between 3.4
and 3.13-rc4 and have no intuitive feeling for when this might have been
introduced. I'm also not sure if we used to do anything like send IPIs
to reschedule tasks or balance tasks between idle cores that changed
recently. There was also a truckload of nohz changes in that window that
I'm not familiar with that are potentially responsible. Should have
answers soon enough.
--
Mel Gorman
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-12-16 12:59 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-12-13 20:01 Mel Gorman
2013-12-13 20:01 ` [PATCH 1/4] x86: mm: Clean up inconsistencies when flushing TLB ranges Mel Gorman
2013-12-13 20:01 ` [PATCH 2/4] x86: mm: Account for TLB flushes only when debugging Mel Gorman
2013-12-13 20:01 ` [PATCH 3/4] x86: mm: Change tlb_flushall_shift for IvyBridge Mel Gorman
2013-12-13 20:01 ` [PATCH 4/4] x86: mm: Eliminate redundant page table walk during TLB range flushing Mel Gorman
2013-12-13 21:16 ` [PATCH 0/4] Fix ebizzy performance regression due to X86 TLB range flush v2 Linus Torvalds
2013-12-13 22:38 ` H. Peter Anvin
2013-12-16 10:39 ` Mel Gorman
2013-12-16 17:17 ` Linus Torvalds
2013-12-17 9:55 ` Mel Gorman
2013-12-15 15:55 ` Mel Gorman
2013-12-15 16:17 ` Mel Gorman
2013-12-15 18:34 ` Linus Torvalds
2013-12-16 11:16 ` Mel Gorman
2013-12-16 10:24 ` Ingo Molnar
2013-12-16 12:59 ` Mel Gorman [this message]
2013-12-16 13:44 ` Ingo Molnar
2013-12-17 9:21 ` Mel Gorman
2013-12-17 9:26 ` Peter Zijlstra
2013-12-17 11:00 ` Ingo Molnar
2013-12-17 14:32 ` Mel Gorman
2013-12-17 14:42 ` Ingo Molnar
2013-12-17 17:54 ` Mel Gorman
2013-12-18 10:24 ` Ingo Molnar
2013-12-19 14:24 ` Mel Gorman
2013-12-19 16:49 ` Ingo Molnar
2013-12-20 11:13 ` Mel Gorman
2013-12-20 11:18 ` Ingo Molnar
2013-12-20 12:00 ` Mel Gorman
2013-12-20 12:20 ` Ingo Molnar
2013-12-20 13:55 ` Mel Gorman
2013-12-18 7:28 ` Fengguang Wu
2013-12-19 14:34 ` Mel Gorman
2013-12-20 15:51 ` Fengguang Wu
2013-12-20 16:44 ` Mel Gorman
2013-12-21 15:49 ` Fengguang Wu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131216125923.GS11295@suse.de \
--to=mgorman@suse.de \
--cc=a.p.zijlstra@chello.nl \
--cc=akpm@linux-foundation.org \
--cc=alex.shi@linaro.org \
--cc=fengguang.wu@intel.com \
--cc=hpa@zytor.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@kernel.org \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox