[Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Chris Mason]

Hi everyone,

We're in the middle of upgrading the tiers here from older kernels 
(2.6.38, 3.2) into 3.10 and higher.

I've been doing this upgrade game for a number of years now, with 
different business cards taped to my forehead and with different target 
workloads.

The result is always the same...if I'm really lucky the system isn't 
slower, but usually I'm left with a steaming pile of 10-30% regressions.

The KS dates should put us right at the end of our regression hunt, I 
can talk through the main problems we hit, how (if) we fixed them and 
hopefully offer tests to keep them from coming back.

We've split the tiers up between Josef Bacik, Jens Axboe, myself and a 
few others.  I'd nominate Josef and Jens to come share their findings as 
well.

Another topic here is controlling preemption from userland without 
needing to go full real time.  CPU intensive in-memory databases are 
leaving a ton of performance on the floor in context switches.  I'm 
hoping to experiment with better preemption controls and the userland 
RCU project to improve things.

-chris

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Davidlohr Bueso]

On Mon, 2014-05-12 at 10:32 -0400, Chris Mason wrote:
> Hi everyone,
> 
> We're in the middle of upgrading the tiers here from older kernels 
> (2.6.38, 3.2) into 3.10 and higher.
> 
> I've been doing this upgrade game for a number of years now, with 
> different business cards taped to my forehead and with different target 
> workloads.
> 
> The result is always the same...if I'm really lucky the system isn't 
> slower, but usually I'm left with a steaming pile of 10-30% regressions.

Please consider me for this discussion. At HP we've been working on
different performance issues in the kernel, including a lot of locking
improvements and optimizations, which have boosted performance for a
wide variety of workloads. There are plenty of really interesting
results and I'd be happy to share some of them and other findings.

Furthermore, I'm always on the lookout for new performance issues so I'd
be interested in knowing some of the seen regressions and if they are
still present today (lots of changes since 3.10).

> The KS dates should put us right at the end of our regression hunt, I 
> can talk through the main problems we hit, how (if) we fixed them and 
> hopefully offer tests to keep them from coming back.
> 
> We've split the tiers up between Josef Bacik, Jens Axboe, myself and a 
> few others.  I'd nominate Josef and Jens to come share their findings as 
> well.

I'd also like to include PeterZ and Ingo.

> Another topic here is controlling preemption from userland without 
> needing to go full real time.  CPU intensive in-memory databases are 
> leaving a ton of performance on the floor in context switches.  I'm 
> hoping to experiment with better preemption controls and the userland 
> RCU project to improve things.

Hmm specifically for this, it would also be good to have Khalid Aziz in
the discussion. He's been trying to add userspace controlled preemption
for a while now.

Thanks,
Davidlohr

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Jan Kara]

  Hello,

On Mon 12-05-14 10:32:27, Chris Mason wrote:
> We're in the middle of upgrading the tiers here from older kernels
> (2.6.38, 3.2) into 3.10 and higher.
> 
> I've been doing this upgrade game for a number of years now, with
> different business cards taped to my forehead and with different
> target workloads.
> 
> The result is always the same...if I'm really lucky the system isn't
> slower, but usually I'm left with a steaming pile of 10-30%
> regressions.
  I'd be interested in this discussion as well. We are in a similar
situation in SUSE now when moving our enterprise kernel from 3.0 to 3.12
base. So me, Jiri Kosina, or Mel Gorman can fill in things we found and you
didn't ;) Plus we are slowly working on setting up some more systematic and
continuous way of tracking performance in kernels so sharing experiences
with what is useful to run would be interesting as well.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Andy Lutomirski]

On Mon, May 12, 2014 at 7:32 AM, Chris Mason <clm@fb.com> wrote:
> Hi everyone,
>
> We're in the middle of upgrading the tiers here from older kernels (2.6.38,
> 3.2) into 3.10 and higher.
>
> I've been doing this upgrade game for a number of years now, with different
> business cards taped to my forehead and with different target workloads.
>
> The result is always the same...if I'm really lucky the system isn't slower,
> but usually I'm left with a steaming pile of 10-30% regressions.
>
> The KS dates should put us right at the end of our regression hunt, I can
> talk through the main problems we hit, how (if) we fixed them and hopefully
> offer tests to keep them from coming back.
>
> We've split the tiers up between Josef Bacik, Jens Axboe, myself and a few
> others.  I'd nominate Josef and Jens to come share their findings as well.
>
> Another topic here is controlling preemption from userland without needing
> to go full real time.  CPU intensive in-memory databases are leaving a ton
> of performance on the floor in context switches.  I'm hoping to experiment
> with better preemption controls and the userland RCU project to improve
> things.

I wonder how much of this overhead is the actual interrupt + context
switch and how much is overhead due to caching effects.  If the
former, there may be value in having a simple benchmark for how long
preemption takes and running it periodically.

--Andy

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Greg KH]

On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> Hi everyone,
> 
> We're in the middle of upgrading the tiers here from older kernels 
> (2.6.38, 3.2) into 3.10 and higher.
> 
> I've been doing this upgrade game for a number of years now, with 
> different business cards taped to my forehead and with different target 
> workloads.
> 
> The result is always the same...if I'm really lucky the system isn't 
> slower, but usually I'm left with a steaming pile of 10-30% regressions.

How long have we been having this discussion?  8 years?  It's not like
people don't know that performance testing needs to be constantly
happening, we've been saying that for a long time.  It's just that no
one seems to listen to us :(

And that is the larger problem, what can we do about that issue.
Honestly, I don't think much, as it takes money from companies to commit
to do this work, which no one seems to ever want to do.  What make this
year the year that something different happens?

greg k-h

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Josh Triplett]

On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> Hi everyone,
> 
> We're in the middle of upgrading the tiers here from older kernels (2.6.38,
> 3.2) into 3.10 and higher.
> 
> I've been doing this upgrade game for a number of years now, with different
> business cards taped to my forehead and with different target workloads.
> 
> The result is always the same...if I'm really lucky the system isn't slower,
> but usually I'm left with a steaming pile of 10-30% regressions.

How automated are your benchmark workloads, how long do they take, and
how consistent are they from run to run (on a system running nothing
else)?  What about getting them into Fengguang Wu's automated patch
checker, or a similar system that checks every patch or pull rather than
just full releases?  If we had feedback at the time of patch submission
that a specific patch made the kernel x% slower for a specific
well-defined workload, that would prove much easier to act on than just
a comparison of 3.x and 3.y.

- Josh Triplett

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Davidlohr Bueso]

On Mon, 2014-05-12 at 16:54 -0700, Josh Triplett wrote:
> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> > Hi everyone,
> > 
> > We're in the middle of upgrading the tiers here from older kernels (2.6.38,
> > 3.2) into 3.10 and higher.
> > 
> > I've been doing this upgrade game for a number of years now, with different
> > business cards taped to my forehead and with different target workloads.
> > 
> > The result is always the same...if I'm really lucky the system isn't slower,
> > but usually I'm left with a steaming pile of 10-30% regressions.
> 
> How automated are your benchmark workloads, how long do they take, and
> how consistent are they from run to run (on a system running nothing
> else)?  What about getting them into Fengguang Wu's automated patch
> checker, or a similar system that checks every patch or pull rather than
> just full releases?  If we had feedback at the time of patch submission
> that a specific patch made the kernel x% slower for a specific
> well-defined workload, that would prove much easier to act on than just
> a comparison of 3.x and 3.y.

This sounds ideal, but reality is very very different.

Fengguang's scripts are quite nice and work for a number of scenarios,
but cannot possibly cover everything. And the regressions Chris mentions
are quite common, depending what and where you're looking at. Just
consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not
talking about pgbench only). Or just about anything that's not synthetic
and easy to setup (ie: Hadoop). Subtle architecture specific changes
(ie: non x86) are also beyond this scope and can trigger major
performance regressions. And even common benchmarks and systems such as
aim7 (which I know Fengguang runs) and x86 can bypass the automated
checks, just look at https://lkml.org/lkml/2014/3/17/587.
There are just too many variables to control.

That said, I do agree that we could do better, and yeah, adding more
workloads to Fengguang's scripts are always a good thing -- perhaps even
adding stuff from perf-bench.

Thanks,
Davidlohr

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Chris Mason]

On 05/12/2014 07:16 PM, Greg KH wrote:
> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
>> Hi everyone,
>>
>> We're in the middle of upgrading the tiers here from older kernels 
>> (2.6.38, 3.2) into 3.10 and higher.
>>
>> I've been doing this upgrade game for a number of years now, with 
>> different business cards taped to my forehead and with different target 
>> workloads.
>>
>> The result is always the same...if I'm really lucky the system isn't 
>> slower, but usually I'm left with a steaming pile of 10-30% regressions.
> 
> How long have we been having this discussion?  8 years?  It's not like
> people don't know that performance testing needs to be constantly
> happening, we've been saying that for a long time.  It's just that no
> one seems to listen to us :(
> 

Yes and no.  Intel listened, and I think they have had a huge positive
impact here.  Others have as well, maybe not as consistently, but still.

I do find it really interesting that even with huge improvements all
over the place, we have a very hard time upgrading production workloads
without hitting big regressions.

Sometimes it is just a few .config entries, and sometimes we paper over
it with improvements in other areas, but it's almost always worse.

> And that is the larger problem, what can we do about that issue.
> Honestly, I don't think much, as it takes money from companies to commit
> to do this work, which no one seems to ever want to do.  What make this
> year the year that something different happens?

I can't promise different.  At best we'll tease out some new benchmarks
and try to get them into a format that get run often.

-chris

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Jan Kara]

On Tue 13-05-14 01:16:48, Greg KH wrote:
> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> > Hi everyone,
> > 
> > We're in the middle of upgrading the tiers here from older kernels 
> > (2.6.38, 3.2) into 3.10 and higher.
> > 
> > I've been doing this upgrade game for a number of years now, with 
> > different business cards taped to my forehead and with different target 
> > workloads.
> > 
> > The result is always the same...if I'm really lucky the system isn't 
> > slower, but usually I'm left with a steaming pile of 10-30% regressions.
> 
> How long have we been having this discussion?  8 years?  It's not like
> people don't know that performance testing needs to be constantly
> happening, we've been saying that for a long time.  It's just that no
> one seems to listen to us :(
  I agree with you that this is coming over and over again for quite a few
years.

> And that is the larger problem, what can we do about that issue.
> Honestly, I don't think much, as it takes money from companies to commit
> to do this work, which no one seems to ever want to do.  What make this
> year the year that something different happens?
  So what I found interesting about the topic is:
1) What has regressed this time in particular? This might be interesting
for a broader audience I believe. With my SUSE hat on, I can learn which
problems should I expect our customers to see that we didn't find during
our testing.  With my community hat on, I can possibly learn some things
which are easy enough to test for...

2) We are looking into some more continuous testing in SUSE so learning
what others have tested would be interesting for me.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Li Zefan]

On 2014/5/13 9:43, Chris Mason wrote:
> On 05/12/2014 07:16 PM, Greg KH wrote:
>> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
>>> Hi everyone,
>>>
>>> We're in the middle of upgrading the tiers here from older kernels 
>>> (2.6.38, 3.2) into 3.10 and higher.
>>>
>>> I've been doing this upgrade game for a number of years now, with 
>>> different business cards taped to my forehead and with different target 
>>> workloads.
>>>
>>> The result is always the same...if I'm really lucky the system isn't 
>>> slower, but usually I'm left with a steaming pile of 10-30% regressions.
>>
>> How long have we been having this discussion?  8 years?  It's not like
>> people don't know that performance testing needs to be constantly
>> happening, we've been saying that for a long time.  It's just that no
>> one seems to listen to us :(
>>
> 
> Yes and no.  Intel listened, and I think they have had a huge positive
> impact here.  Others have as well, maybe not as consistently, but still.
> 
> I do find it really interesting that even with huge improvements all
> over the place, we have a very hard time upgrading production workloads
> without hitting big regressions.
> 
> Sometimes it is just a few .config entries,

Do you mean some regressions are introduced by new configs i.e. new
features? If so, I don't think that can be strictly called regressions.

> and sometimes we paper over
> it with improvements in other areas, but it's almost always worse.
> 

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Chris Mason]

On 05/13/2014 09:31 PM, Li Zefan wrote:
> On 2014/5/13 9:43, Chris Mason wrote:
>> On 05/12/2014 07:16 PM, Greg KH wrote:
>>> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
>>>> Hi everyone,
>>>>
>>>> We're in the middle of upgrading the tiers here from older kernels 
>>>> (2.6.38, 3.2) into 3.10 and higher.
>>>>
>>>> I've been doing this upgrade game for a number of years now, with 
>>>> different business cards taped to my forehead and with different target 
>>>> workloads.
>>>>
>>>> The result is always the same...if I'm really lucky the system isn't 
>>>> slower, but usually I'm left with a steaming pile of 10-30% regressions.
>>>
>>> How long have we been having this discussion?  8 years?  It's not like
>>> people don't know that performance testing needs to be constantly
>>> happening, we've been saying that for a long time.  It's just that no
>>> one seems to listen to us :(
>>>
>>
>> Yes and no.  Intel listened, and I think they have had a huge positive
>> impact here.  Others have as well, maybe not as consistently, but still.
>>
>> I do find it really interesting that even with huge improvements all
>> over the place, we have a very hard time upgrading production workloads
>> without hitting big regressions.
>>
>> Sometimes it is just a few .config entries,
> 
> Do you mean some regressions are introduced by new configs i.e. new
> features? If so, I don't think that can be strictly called regressions.
> 

Agreed, at least unless the .config choice that makes it slow is the new
default.  +/- slub, I haven't hit this case.

-chris

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Paul E. McKenney]

On Mon, May 12, 2014 at 04:54:35PM -0700, Josh Triplett wrote:
> On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> > Hi everyone,
> > 
> > We're in the middle of upgrading the tiers here from older kernels (2.6.38,
> > 3.2) into 3.10 and higher.
> > 
> > I've been doing this upgrade game for a number of years now, with different
> > business cards taped to my forehead and with different target workloads.
> > 
> > The result is always the same...if I'm really lucky the system isn't slower,
> > but usually I'm left with a steaming pile of 10-30% regressions.
> 
> How automated are your benchmark workloads, how long do they take, and
> how consistent are they from run to run (on a system running nothing
> else)?  What about getting them into Fengguang Wu's automated patch
> checker, or a similar system that checks every patch or pull rather than
> just full releases?  If we had feedback at the time of patch submission
> that a specific patch made the kernel x% slower for a specific
> well-defined workload, that would prove much easier to act on than just
> a comparison of 3.x and 3.y.

Good point on Fengguang's checker!  I do sometimes get emails from him
noting changes in performance, conveniently bisected down to the commit.

							Thanx, Paul

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Fengguang Wu]

On Mon, May 12, 2014 at 05:31:21PM -0700, Davidlohr Bueso wrote:
> On Mon, 2014-05-12 at 16:54 -0700, Josh Triplett wrote:
> > On Mon, May 12, 2014 at 10:32:27AM -0400, Chris Mason wrote:
> > > Hi everyone,
> > > 
> > > We're in the middle of upgrading the tiers here from older kernels (2.6.38,
> > > 3.2) into 3.10 and higher.
> > > 
> > > I've been doing this upgrade game for a number of years now, with different
> > > business cards taped to my forehead and with different target workloads.
> > > 
> > > The result is always the same...if I'm really lucky the system isn't slower,
> > > but usually I'm left with a steaming pile of 10-30% regressions.
> > 
> > How automated are your benchmark workloads, how long do they take, and
> > how consistent are they from run to run (on a system running nothing
> > else)?  What about getting them into Fengguang Wu's automated patch
> > checker, or a similar system that checks every patch or pull rather than
> > just full releases?  If we had feedback at the time of patch submission
> > that a specific patch made the kernel x% slower for a specific
> > well-defined workload, that would prove much easier to act on than just
> > a comparison of 3.x and 3.y.
> 
> This sounds ideal, but reality is very very different.
> 
> Fengguang's scripts are quite nice and work for a number of scenarios,
> but cannot possibly cover everything.

Sorry for being late.. Yup, test coverage is a huge challenge and
I believe collaborations are the key to make substantial progresses.

Intel OTC has been running a LKP (Linux Kernel Performance) project
which does boot, functional, performance and power tests over the
community kernel git trees. Some diligent hackers (Hi Paul!) can
occasionally trigger our regression reports. We believe it could
potentially be a tool for more developers to evaluate performance/power
of their wip patches, in a more direct and manageable way.

So we are excited to share LKP test cases with the community in GPLv2:

  git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests

It's still missing the documents part -- when ready, I'll make a
public announce in LKML. Basically it enables a kernel developer to
run LKP tests in his own test box and generate/compare test results
like this:

        https://lists.01.org/pipermail/lkp/2014-July/000324.html

The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are
inspired by Mel Gorman's mmtests suite and they are really helpful in
catching&analyzing the subtle impacts a patch might bring to the system.

> And the regressions Chris mentions
> are quite common, depending what and where you're looking at. Just
> consider proprietary tools and benchmarks (ie: Oracle -- and no, I'm not
> talking about pgbench only). Or just about anything that's not synthetic
> and easy to setup (ie: Hadoop). Subtle architecture specific changes
> (ie: non x86) are also beyond this scope and can trigger major
> performance regressions. And even common benchmarks and systems such as
> aim7 (which I know Fengguang runs) and x86 can bypass the automated
> checks, just look at https://lkml.org/lkml/2014/3/17/587.
> There are just too many variables to control.

Yes, there are often the need to test combinations of parameters.
In LKP, we make it convenient to define "matrix" test jobs like:

fio:
  rw:
  - randwrite
  - randrw
  ioengine:
  - sync
  - mmap
  bs:
  - 4k
  - 64k

Which will be split into 2*2*2 unit jobs for execution.  For example,
the first unit job is:

fio:
  rw: randwrite
  ioengine: sync
  bs: 4k

> That said, I do agree that we could do better, and yeah, adding more
> workloads to Fengguang's scripts are always a good thing -- perhaps even
> adding stuff from perf-bench.

You are very welcome to add new cases, monitors or setup scripts!
Depending on their nature and resource requirement, we may choose
the adequate policy to run them in our LKP test infrastructure --
which works 7x24 on the fresh new code in 400+ kernel git trees. By
feeding it more test cases, we may reasonably safeguard more kernel
code and use scenarios from _silent_ regressions in future.

Thanks,
Fengguang

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Christoph Lameter]

On Thu, 14 Aug 2014, Fengguang Wu wrote:

> The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are
> inspired by Mel Gorman's mmtests suite and they are really helpful in
> catching&analyzing the subtle impacts a patch might bring to the system.

Mel also has some interesting tests in the suite like the page fault test
and aim9 among numerous others. Could those also be added?

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Fengguang Wu]

On Thu, Aug 14, 2014 at 12:17:38PM -0500, Christoph Lameter wrote:
> On Thu, 14 Aug 2014, Fengguang Wu wrote:
> 
> > The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are
> > inspired by Mel Gorman's mmtests suite and they are really helpful in
> > catching&analyzing the subtle impacts a patch might bring to the system.
> 
> Mel also has some interesting tests in the suite like the page fault test
> and aim9 among numerous others. Could those also be added?

Sure. I just added aim9 to the TODO list.

pft is already included in lkp-tests:

% ls */pft
jobs/pft.yaml  pack/pft  pack/pft.patch  stats/pft  tests/pft

Thanks,
Fengguang

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Christoph Lameter]

On Fri, 15 Aug 2014, Fengguang Wu wrote:

> On Thu, Aug 14, 2014 at 12:17:38PM -0500, Christoph Lameter wrote:
> > On Thu, 14 Aug 2014, Fengguang Wu wrote:
> >
> > > The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are
> > > inspired by Mel Gorman's mmtests suite and they are really helpful in
> > > catching&analyzing the subtle impacts a patch might bring to the system.
> >
> > Mel also has some interesting tests in the suite like the page fault test
> > and aim9 among numerous others. Could those also be added?
>
> Sure. I just added aim9 to the TODO list.
>
> pft is already included in lkp-tests:
>
> % ls */pft
> jobs/pft.yaml  pack/pft  pack/pft.patch  stats/pft  tests/pft

Are the test results somewhere publicly available?

[Ksummit-discuss] 0day kernel performance/power test service

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 3698 bytes --]

On Fri, Aug 15, 2014 at 09:07:47AM -0500, Christoph Lameter wrote:
> On Fri, 15 Aug 2014, Fengguang Wu wrote:
> 
> > On Thu, Aug 14, 2014 at 12:17:38PM -0500, Christoph Lameter wrote:
> > > On Thu, 14 Aug 2014, Fengguang Wu wrote:
> > >
> > > > The proc-vmstat, perf-profile, cpuidle, turbostat etc. "monitors" are
> > > > inspired by Mel Gorman's mmtests suite and they are really helpful in
> > > > catching&analyzing the subtle impacts a patch might bring to the system.
> > >
> > > Mel also has some interesting tests in the suite like the page fault test
> > > and aim9 among numerous others. Could those also be added?
> >
> > Sure. I just added aim9 to the TODO list.
> >
> > pft is already included in lkp-tests:
> >
> > % ls */pft
> > jobs/pft.yaml  pack/pft  pack/pft.patch  stats/pft  tests/pft
> 
> Are the test results somewhere publicly available?

Not available yet.. Sorry! For now I'm afraid I can only send tar
balls of raw results on your request.

btw, we'll offer an experimental 0day performance/power testing
service for the core maintainer trees. Just send me a lkp-tests patch
(attached an example for ext4:dev) to selectively run a set of test
jobs on a set of machines for your tree:branch. Future git push will
then trigger the test set and you'll receive the report that compares
the branch BASE:HEAD performance in 24 hours.

The patch is basically selecting a combination of test jobs and machines.
All the available test jobs and machines are included in lkp-tests. You
are also welcome to add new jobs to the pool.

wfg /c/lkp-tests% ls hosts
bay         ivb43     lkp-nex04  lkp-sb03   lkp-t410   loslunas   vm-kbuild-2G          vm-kbuild-yocto-x86_64  vm-vp-quantal-x86_64  xps2
bens        ivb44     lkp-nex05  lkp-sbx04  lkp-ws02   nhm4       vm-kbuild-4G          vm-vp-1G                wsm
brickland1  lkp-ne02  lkp-nex06  lkp-snb01  lkp-wsx01  nhm-white  vm-kbuild-yocto-i386  vm-vp-2G                xbm
brickland3  lkp-ne04  lkp-sb02   lkp-st02   lkp-wsx02  snb-drag   vm-kbuild-yocto-ia32  vm-vp-quantal-i386      xps

wfg /c/lkp-tests% ls jobs
blogbench.yaml           dd-write.yaml                          hackbench-8.yaml       netperf.yaml                tbench.yaml
boot.yaml                debug.yaml                             hackbench-cpuset.yaml  nfs-iozone.yaml             tcrypt.yaml
borrow-1d.yaml           DEFAULTS                               hackbench-perf.yaml    nuttcp.yaml                 thrulay.yaml
borrow-1h.yaml           ebizzy.yaml                            hackbench.yaml         oltp.yaml                   tlbflush.yaml
dbench.yaml              fileio.yaml                            idle.yaml              packetdrill.yaml            unixbench.yaml
dd-write-11hdd.yaml      fio-jbod-12hdd-randwrite-sync-4k.yaml  iozone.yaml            perf-bench-numa-mem.yaml    vm-scalability-numa.yaml
dd-write-1hdd-fuse.yaml  fio-jbod-12hdd.yaml                    iperf.yaml             perf-bench-sched-pipe.yaml  vm-scalability.yaml
dd-write-1hdd.yaml       fsmark.yaml                            kbuild.yaml            pft.yaml                    will-it-scale.yaml
dd-write-1ssd.yaml       ftq.yaml                               kernel_selftests.yaml  piglit.yaml                 xfstests-btrfs.yaml
dd-write-4hdd-fuse.yaml  fwq.yaml                               linpack.yaml           pigz.yaml                   xfstests-ext4.yaml
dd-write-4hdd.yaml       glbenchmark.yaml                       ltp.yaml               qperf.yaml                  xfstests-generic.yaml
dd-write-hdd-usb.yaml    hackbench-6.yaml                       nepim.yaml             sockperf.yaml               xfstests-xfs.yaml

Thanks,
Fengguang

[-- Attachment #2: 0001-define-0day-performance-test-set-for-ext4-dev.patch --]
[-- Type: text/x-diff, Size: 2920 bytes --]

>From 8cd384c2f34f593d829457e7043ee6cca204bda3 Mon Sep 17 00:00:00 2001
From: Fengguang Wu <fengguang.wu@intel.com>
Date: Sat, 16 Aug 2014 09:12:35 +0800
Subject: [PATCH] define 0day performance test set for ext4/dev

This set of test jobs will be auto triggered by git push and
email report should be sent to the committer in 1 day.

CC: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
 allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml           | 27 ++++++++++++++++++++++
 allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml   | 27 ++++++++++++++++++++++
 .../vm-kbuild-2G/xfstests-generic-ext4.yaml        | 16 +++++++++++++
 allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml         |  1 +
 4 files changed, 71 insertions(+)
 create mode 100644 allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml
 create mode 100644 allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml
 create mode 100644 allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml
 create mode 120000 allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml

diff --git a/allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml b/allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml
new file mode 100644
index 0000000..ca2e3ea
--- /dev/null
+++ b/allot/ext4:dev/lkp-ne04/fsmark-ext4.yaml
@@ -0,0 +1,27 @@
+testcase: fsmark
+
+iterations: 1x
+nr_threads: 32t
+
+disk: 1HDD
+fs:
+- ext4
+
+fsmark:
+  filesize: 16MB
+  test_size: 60G
+  sync_method:
+  - NoSync
+  - fsyncBeforeClose
+  nr_directories: 16d
+  nr_files_per_directory: 256f
+
+---
+fsmark:
+  filesize:
+  - 8K
+  - 9B
+  test_size: 400M
+  sync_method: fsyncBeforeClose
+  nr_directories: 16d
+  nr_files_per_directory: 256f
diff --git a/allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml b/allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml
new file mode 100644
index 0000000..d59f3dd
--- /dev/null
+++ b/allot/ext4:dev/lkp-st02/dd-write-11hdd-ext4.yaml
@@ -0,0 +1,27 @@
+testcase: dd-write
+
+runtime: 5m
+
+disk: 11HDD
+md:
+- JBOD
+- RAID5
+iosched:
+- cfq
+fs:
+- ext4
+
+monitors:
+  perf-stat:
+  perf-profile:
+  ftrace:
+  - events:
+      balance_dirty_pages
+      bdi_dirty_ratelimit
+      global_dirty_state
+      writeback_single_inode
+
+nr_threads:
+- 100dd
+
+dd:
diff --git a/allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml b/allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml
new file mode 100644
index 0000000..2561003
--- /dev/null
+++ b/allot/ext4:dev/vm-kbuild-2G/xfstests-generic-ext4.yaml
@@ -0,0 +1,16 @@
+testcase: xfstests
+
+default_monitors:
+
+disk: 4HDD
+
+fs:
+- ext4
+
+xfstests:
+  test:
+  - generic-113
+  - generic-slow1
+  - generic-slow2
+  - generic-mid
+  - generic-quick
diff --git a/allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml b/allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml
new file mode 120000
index 0000000..e3ea5c2
--- /dev/null
+++ b/allot/ext4:dev/vm-vp-1G/xfstests-ext4.yaml
@@ -0,0 +1 @@
+../../../jobs/xfstests-ext4.yaml
\ No newline at end of file
-- 
2.1.0.rc1

Re: [Ksummit-discuss] [TOPIC] Application performance: regressions, controlling preemption

[Fengguang Wu]

Hi Chris,

> The KS dates should put us right at the end of our regression hunt, I can
> talk through the main problems we hit, how (if) we fixed them and hopefully
> offer tests to keep them from coming back.

It'd be sweet if your tests (and others') can be added to

        git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests

Which runs in Intel actively to keep the relevant regressions from
coming back.

Currently the repository has test cases

wfg /c/lkp-tests% ls tests
aim7       debug-test  fio-jbod  glbenchmark  kbuild            nepim    packetdrill            piglit  sockperf  tlbflush        wrapper
blogbench  ebizzy      fsmark    hackbench    kernel_selftests  netperf  perf-bench-numa-mem    pigz    tbench    unixbench       xfstests
dbench     fileio      ftq       iozone       linpack           nuttcp   perf-bench-sched-pipe  qperf   tcrypt    vm-scalability
dd         fio         fwq       iperf        ltp               oltp     pft                    sleep   thrulay   will-it-scale

Which is not only too few in number, but also rather limited in test
parameters and system setups. For example, if run them in different
cgroup/numa/governor/network/... setups, or even with different
kernel configs, the results may become quite different.

Thanks,
Fengguang