Re: 2.6.4-mm2

Re: 2.6.4-mm2

[Mary Edie Meredith]

[was "Poor DBT-3 pgsql 8way numbers on recent 2.6 mm kernels" on
linux-mm]

Andrew,

This same patch (02) applied in STP (plm 2780) when run against
dbt3-pgsql DSS workload displays the performance problem with the
throughput numbers that I reported on linux-mm on our 8way systems,
where the previous patch (plm 2777 -01) does not.  

Here is the data (patches applied to 2.6.5-rc1)

PLM.....CPUs.Runid..Thruput Metric (bigger is better)
2777(01)  8  290298  138.22  (base  )
2779(02)  8  290304  88.57   (-35.9%)

The 8way is a 700MHz (1024k processor cache) with 8GB of memory.

Original message on linux-mm:
http://marc.theaimsgroup.com/?l=linux-mm&m=107913089923436&w=2

Results from runid 290298 (the good result);
http://khack.osdl.org/stp/290298/  (top level)

Results from runid 290304 (the bad result):
http://khack.osdl.org/stp/290305/  (top level)
For sar results see "Raw data" section everything labeled as
"thruput.sar."
http://khack.osdl.org/stp/290305/profile/after_throughput_test_1-tick.top20  (profile of throughput phase of the test)
http://khack.osdl.org/stp/290305/results/plot/thuput.vmstat.txt
(vmstat of thoughput phase of the test)

On Fri, 2004-03-19 at 18:39, Mark Wong wrote:
> On Thu, Mar 18, 2004 at 07:41:50PM -0800, Andrew Morton wrote:
> > Andrew Morton <akpm@osdl.org> wrote:
> > >
> > > Mark, if it's OK I'll run up some kernels for you to test.
> > 
> > At
> > 
> > 	http://www.zip.com.au/~akpm/linux/patches/markw/
> 
> Ok, looks like I take the first hit with the 02 patch.  Here's re-summary:
> 
> kernel          16 kb   32 kb   64 kb   128 kb  256 kb  512 kb
> 2.6.3                           2308    2335    2348    2334
> 2.6.4-mm2       2028    2048    2074    2096    2082    2078
> 2.6.5-rc1-01                                            2394
> 2.6.5-rc1-02                                            2117
> 2.6.5-rc1-mm2                                           2036
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Mary Edie Meredith 
maryedie@osdl.org
503-626-2455 x42
Open Source Development Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: 2.6.4-mm2

[Andrew Morton]

Mary Edie Meredith <maryedie@osdl.org> wrote:
>
> [was "Poor DBT-3 pgsql 8way numbers on recent 2.6 mm kernels" on
> linux-mm]
> 
> Andrew,
> 
> This same patch (02) applied in STP (plm 2780) when run against
> dbt3-pgsql DSS workload displays the performance problem with the
> throughput numbers that I reported on linux-mm on our 8way systems,
> where the previous patch (plm 2777 -01) does not.  
> 
> Here is the data (patches applied to 2.6.5-rc1)
> 
> PLM.....CPUs.Runid..Thruput Metric (bigger is better)
> 2777(01)  8  290298  138.22  (base  )
> 2779(02)  8  290304  88.57   (-35.9%)

36% regression due to the CPU scheduler changes?  ow.

And that machine is a PIII, so presumably the setting of CONFIG_SCHED_SMT
makes no difference.

>From a quick look at the material you have there it appears that this
workload also is very I/O bound.  It's a little surprising that the CPU
scheduler could make so much difference.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: 2.6.4-mm2

[Mary Edie Meredith]

On Mon, 2004-03-22 at 16:27, Andrew Morton wrote:
> Mary Edie Meredith <maryedie@osdl.org> wrote:
> >
> > [was "Poor DBT-3 pgsql 8way numbers on recent 2.6 mm kernels" on
> > linux-mm]
> > 
> > Andrew,
> > 
> > This same patch (02) applied in STP (plm 2780) when run against
> > dbt3-pgsql DSS workload displays the performance problem with the
> > throughput numbers that I reported on linux-mm on our 8way systems,
> > where the previous patch (plm 2777 -01) does not.  
> > 
> > Here is the data (patches applied to 2.6.5-rc1)
> > 
> > PLM.....CPUs.Runid..Thruput Metric (bigger is better)
> > 2777(01)  8  290298  138.22  (base  )
> > 2779(02)  8  290304  88.57   (-35.9%)
> 
> 36% regression due to the CPU scheduler changes?  ow.
> 
> And that machine is a PIII, so presumably the setting of CONFIG_SCHED_SMT
> makes no difference.
> 
> >From a quick look at the material you have there it appears that this
> workload also is very I/O bound.  It's a little surprising that the CPU
> scheduler could make so much difference.
I'm not sure why you think this is IO bound. For 
the throughput phase of the test (from which the 
metric above is taken) there is very little physical 
IO except at the start when the updates occur.  They
finish in a few minutes, after which there is very
little.

http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat_io.png
http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat.txt
Perhaps you were looking at the start or at some other
part of the test?

The power test (single stream phase) does not display 
any performance hit at all compared to the baseline.
The throughput test runs eight streams (processes)
and does display the problem.  Furthermore the problem
is worse on 8 ways than on 4 ways.  It seems reasonable 
to me that this could be due to a task schedule issue.

Am I missing something?

> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Mary Edie Meredith 
maryedie@osdl.org
503-626-2455 x42
Open Source Development Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: 2.6.4-mm2

[Andrew Morton]

Mary Edie Meredith <maryedie@osdl.org> wrote:
>
> > 36% regression due to the CPU scheduler changes?  ow.
>  > 
>  > And that machine is a PIII, so presumably the setting of CONFIG_SCHED_SMT
>  > makes no difference.
>  > 
>  > >From a quick look at the material you have there it appears that this
>  > workload also is very I/O bound.  It's a little surprising that the CPU
>  > scheduler could make so much difference.
>  I'm not sure why you think this is IO bound. For 
>  the throughput phase of the test (from which the 
>  metric above is taken) there is very little physical 
>  IO except at the start when the updates occur.  They
>  finish in a few minutes, after which there is very
>  little.
> 
>  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat_io.png
>  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat.txt

There seems to be a large amount of idle time in the profiles and in the
vmstat trace.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: 2.6.4-mm2

[Mary Edie Meredith]

On Tue, 2004-03-23 at 11:32, Andrew Morton wrote:
> Mary Edie Meredith <maryedie@osdl.org> wrote:
> >
> > > 36% regression due to the CPU scheduler changes?  ow.
> >  > 
> >  > And that machine is a PIII, so presumably the setting of CONFIG_SCHED_SMT
> >  > makes no difference.
> >  > 
> >  > >From a quick look at the material you have there it appears that this
> >  > workload also is very I/O bound.  It's a little surprising that the CPU
> >  > scheduler could make so much difference.
> >  I'm not sure why you think this is IO bound. For 
> >  the throughput phase of the test (from which the 
> >  metric above is taken) there is very little physical 
> >  IO except at the start when the updates occur.  They
> >  finish in a few minutes, after which there is very
> >  little.
> > 
> >  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat_io.png
> >  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat.txt
> 
> There seems to be a large amount of idle time in the profiles and in the
> vmstat trace.
Yes.  There is considerably more idle time in the bad run:
Good one:
http://khack.osdl.org/stp/290298/results/plot/thuput.sar_cpu_all.png
Bad one:
http://khack.osdl.org/stp/290304/results/plot/thuput.sar_cpu_all.png

I am concerned with the drop in CPU utilization relative to
the other run.     

-- 
Mary Edie Meredith 
maryedie@osdl.org
503-626-2455 x42
Open Source Development Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: 2.6.4-mm2

[Mary Edie Meredith]

The 2.6.5-rc2-mm5 kernel has vastly improved the 
performance issue with dbt3-pgsql throughput numbers
(bigger is better):

Runid...Metric.PLM..Kernel........diff%
290357  141.84 2788 2.6.5-rc2.....  base 
290576   91.18 2814 2.6.5-rc2-mm2 -35.72
290856   60.02 2842 2.6.5-rc2-mm4 -57.68
290953  134.10 2849 2.6.5-rc2-mm5  -5.46 <-------

Thanks to Nick and Ingo. 

On Tue, 2004-03-23 at 16:07, Mary Edie Meredith wrote:
> On Tue, 2004-03-23 at 11:32, Andrew Morton wrote:
> > Mary Edie Meredith <maryedie@osdl.org> wrote:
> > >
> > > > 36% regression due to the CPU scheduler changes?  ow.
> > >  > 
> > >  > And that machine is a PIII, so presumably the setting of CONFIG_SCHED_SMT
> > >  > makes no difference.
> > >  > 
> > >  > >From a quick look at the material you have there it appears that this
> > >  > workload also is very I/O bound.  It's a little surprising that the CPU
> > >  > scheduler could make so much difference.
> > >  I'm not sure why you think this is IO bound. For 
> > >  the throughput phase of the test (from which the 
> > >  metric above is taken) there is very little physical 
> > >  IO except at the start when the updates occur.  They
> > >  finish in a few minutes, after which there is very
> > >  little.
> > > 
> > >  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat_io.png
> > >  http://khack.osdl.org/stp/290304/results/plot/thuput.vmstat.txt
> > 
> > There seems to be a large amount of idle time in the profiles and in the
> > vmstat trace.
> Yes.  There is considerably more idle time in the bad run:
> Good one:
> http://khack.osdl.org/stp/290298/results/plot/thuput.sar_cpu_all.png
> Bad one:
> http://khack.osdl.org/stp/290304/results/plot/thuput.sar_cpu_all.png
> 
> I am concerned with the drop in CPU utilization relative to
> the other run.     
-- 
Mary Edie Meredith 
maryedie@osdl.org
503-626-2455 x42
Open Source Development Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>