Re: Poor DBT-3 pgsql 8way numbers on recent 2.6 mm kernels

[Mary Edie Meredith <maryedie@osdl.org>]

Ram, it took a while to implement your suggestion.  Your
patch was again 2.6.3-rc1-mm1.  Unfortunately, the series
of mm kernels from  2.6.3-rc1-mm1 thru 2.6.3-mm3 failed
to run on STP.  

Judith made a patch (PLM 2766) by reverting
the patch you referenced below using 2.6.5-rc1-mm1 
(PLM 2760) as the original.  It compiled and ran without error.  The
performance did not significantly improve. So I think we can conclude
that readahead is not the problem.


Here is the data (bigger is better on metric):

PLM..Kernel........Runid..CPUs..Thruput Metric 
2760 2.6.5-rc1-mm1 290149   8   86.82  
2766 2.6.5-rc1-mm1*290197   8   88.70 *(rev-readahead)
2760 2.6.5-rc1-mm1 290120   4   114.41 (worse than 8)
2757 2.6.5-rc1     290064   4   122.74  (baseline 4way) 
(8way run on 2.6.5-rc1 hasn't completed yet)
2679 2.6.4 base    289421   8   137.2   (baseline 8way)

Meantime I attempted to do a binary search to 
find the point where the mm kernel performance
went bad.  It unfortunately appears to have
occurred during the period of time that the
mm kernels did not run on STP:

(These are all 8-way results)
PLM..Kernel........RUNID...Thruput Metric
2656 2.6.3-mm4     288850   87.82
[2.6.3-rc1-mm1 thru 2.6.3-mm3 fail to run on STP]
2603 2.6.2-mm1     290003   115.24
2582 2.6.2-rc2-mm1 290005   115.85
2564 2.6.1-mm5     289381   124.02

So there is a little hit between 2.6.1-mm5 and 2.6.2-rc2-mm1
but a very big hit between 2.6.2.mm1 and 2.6.3-mm4.

Cliff is on vacation so it may take me a while to 
track down patches to rix 2.6.3 mm kernels.  I see 
a patch he tried with reaim on 2.6.3-mm1 (PLM2654) 
so I'll give that a try.

It may take a while but I'll report back.  

Another thing I may not have mentioned before is that
we use LVM in this workload.  We are also using LVM for
our dbt2 (OLTP) postgreSQL workload.  Markm is doing
some runs to see how the latest mm kernel compares 
with baseline.  

Thanks.

On Mon, 2004-03-15 at 11:33, Ram Pai wrote:
> On Mon, 2004-03-15 at 08:45, Mary Edie Meredith wrote:
>  
> > 
> > 
> > > And if that is indeed the case I'd be suspecting the CPU scheduler.  But
> > > then, Meredith's profiles show almost completely idle CPUs.
> > > 
> > > The simplest way to hunt this down is the old binary-search-through-the-patches process.  But that requires some test which takes just a few minutes.
> > 
> > If you are referring to a binary search to find when the
> > performance changed, I can do this with STP.  It may take 
> > some time, but I'm willing.  I didnt want to do that if 
> > the problem was a known problem.  
> 
> Based on your data, I dont think readahead patch is responsible. However
> since you are seeing this only on mm kernel there is a small needle of
> suspicion on the readahead patch.
> 
> How about reverting only the readahaed patch in mm tree and trying it
> out? 
> 
> http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.3-rc1/2.6.3-rc1-mm1/broken-out/adaptive-lazy-readahead.patch
> 
> My DSS workload benchmarks always touches the disk because I have only
> 4GB memory configured. I will give a try with 8GB memory and see if I
> see any of your behavior. (I wont be able to put all my database in
> memory)...
> 
> RP
> 
> 
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>
> > 
-- 
Mary Edie Meredith 
maryedie@osdl.org
503-626-2455 x42
Open Source Development Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>