From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with SMTP id 1527F6B016A for ; Mon, 22 Aug 2011 16:28:10 -0400 (EDT) Message-ID: Content-Type: multipart/alternative; boundary="_c058eb12-408f-4944-b239-23bdedb827c3_" From: Mark Petersen Subject: RE: [Bugme-new] [Bug 41552] New: Performance of writing and reading from multiple drives decreases by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond) Date: Mon, 22 Aug 2011 15:28:08 -0500 In-Reply-To: <20110822195651.GB15087@redhat.com> References: ,<20110822122443.c04839c8.akpm@linux-foundation.org>,,<20110822195651.GB15087@redhat.com> MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: vgoyal@redhat.com Cc: akpm@linux-foundation.org, bugme-daemon@bugzilla.kernel.org, axboe@kernel.dk, linux-mm@kvack.org, linux-scsi@vger.kernel.org --_c058eb12-408f-4944-b239-23bdedb827c3_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable The writes we are performing are SCSI commands directly to the device=2C on= e sector at a time. The only thing we changed between our updates was the = Kernel itself=2C which we leave everything in there at its default value if= it is enabled (we disable a great many things we don't need). The latest = version I tried that still showed the issue was v3.0.1. Thanks=2C Mark > Date: Mon=2C 22 Aug 2011 15:56:51 -0400 > From: vgoyal@redhat.com > To: mpete_06@hotmail.com > CC: akpm@linux-foundation.org=3B bugme-daemon@bugzilla.kernel.org=3B axbo= e@kernel.dk=3B linux-mm@kvack.org=3B linux-scsi@vger.kernel.org > Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and read= ing from multiple drives decreases by 40% when going from Linux Kernel 2.6.= 36.4 to 2.6.37 (and beyond) >=20 > On Mon=2C Aug 22=2C 2011 at 02:49:56PM -0500=2C Mark Petersen wrote: > >=20 > > The majority of the slowdown we found is coming during the writing as w= e were doing limited reading for the purpose of the testing. It may be tha= t it happens in both areas=2C but we did not do extensive testing with the = reading portion of it. >=20 > What kind of writes these are? Write slowdown by 40%. Somehow now a days > barriers/flush/fua comes to my mind. Any changes there w.r.t your setup? >=20 > Recently Jeff moyer and Mike Snitzer had discovered and fixed a slowdown > in a dm-multipath and disks not having write caches. I guess that's not > your setup. Though mentioning it does not harm. >=20 > Thanks > Vivek >=20 > =20 > >=20 > > > Date: Mon=2C 22 Aug 2011 12:24:43 -0700 > > > From: akpm@linux-foundation.org > > > To: mpete_06@hotmail.com > > > CC: bugme-daemon@bugzilla.kernel.org=3B axboe@kernel.dk=3B vgoyal@red= hat.com=3B linux-mm@kvack.org=3B linux-scsi@vger.kernel.org > > > Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and = reading from multiple drives decreases by 40% when going from Linux Kernel = 2.6.36.4 to 2.6.37 (and beyond) > > >=20 > > >=20 > > > (switched to email. Please respond via emailed reply-to-all=2C not v= ia the > > > bugzilla web interface). > > >=20 > > > On Mon=2C 22 Aug 2011 15:20:41 GMT > > > bugzilla-daemon@bugzilla.kernel.org wrote: > > >=20 > > > > https://bugzilla.kernel.org/show_bug.cgi?id=3D41552 > > > >=20 > > > > Summary: Performance of writing and reading from multipl= e > > > > drives decreases by 40% when going from Linux K= ernel > > > > 2.6.36.4 to 2.6.37 (and beyond) > > > > Product: IO/Storage > > > > Version: 2.5 > > > > Kernel Version: 2.6.37 > > > > Platform: All > > > > OS/Version: Linux > > > > Tree: Mainline > > > > Status: NEW > > > > Severity: normal > > > > Priority: P1 > > > > Component: SCSI > > > > AssignedTo: linux-scsi@vger.kernel.org > > > > ReportedBy: mpete_06@hotmail.com > > > > Regression: No > > > >=20 > > > >=20 > > > > We have an application that will write and read from every sector o= n a drive.=20 > > > > The application can perform these tasks on multiple drives at the s= ame time.=20 > > > > It is designed to run on top of the Linux Kernel=2C which we period= ically update > > > > so that we can get the latest device drivers. When performing the = last update > > > > from 2.6.33.2 to 2.6.37=2C we found that the performance of a set o= f drives > > > > decreased by some 40% (took 3 hours and 11 minutes to write and rea= d from 5 > > > > drives on 2.6.37 versus 2 hours and 12 minutes on 2.6.33.3). I was= able to > > > > determine that the issue was in the 2.6.37 Kernel as I was able to = run it with > > > > the 2.6.36.4 kernel=2C and it had the better performance. After s= eeing that I/O > > > > throttling was introduced in the 2.6.37 Kernel=2C I naturally suspe= cted that.=20 > > > > However=2C by default=2C all the throttling was turned off (I attac= hed the actual > > > > .config that was used to build the kernel). I then tried to turn o= n the > > > > throttling and set it to a high number to see what would happen. W= hen I did > > > > that=2C I was able to reduce the time from 3 hours and 11 minutes t= o 2 hours and > > > > 50 minutes. There seems to be something there that changed that is= impacting > > > > performance on multiple drives. When we do this same test with onl= y one drive=2C > > > > the performance is identical between the systems. This issue still= occurs on > > > > Kernel 3.0.2. > > > >=20 > > >=20 > > > Are you able to determine whether this regression is due to slower > > > reading=2C to slower writing or to both? > > >=20 > > > Thanks. > > =20 = --_c058eb12-408f-4944-b239-23bdedb827c3_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
The writes we are performing are SCSI commands directly to the device=2C on= e sector at a time. =3B The only thing we changed between our updates w= as the Kernel itself=2C which we leave everything in there at its default v= alue if it is enabled (we disable a great many things we don't need). = =3B The latest version I tried that still showed the issue was v3.0.1.
<= br>Thanks=2C
Mark

>=3B Date: Mon=2C 22 Aug 2011 15:56:51 -= 0400
>=3B From: vgoyal@redhat.com
>=3B To: mpete_06@hotmail.com>=3B CC: akpm@linux-foundation.org=3B bugme-daemon@bugzilla.kernel.org= =3B axboe@kernel.dk=3B linux-mm@kvack.org=3B linux-scsi@vger.kernel.org
= >=3B Subject: Re: [Bugme-new] [Bug 41552] New: Performance of writing and= reading from multiple drives decreases by 40% when going from Linux Kernel= 2.6.36.4 to 2.6.37 (and beyond)
>=3B
>=3B On Mon=2C Aug 22=2C 2= 011 at 02:49:56PM -0500=2C Mark Petersen wrote:
>=3B >=3B
>=3B= >=3B The majority of the slowdown we found is coming during the writing = as we were doing limited reading for the purpose of the testing. It may be= that it happens in both areas=2C but we did not do extensive testing with = the reading portion of it.
>=3B
>=3B What kind of writes these a= re? Write slowdown by 40%. Somehow now a days
>=3B barriers/flush/fua = comes to my mind. Any changes there w.r.t your setup?
>=3B
>=3B = Recently Jeff moyer and Mike Snitzer had discovered and fixed a slowdown>=3B in a dm-multipath and disks not having write caches. I guess that's= not
>=3B your setup. Though mentioning it does not harm.
>=3B >=3B Thanks
>=3B Vivek
>=3B
>=3B
>=3B >=3B >=3B >=3B >=3B Date: Mon=2C 22 Aug 2011 12:24:43 -0700
>=3B >= =3B >=3B From: akpm@linux-foundation.org
>=3B >=3B >=3B To: mpet= e_06@hotmail.com
>=3B >=3B >=3B CC: bugme-daemon@bugzilla.kernel.o= rg=3B axboe@kernel.dk=3B vgoyal@redhat.com=3B linux-mm@kvack.org=3B linux-s= csi@vger.kernel.org
>=3B >=3B >=3B Subject: Re: [Bugme-new] [Bug 4= 1552] New: Performance of writing and reading from multiple drives decrease= s by 40% when going from Linux Kernel 2.6.36.4 to 2.6.37 (and beyond)
&g= t=3B >=3B >=3B
>=3B >=3B >=3B
>=3B >=3B >=3B (switc= hed to email. Please respond via emailed reply-to-all=2C not via the
&g= t=3B >=3B >=3B bugzilla web interface).
>=3B >=3B >=3B
>= =3B >=3B >=3B On Mon=2C 22 Aug 2011 15:20:41 GMT
>=3B >=3B >= =3B bugzilla-daemon@bugzilla.kernel.org wrote:
>=3B >=3B >=3B
= >=3B >=3B >=3B >=3B https://bugzilla.kernel.org/show_bug.cgi?id=3D4= 1552
>=3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B = Summary: Performance of writing and reading from multiple
>=3B >= =3B >=3B >=3B drives decreases by 40% when going fr= om Linux Kernel
>=3B >=3B >=3B >=3B 2.6.36.4= to 2.6.37 (and beyond)
>=3B >=3B >=3B >=3B Product: = IO/Storage
>=3B >=3B >=3B >=3B Version: 2.5
>=3B= >=3B >=3B >=3B Kernel Version: 2.6.37
>=3B >=3B >=3B &g= t=3B Platform: All
>=3B >=3B >=3B >=3B OS/Vers= ion: Linux
>=3B >=3B >=3B >=3B Tree: Mainline
&= gt=3B >=3B >=3B >=3B Status: NEW
>=3B >=3B >=3B = >=3B Severity: normal
>=3B >=3B >=3B >=3B = Priority: P1
>=3B >=3B >=3B >=3B Component: SCSI
>= =3B >=3B >=3B >=3B AssignedTo: linux-scsi@vger.kernel.org
= >=3B >=3B >=3B >=3B ReportedBy: mpete_06@hotmail.com
>= =3B >=3B >=3B >=3B Regression: No
>=3B >=3B >=3B >= =3B
>=3B >=3B >=3B >=3B
>=3B >=3B >=3B >=3B We have= an application that will write and read from every sector on a drive.
= >=3B >=3B >=3B >=3B The application can perform these tasks on mult= iple drives at the same time.
>=3B >=3B >=3B >=3B It is designe= d to run on top of the Linux Kernel=2C which we periodically update
>= =3B >=3B >=3B >=3B so that we can get the latest device drivers. Whe= n performing the last update
>=3B >=3B >=3B >=3B from 2.6.33.2 t= o 2.6.37=2C we found that the performance of a set of drives
>=3B >= =3B >=3B >=3B decreased by some 40% (took 3 hours and 11 minutes to wri= te and read from 5
>=3B >=3B >=3B >=3B drives on 2.6.37 versus 2= hours and 12 minutes on 2.6.33.3). I was able to
>=3B >=3B >=3B = >=3B determine that the issue was in the 2.6.37 Kernel as I was able to r= un it with
>=3B >=3B >=3B >=3B the 2.6.36.4 kernel=2C and it had= the better performance. After seeing that I/O
>=3B >=3B >=3B &g= t=3B throttling was introduced in the 2.6.37 Kernel=2C I naturally suspecte= d that.
>=3B >=3B >=3B >=3B However=2C by default=2C all the th= rottling was turned off (I attached the actual
>=3B >=3B >=3B >= =3B .config that was used to build the kernel). I then tried to turn on th= e
>=3B >=3B >=3B >=3B throttling and set it to a high number to = see what would happen. When I did
>=3B >=3B >=3B >=3B that=2C I= was able to reduce the time from 3 hours and 11 minutes to 2 hours and
= >=3B >=3B >=3B >=3B 50 minutes. There seems to be something there = that changed that is impacting
>=3B >=3B >=3B >=3B performance o= n multiple drives. When we do this same test with only one drive=2C
>= =3B >=3B >=3B >=3B the performance is identical between the systems. = This issue still occurs on
>=3B >=3B >=3B >=3B Kernel 3.0.2.>=3B >=3B >=3B >=3B
>=3B >=3B >=3B
>=3B >=3B >= =3B Are you able to determine whether this regression is due to slower
&= gt=3B >=3B >=3B reading=2C to slower writing or to both?
>=3B >= =3B >=3B
>=3B >=3B >=3B Thanks.
>=3B >=3B
= --_c058eb12-408f-4944-b239-23bdedb827c3_-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org