Re: [PATCH] readahead:add blk_run_backing_dev

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

(cc to Wu and linux-mm)

> Hi.
> 
> I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
> so readahead I/O is unpluged to improve throughput.
> 
> Following is the test result with dd.
> 
> #dd if=testdir/testfile of=/dev/null bs=16384
> 
> -2.6.30-rc6
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> 
> -2.6.30-rc6-patched
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> 
> Sequential read performance on a big file was improved.
> Please merge my patch.

I guess the improvement depend on readahead window size.
Have you mesure random access workload?

> 
> Thanks.
> 
> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> 
> diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c linux-2.6.30-rc6.unplug/mm/readahead.c
> --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
> +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
> @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>  
>  	/* do read-ahead */
>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> +
> +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>  }
>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 10:07 09/05/20, KOSAKI Motohiro wrote:
>(cc to Wu and linux-mm)
>
>> Hi.
>> 
>> I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
>> so readahead I/O is unpluged to improve throughput.
>> 
>> Following is the test result with dd.
>> 
>> #dd if=testdir/testfile of=/dev/null bs=16384
>> 
>> -2.6.30-rc6
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> 
>> -2.6.30-rc6-patched
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> 
>> Sequential read performance on a big file was improved.
>> Please merge my patch.
>
>I guess the improvement depend on readahead window size.
>Have you mesure random access workload?

I tried with iozone. But there was no difference.

>
>> 
>> Thanks.
>> 
>> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
>> 
>> diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
>linux-2.6.30-rc6.unplug/mm/readahead.c
>> --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
>> +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
>> @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>>  
>>  	/* do read-ahead */
>>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> +
>> +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>>  }
>>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Mon, May 18, 2009 at 07:53:00PM +0200, Jens Axboe wrote:
> On Mon, May 18 2009, Hisashi Hifumi wrote:
> > Hi.
> > 
> > I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
> > so readahead I/O is unpluged to improve throughput.
> > 
> > Following is the test result with dd.
> > 
> > #dd if=testdir/testfile of=/dev/null bs=16384
> > 
> > -2.6.30-rc6
> > 1048576+0 records in
> > 1048576+0 records out
> > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> > 
> > -2.6.30-rc6-patched
> > 1048576+0 records in
> > 1048576+0 records out
> > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> > 
> > Sequential read performance on a big file was improved.
> > Please merge my patch.
> > 
> > Thanks.
> > 
> > Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> > 
> > diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c linux-2.6.30-rc6.unplug/mm/readahead.c
> > --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
> > +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
> > @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
> >  
> >  	/* do read-ahead */
> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> > +
> > +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >  }
> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> 
> I'm surprised this makes much of a difference. It seems correct to me to
> NOT unplug the device, since it will get unplugged when someone ends up
> actually waiting for a page. And that will then kick off the remaining
> IO as well. For this dd case, you'll be hitting lock_page() for the
> readahead page really soon, definitely not long enough to warrant such a
> big difference in speed.

The possible timing change of this patch is (assuming readahead size=100):

T0   read(100), which triggers readahead(200, 100)
T1   read(101)
T2   read(102)
...
T100 read(200), find_get_page(200) => readahead(300, 100)
                lock_page(200) => implicit unplug

The readahead(200, 100) submitted at time T0 *might* be delayed to the
unplug time of T100.

But that is only a possibility. In normal cases, the read(200) would
be blocking and there will be a lock_page(200) that will immediately
unplug device for readahead(300, 100).

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 20, 2009 at 09:43:18AM +0800, Hisashi Hifumi wrote:
> 
> At 10:07 09/05/20, KOSAKI Motohiro wrote:
> >(cc to Wu and linux-mm)
> >
> >> Hi.
> >> 
> >> I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
> >> so readahead I/O is unpluged to improve throughput.
> >> 
> >> Following is the test result with dd.
> >> 
> >> #dd if=testdir/testfile of=/dev/null bs=16384
> >> 
> >> -2.6.30-rc6
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> 
> >> -2.6.30-rc6-patched
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> 
> >> Sequential read performance on a big file was improved.
> >> Please merge my patch.
> >
> >I guess the improvement depend on readahead window size.
> >Have you mesure random access workload?
> 
> I tried with iozone. But there was no difference.

It does not impact random IO because the patch only modified the
*async* readahead path, and random IO is obviously *sync* ones.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:51 09/05/20, Wu Fengguang wrote:
>On Mon, May 18, 2009 at 07:53:00PM +0200, Jens Axboe wrote:
>> On Mon, May 18 2009, Hisashi Hifumi wrote:
>> > Hi.
>> > 
>> > I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
>> > so readahead I/O is unpluged to improve throughput.
>> > 
>> > Following is the test result with dd.
>> > 
>> > #dd if=testdir/testfile of=/dev/null bs=16384
>> > 
>> > -2.6.30-rc6
>> > 1048576+0 records in
>> > 1048576+0 records out
>> > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> > 
>> > -2.6.30-rc6-patched
>> > 1048576+0 records in
>> > 1048576+0 records out
>> > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> > 
>> > Sequential read performance on a big file was improved.
>> > Please merge my patch.
>> > 
>> > Thanks.
>> > 
>> > Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
>> > 
>> > diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
>linux-2.6.30-rc6.unplug/mm/readahead.c
>> > --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
>> > +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 
>13:00:42.000000000 +0900
>> > @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>> >  
>> >  	/* do read-ahead */
>> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> > +
>> > +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>> >  }
>> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
>> 
>> I'm surprised this makes much of a difference. It seems correct to me to
>> NOT unplug the device, since it will get unplugged when someone ends up
>> actually waiting for a page. And that will then kick off the remaining
>> IO as well. For this dd case, you'll be hitting lock_page() for the
>> readahead page really soon, definitely not long enough to warrant such a
>> big difference in speed.
>
>The possible timing change of this patch is (assuming readahead size=100):
>
>T0   read(100), which triggers readahead(200, 100)
>T1   read(101)
>T2   read(102)
>...
>T100 read(200), find_get_page(200) => readahead(300, 100)
>                lock_page(200) => implicit unplug
>
>The readahead(200, 100) submitted at time T0 *might* be delayed to the
>unplug time of T100.
>
>But that is only a possibility. In normal cases, the read(200) would
>be blocking and there will be a lock_page(200) that will immediately
>unplug device for readahead(300, 100).


Hi Andrew.
Following patch improves sequential read performance and does not harm
other performance.
Please merge my patch.
Comments?
Thanks.

#dd if=testdir/testfile of=/dev/null bs=16384
-2.6.30-rc6
1048576+0 records in
1048576+0 records out
17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s

-2.6.30-rc6-patched
1048576+0 records in
1048576+0 records out
17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s

Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>

diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c linux-2.6.30-rc6.unplug/mm/readahead.c
--- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
+++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
@@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
 
 	/* do read-ahead */
 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
+
+	blk_run_backing_dev(mapping->backing_dev_info, NULL);
 }
 EXPORT_SYMBOL_GPL(page_cache_async_readahead);



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Thu, May 21, 2009 at 02:01:47PM +0800, Hisashi Hifumi wrote:
> 
> At 11:51 09/05/20, Wu Fengguang wrote:
> >On Mon, May 18, 2009 at 07:53:00PM +0200, Jens Axboe wrote:
> >> On Mon, May 18 2009, Hisashi Hifumi wrote:
> >> > Hi.
> >> > 
> >> > I wrote a patch that adds blk_run_backing_dev on page_cache_async_readahead
> >> > so readahead I/O is unpluged to improve throughput.
> >> > 
> >> > Following is the test result with dd.
> >> > 
> >> > #dd if=testdir/testfile of=/dev/null bs=16384
> >> > 
> >> > -2.6.30-rc6
> >> > 1048576+0 records in
> >> > 1048576+0 records out
> >> > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> > 
> >> > -2.6.30-rc6-patched
> >> > 1048576+0 records in
> >> > 1048576+0 records out
> >> > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> > 
> >> > Sequential read performance on a big file was improved.
> >> > Please merge my patch.
> >> > 
> >> > Thanks.
> >> > 
> >> > Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> >> > 
> >> > diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
> >linux-2.6.30-rc6.unplug/mm/readahead.c
> >> > --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
> >> > +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 
> >13:00:42.000000000 +0900
> >> > @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
> >> >  
> >> >  	/* do read-ahead */
> >> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> >> > +
> >> > +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >> >  }
> >> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> >> 
> >> I'm surprised this makes much of a difference. It seems correct to me to
> >> NOT unplug the device, since it will get unplugged when someone ends up
> >> actually waiting for a page. And that will then kick off the remaining
> >> IO as well. For this dd case, you'll be hitting lock_page() for the
> >> readahead page really soon, definitely not long enough to warrant such a
> >> big difference in speed.
> >
> >The possible timing change of this patch is (assuming readahead size=100):
> >
> >T0   read(100), which triggers readahead(200, 100)
> >T1   read(101)
> >T2   read(102)
> >...
> >T100 read(200), find_get_page(200) => readahead(300, 100)
> >                lock_page(200) => implicit unplug
> >
> >The readahead(200, 100) submitted at time T0 *might* be delayed to the
> >unplug time of T100.
> >
> >But that is only a possibility. In normal cases, the read(200) would
> >be blocking and there will be a lock_page(200) that will immediately
> >unplug device for readahead(300, 100).
> 
> 
> Hi Andrew.
> Following patch improves sequential read performance and does not harm
> other performance.
> Please merge my patch.
> Comments?
> Thanks.
> 
> #dd if=testdir/testfile of=/dev/null bs=16384
> -2.6.30-rc6
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> 
> -2.6.30-rc6-patched
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> 
> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> 
> diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c linux-2.6.30-rc6.unplug/mm/readahead.c
> --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
> +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
> @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>  
>  	/* do read-ahead */
>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> +
> +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>  }
>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> 
> 

Hi Hisashi,

I wonder if the following updated patch can achieve the same
performance.  Can you try testing this out?

Thanks,
Fengguang
---

diff --git a/mm/readahead.c b/mm/readahead.c
index 133b6d5..fd3df66 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -490,5 +490,8 @@ page_cache_async_readahead(struct address_space *mapping,
 
 	/* do read-ahead */
 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
+
+	if (PageUptodate(page))
+		blk_run_backing_dev(mapping->backing_dev_info, NULL);		
 }
 EXPORT_SYMBOL_GPL(page_cache_async_readahead);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 10:05 09/05/22, Wu Fengguang wrote:
>On Thu, May 21, 2009 at 02:01:47PM +0800, Hisashi Hifumi wrote:
>> 
>> At 11:51 09/05/20, Wu Fengguang wrote:
>> >On Mon, May 18, 2009 at 07:53:00PM +0200, Jens Axboe wrote:
>> >> On Mon, May 18 2009, Hisashi Hifumi wrote:
>> >> > Hi.
>> >> > 
>> >> > I wrote a patch that adds blk_run_backing_dev on 
>page_cache_async_readahead
>> >> > so readahead I/O is unpluged to improve throughput.
>> >> > 
>> >> > Following is the test result with dd.
>> >> > 
>> >> > #dd if=testdir/testfile of=/dev/null bs=16384
>> >> > 
>> >> > -2.6.30-rc6
>> >> > 1048576+0 records in
>> >> > 1048576+0 records out
>> >> > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> >> > 
>> >> > -2.6.30-rc6-patched
>> >> > 1048576+0 records in
>> >> > 1048576+0 records out
>> >> > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> >> > 
>> >> > Sequential read performance on a big file was improved.
>> >> > Please merge my patch.
>> >> > 
>> >> > Thanks.
>> >> > 
>> >> > Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
>> >> > 
>> >> > diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
>> >linux-2.6.30-rc6.unplug/mm/readahead.c
>> >> > --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 
>10:46:15.000000000 +0900
>> >> > +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 
>> >13:00:42.000000000 +0900
>> >> > @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>> >> >  
>> >> >  	/* do read-ahead */
>> >> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> >> > +
>> >> > +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>> >> >  }
>> >> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
>> >> 
>> >> I'm surprised this makes much of a difference. It seems correct to me to
>> >> NOT unplug the device, since it will get unplugged when someone ends up
>> >> actually waiting for a page. And that will then kick off the remaining
>> >> IO as well. For this dd case, you'll be hitting lock_page() for the
>> >> readahead page really soon, definitely not long enough to warrant such a
>> >> big difference in speed.
>> >
>> >The possible timing change of this patch is (assuming readahead size=100):
>> >
>> >T0   read(100), which triggers readahead(200, 100)
>> >T1   read(101)
>> >T2   read(102)
>> >...
>> >T100 read(200), find_get_page(200) => readahead(300, 100)
>> >                lock_page(200) => implicit unplug
>> >
>> >The readahead(200, 100) submitted at time T0 *might* be delayed to the
>> >unplug time of T100.
>> >
>> >But that is only a possibility. In normal cases, the read(200) would
>> >be blocking and there will be a lock_page(200) that will immediately
>> >unplug device for readahead(300, 100).
>> 
>> 
>> Hi Andrew.
>> Following patch improves sequential read performance and does not harm
>> other performance.
>> Please merge my patch.
>> Comments?
>> Thanks.
>> 
>> #dd if=testdir/testfile of=/dev/null bs=16384
>> -2.6.30-rc6
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> 
>> -2.6.30-rc6-patched
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> 
>> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
>> 
>> diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
>linux-2.6.30-rc6.unplug/mm/readahead.c
>> --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
>> +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
>> @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
>>  
>>  	/* do read-ahead */
>>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> +
>> +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
>>  }
>>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
>> 
>> 
>
>Hi Hisashi,
>
>I wonder if the following updated patch can achieve the same
>performance.  Can you try testing this out?
>
>Thanks,
>Fengguang
>---
>
>diff --git a/mm/readahead.c b/mm/readahead.c
>index 133b6d5..fd3df66 100644
>--- a/mm/readahead.c
>+++ b/mm/readahead.c
>@@ -490,5 +490,8 @@ page_cache_async_readahead(struct address_space *mapping,
> 
> 	/* do read-ahead */
> 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>+
>+	if (PageUptodate(page))
>+		blk_run_backing_dev(mapping->backing_dev_info, NULL);		
> }
> EXPORT_SYMBOL_GPL(page_cache_async_readahead);

Hi.
I tested above patch, and I got same performance number.
I wonder why if (PageUptodate(page)) check is there...


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Fri, May 22, 2009 at 09:44:59AM +0800, Hisashi Hifumi wrote:
> 
> At 10:05 09/05/22, Wu Fengguang wrote:
> >On Thu, May 21, 2009 at 02:01:47PM +0800, Hisashi Hifumi wrote:
> >> 
> >> At 11:51 09/05/20, Wu Fengguang wrote:
> >> >On Mon, May 18, 2009 at 07:53:00PM +0200, Jens Axboe wrote:
> >> >> On Mon, May 18 2009, Hisashi Hifumi wrote:
> >> >> > Hi.
> >> >> > 
> >> >> > I wrote a patch that adds blk_run_backing_dev on 
> >page_cache_async_readahead
> >> >> > so readahead I/O is unpluged to improve throughput.
> >> >> > 
> >> >> > Following is the test result with dd.
> >> >> > 
> >> >> > #dd if=testdir/testfile of=/dev/null bs=16384
> >> >> > 
> >> >> > -2.6.30-rc6
> >> >> > 1048576+0 records in
> >> >> > 1048576+0 records out
> >> >> > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> >> > 
> >> >> > -2.6.30-rc6-patched
> >> >> > 1048576+0 records in
> >> >> > 1048576+0 records out
> >> >> > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> >> > 
> >> >> > Sequential read performance on a big file was improved.
> >> >> > Please merge my patch.
> >> >> > 
> >> >> > Thanks.
> >> >> > 
> >> >> > Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> >> >> > 
> >> >> > diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
> >> >linux-2.6.30-rc6.unplug/mm/readahead.c
> >> >> > --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 
> >10:46:15.000000000 +0900
> >> >> > +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 
> >> >13:00:42.000000000 +0900
> >> >> > @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
> >> >> >  
> >> >> >  	/* do read-ahead */
> >> >> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> >> >> > +
> >> >> > +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >> >> >  }
> >> >> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> >> >> 
> >> >> I'm surprised this makes much of a difference. It seems correct to me to
> >> >> NOT unplug the device, since it will get unplugged when someone ends up
> >> >> actually waiting for a page. And that will then kick off the remaining
> >> >> IO as well. For this dd case, you'll be hitting lock_page() for the
> >> >> readahead page really soon, definitely not long enough to warrant such a
> >> >> big difference in speed.
> >> >
> >> >The possible timing change of this patch is (assuming readahead size=100):
> >> >
> >> >T0   read(100), which triggers readahead(200, 100)
> >> >T1   read(101)
> >> >T2   read(102)
> >> >...
> >> >T100 read(200), find_get_page(200) => readahead(300, 100)
> >> >                lock_page(200) => implicit unplug
> >> >
> >> >The readahead(200, 100) submitted at time T0 *might* be delayed to the
> >> >unplug time of T100.
> >> >
> >> >But that is only a possibility. In normal cases, the read(200) would
> >> >be blocking and there will be a lock_page(200) that will immediately
> >> >unplug device for readahead(300, 100).
> >> 
> >> 
> >> Hi Andrew.
> >> Following patch improves sequential read performance and does not harm
> >> other performance.
> >> Please merge my patch.
> >> Comments?
> >> Thanks.
> >> 
> >> #dd if=testdir/testfile of=/dev/null bs=16384
> >> -2.6.30-rc6
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> 
> >> -2.6.30-rc6-patched
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> 
> >> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> >> 
> >> diff -Nrup linux-2.6.30-rc6.org/mm/readahead.c 
> >linux-2.6.30-rc6.unplug/mm/readahead.c
> >> --- linux-2.6.30-rc6.org/mm/readahead.c	2009-05-18 10:46:15.000000000 +0900
> >> +++ linux-2.6.30-rc6.unplug/mm/readahead.c	2009-05-18 13:00:42.000000000 +0900
> >> @@ -490,5 +490,7 @@ page_cache_async_readahead(struct addres
> >>  
> >>  	/* do read-ahead */
> >>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> >> +
> >> +	blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >>  }
> >>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> >> 
> >> 
> >
> >Hi Hisashi,
> >
> >I wonder if the following updated patch can achieve the same
> >performance.  Can you try testing this out?
> >
> >Thanks,
> >Fengguang
> >---
> >
> >diff --git a/mm/readahead.c b/mm/readahead.c
> >index 133b6d5..fd3df66 100644
> >--- a/mm/readahead.c
> >+++ b/mm/readahead.c
> >@@ -490,5 +490,8 @@ page_cache_async_readahead(struct address_space *mapping,
> > 
> > 	/* do read-ahead */
> > 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> >+
> >+	if (PageUptodate(page))
> >+		blk_run_backing_dev(mapping->backing_dev_info, NULL);		
> > }
> > EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> 
> Hi.
> I tested above patch, and I got same performance number.
> I wonder why if (PageUptodate(page)) check is there...

Thanks!  This is an interesting micro timing behavior that
demands some research work.  The above check is to confirm if it's
the PageUptodate() case that makes the difference. So why that case
happens so frequently so as to impact the performance? Will it also
happen in NFS?

The problem is readahead IO pipeline is not running smoothly, which is
undesirable and not well understood for now.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Andrew Morton]

On Fri, 22 May 2009 10:33:23 +0800
Wu Fengguang <fengguang.wu@intel.com> wrote:

> > I tested above patch, and I got same performance number.
> > I wonder why if (PageUptodate(page)) check is there...
> 
> Thanks!  This is an interesting micro timing behavior that
> demands some research work.  The above check is to confirm if it's
> the PageUptodate() case that makes the difference. So why that case
> happens so frequently so as to impact the performance? Will it also
> happen in NFS?
> 
> The problem is readahead IO pipeline is not running smoothly, which is
> undesirable and not well understood for now.

The patch causes a remarkably large performance increase.  A 9%
reduction in time for a linear read?  I'd be surprised if the workload
even consumed 9% of a CPU, so where on earth has the kernel gone to?

Have you been able to reproduce this in your testing?

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 08:42 09/05/27, Andrew Morton wrote:
>On Fri, 22 May 2009 10:33:23 +0800
>Wu Fengguang <fengguang.wu@intel.com> wrote:
>
>> > I tested above patch, and I got same performance number.
>> > I wonder why if (PageUptodate(page)) check is there...
>> 
>> Thanks!  This is an interesting micro timing behavior that
>> demands some research work.  The above check is to confirm if it's
>> the PageUptodate() case that makes the difference. So why that case
>> happens so frequently so as to impact the performance? Will it also
>> happen in NFS?
>> 
>> The problem is readahead IO pipeline is not running smoothly, which is
>> undesirable and not well understood for now.
>
>The patch causes a remarkably large performance increase.  A 9%
>reduction in time for a linear read? I'd be surprised if the workload

Hi Andrew.
Yes, I tested this with dd.

>even consumed 9% of a CPU, so where on earth has the kernel gone to?
>
>Have you been able to reproduce this in your testing?

Yes, this test on my environment is reproducible.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 07:42:52AM +0800, Andrew Morton wrote:
> On Fri, 22 May 2009 10:33:23 +0800
> Wu Fengguang <fengguang.wu@intel.com> wrote:
> 
> > > I tested above patch, and I got same performance number.
> > > I wonder why if (PageUptodate(page)) check is there...
> > 
> > Thanks!  This is an interesting micro timing behavior that
> > demands some research work.  The above check is to confirm if it's
> > the PageUptodate() case that makes the difference. So why that case
> > happens so frequently so as to impact the performance? Will it also
> > happen in NFS?
> > 
> > The problem is readahead IO pipeline is not running smoothly, which is
> > undesirable and not well understood for now.
> 
> The patch causes a remarkably large performance increase.  A 9%
> reduction in time for a linear read?  I'd be surprised if the workload
> even consumed 9% of a CPU, so where on earth has the kernel gone to?
> 
> Have you been able to reproduce this in your testing?

No I cannot reproduce it on raw partition and ext4fs.

The commands I run:

        # echo 1 > /proc/sys/vm/drop_caches
        # dd if=/dev/sda1 of=/dev/null bs=16384 count=100000 # sda1 is not mounted

The results are almost identical:

before:
        1638400000 bytes (1.6 GB) copied, 31.3073 s, 52.3 MB/s
        1638400000 bytes (1.6 GB) copied, 31.3393 s, 52.3 MB/s
after:
        1638400000 bytes (1.6 GB) copied, 31.3216 s, 52.3 MB/s
        1638400000 bytes (1.6 GB) copied, 31.3762 s, 52.2 MB/s

My kernel is
        Linux hp 2.6.30-rc6 #281 SMP Wed May 27 09:32:37 CST 2009 x86_64 GNU/Linux

The readahead size is the default one:
        # blockdev --getra  /dev/sda    
        256

I tried another ext4 directory with many ~100MB files(vmlinux-2.6.*) in it:

        # time tar cf - /hp/boot | cat > /dev/null

before:
        tar cf - /hp/boot  0.22s user 5.63s system 21% cpu 26.750 total
        tar cf - /hp/boot  0.26s user 5.53s system 21% cpu 26.620 total
after:
        tar cf - /hp/boot  0.18s user 5.57s system 21% cpu 26.719 total
        tar cf - /hp/boot  0.22s user 5.32s system 21% cpu 26.321 total

Another round with 1MB readahead size:

before:
        tar cf - /hp/boot  0.24s user 4.70s system 19% cpu 25.689 total
        tar cf - /hp/boot  0.22s user 4.99s system 20% cpu 25.634 total
after:
        tar cf - /hp/boot  0.18s user 4.89s system 19% cpu 25.599 total
        tar cf - /hp/boot  0.18s user 4.97s system 20% cpu 25.645 total

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> 
> At 08:42 09/05/27, Andrew Morton wrote:
> >On Fri, 22 May 2009 10:33:23 +0800
> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >
> >> > I tested above patch, and I got same performance number.
> >> > I wonder why if (PageUptodate(page)) check is there...
> >> 
> >> Thanks!  This is an interesting micro timing behavior that
> >> demands some research work.  The above check is to confirm if it's
> >> the PageUptodate() case that makes the difference. So why that case
> >> happens so frequently so as to impact the performance? Will it also
> >> happen in NFS?
> >> 
> >> The problem is readahead IO pipeline is not running smoothly, which is
> >> undesirable and not well understood for now.
> >
> >The patch causes a remarkably large performance increase.  A 9%
> >reduction in time for a linear read? I'd be surprised if the workload
> 
> Hi Andrew.
> Yes, I tested this with dd.
> 
> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >
> >Have you been able to reproduce this in your testing?
> 
> Yes, this test on my environment is reproducible.

Hisashi, does your environment have some special configurations?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:09 09/05/27, Wu Fengguang wrote:
>On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> 
>> At 08:42 09/05/27, Andrew Morton wrote:
>> >On Fri, 22 May 2009 10:33:23 +0800
>> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >
>> >> > I tested above patch, and I got same performance number.
>> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> 
>> >> Thanks!  This is an interesting micro timing behavior that
>> >> demands some research work.  The above check is to confirm if it's
>> >> the PageUptodate() case that makes the difference. So why that case
>> >> happens so frequently so as to impact the performance? Will it also
>> >> happen in NFS?
>> >> 
>> >> The problem is readahead IO pipeline is not running smoothly, which is
>> >> undesirable and not well understood for now.
>> >
>> >The patch causes a remarkably large performance increase.  A 9%
>> >reduction in time for a linear read? I'd be surprised if the workload
>> 
>> Hi Andrew.
>> Yes, I tested this with dd.
>> 
>> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
>> >
>> >Have you been able to reproduce this in your testing?
>> 
>> Yes, this test on my environment is reproducible.
>
>Hisashi, does your environment have some special configurations?

Hi.
My testing environment is as follows:
Hardware: HP DL580 
CPU:Xeon 3.2GHz *4 HT enabled
Memory:8GB
Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)

I did dd to this disk-array and got improved performance number.

I noticed that when a disk is just one HDD, performance improvement
is very small.

Thanks.




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >
> >> >Have you been able to reproduce this in your testing?
> >> 
> >> Yes, this test on my environment is reproducible.
> >
> >Hisashi, does your environment have some special configurations?
> 
> Hi.
> My testing environment is as follows:
> Hardware: HP DL580 
> CPU:Xeon 3.2GHz *4 HT enabled
> Memory:8GB
> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> 
> I did dd to this disk-array and got improved performance number.
> 
> I noticed that when a disk is just one HDD, performance improvement
> is very small.

thas's odd.

Why your patch depend on transfer rate difference?





--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Andrew Morton]

On Wed, 27 May 2009 11:21:53 +0900 Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp> wrote:

> 
> At 11:09 09/05/27, Wu Fengguang wrote:
> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >> 
> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >
> >> >> > I tested above patch, and I got same performance number.
> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >> 
> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> demands some research work.  The above check is to confirm if it's
> >> >> the PageUptodate() case that makes the difference. So why that case
> >> >> happens so frequently so as to impact the performance? Will it also
> >> >> happen in NFS?
> >> >> 
> >> >> The problem is readahead IO pipeline is not running smoothly, which is
> >> >> undesirable and not well understood for now.
> >> >
> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >reduction in time for a linear read? I'd be surprised if the workload
> >> 
> >> Hi Andrew.
> >> Yes, I tested this with dd.
> >> 
> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >
> >> >Have you been able to reproduce this in your testing?
> >> 
> >> Yes, this test on my environment is reproducible.
> >
> >Hisashi, does your environment have some special configurations?
> 
> Hi.
> My testing environment is as follows:
> Hardware: HP DL580 
> CPU:Xeon 3.2GHz *4 HT enabled
> Memory:8GB
> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> 
> I did dd to this disk-array and got improved performance number.
> 
> I noticed that when a disk is just one HDD, performance improvement
> is very small.
> 

Ah.  So it's likely to be some strange interaction with the RAID setup.

I assume that you're using the SANNet 2's "hardware raid"?  Or is the
array set up as jbod and you're using kernel raid0?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
>
> At 11:09 09/05/27, Wu Fengguang wrote:
> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >>
> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >
> >> >> > I tested above patch, and I got same performance number.
> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >>
> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> demands some research work.  The above check is to confirm if it's
> >> >> the PageUptodate() case that makes the difference. So why that case
> >> >> happens so frequently so as to impact the performance? Will it also
> >> >> happen in NFS?
> >> >>
> >> >> The problem is readahead IO pipeline is not running smoothly, which is
> >> >> undesirable and not well understood for now.
> >> >
> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >reduction in time for a linear read? I'd be surprised if the workload
> >>
> >> Hi Andrew.
> >> Yes, I tested this with dd.
> >>
> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >
> >> >Have you been able to reproduce this in your testing?
> >>
> >> Yes, this test on my environment is reproducible.
> >
> >Hisashi, does your environment have some special configurations?
>
> Hi.
> My testing environment is as follows:
> Hardware: HP DL580
> CPU:Xeon 3.2GHz *4 HT enabled
> Memory:8GB
> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)

This is a big hardware RAID. What's the readahead size?

The numbers look too small for a 7 disk RAID:

        > #dd if=testdir/testfile of=/dev/null bs=16384
        >
        > -2.6.30-rc6
        > 1048576+0 records in
        > 1048576+0 records out
        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
        >
        > -2.6.30-rc6-patched
        > 1048576+0 records in
        > 1048576+0 records out
        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s

I'd suggest you to configure the array properly before coming back to
measuring the impact of this patch.

> I did dd to this disk-array and got improved performance number.
>
> I noticed that when a disk is just one HDD, performance improvement
> is very small.

OK. You should mention the single disk and RAID performance earlier.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:36 09/05/27, Andrew Morton wrote:
>On Wed, 27 May 2009 11:21:53 +0900 Hisashi Hifumi 
><hifumi.hisashi@oss.ntt.co.jp> wrote:
>
>> 
>> At 11:09 09/05/27, Wu Fengguang wrote:
>> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> >> 
>> >> At 08:42 09/05/27, Andrew Morton wrote:
>> >> >On Fri, 22 May 2009 10:33:23 +0800
>> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> >
>> >> >> > I tested above patch, and I got same performance number.
>> >> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> >> 
>> >> >> Thanks!  This is an interesting micro timing behavior that
>> >> >> demands some research work.  The above check is to confirm if it's
>> >> >> the PageUptodate() case that makes the difference. So why that case
>> >> >> happens so frequently so as to impact the performance? Will it also
>> >> >> happen in NFS?
>> >> >> 
>> >> >> The problem is readahead IO pipeline is not running smoothly, which is
>> >> >> undesirable and not well understood for now.
>> >> >
>> >> >The patch causes a remarkably large performance increase.  A 9%
>> >> >reduction in time for a linear read? I'd be surprised if the workload
>> >> 
>> >> Hi Andrew.
>> >> Yes, I tested this with dd.
>> >> 
>> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
>> >> >
>> >> >Have you been able to reproduce this in your testing?
>> >> 
>> >> Yes, this test on my environment is reproducible.
>> >
>> >Hisashi, does your environment have some special configurations?
>> 
>> Hi.
>> My testing environment is as follows:
>> Hardware: HP DL580 
>> CPU:Xeon 3.2GHz *4 HT enabled
>> Memory:8GB
>> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>> 
>> I did dd to this disk-array and got improved performance number.
>> 
>> I noticed that when a disk is just one HDD, performance improvement
>> is very small.
>> 
>
>Ah.  So it's likely to be some strange interaction with the RAID setup.
>
>I assume that you're using the SANNet 2's "hardware raid"?  Or is the
>array set up as jbod and you're using kernel raid0?

I used SANNet 2's "hardware raid".

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:36 09/05/27, Wu Fengguang wrote:
>On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
>>
>> At 11:09 09/05/27, Wu Fengguang wrote:
>> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> >>
>> >> At 08:42 09/05/27, Andrew Morton wrote:
>> >> >On Fri, 22 May 2009 10:33:23 +0800
>> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> >
>> >> >> > I tested above patch, and I got same performance number.
>> >> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> >>
>> >> >> Thanks!  This is an interesting micro timing behavior that
>> >> >> demands some research work.  The above check is to confirm if it's
>> >> >> the PageUptodate() case that makes the difference. So why that case
>> >> >> happens so frequently so as to impact the performance? Will it also
>> >> >> happen in NFS?
>> >> >>
>> >> >> The problem is readahead IO pipeline is not running smoothly, which is
>> >> >> undesirable and not well understood for now.
>> >> >
>> >> >The patch causes a remarkably large performance increase.  A 9%
>> >> >reduction in time for a linear read? I'd be surprised if the workload
>> >>
>> >> Hi Andrew.
>> >> Yes, I tested this with dd.
>> >>
>> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
>> >> >
>> >> >Have you been able to reproduce this in your testing?
>> >>
>> >> Yes, this test on my environment is reproducible.
>> >
>> >Hisashi, does your environment have some special configurations?
>>
>> Hi.
>> My testing environment is as follows:
>> Hardware: HP DL580
>> CPU:Xeon 3.2GHz *4 HT enabled
>> Memory:8GB
>> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>
>This is a big hardware RAID. What's the readahead size?
>
>The numbers look too small for a 7 disk RAID:
>
>        > #dd if=testdir/testfile of=/dev/null bs=16384
>        >
>        > -2.6.30-rc6
>        > 1048576+0 records in
>        > 1048576+0 records out
>        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>        >
>        > -2.6.30-rc6-patched
>        > 1048576+0 records in
>        > 1048576+0 records out
>        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>
>I'd suggest you to configure the array properly before coming back to
>measuring the impact of this patch.


I created 16GB file to this disk array, and mounted to testdir, dd to this directory.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
> 
> At 11:36 09/05/27, Wu Fengguang wrote:
> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
> >>
> >> At 11:09 09/05/27, Wu Fengguang wrote:
> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >> >>
> >> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >> >
> >> >> >> > I tested above patch, and I got same performance number.
> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >> >>
> >> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> >> demands some research work.  The above check is to confirm if it's
> >> >> >> the PageUptodate() case that makes the difference. So why that case
> >> >> >> happens so frequently so as to impact the performance? Will it also
> >> >> >> happen in NFS?
> >> >> >>
> >> >> >> The problem is readahead IO pipeline is not running smoothly, which is
> >> >> >> undesirable and not well understood for now.
> >> >> >
> >> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >> >reduction in time for a linear read? I'd be surprised if the workload
> >> >>
> >> >> Hi Andrew.
> >> >> Yes, I tested this with dd.
> >> >>
> >> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >> >
> >> >> >Have you been able to reproduce this in your testing?
> >> >>
> >> >> Yes, this test on my environment is reproducible.
> >> >
> >> >Hisashi, does your environment have some special configurations?
> >>
> >> Hi.
> >> My testing environment is as follows:
> >> Hardware: HP DL580
> >> CPU:Xeon 3.2GHz *4 HT enabled
> >> Memory:8GB
> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> >
> >This is a big hardware RAID. What's the readahead size?
> >
> >The numbers look too small for a 7 disk RAID:
> >
> >        > #dd if=testdir/testfile of=/dev/null bs=16384
> >        >
> >        > -2.6.30-rc6
> >        > 1048576+0 records in
> >        > 1048576+0 records out
> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >        >
> >        > -2.6.30-rc6-patched
> >        > 1048576+0 records in
> >        > 1048576+0 records out
> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >
> >I'd suggest you to configure the array properly before coming back to
> >measuring the impact of this patch.
> 
> 
> I created 16GB file to this disk array, and mounted to testdir, dd to this directory.

I mean, you should get >300MB/s throughput with 7 disks, and you
should seek ways to achieve that before testing out this patch :-)

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:57 09/05/27, Wu Fengguang wrote:
>On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
>> 
>> At 11:36 09/05/27, Wu Fengguang wrote:
>> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
>> >>
>> >> At 11:09 09/05/27, Wu Fengguang wrote:
>> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> >> >>
>> >> >> At 08:42 09/05/27, Andrew Morton wrote:
>> >> >> >On Fri, 22 May 2009 10:33:23 +0800
>> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> >> >
>> >> >> >> > I tested above patch, and I got same performance number.
>> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> >> >>
>> >> >> >> Thanks!  This is an interesting micro timing behavior that
>> >> >> >> demands some research work.  The above check is to confirm if it's
>> >> >> >> the PageUptodate() case that makes the difference. So why that case
>> >> >> >> happens so frequently so as to impact the performance? Will it also
>> >> >> >> happen in NFS?
>> >> >> >>
>> >> >> >> The problem is readahead IO pipeline is not running smoothly, which is
>> >> >> >> undesirable and not well understood for now.
>> >> >> >
>> >> >> >The patch causes a remarkably large performance increase.  A 9%
>> >> >> >reduction in time for a linear read? I'd be surprised if the workload
>> >> >>
>> >> >> Hi Andrew.
>> >> >> Yes, I tested this with dd.
>> >> >>
>> >> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
>> >> >> >
>> >> >> >Have you been able to reproduce this in your testing?
>> >> >>
>> >> >> Yes, this test on my environment is reproducible.
>> >> >
>> >> >Hisashi, does your environment have some special configurations?
>> >>
>> >> Hi.
>> >> My testing environment is as follows:
>> >> Hardware: HP DL580
>> >> CPU:Xeon 3.2GHz *4 HT enabled
>> >> Memory:8GB
>> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>> >
>> >This is a big hardware RAID. What's the readahead size?
>> >
>> >The numbers look too small for a 7 disk RAID:
>> >
>> >        > #dd if=testdir/testfile of=/dev/null bs=16384
>> >        >
>> >        > -2.6.30-rc6
>> >        > 1048576+0 records in
>> >        > 1048576+0 records out
>> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> >        >
>> >        > -2.6.30-rc6-patched
>> >        > 1048576+0 records in
>> >        > 1048576+0 records out
>> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> >
>> >I'd suggest you to configure the array properly before coming back to
>> >measuring the impact of this patch.
>> 
>> 
>> I created 16GB file to this disk array, and mounted to testdir, dd to 
>this directory.
>
>I mean, you should get >300MB/s throughput with 7 disks, and you
>should seek ways to achieve that before testing out this patch :-)

Throughput number of storage array is very from one product to another.
On my hardware environment I think this number is valid and
my patch is effective.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> >> >The numbers look too small for a 7 disk RAID:
> >> >
> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
> >> >        >
> >> >        > -2.6.30-rc6
> >> >        > 1048576+0 records in
> >> >        > 1048576+0 records out
> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> >        >
> >> >        > -2.6.30-rc6-patched
> >> >        > 1048576+0 records in
> >> >        > 1048576+0 records out
> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> >
> >> >I'd suggest you to configure the array properly before coming back to
> >> >measuring the impact of this patch.
> >> 
> >> 
> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
> >this directory.
> >
> >I mean, you should get >300MB/s throughput with 7 disks, and you
> >should seek ways to achieve that before testing out this patch :-)
> 
> Throughput number of storage array is very from one product to another.
> On my hardware environment I think this number is valid and
> my patch is effective.

Hifumi-san, if you really want to merge, you should reproduce this
issue on typical hardware, I think.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 10:36:01AM +0800, Andrew Morton wrote:
> On Wed, 27 May 2009 11:21:53 +0900 Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp> wrote:
> 
> > 
> > At 11:09 09/05/27, Wu Fengguang wrote:
> > >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> > >> 
> > >> At 08:42 09/05/27, Andrew Morton wrote:
> > >> >On Fri, 22 May 2009 10:33:23 +0800
> > >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> > >> >
> > >> >> > I tested above patch, and I got same performance number.
> > >> >> > I wonder why if (PageUptodate(page)) check is there...
> > >> >> 
> > >> >> Thanks!  This is an interesting micro timing behavior that
> > >> >> demands some research work.  The above check is to confirm if it's
> > >> >> the PageUptodate() case that makes the difference. So why that case
> > >> >> happens so frequently so as to impact the performance? Will it also
> > >> >> happen in NFS?
> > >> >> 
> > >> >> The problem is readahead IO pipeline is not running smoothly, which is
> > >> >> undesirable and not well understood for now.
> > >> >
> > >> >The patch causes a remarkably large performance increase.  A 9%
> > >> >reduction in time for a linear read? I'd be surprised if the workload
> > >> 
> > >> Hi Andrew.
> > >> Yes, I tested this with dd.
> > >> 
> > >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> > >> >
> > >> >Have you been able to reproduce this in your testing?
> > >> 
> > >> Yes, this test on my environment is reproducible.
> > >
> > >Hisashi, does your environment have some special configurations?
> > 
> > Hi.
> > My testing environment is as follows:
> > Hardware: HP DL580 
> > CPU:Xeon 3.2GHz *4 HT enabled
> > Memory:8GB
> > Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> > 
> > I did dd to this disk-array and got improved performance number.
> > 
> > I noticed that when a disk is just one HDD, performance improvement
> > is very small.
> > 
> 
> Ah.  So it's likely to be some strange interaction with the RAID setup.

The normal case is, if page N become uptodate at time T(N), then
T(N) <= T(N+1) holds. But for RAID, the data arrival time depends on
runtime status of individual disks, which breaks that formula. So
in do_generic_file_read(), just after submitting the async readahead IO
request, the current page may well be uptodate, so the page won't be locked,
and the block device won't be implicitly unplugged:

               if (PageReadahead(page))
                        page_cache_async_readahead()
                if (!PageUptodate(page))
                                goto page_not_up_to_date;
                //...
page_not_up_to_date:
                lock_page_killable(page);


Therefore explicit unplugging can help, so

        Acked-by: Wu Fengguang <fengguang.wu@intel.com> 

The only question is, shall we avoid the double unplug by doing this?

---
 mm/readahead.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

--- linux.orig/mm/readahead.c
+++ linux/mm/readahead.c
@@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
 
 	/* do read-ahead */
 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
+
+	/*
+	 * Normally the current page is !uptodate and lock_page() will be
+	 * immediately called to implicitly unplug the device. However this
+	 * is not always true for RAID conifgurations, where data arrives
+	 * not strictly in their submission order. In this case we need to
+	 * explicitly kick off the IO.
+	 */
+	if (PageUptodate(page))
+		blk_run_backing_dev(mapping->backing_dev_info, NULL);
 }
 EXPORT_SYMBOL_GPL(page_cache_async_readahead);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> > Ah.  So it's likely to be some strange interaction with the RAID setup.
> 
> The normal case is, if page N become uptodate at time T(N), then
> T(N) <= T(N+1) holds. But for RAID, the data arrival time depends on
> runtime status of individual disks, which breaks that formula. So
> in do_generic_file_read(), just after submitting the async readahead IO
> request, the current page may well be uptodate, so the page won't be locked,
> and the block device won't be implicitly unplugged:

Hifumi-san, Can you get blktrace data and confirm Wu's assumption?


> 
>                if (PageReadahead(page))
>                         page_cache_async_readahead()
>                 if (!PageUptodate(page))
>                                 goto page_not_up_to_date;
>                 //...
> page_not_up_to_date:
>                 lock_page_killable(page);
> 
> 
> Therefore explicit unplugging can help, so
> 
>         Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
> 
> The only question is, shall we avoid the double unplug by doing this?
> 
> ---
>  mm/readahead.c |   10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> --- linux.orig/mm/readahead.c
> +++ linux/mm/readahead.c
> @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
>  
>  	/* do read-ahead */
>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> +
> +	/*
> +	 * Normally the current page is !uptodate and lock_page() will be
> +	 * immediately called to implicitly unplug the device. However this
> +	 * is not always true for RAID conifgurations, where data arrives
> +	 * not strictly in their submission order. In this case we need to
> +	 * explicitly kick off the IO.
> +	 */
> +	if (PageUptodate(page))
> +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
>  }
>  EXPORT_SYMBOL_GPL(page_cache_async_readahead);



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 12:06:12PM +0800, KOSAKI Motohiro wrote:
> > > Ah.  So it's likely to be some strange interaction with the RAID setup.
> > 
> > The normal case is, if page N become uptodate at time T(N), then
> > T(N) <= T(N+1) holds. But for RAID, the data arrival time depends on
> > runtime status of individual disks, which breaks that formula. So
> > in do_generic_file_read(), just after submitting the async readahead IO
> > request, the current page may well be uptodate, so the page won't be locked,
> > and the block device won't be implicitly unplugged:
> 
> Hifumi-san, Can you get blktrace data and confirm Wu's assumption?

To make the reasoning more obvious:

Assume we just submitted readahead IO request for pages N ~ N+M, then
        T(N) <= T(N+1)
        T(N) <= T(N+2)
        T(N) <= T(N+3)
        ...
        T(N) <= T(N+M)   (M = readahead size)
So if the reader is going to block on any page in the above chunk,
it is going to first block on page N.

With RAID (and NFS to some degree), there is no strict ordering,
so the reader is more likely to block on some random pages.

In the first case, the effective async_size = M, in the second case,
the effective async_size <= M. The more async_size, the more degree of
readahead pipeline, hence the more low level IO latencies are hidden
to the application.

Thanks,
Fengguang

> 
> > 
> >                if (PageReadahead(page))
> >                         page_cache_async_readahead()
> >                 if (!PageUptodate(page))
> >                                 goto page_not_up_to_date;
> >                 //...
> > page_not_up_to_date:
> >                 lock_page_killable(page);
> > 
> > 
> > Therefore explicit unplugging can help, so
> > 
> >         Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
> > 
> > The only question is, shall we avoid the double unplug by doing this?
> > 
> > ---
> >  mm/readahead.c |   10 ++++++++++
> >  1 file changed, 10 insertions(+)
> > 
> > --- linux.orig/mm/readahead.c
> > +++ linux/mm/readahead.c
> > @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
> >  
> >  	/* do read-ahead */
> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> > +
> > +	/*
> > +	 * Normally the current page is !uptodate and lock_page() will be
> > +	 * immediately called to implicitly unplug the device. However this
> > +	 * is not always true for RAID conifgurations, where data arrives
> > +	 * not strictly in their submission order. In this case we need to
> > +	 * explicitly kick off the IO.
> > +	 */
> > +	if (PageUptodate(page))
> > +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >  }
> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 13:36 09/05/27, Wu Fengguang wrote:
>On Wed, May 27, 2009 at 12:06:12PM +0800, KOSAKI Motohiro wrote:
>> > > Ah.  So it's likely to be some strange interaction with the RAID setup.
>> > 
>> > The normal case is, if page N become uptodate at time T(N), then
>> > T(N) <= T(N+1) holds. But for RAID, the data arrival time depends on
>> > runtime status of individual disks, which breaks that formula. So
>> > in do_generic_file_read(), just after submitting the async readahead IO
>> > request, the current page may well be uptodate, so the page won't be locked,
>> > and the block device won't be implicitly unplugged:
>> 
>> Hifumi-san, Can you get blktrace data and confirm Wu's assumption?
>
>To make the reasoning more obvious:
>
>Assume we just submitted readahead IO request for pages N ~ N+M, then
>        T(N) <= T(N+1)
>        T(N) <= T(N+2)
>        T(N) <= T(N+3)
>        ...
>        T(N) <= T(N+M)   (M = readahead size)
>So if the reader is going to block on any page in the above chunk,
>it is going to first block on page N.
>
>With RAID (and NFS to some degree), there is no strict ordering,
>so the reader is more likely to block on some random pages.
>
>In the first case, the effective async_size = M, in the second case,
>the effective async_size <= M. The more async_size, the more degree of
>readahead pipeline, hence the more low level IO latencies are hidden
>to the application.

I got your explanation especially about RAID specific matters.

>
>Thanks,
>Fengguang
>
>> 
>> > 
>> >                if (PageReadahead(page))
>> >                         page_cache_async_readahead()
>> >                 if (!PageUptodate(page))
>> >                                 goto page_not_up_to_date;
>> >                 //...
>> > page_not_up_to_date:
>> >                 lock_page_killable(page);
>> > 
>> > 
>> > Therefore explicit unplugging can help, so
>> > 
>> >         Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
>> > 
>> > The only question is, shall we avoid the double unplug by doing this?
>> > 
>> > ---
>> >  mm/readahead.c |   10 ++++++++++
>> >  1 file changed, 10 insertions(+)
>> > 
>> > --- linux.orig/mm/readahead.c
>> > +++ linux/mm/readahead.c
>> > @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
>> >  
>> >  	/* do read-ahead */
>> >  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> > +
>> > +	/*
>> > +	* Normally the current page is !uptodate and lock_page() will be
>> > +	* immediately called to implicitly unplug the device. However this
>> > +	* is not always true for RAID conifgurations, where data arrives
>> > +	* not strictly in their submission order. In this case we need to
>> > +	* explicitly kick off the IO.
>> > +	*/
>> > +	if (PageUptodate(page))
>> > +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
>> >  }
>> >  EXPORT_SYMBOL_GPL(page_cache_async_readahead);

I am for this to avoid double unplug.
Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

>To make the reasoning more obvious:
>
>Assume we just submitted readahead IO request for pages N ~ N+M, then
>        T(N) <= T(N+1)
>        T(N) <= T(N+2)
>        T(N) <= T(N+3)
>        ...
>        T(N) <= T(N+M)   (M = readahead size)
>So if the reader is going to block on any page in the above chunk,
>it is going to first block on page N.
>
>With RAID (and NFS to some degree), there is no strict ordering,
>so the reader is more likely to block on some random pages.
>
>In the first case, the effective async_size = M, in the second case,
>the effective async_size <= M. The more async_size, the more degree of
>readahead pipeline, hence the more low level IO latencies are hidden
>to the application.
>
>Thanks,
>Fengguang
>
>> 
>> > 
>> >                if (PageReadahead(page))
>> >                         page_cache_async_readahead()
>> >                 if (!PageUptodate(page))
>> >                                 goto page_not_up_to_date;
>> >                 //...
>> > page_not_up_to_date:
>> >                 lock_page_killable(page);
>> > 
>> > 
>> > Therefore explicit unplugging can help, so
>> > 
>> >         Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
>> > 
>> > The only question is, shall we avoid the double unplug by doing this?
>> > 


Hi Andrew.
Please merge following patch.
Thanks.

---

I added blk_run_backing_dev on page_cache_async_readahead
so readahead I/O is unpluged to improve throughput on 
especially RAID environment. 

Following is the test result with dd.

#dd if=testdir/testfile of=/dev/null bs=16384

-2.6.30-rc6
1048576+0 records in
1048576+0 records out
17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s

-2.6.30-rc6-patched
1048576+0 records in
1048576+0 records out
17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s

My testing environment is as follows:
Hardware: HP DL580 
CPU:Xeon 3.2GHz *4 HT enabled
Memory:8GB
Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)

The normal case is, if page N become uptodate at time T(N), then
T(N) <= T(N+1) holds. With RAID (and NFS to some degree), there 
is no strict ordering, the data arrival time depends on
runtime status of individual disks, which breaks that formula. So
in do_generic_file_read(), just after submitting the async readahead IO
request, the current page may well be uptodate, so the page won't be locked,
and the block device won't be implicitly unplugged:

               if (PageReadahead(page))
                        page_cache_async_readahead()
                if (!PageUptodate(page))
                                goto page_not_up_to_date;
                //...
page_not_up_to_date:
                lock_page_killable(page);

Therefore explicit unplugging can help.

Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
Acked-by: Wu Fengguang <fengguang.wu@intel.com> 


 mm/readahead.c |   10 ++++++++++
 1 file changed, 10 insertions(+)

--- linux.orig/mm/readahead.c
+++ linux/mm/readahead.c
@@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
 
 	/* do read-ahead */
 	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
+
+	/*
+	* Normally the current page is !uptodate and lock_page() will be
+	* immediately called to implicitly unplug the device. However this
+	* is not always true for RAID conifgurations, where data arrives
+	* not strictly in their submission order. In this case we need to
+	* explicitly kick off the IO.
+	*/
+	if (PageUptodate(page))
+		blk_run_backing_dev(mapping->backing_dev_info, NULL);
 }
 EXPORT_SYMBOL_GPL(page_cache_async_readahead); 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> Hi Andrew.
> Please merge following patch.
> Thanks.
> 
> ---
> 
> I added blk_run_backing_dev on page_cache_async_readahead
> so readahead I/O is unpluged to improve throughput on 
> especially RAID environment. 
> 
> Following is the test result with dd.
> 
> #dd if=testdir/testfile of=/dev/null bs=16384
> 
> -2.6.30-rc6
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> 
> -2.6.30-rc6-patched
> 1048576+0 records in
> 1048576+0 records out
> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> 
> My testing environment is as follows:
> Hardware: HP DL580 
> CPU:Xeon 3.2GHz *4 HT enabled
> Memory:8GB
> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> 
> The normal case is, if page N become uptodate at time T(N), then
> T(N) <= T(N+1) holds. With RAID (and NFS to some degree), there 
> is no strict ordering, the data arrival time depends on
> runtime status of individual disks, which breaks that formula. So
> in do_generic_file_read(), just after submitting the async readahead IO
> request, the current page may well be uptodate, so the page won't be locked,
> and the block device won't be implicitly unplugged:

Please attach blktrace analysis ;)


> 
>                if (PageReadahead(page))
>                         page_cache_async_readahead()
>                 if (!PageUptodate(page))
>                                 goto page_not_up_to_date;
>                 //...
> page_not_up_to_date:
>                 lock_page_killable(page);
> 
> Therefore explicit unplugging can help.
> 
> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
> 
> 
>  mm/readahead.c |   10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> --- linux.orig/mm/readahead.c
> +++ linux/mm/readahead.c
> @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
>  
>  	/* do read-ahead */
>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> +
> +	/*
> +	* Normally the current page is !uptodate and lock_page() will be
> +	* immediately called to implicitly unplug the device. However this
> +	* is not always true for RAID conifgurations, where data arrives
> +	* not strictly in their submission order. In this case we need to
> +	* explicitly kick off the IO.
> +	*/
> +	if (PageUptodate(page))
> +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
>  }
>  EXPORT_SYMBOL_GPL(page_cache_async_readahead); 
> 



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:23 09/05/28, KOSAKI Motohiro wrote:
>> Hi Andrew.
>> Please merge following patch.
>> Thanks.
>> 
>> ---
>> 
>> I added blk_run_backing_dev on page_cache_async_readahead
>> so readahead I/O is unpluged to improve throughput on 
>> especially RAID environment. 
>> 
>> Following is the test result with dd.
>> 
>> #dd if=testdir/testfile of=/dev/null bs=16384
>> 
>> -2.6.30-rc6
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> 
>> -2.6.30-rc6-patched
>> 1048576+0 records in
>> 1048576+0 records out
>> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> 
>> My testing environment is as follows:
>> Hardware: HP DL580 
>> CPU:Xeon 3.2GHz *4 HT enabled
>> Memory:8GB
>> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>> 
>> The normal case is, if page N become uptodate at time T(N), then
>> T(N) <= T(N+1) holds. With RAID (and NFS to some degree), there 
>> is no strict ordering, the data arrival time depends on
>> runtime status of individual disks, which breaks that formula. So
>> in do_generic_file_read(), just after submitting the async readahead IO
>> request, the current page may well be uptodate, so the page won't be locked,
>> and the block device won't be implicitly unplugged:
>
>Please attach blktrace analysis ;)

Hi, Motohiro.

I've got blktrace output for both with and without the patch, 
but I just did not clarify the reason for throuput improvement
from this result.

I do not notice any difference except around unplug behavior by dd.
Comments?

-2.6.30-rc6
  8,0    3   177784    50.001437357     0  C   R 8717567 + 512 [0]
  8,0    3   177785    50.001635405  4148  A   R 8718079 + 256 <- (8,1) 8718016
  8,0    3   177786    50.001635675  4148  Q   R 8718079 + 256 [dd]
  8,0    3   177787    50.001637517  4148  G   R 8718079 + 256 [dd]
  8,0    3   177788    50.001638954  4148  P   N [dd]
  8,0    3   177789    50.001639290  4148  I   R 8718079 + 256 [dd]
  8,0    3   177790    50.001765339  4148  A   R 8718335 + 256 <- (8,1) 8718272
  8,0    3   177791    50.001765699  4148  Q   R 8718335 + 256 [dd]
  8,0    3   177792    50.001766971  4148  M   R 8718335 + 256 [dd]
  8,0    3   177793    50.001768243  4148  U   N [dd] 1
  8,0    3   177794    50.001769464  4148  D   R 8718079 + 512 [dd]
  8,0    3   177795    50.003815034     0  C   R 8718079 + 512 [0]
  8,0    3   177796    50.004008636  4148  A   R 8718591 + 256 <- (8,1) 8718528
  8,0    3   177797    50.004008951  4148  Q   R 8718591 + 256 [dd]
  8,0    3   177798    50.004010787  4148  G   R 8718591 + 256 [dd]
  8,0    3   177799    50.004012089  4148  P   N [dd]
  8,0    3   177800    50.004012641  4148  I   R 8718591 + 256 [dd]
  8,0    3   177801    50.004139944  4148  A   R 8718847 + 256 <- (8,1) 8718784
  8,0    3   177802    50.004140298  4148  Q   R 8718847 + 256 [dd]
  8,0    3   177803    50.004141393  4148  M   R 8718847 + 256 [dd]
  8,0    3   177804    50.004142815  4148  U   N [dd] 1
  8,0    3   177805    50.004144003  4148  D   R 8718591 + 512 [dd]
  8,0    3   177806    50.007151480     0  C   R 8718591 + 512 [0]
  8,0    3   177807    50.007344467  4148  A   R 8719103 + 256 <- (8,1) 8719040
  8,0    3   177808    50.007344779  4148  Q   R 8719103 + 256 [dd]
  8,0    3   177809    50.007346636  4148  G   R 8719103 + 256 [dd]
  8,0    3   177810    50.007347821  4148  P   N [dd]
  8,0    3   177811    50.007348346  4148  I   R 8719103 + 256 [dd]
  8,0    3   177812    50.007480827  4148  A   R 8719359 + 256 <- (8,1) 8719296
  8,0    3   177813    50.007481187  4148  Q   R 8719359 + 256 [dd]
  8,0    3   177814    50.007482669  4148  M   R 8719359 + 256 [dd]
  8,0    3   177815    50.007483965  4148  U   N [dd] 1
  8,0    3   177816    50.007485171  4148  D   R 8719103 + 512 [dd]
  8,0    3   177817    50.009885672     0  C   R 8719103 + 512 [0]
  8,0    3   177818    50.010077696  4148  A   R 8719615 + 256 <- (8,1) 8719552
  8,0    3   177819    50.010078008  4148  Q   R 8719615 + 256 [dd]
  8,0    3   177820    50.010079841  4148  G   R 8719615 + 256 [dd]
  8,0    3   177821    50.010081227  4148  P   N [dd]
  8,0    3   177822    50.010081560  4148  I   R 8719615 + 256 [dd]
  8,0    3   177823    50.010208686  4148  A   R 8719871 + 256 <- (8,1) 8719808
  8,0    3   177824    50.010209046  4148  Q   R 8719871 + 256 [dd]
  8,0    3   177825    50.010210366  4148  M   R 8719871 + 256 [dd]
  8,0    3   177826    50.010211686  4148  U   N [dd] 1
  8,0    3   177827    50.010212916  4148  D   R 8719615 + 512 [dd]
  8,0    3   177828    50.013880081     0  C   R 8719615 + 512 [0]
  8,0    3   177829    50.014071235  4148  A   R 8720127 + 256 <- (8,1) 8720064
  8,0    3   177830    50.014071544  4148  Q   R 8720127 + 256 [dd]
  8,0    3   177831    50.014073332  4148  G   R 8720127 + 256 [dd]
  8,0    3   177832    50.014074517  4148  P   N [dd]
  8,0    3   177833    50.014075084  4148  I   R 8720127 + 256 [dd]
  8,0    3   177834    50.014201763  4148  A   R 8720383 + 256 <- (8,1) 8720320
  8,0    3   177835    50.014202123  4148  Q   R 8720383 + 256 [dd]
  8,0    3   177836    50.014203608  4148  M   R 8720383 + 256 [dd]
  8,0    3   177837    50.014204889  4148  U   N [dd] 1
  8,0    3   177838    50.014206095  4148  D   R 8720127 + 512 [dd]
  8,0    3   177839    50.017545281     0  C   R 8720127 + 512 [0]
  8,0    3   177840    50.017741679  4148  A   R 8720639 + 256 <- (8,1) 8720576
  8,0    3   177841    50.017742006  4148  Q   R 8720639 + 256 [dd]
  8,0    3   177842    50.017743848  4148  G   R 8720639 + 256 [dd]
  8,0    3   177843    50.017745318  4148  P   N [dd]
  8,0    3   177844    50.017745672  4148  I   R 8720639 + 256 [dd]
  8,0    3   177845    50.017876956  4148  A   R 8720895 + 256 <- (8,1) 8720832
  8,0    3   177846    50.017877286  4148  Q   R 8720895 + 256 [dd]
  8,0    3   177847    50.017878615  4148  M   R 8720895 + 256 [dd]
  8,0    3   177848    50.017880082  4148  U   N [dd] 1
  8,0    3   177849    50.017881339  4148  D   R 8720639 + 512 [dd]
  8,0    3   177850    50.020674534     0  C   R 8720639 + 512 [0]
  8,0    3   177851    50.020864689  4148  A   R 8721151 + 256 <- (8,1) 8721088
  8,0    3   177852    50.020865007  4148  Q   R 8721151 + 256 [dd]
  8,0    3   177853    50.020866900  4148  G   R 8721151 + 256 [dd]
  8,0    3   177854    50.020868283  4148  P   N [dd]
  8,0    3   177855    50.020868628  4148  I   R 8721151 + 256 [dd]
  8,0    3   177856    50.020997302  4148  A   R 8721407 + 256 <- (8,1) 8721344
  8,0    3   177857    50.020997662  4148  Q   R 8721407 + 256 [dd]
  8,0    3   177858    50.020998976  4148  M   R 8721407 + 256 [dd]
  8,0    3   177859    50.021000305  4148  U   N [dd] 1
  8,0    3   177860    50.021001520  4148  D   R 8721151 + 512 [dd]
  8,0    3   177861    50.024269136     0  C   R 8721151 + 512 [0]
  8,0    3   177862    50.024460931  4148  A   R 8721663 + 256 <- (8,1) 8721600
  8,0    3   177863    50.024461337  4148  Q   R 8721663 + 256 [dd]
  8,0    3   177864    50.024463175  4148  G   R 8721663 + 256 [dd]
  8,0    3   177865    50.024464537  4148  P   N [dd]
  8,0    3   177866    50.024464871  4148  I   R 8721663 + 256 [dd]
  8,0    3   177867    50.024597943  4148  A   R 8721919 + 256 <- (8,1) 8721856
  8,0    3   177868    50.024598213  4148  Q   R 8721919 + 256 [dd]
  8,0    3   177869    50.024599323  4148  M   R 8721919 + 256 [dd]
  8,0    3   177870    50.024600751  4148  U   N [dd] 1
  8,0    3   177871    50.024602104  4148  D   R 8721663 + 512 [dd]
  8,0    3   177872    50.026966145     0  C   R 8721663 + 512 [0]
  8,0    3   177873    50.027157245  4148  A   R 8722175 + 256 <- (8,1) 8722112
  8,0    3   177874    50.027157563  4148  Q   R 8722175 + 256 [dd]
  8,0    3   177875    50.027159351  4148  G   R 8722175 + 256 [dd]
  8,0    3   177876    50.027160731  4148  P   N [dd]
  8,0    3   177877    50.027161064  4148  I   R 8722175 + 256 [dd]
  8,0    3   177878    50.027288745  4148  A   R 8722431 + 256 <- (8,1) 8722368
  8,0    3   177879    50.027289105  4148  Q   R 8722431 + 256 [dd]
  8,0    3   177880    50.027290206  4148  M   R 8722431 + 256 [dd]
  8,0    3   177881    50.027291697  4148  U   N [dd] 1
  8,0    3   177882    50.027293119  4148  D   R 8722175 + 512 [dd]
  8,0    3   177883    50.030406105     0  C   R 8722175 + 512 [0]
  8,0    3   177884    50.030600613  4148  A   R 8722687 + 256 <- (8,1) 8722624
  8,0    3   177885    50.030601199  4148  Q   R 8722687 + 256 [dd]
  8,0    3   177886    50.030603269  4148  G   R 8722687 + 256 [dd]
  8,0    3   177887    50.030604463  4148  P   N [dd]
  8,0    3   177888    50.030604799  4148  I   R 8722687 + 256 [dd]
  8,0    3   177889    50.030731757  4148  A   R 8722943 + 256 <- (8,1) 8722880
  8,0    3   177890    50.030732117  4148  Q   R 8722943 + 256 [dd]
  8,0    3   177891    50.030733397  4148  M   R 8722943 + 256 [dd]
  8,0    3   177892    50.030734882  4148  U   N [dd] 1
  8,0    3   177893    50.030736109  4148  D   R 8722687 + 512 [dd]
  8,0    3   177894    50.032916699     0  C   R 8722687 + 512 [0]
  8,0    3   177895    50.033176618  4148  A   R 8723199 + 256 <- (8,1) 8723136
  8,0    3   177896    50.033177218  4148  Q   R 8723199 + 256 [dd]
  8,0    3   177897    50.033181433  4148  G   R 8723199 + 256 [dd]
  8,0    3   177898    50.033184757  4148  P   N [dd]
  8,0    3   177899    50.033185642  4148  I   R 8723199 + 256 [dd]
  8,0    3   177900    50.033371264  4148  A   R 8723455 + 256 <- (8,1) 8723392
  8,0    3   177901    50.033371717  4148  Q   R 8723455 + 256 [dd]
  8,0    3   177902    50.033374015  4148  M   R 8723455 + 256 [dd]
  8,0    3   177903    50.033376814  4148  U   N [dd] 1
  8,0    3   177904    50.033380126  4148  D   R 8723199 + 512 [dd]
  8,0    3   177905    50.036715133     0  C   R 8723199 + 512 [0]
  8,0    3   177906    50.036971296  4148  A   R 8723711 + 256 <- (8,1) 8723648
  8,0    3   177907    50.036972136  4148  Q   R 8723711 + 256 [dd]
  8,0    3   177908    50.036975673  4148  G   R 8723711 + 256 [dd]
  8,0    3   177909    50.036978277  4148  P   N [dd]
  8,0    3   177910    50.036979450  4148  I   R 8723711 + 256 [dd]
  8,0    3   177911    50.037162429  4148  A   R 8723967 + 256 <- (8,1) 8723904
  8,0    3   177912    50.037162840  4148  Q   R 8723967 + 256 [dd]
  8,0    3   177913    50.037164967  4148  M   R 8723967 + 256 [dd]
  8,0    3   177914    50.037167223  4148  U   N [dd] 1
  8,0    3   177915    50.037170001  4148  D   R 8723711 + 512 [dd]
  8,0    3   177916    50.040521790     0  C   R 8723711 + 512 [0]
  8,0    3   177917    50.040729738  4148  A   R 8724223 + 256 <- (8,1) 8724160
  8,0    3   177918    50.040730200  4148  Q   R 8724223 + 256 [dd]
  8,0    3   177919    50.040732060  4148  G   R 8724223 + 256 [dd]
  8,0    3   177920    50.040733551  4148  P   N [dd]
  8,0    3   177921    50.040734109  4148  I   R 8724223 + 256 [dd]
  8,0    3   177922    50.040860173  4148  A   R 8724479 + 160 <- (8,1) 8724416
  8,0    3   177923    50.040860536  4148  Q   R 8724479 + 160 [dd]
  8,0    3   177924    50.040861517  4148  M   R 8724479 + 160 [dd]
  8,0    3   177925    50.040872542  4148  A   R 1055943 + 8 <- (8,1) 1055880
  8,0    3   177926    50.040872800  4148  Q   R 1055943 + 8 [dd]
  8,0    3   177927    50.040874849  4148  G   R 1055943 + 8 [dd]
  8,0    3   177928    50.040875485  4148  I   R 1055943 + 8 [dd]
  8,0    3   177929    50.040877045  4148  U   N [dd] 2
  8,0    3   177930    50.040878625  4148  D   R 8724223 + 416 [dd]
  8,0    3   177931    50.040895335  4148  D   R 1055943 + 8 [dd]
  8,0    3   177932    50.044383267     0  C   R 8724223 + 416 [0]
  8,0    3   177933    50.044704725     0  C   R 1055943 + 8 [0]
  8,0    3   177934    50.044749068  4148  A   R 8724639 + 96 <- (8,1) 8724576
  8,0    3   177935    50.044749472  4148  Q   R 8724639 + 96 [dd]
  8,0    3   177936    50.044752184  4148  G   R 8724639 + 96 [dd]
  8,0    3   177937    50.044753552  4148  P   N [dd]
  8,0    3   177938    50.044754032  4148  I   R 8724639 + 96 [dd]
  8,0    3   177939    50.044896095  4148  A   R 8724735 + 256 <- (8,1) 8724672
  8,0    3   177940    50.044896443  4148  Q   R 8724735 + 256 [dd]
  8,0    3   177941    50.044897538  4148  M   R 8724735 + 256 [dd]
  8,0    3   177942    50.044948546  4148  U   N [dd] 1
  8,0    3   177943    50.044950001  4148  D   R 8724639 + 352 [dd]
  8,0    3   177944    50.047150137     0  C   R 8724639 + 352 [0]
  8,0    3   177945    50.047294824  4148  A   R 8724991 + 256 <- (8,1) 8724928
  8,0    3   177946    50.047295142  4148  Q   R 8724991 + 256 [dd]
  8,0    3   177947    50.047296978  4148  G   R 8724991 + 256 [dd]
  8,0    3   177948    50.047298301  4148  P   N [dd]
  8,0    3   177949    50.047298637  4148  I   R 8724991 + 256 [dd]
  8,0    3   177950    50.047429027  4148  A   R 8725247 + 256 <- (8,1) 8725184
  8,0    3   177951    50.047429387  4148  Q   R 8725247 + 256 [dd]
  8,0    3   177952    50.047430479  4148  M   R 8725247 + 256 [dd]
  8,0    3   177953    50.047431736  4148  U   N [dd] 1
  8,0    3   177954    50.047432951  4148  D   R 8724991 + 512 [dd]
  8,0    3   177955    50.050313976     0  C   R 8724991 + 512 [0]
  8,0    3   177956    50.050507961  4148  A   R 8725503 + 256 <- (8,1) 8725440
  8,0    3   177957    50.050508273  4148  Q   R 8725503 + 256 [dd]
  8,0    3   177958    50.050510139  4148  G   R 8725503 + 256 [dd]
  8,0    3   177959    50.050511522  4148  P   N [dd]
  8,0    3   177960    50.050512062  4148  I   R 8725503 + 256 [dd]
  8,0    3   177961    50.050645393  4148  A   R 8725759 + 256 <- (8,1) 8725696
  8,0    3   177962    50.050645867  4148  Q   R 8725759 + 256 [dd]
  8,0    3   177963    50.050647171  4148  M   R 8725759 + 256 [dd]
  8,0    3   177964    50.050648593  4148  U   N [dd] 1
  8,0    3   177965    50.050649985  4148  D   R 8725503 + 512 [dd]
  8,0    3   177966    50.053380250     0  C   R 8725503 + 512 [0]
  8,0    3   177967    50.053576324  4148  A   R 8726015 + 256 <- (8,1) 8725952
  8,0    3   177968    50.053576615  4148  Q   R 8726015 + 256 [dd]
  8,0    3   177969    50.053578994  4148  G   R 8726015 + 256 [dd]
  8,0    3   177970    50.053580173  4148  P   N [dd]
  8,0    3   177971    50.053580509  4148  I   R 8726015 + 256 [dd]
  8,0    3   177972    50.053711503  4148  A   R 8726271 + 256 <- (8,1) 8726208
  8,0    3   177973    50.053712001  4148  Q   R 8726271 + 256 [dd]
  8,0    3   177974    50.053713332  4148  M   R 8726271 + 256 [dd]
  8,0    3   177975    50.053714583  4148  U   N [dd] 1
  8,0    3   177976    50.053715768  4148  D   R 8726015 + 512 [dd]
  8,0    3   177977    50.056970395     0  C   R 8726015 + 512 [0]
  8,0    3   177978    50.057161408  4148  A   R 8726527 + 256 <- (8,1) 8726464
  8,0    3   177979    50.057161726  4148  Q   R 8726527 + 256 [dd]
  8,0    3   177980    50.057163718  4148  G   R 8726527 + 256 [dd]
  8,0    3   177981    50.057165098  4148  P   N [dd]
  8,0    3   177982    50.057165431  4148  I   R 8726527 + 256 [dd]
  8,0    3   177983    50.057294630  4148  A   R 8726783 + 256 <- (8,1) 8726720
  8,0    3   177984    50.057294990  4148  Q   R 8726783 + 256 [dd]
  8,0    3   177985    50.057296070  4148  M   R 8726783 + 256 [dd]
  8,0    3   177986    50.057297402  4148  U   N [dd] 1
  8,0    3   177987    50.057298899  4148  D   R 8726527 + 512 [dd]
  8,0    3   177988    50.060326743     0  C   R 8726527 + 512 [0]
  8,0    3   177989    50.060523768  4148  A   R 8727039 + 256 <- (8,1) 8726976
  8,0    3   177990    50.060524095  4148  Q   R 8727039 + 256 [dd]
  8,0    3   177991    50.060525910  4148  G   R 8727039 + 256 [dd]
  8,0    3   177992    50.060527239  4148  P   N [dd]
  8,0    3   177993    50.060527575  4148  I   R 8727039 + 256 [dd]
  8,0    3   177994    50.060662280  4148  A   R 8727295 + 256 <- (8,1) 8727232
  8,0    3   177995    50.060662778  4148  Q   R 8727295 + 256 [dd]
  8,0    3   177996    50.060663993  4148  M   R 8727295 + 256 [dd]
  8,0    3   177997    50.060665403  4148  U   N [dd] 1
  8,0    3   177998    50.060666999  4148  D   R 8727039 + 512 [dd]
  8,0    3   177999    50.063922341     0  C   R 8727039 + 512 [0]
  8,0    3   178000    50.064113177  4148  A   R 8727551 + 256 <- (8,1) 8727488
  8,0    3   178001    50.064113492  4148  Q   R 8727551 + 256 [dd]
  8,0    3   178002    50.064115373  4148  G   R 8727551 + 256 [dd]

-2.6.30-rc6-patched
  8,0    3   257297    50.000760847     0  C   R 9480703 + 256 [0]
  8,0    3   257298    50.000944399  4139  A   R 9481215 + 256 <- (8,1) 9481152
  8,0    3   257299    50.000944693  4139  Q   R 9481215 + 256 [dd]
  8,0    3   257300    50.000946541  4139  G   R 9481215 + 256 [dd]
  8,0    3   257301    50.000947954  4139  P   N [dd]
  8,0    3   257302    50.000948368  4139  I   R 9481215 + 256 [dd]
  8,0    3   257303    50.000948920  4139  U   N [dd] 2
  8,0    3   257304    50.000950003  4139  D   R 9481215 + 256 [dd]
  8,0    3   257305    50.000962541  4139  U   N [dd] 2
  8,0    3   257306    50.003034240     0  C   R 9480959 + 256 [0]
  8,0    3   257307    50.003076338     0  C   R 9481215 + 256 [0]
  8,0    3   257308    50.003258111  4139  A   R 9481471 + 256 <- (8,1) 9481408
  8,0    3   257309    50.003258402  4139  Q   R 9481471 + 256 [dd]
  8,0    3   257310    50.003260190  4139  G   R 9481471 + 256 [dd]
  8,0    3   257311    50.003261399  4139  P   N [dd]
  8,0    3   257312    50.003261768  4139  I   R 9481471 + 256 [dd]
  8,0    3   257313    50.003262335  4139  U   N [dd] 1
  8,0    3   257314    50.003263406  4139  D   R 9481471 + 256 [dd]
  8,0    3   257315    50.003430472  4139  A   R 9481727 + 256 <- (8,1) 9481664
  8,0    3   257316    50.003430748  4139  Q   R 9481727 + 256 [dd]
  8,0    3   257317    50.003433065  4139  G   R 9481727 + 256 [dd]
  8,0    3   257318    50.003434343  4139  P   N [dd]
  8,0    3   257319    50.003434658  4139  I   R 9481727 + 256 [dd]
  8,0    3   257320    50.003435138  4139  U   N [dd] 2
  8,0    3   257321    50.003436083  4139  D   R 9481727 + 256 [dd]
  8,0    3   257322    50.003447795  4139  U   N [dd] 2
  8,0    3   257323    50.004774693     0  C   R 9481471 + 256 [0]
  8,0    3   257324    50.004959499  4139  A   R 9481983 + 256 <- (8,1) 9481920
  8,0    3   257325    50.004959790  4139  Q   R 9481983 + 256 [dd]
  8,0    3   257326    50.004961590  4139  G   R 9481983 + 256 [dd]
  8,0    3   257327    50.004962793  4139  P   N [dd]
  8,0    3   257328    50.004963153  4139  I   R 9481983 + 256 [dd]
  8,0    3   257329    50.004964098  4139  U   N [dd] 2
  8,0    3   257330    50.004965184  4139  D   R 9481983 + 256 [dd]
  8,0    3   257331    50.004978967  4139  U   N [dd] 2
  8,0    3   257332    50.006865854     0  C   R 9481727 + 256 [0]
  8,0    3   257333    50.007052043  4139  A   R 9482239 + 256 <- (8,1) 9482176
  8,0    3   257334    50.007052331  4139  Q   R 9482239 + 256 [dd]
  8,0    3   257335    50.007054146  4139  G   R 9482239 + 256 [dd]
  8,0    3   257336    50.007055355  4139  P   N [dd]
  8,0    3   257337    50.007055724  4139  I   R 9482239 + 256 [dd]
  8,0    3   257338    50.007056438  4139  U   N [dd] 2
  8,0    3   257339    50.007057605  4139  D   R 9482239 + 256 [dd]
  8,0    3   257340    50.007069963  4139  U   N [dd] 2
  8,0    3   257341    50.008250294     0  C   R 9481983 + 256 [0]
  8,0    3   257342    50.008431589  4139  A   R 9482495 + 256 <- (8,1) 9482432
  8,0    3   257343    50.008431881  4139  Q   R 9482495 + 256 [dd]
  8,0    3   257344    50.008433921  4139  G   R 9482495 + 256 [dd]
  8,0    3   257345    50.008435097  4139  P   N [dd]
  8,0    3   257346    50.008435466  4139  I   R 9482495 + 256 [dd]
  8,0    3   257347    50.008436213  4139  U   N [dd] 2
  8,0    3   257348    50.008437296  4139  D   R 9482495 + 256 [dd]
  8,0    3   257349    50.008450034  4139  U   N [dd] 2
  8,0    3   257350    50.010008843     0  C   R 9482239 + 256 [0]
  8,0    3   257351    50.010135287  4139  C   R 9482495 + 256 [0]
  8,0    3   257352    50.010226816  4139  A   R 9482751 + 256 <- (8,1) 9482688
  8,0    3   257353    50.010227107  4139  Q   R 9482751 + 256 [dd]
  8,0    3   257354    50.010229363  4139  G   R 9482751 + 256 [dd]
  8,0    3   257355    50.010230728  4139  P   N [dd]
  8,0    3   257356    50.010231097  4139  I   R 9482751 + 256 [dd]
  8,0    3   257357    50.010231655  4139  U   N [dd] 1
  8,0    3   257358    50.010232696  4139  D   R 9482751 + 256 [dd]
  8,0    3   257359    50.010380946  4139  A   R 9483007 + 256 <- (8,1) 9482944
  8,0    3   257360    50.010381264  4139  Q   R 9483007 + 256 [dd]
  8,0    3   257361    50.010383358  4139  G   R 9483007 + 256 [dd]
  8,0    3   257362    50.010384429  4139  P   N [dd]
  8,0    3   257363    50.010384741  4139  I   R 9483007 + 256 [dd]
  8,0    3   257364    50.010385395  4139  U   N [dd] 2
  8,0    3   257365    50.010386364  4139  D   R 9483007 + 256 [dd]
  8,0    3   257366    50.010397869  4139  U   N [dd] 2
  8,0    3   257367    50.014210132     0  C   R 9482751 + 256 [0]
  8,0    3   257368    50.014252938     0  C   R 9483007 + 256 [0]
  8,0    3   257369    50.014430811  4139  A   R 9483263 + 256 <- (8,1) 9483200
  8,0    3   257370    50.014431105  4139  Q   R 9483263 + 256 [dd]
  8,0    3   257371    50.014433139  4139  G   R 9483263 + 256 [dd]
  8,0    3   257372    50.014434520  4139  P   N [dd]
  8,0    3   257373    50.014435110  4139  I   R 9483263 + 256 [dd]
  8,0    3   257374    50.014435674  4139  U   N [dd] 1
  8,0    3   257375    50.014436770  4139  D   R 9483263 + 256 [dd]
  8,0    3   257376    50.014592117  4139  A   R 9483519 + 256 <- (8,1) 9483456
  8,0    3   257377    50.014592573  4139  Q   R 9483519 + 256 [dd]
  8,0    3   257378    50.014594391  4139  G   R 9483519 + 256 [dd]
  8,0    3   257379    50.014595504  4139  P   N [dd]
  8,0    3   257380    50.014595876  4139  I   R 9483519 + 256 [dd]
  8,0    3   257381    50.014596366  4139  U   N [dd] 2
  8,0    3   257382    50.014597368  4139  D   R 9483519 + 256 [dd]
  8,0    3   257383    50.014609521  4139  U   N [dd] 2
  8,0    3   257384    50.015937813     0  C   R 9483263 + 256 [0]
  8,0    3   257385    50.016124825  4139  A   R 9483775 + 256 <- (8,1) 9483712
  8,0    3   257386    50.016125116  4139  Q   R 9483775 + 256 [dd]
  8,0    3   257387    50.016127162  4139  G   R 9483775 + 256 [dd]
  8,0    3   257388    50.016128569  4139  P   N [dd]
  8,0    3   257389    50.016128983  4139  I   R 9483775 + 256 [dd]
  8,0    3   257390    50.016129538  4139  U   N [dd] 2
  8,0    3   257391    50.016130627  4139  D   R 9483775 + 256 [dd]
  8,0    3   257392    50.016143077  4139  U   N [dd] 2
  8,0    3   257393    50.016925304     0  C   R 9483519 + 256 [0]
  8,0    3   257394    50.017111307  4139  A   R 9484031 + 256 <- (8,1) 9483968
  8,0    3   257395    50.017111598  4139  Q   R 9484031 + 256 [dd]
  8,0    3   257396    50.017113410  4139  G   R 9484031 + 256 [dd]
  8,0    3   257397    50.017114835  4139  P   N [dd]
  8,0    3   257398    50.017115213  4139  I   R 9484031 + 256 [dd]
  8,0    3   257399    50.017115765  4139  U   N [dd] 2
  8,0    3   257400    50.017116839  4139  D   R 9484031 + 256 [dd]
  8,0    3   257401    50.017129023  4139  U   N [dd] 2
  8,0    3   257402    50.017396693     0  C   R 9483775 + 256 [0]
  8,0    3   257403    50.017584595  4139  A   R 9484287 + 256 <- (8,1) 9484224
  8,0    3   257404    50.017585018  4139  Q   R 9484287 + 256 [dd]
  8,0    3   257405    50.017586866  4139  G   R 9484287 + 256 [dd]
  8,0    3   257406    50.017587997  4139  P   N [dd]
  8,0    3   257407    50.017588393  4139  I   R 9484287 + 256 [dd]
  8,0    3   257408    50.017589105  4139  U   N [dd] 2
  8,0    3   257409    50.017590173  4139  D   R 9484287 + 256 [dd]
  8,0    3   257410    50.017602614  4139  U   N [dd] 2
  8,0    3   257411    50.020578876     0  C   R 9484031 + 256 [0]
  8,0    3   257412    50.020721857  4139  C   R 9484287 + 256 [0]
  8,0    3   257413    50.020803183  4139  A   R 9484543 + 256 <- (8,1) 9484480
  8,0    3   257414    50.020803507  4139  Q   R 9484543 + 256 [dd]
  8,0    3   257415    50.020805256  4139  G   R 9484543 + 256 [dd]
  8,0    3   257416    50.020806672  4139  P   N [dd]
  8,0    3   257417    50.020807065  4139  I   R 9484543 + 256 [dd]
  8,0    3   257418    50.020807668  4139  U   N [dd] 1
  8,0    3   257419    50.020808733  4139  D   R 9484543 + 256 [dd]
  8,0    3   257420    50.020957132  4139  A   R 9484799 + 256 <- (8,1) 9484736
  8,0    3   257421    50.020957423  4139  Q   R 9484799 + 256 [dd]
  8,0    3   257422    50.020959205  4139  G   R 9484799 + 256 [dd]
  8,0    3   257423    50.020960276  4139  P   N [dd]
  8,0    3   257424    50.020960594  4139  I   R 9484799 + 256 [dd]
  8,0    3   257425    50.020961062  4139  U   N [dd] 2
  8,0    3   257426    50.020961959  4139  D   R 9484799 + 256 [dd]
  8,0    3   257427    50.020974191  4139  U   N [dd] 2
  8,0    3   257428    50.023987847     0  C   R 9484543 + 256 [0]
  8,0    3   257429    50.024093062  4139  C   R 9484799 + 256 [0]
  8,0    3   257430    50.024207161  4139  A   R 9485055 + 256 <- (8,1) 9484992
  8,0    3   257431    50.024207434  4139  Q   R 9485055 + 256 [dd]
  8,0    3   257432    50.024209567  4139  G   R 9485055 + 256 [dd]
  8,0    3   257433    50.024210728  4139  P   N [dd]
  8,0    3   257434    50.024211097  4139  I   R 9485055 + 256 [dd]
  8,0    3   257435    50.024211661  4139  U   N [dd] 1
  8,0    3   257436    50.024212693  4139  D   R 9485055 + 256 [dd]
  8,0    3   257437    50.024359266  4139  A   R 9485311 + 256 <- (8,1) 9485248
  8,0    3   257438    50.024359584  4139  Q   R 9485311 + 256 [dd]
  8,0    3   257439    50.024361720  4139  G   R 9485311 + 256 [dd]
  8,0    3   257440    50.024362794  4139  P   N [dd]
  8,0    3   257441    50.024363106  4139  I   R 9485311 + 256 [dd]
  8,0    3   257442    50.024363760  4139  U   N [dd] 2
  8,0    3   257443    50.024364759  4139  D   R 9485311 + 256 [dd]
  8,0    3   257444    50.024376535  4139  U   N [dd] 2
  8,0    3   257445    50.026532544     0  C   R 9485055 + 256 [0]
  8,0    3   257446    50.026714236  4139  A   R 9485567 + 256 <- (8,1) 9485504
  8,0    3   257447    50.026714524  4139  Q   R 9485567 + 256 [dd]
  8,0    3   257448    50.026716354  4139  G   R 9485567 + 256 [dd]
  8,0    3   257449    50.026717791  4139  P   N [dd]
  8,0    3   257450    50.026718175  4139  I   R 9485567 + 256 [dd]
  8,0    3   257451    50.026718778  4139  U   N [dd] 2
  8,0    3   257452    50.026719876  4139  D   R 9485567 + 256 [dd]
  8,0    3   257453    50.026736383  4139  U   N [dd] 2
  8,0    3   257454    50.028531879     0  C   R 9485311 + 256 [0]
  8,0    3   257455    50.028684347  4139  C   R 9485567 + 256 [0]
  8,0    3   257456    50.028758787  4139  A   R 9485823 + 256 <- (8,1) 9485760
  8,0    3   257457    50.028759069  4139  Q   R 9485823 + 256 [dd]
  8,0    3   257458    50.028760884  4139  G   R 9485823 + 256 [dd]
  8,0    3   257459    50.028762099  4139  P   N [dd]
  8,0    3   257460    50.028762447  4139  I   R 9485823 + 256 [dd]
  8,0    3   257461    50.028763038  4139  U   N [dd] 1
  8,0    3   257462    50.028764268  4139  D   R 9485823 + 256 [dd]
  8,0    3   257463    50.028909841  4139  A   R 9486079 + 256 <- (8,1) 9486016
  8,0    3   257464    50.028910156  4139  Q   R 9486079 + 256 [dd]
  8,0    3   257465    50.028911896  4139  G   R 9486079 + 256 [dd]
  8,0    3   257466    50.028912964  4139  P   N [dd]
  8,0    3   257467    50.028913270  4139  I   R 9486079 + 256 [dd]
  8,0    3   257468    50.028913912  4139  U   N [dd] 2
  8,0    3   257469    50.028914878  4139  D   R 9486079 + 256 [dd]
  8,0    3   257470    50.028927497  4139  U   N [dd] 2
  8,0    3   257471    50.031158357     0  C   R 9485823 + 256 [0]
  8,0    3   257472    50.031292365  4139  C   R 9486079 + 256 [0]
  8,0    3   257473    50.031369697  4139  A   R 9486335 + 160 <- (8,1) 9486272
  8,0    3   257474    50.031369988  4139  Q   R 9486335 + 160 [dd]
  8,0    3   257475    50.031371779  4139  G   R 9486335 + 160 [dd]
  8,0    3   257476    50.031372850  4139  P   N [dd]
  8,0    3   257477    50.031373198  4139  I   R 9486335 + 160 [dd]
  8,0    3   257478    50.031384931  4139  A   R 1056639 + 8 <- (8,1) 1056576
  8,0    3   257479    50.031385201  4139  Q   R 1056639 + 8 [dd]
  8,0    3   257480    50.031388480  4139  G   R 1056639 + 8 [dd]
  8,0    3   257481    50.031388904  4139  I   R 1056639 + 8 [dd]
  8,0    3   257482    50.031390362  4139  U   N [dd] 2
  8,0    3   257483    50.031391523  4139  D   R 9486335 + 160 [dd]
  8,0    3   257484    50.031403403  4139  D   R 1056639 + 8 [dd]
  8,0    3   257485    50.033630747     0  C   R 1056639 + 8 [0]
  8,0    3   257486    50.033690300  4139  A   R 9486495 + 96 <- (8,1) 9486432
  8,0    3   257487    50.033690810  4139  Q   R 9486495 + 96 [dd]
  8,0    3   257488    50.033694581  4139  G   R 9486495 + 96 [dd]
  8,0    3   257489    50.033696739  4139  P   N [dd]
  8,0    3   257490    50.033697357  4139  I   R 9486495 + 96 [dd]
  8,0    3   257491    50.033698611  4139  U   N [dd] 2
  8,0    3   257492    50.033700945  4139  D   R 9486495 + 96 [dd]
  8,0    3   257493    50.033727763  4139  C   R 9486335 + 160 [0]
  8,0    3   257494    50.033996024  4139  A   R 9486591 + 256 <- (8,1) 9486528
  8,0    3   257495    50.033996396  4139  Q   R 9486591 + 256 [dd]
  8,0    3   257496    50.034000030  4139  G   R 9486591 + 256 [dd]
  8,0    3   257497    50.034002268  4139  P   N [dd]
  8,0    3   257498    50.034002820  4139  I   R 9486591 + 256 [dd]
  8,0    3   257499    50.034003924  4139  U   N [dd] 2
  8,0    3   257500    50.034006201  4139  D   R 9486591 + 256 [dd]
  8,0    3   257501    50.034091438  4139  U   N [dd] 2
  8,0    3   257502    50.034637372     0  C   R 9486495 + 96 [0]
  8,0    3   257503    50.034841508  4139  A   R 9486847 + 256 <- (8,1) 9486784
  8,0    3   257504    50.034842072  4139  Q   R 9486847 + 256 [dd]
  8,0    3   257505    50.034846117  4139  G   R 9486847 + 256 [dd]
  8,0    3   257506    50.034848676  4139  P   N [dd]
  8,0    3   257507    50.034849384  4139  I   R 9486847 + 256 [dd]
  8,0    3   257508    50.034850545  4139  U   N [dd] 2
  8,0    3   257509    50.034852795  4139  D   R 9486847 + 256 [dd]
  8,0    3   257510    50.034875503  4139  U   N [dd] 2
  8,0    3   257511    50.035370009     0  C   R 9486591 + 256 [0]
  8,0    3   257512    50.035622315  4139  A   R 9487103 + 256 <- (8,1) 9487040
  8,0    3   257513    50.035622954  4139  Q   R 9487103 + 256 [dd]
  8,0    3   257514    50.035627101  4139  G   R 9487103 + 256 [dd]
  8,0    3   257515    50.035629510  4139  P   N [dd]
  8,0    3   257516    50.035630143  4139  I   R 9487103 + 256 [dd]
  8,0    3   257517    50.035631058  4139  U   N [dd] 2
  8,0    3   257518    50.035632657  4139  D   R 9487103 + 256 [dd]
  8,0    3   257519    50.035656358  4139  U   N [dd] 2
  8,0    3   257520    50.036703329     0  C   R 9486847 + 256 [0]
  8,0    3   257521    50.036963604  4139  A   R 9487359 + 256 <- (8,1) 9487296
  8,0    3   257522    50.036964057  4139  Q   R 9487359 + 256 [dd]
  8,0    3   257523    50.036967636  4139  G   R 9487359 + 256 [dd]
  8,0    3   257524    50.036969710  4139  P   N [dd]
  8,0    3   257525    50.036970586  4139  I   R 9487359 + 256 [dd]
  8,0    3   257526    50.036971684  4139  U   N [dd] 2
  8,0    3   257527    50.036973631  4139  D   R 9487359 + 256 [dd]
  8,0    3   257528    50.036995034  4139  U   N [dd] 2
  8,0    3   257529    50.038904428     0  C   R 9487103 + 256 [0]
  8,0    3   257530    50.039161508  4139  A   R 9487615 + 256 <- (8,1) 9487552
  8,0    3   257531    50.039161934  4139  Q   R 9487615 + 256 [dd]
  8,0    3   257532    50.039165834  4139  G   R 9487615 + 256 [dd]
  8,0    3   257533    50.039168561  4139  P   N [dd]
  8,0    3   257534    50.039169353  4139  I   R 9487615 + 256 [dd]
  8,0    3   257535    50.039170343  4139  U   N [dd] 2
  8,0    3   257536    50.039171645  4139  D   R 9487615 + 256 [dd]
  8,0    3   257537    50.039193195  4139  U   N [dd] 2
  8,0    3   257538    50.040570003     0  C   R 9487359 + 256 [0]
  8,0    3   257539    50.040842161  4139  A   R 9487871 + 256 <- (8,1) 9487808
  8,0    3   257540    50.040842827  4139  Q   R 9487871 + 256 [dd]
  8,0    3   257541    50.040846803  4139  G   R 9487871 + 256 [dd]
  8,0    3   257542    50.040849902  4139  P   N [dd]
  8,0    3   257543    50.040850715  4139  I   R 9487871 + 256 [dd]
  8,0    3   257544    50.040851642  4139  U   N [dd] 2
  8,0    3   257545    50.040853658  4139  D   R 9487871 + 256 [dd]
  8,0    3   257546    50.040876270  4139  U   N [dd] 2
  8,0    3   257547    50.042081391     0  C   R 9487615 + 256 [0]
  8,0    3   257548    50.042215837  4139  C   R 9487871 + 256 [0]
  8,0    3   257549    50.042316192  4139  A   R 9488127 + 256 <- (8,1) 9488064
  8,0    3   257550    50.042316633  4139  Q   R 9488127 + 256 [dd]
  8,0    3   257551    50.042319213  4139  G   R 9488127 + 256 [dd]
  8,0    3   257552    50.042320803  4139  P   N [dd]
  8,0    3   257553    50.042321412  4139  I   R 9488127 + 256 [dd]
  8,0    3   257554    50.042322219  4139  U   N [dd] 1
  8,0    3   257555    50.042323362  4139  D   R 9488127 + 256 [dd]
  8,0    3   257556    50.042484350  4139  A   R 9488383 + 256 <- (8,1) 9488320
  8,0    3   257557    50.042484602  4139  Q   R 9488383 + 256 [dd]
  8,0    3   257558    50.042486744  4139  G   R 9488383 + 256 [dd]
  8,0    3   257559    50.042487908  4139  P   N [dd]
  8,0    3   257560    50.042488223  4139  I   R 9488383 + 256 [dd]
  8,0    3   257561    50.042488754  4139  U   N [dd] 2
  8,0    3   257562    50.042489927  4139  D   R 9488383 + 256 [dd]
  8,0    3   257563    50.042502678  4139  U   N [dd] 2
  8,0    3   257564    50.045166592     0  C   R 9488127 + 256 [0]
  8,0    3   257565    50.045355163  4139  A   R 9488639 + 256 <- (8,1) 9488576
  8,0    3   257566    50.045355493  4139  Q   R 9488639 + 256 [dd]
  8,0    3   257567    50.045357497  4139  G   R 9488639 + 256 [dd]
  8,0    3   257568    50.045358673  4139  P   N [dd]
  8,0    3   257569    50.045359267  4139  I   R 9488639 + 256 [dd]
  8,0    3   257570    50.045359831  4139  U   N [dd] 2
  8,0    3   257571    50.045360911  4139  D   R 9488639 + 256 [dd]
  8,0    3   257572    50.045373959  4139  U   N [dd] 2
  8,0    3   257573    50.046450730     0  C   R 9488383 + 256 [0]
  8,0    3   257574    50.046641639  4139  A   R 9488895 + 256 <- (8,1) 9488832
  8,0    3   257575    50.046642086  4139  Q   R 9488895 + 256 [dd]
  8,0    3   257576    50.046643937  4139  G   R 9488895 + 256 [dd]
  8,0    3   257577    50.046645092  4139  P   N [dd]
  8,0    3   257578    50.046645527  4139  I   R 9488895 + 256 [dd]
  8,0    3   257579    50.046646244  4139  U   N [dd] 2
  8,0    3   257580    50.046647327  4139  D   R 9488895 + 256 [dd]
  8,0    3   257581    50.046660234  4139  U   N [dd] 2
  8,0    3   257582    50.047826305     0  C   R 9488639 + 256 [0]
  8,0    3   257583    50.048011468  4139  A   R 9489151 + 256 <- (8,1) 9489088
  8,0    3   257584    50.048011762  4139  Q   R 9489151 + 256 [dd]
  8,0    3   257585    50.048013793  4139  G   R 9489151 + 256 [dd]
  8,0    3   257586    50.048014966  4139  P   N [dd]
  8,0    3   257587    50.048015380  4139  I   R 9489151 + 256 [dd]
  8,0    3   257588    50.048016112  4139  U   N [dd] 2
  8,0    3   257589    50.048017202  4139  D   R 9489151 + 256 [dd]
  8,0    3   257590    50.048029553  4139  U   N [dd] 2
  8,0    3   257591    50.049319830     0  C   R 9488895 + 256 [0]
  8,0    3   257592    50.049446089  4139  C   R 9489151 + 256 [0]
  8,0    3   257593    50.049545199  4139  A   R 9489407 + 256 <- (8,1) 9489344
  8,0    3   257594    50.049545628  4139  Q   R 9489407 + 256 [dd]
  8,0    3   257595    50.049547512  4139  G   R 9489407 + 256 [dd]
  8,0    3   257596    50.049548886  4139  P   N [dd]
  8,0    3   257597    50.049549318  4139  I   R 9489407 + 256 [dd]
  8,0    3   257598    50.049550047  4139  U   N [dd] 1
  8,0    3   257599    50.049551241  4139  D   R 9489407 + 256 [dd]
  8,0    3   257600    50.049699283  4139  A   R 9489663 + 256 <- (8,1) 9489600
  8,0    3   257601    50.049699556  4139  Q   R 9489663 + 256 [dd]
  8,0    3   257602    50.049701266  4139  G   R 9489663 + 256 [dd]
  8,0    3   257603    50.049702310  4139  P   N [dd]
  8,0    3   257604    50.049702656  4139  I   R 9489663 + 256 [dd]
  8,0    3   257605    50.049703118  4139  U   N [dd] 2
  8,0    3   257606    50.049704020  4139  D   R 9489663 + 256 [dd]
  8,0    3   257607    50.049715940  4139  U   N [dd] 2
  8,0    3   257608    50.052662150     0  C   R 9489407 + 256 [0]
  8,0    3   257609    50.052853688  4139  A   R 9489919 + 256 <- (8,1) 9489856
  8,0    3   257610    50.052853985  4139  Q   R 9489919 + 256 [dd]
  8,0    3   257611    50.052855869  4139  G   R 9489919 + 256 [dd]
  8,0    3   257612    50.052857057  4139  P   N [dd]
  8,0    3   257613    50.052857423  4139  I   R 9489919 + 256 [dd]
  8,0    3   257614    50.052858065  4139  U   N [dd] 2
  8,0    3   257615    50.052859164  4139  D   R 9489919 + 256 [dd]
  8,0    3   257616    50.052871806  4139  U   N [dd] 2
  8,0    3   257617    50.053470795     0  C   R 9489663 + 256 [0]
  8,0    3   257618    50.053661719  4139  A   R 9490175 + 256 <- (8,1) 9490112
  8,0    3   257619    50.053662097  4139  Q   R 9490175 + 256 [dd]
  8,0    3   257620    50.053663891  4139  G   R 9490175 + 256 [dd]
  8,0    3   257621    50.053665034  4139  P   N [dd]
  8,0    3   257622    50.053665436  4139  I   R 9490175 + 256 [dd]
  8,0    3   257623    50.053665982  4139  U   N [dd] 2
  8,0    3   257624    50.053667077  4139  D   R 9490175 + 256 [dd]
  8,0    3   257625    50.053679732  4139  U   N [dd] 2
  8,0    3   257626    50.055776383     0  C   R 9489919 + 256 [0]
  8,0    3   257627    50.055915017  4139  C   R 9490175 + 256 [0]
  8,0    3   257628    50.055997812  4139  A   R 9490431 + 256 <- (8,1) 9490368
  8,0    3   257629    50.055998085  4139  Q   R 9490431 + 256 [dd]
  8,0    3   257630    50.055999867  4139  G   R 9490431 + 256 [dd]
  8,0    3   257631    50.056001049  4139  P   N [dd]
  8,0    3   257632    50.056001451  4139  I   R 9490431 + 256 [dd]
  8,0    3   257633    50.056002189  4139  U   N [dd] 1
  8,0    3   257634    50.056003197  4139  D   R 9490431 + 256 [dd]
  8,0    3   257635    50.056149977  4139  A   R 9490687 + 256 <- (8,1) 9490624
  8,0    3   257636    50.056150279  4139  Q   R 9490687 + 256 [dd]
  8,0    3   257637    50.056152047  4139  G   R 9490687 + 256 [dd]
  8,0    3   257638    50.056153109  4139  P   N [dd]
  8,0    3   257639    50.056153442  4139  I   R 9490687 + 256 [dd]
  8,0    3   257640    50.056153904  4139  U   N [dd] 2
  8,0    3   257641    50.056154852  4139  D   R 9490687 + 256 [dd]
  8,0    3   257642    50.056166948  4139  U   N [dd] 2
  8,0    3   257643    50.057600660     0  C   R 9490431 + 256 [0]
  8,0    3   257644    50.057786753  4139  A   R 9490943 + 256 <- (8,1) 9490880
  8,0    3   257645    50.057787050  4139  Q   R 9490943 + 256 [dd]
  8,0    3   257646    50.057788865  4139  G   R 9490943 + 256 [dd]
  8,0    3   257647    50.057790236  4139  P   N [dd]
  8,0    3   257648    50.057790614  4139  I   R 9490943 + 256 [dd]
  8,0    3   257649    50.057791169  4139  U   N [dd] 2
  8,0    3   257650    50.057792246  4139  D   R 9490943 + 256 [dd]
  8,0    3   257651    50.057804469  4139  U   N [dd] 2
  8,0    3   257652    50.060322995     0  C   R 9490687 + 256 [0]
  8,0    3   257653    50.060464005  4139  C   R 9490943 + 256 [0]
  8,0    3   257654    50.060548216  4139  A   R 9491199 + 256 <- (8,1) 9491136
  8,0    3   257655    50.060548696  4139  Q   R 9491199 + 256 [dd]
  8,0    3   257656    50.060550922  4139  G   R 9491199 + 256 [dd]
  8,0    3   257657    50.060552096  4139  P   N [dd]
  8,0    3   257658    50.060552531  4139  I   R 9491199 + 256 [dd]
  8,0    3   257659    50.060553101  4139  U   N [dd] 1
  8,0    3   257660    50.060554100  4139  D   R 9491199 + 256 [dd]
  8,0    3   257661    50.060701569  4139  A   R 9491455 + 256 <- (8,1) 9491392
  8,0    3   257662    50.060701890  4139  Q   R 9491455 + 256 [dd]
  8,0    3   257663    50.060703993  4139  G   R 9491455 + 256 [dd]
  8,0    3   257664    50.060705070  4139  P   N [dd]
  8,0    3   257665    50.060705385  4139  I   R 9491455 + 256 [dd]
  8,0    3   257666    50.060706012  4139  U   N [dd] 2
  8,0    3   257667    50.060706987  4139  D   R 9491455 + 256 [dd]
  8,0    3   257668    50.060718784  4139  U   N [dd] 2
  8,0    3   257669    50.062964966     0  C   R 9491199 + 256 [0]
  8,0    3   257670    50.063102772  4139  C   R 9491455 + 256 [0]
  8,0    3   257671    50.063182666  4139  A   R 9491711 + 256 <- (8,1) 9491648
  8,0    3   257672    50.063182939  4139  Q   R 9491711 + 256 [dd]
  8,0    3   257673    50.063184889  4139  G   R 9491711 + 256 [dd]
  8,0    3   257674    50.063186074  4139  P   N [dd]
  8,0    3   257675    50.063186440  4139  I   R 9491711 + 256 [dd]
  8,0    3   257676    50.063187271  4139  U   N [dd] 1
  8,0    3   257677    50.063188312  4139  D   R 9491711 + 256 [dd]
  8,0    3   257678    50.063340467  4139  A   R 9491967 + 256 <- (8,1) 9491904
  8,0    3   257679    50.063340749  4139  Q   R 9491967 + 256 [dd]
  8,0    3   257680    50.063342529  4139  G   R 9491967 + 256 [dd]
  8,0    3   257681    50.063343597  4139  P   N [dd]
  8,0    3   257682    50.063343915  4139  I   R 9491967 + 256 [dd]
  8,0    3   257683    50.063344374  4139  U   N [dd] 2
  8,0    3   257684    50.063345313  4139  D   R 9491967 + 256 [dd]
  8,0    3   257685    50.063357370  4139  U   N [dd] 2
  8,0    3   257686    50.066605011     0  C   R 9491711 + 256 [0]
  8,0    3   257687    50.066643587     0  C   R 9491967 + 256 [0]
  8,0    3   257688    50.066821310  4139  A   R 9492223 + 256 <- (8,1) 9492160
  8,0    3   257689    50.066821601  4139  Q   R 9492223 + 256 [dd]
  8,0    3   257690    50.066823605  4139  G   R 9492223 + 256 [dd]
  8,0    3   257691    50.066825063  4139  P   N [dd]



>
>
>> 
>>                if (PageReadahead(page))
>>                         page_cache_async_readahead()
>>                 if (!PageUptodate(page))
>>                                 goto page_not_up_to_date;
>>                 //...
>> page_not_up_to_date:
>>                 lock_page_killable(page);
>> 
>> Therefore explicit unplugging can help.
>> 
>> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
>> Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
>> 
>> 
>>  mm/readahead.c |   10 ++++++++++
>>  1 file changed, 10 insertions(+)
>> 
>> --- linux.orig/mm/readahead.c
>> +++ linux/mm/readahead.c
>> @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
>>  
>>  	/* do read-ahead */
>>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
>> +
>> +	/*
>> +	* Normally the current page is !uptodate and lock_page() will be
>> +	* immediately called to implicitly unplug the device. However this
>> +	* is not always true for RAID conifgurations, where data arrives
>> +	* not strictly in their submission order. In this case we need to
>> +	* explicitly kick off the IO.
>> +	*/
>> +	if (PageUptodate(page))
>> +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
>>  }
>>  EXPORT_SYMBOL_GPL(page_cache_async_readahead); 
>> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> 
> At 11:23 09/05/28, KOSAKI Motohiro wrote:
> >> Hi Andrew.
> >> Please merge following patch.
> >> Thanks.
> >> 
> >> ---
> >> 
> >> I added blk_run_backing_dev on page_cache_async_readahead
> >> so readahead I/O is unpluged to improve throughput on 
> >> especially RAID environment. 
> >> 
> >> Following is the test result with dd.
> >> 
> >> #dd if=testdir/testfile of=/dev/null bs=16384
> >> 
> >> -2.6.30-rc6
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> 
> >> -2.6.30-rc6-patched
> >> 1048576+0 records in
> >> 1048576+0 records out
> >> 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> 
> >> My testing environment is as follows:
> >> Hardware: HP DL580 
> >> CPU:Xeon 3.2GHz *4 HT enabled
> >> Memory:8GB
> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> >> 
> >> The normal case is, if page N become uptodate at time T(N), then
> >> T(N) <= T(N+1) holds. With RAID (and NFS to some degree), there 
> >> is no strict ordering, the data arrival time depends on
> >> runtime status of individual disks, which breaks that formula. So
> >> in do_generic_file_read(), just after submitting the async readahead IO
> >> request, the current page may well be uptodate, so the page won't be locked,
> >> and the block device won't be implicitly unplugged:
> >
> >Please attach blktrace analysis ;)
> 
> Hi, Motohiro.
> 
> I've got blktrace output for both with and without the patch, 
> but I just did not clarify the reason for throuput improvement
> from this result.
> 
> I do not notice any difference except around unplug behavior by dd.
> Comments?

Please makes analysis yourself.
following are summarized log of IO completion.

it doesn't contain IO completion inversion.
Why do you think it is RAID specific behavior?

  8,0    3   177784    50.001437357     0  C   R 8717567 + 512 [0]
  8,0    3   177795    50.003815034     0  C   R 8718079 + 512 [0]
  8,0    3   177806    50.007151480     0  C   R 8718591 + 512 [0]
  8,0    3   177817    50.009885672     0  C   R 8719103 + 512 [0]
  8,0    3   177828    50.013880081     0  C   R 8719615 + 512 [0]
  8,0    3   177839    50.017545281     0  C   R 8720127 + 512 [0]
  8,0    3   177850    50.020674534     0  C   R 8720639 + 512 [0]
  8,0    3   177861    50.024269136     0  C   R 8721151 + 512 [0]
  8,0    3   177872    50.026966145     0  C   R 8721663 + 512 [0]
  8,0    3   177883    50.030406105     0  C   R 8722175 + 512 [0]
  8,0    3   177894    50.032916699     0  C   R 8722687 + 512 [0]
  8,0    3   177905    50.036715133     0  C   R 8723199 + 512 [0]
  8,0    3   177916    50.040521790     0  C   R 8723711 + 512 [0]
  8,0    3   177932    50.044383267     0  C   R 8724223 + 416 [0]
  8,0    3   177955    50.050313976     0  C   R 8724991 + 512 [0]
  8,0    3   177966    50.053380250     0  C   R 8725503 + 512 [0]
  8,0    3   177977    50.056970395     0  C   R 8726015 + 512 [0]
  8,0    3   177988    50.060326743     0  C   R 8726527 + 512 [0]
  8,0    3   177999    50.063922341     0  C   R 8727039 + 512 [0]




> 
> -2.6.30-rc6
>   8,0    3   177784    50.001437357     0  C   R 8717567 + 512 [0]
>   8,0    3   177785    50.001635405  4148  A   R 8718079 + 256 <- (8,1) 8718016
>   8,0    3   177786    50.001635675  4148  Q   R 8718079 + 256 [dd]
>   8,0    3   177787    50.001637517  4148  G   R 8718079 + 256 [dd]
>   8,0    3   177788    50.001638954  4148  P   N [dd]
>   8,0    3   177789    50.001639290  4148  I   R 8718079 + 256 [dd]
>   8,0    3   177790    50.001765339  4148  A   R 8718335 + 256 <- (8,1) 8718272
>   8,0    3   177791    50.001765699  4148  Q   R 8718335 + 256 [dd]
>   8,0    3   177792    50.001766971  4148  M   R 8718335 + 256 [dd]
>   8,0    3   177793    50.001768243  4148  U   N [dd] 1
>   8,0    3   177794    50.001769464  4148  D   R 8718079 + 512 [dd]
>   8,0    3   177795    50.003815034     0  C   R 8718079 + 512 [0]
>   8,0    3   177796    50.004008636  4148  A   R 8718591 + 256 <- (8,1) 8718528
>   8,0    3   177797    50.004008951  4148  Q   R 8718591 + 256 [dd]
>   8,0    3   177798    50.004010787  4148  G   R 8718591 + 256 [dd]
>   8,0    3   177799    50.004012089  4148  P   N [dd]
>   8,0    3   177800    50.004012641  4148  I   R 8718591 + 256 [dd]
>   8,0    3   177801    50.004139944  4148  A   R 8718847 + 256 <- (8,1) 8718784
>   8,0    3   177802    50.004140298  4148  Q   R 8718847 + 256 [dd]
>   8,0    3   177803    50.004141393  4148  M   R 8718847 + 256 [dd]
>   8,0    3   177804    50.004142815  4148  U   N [dd] 1
>   8,0    3   177805    50.004144003  4148  D   R 8718591 + 512 [dd]
>   8,0    3   177806    50.007151480     0  C   R 8718591 + 512 [0]
>   8,0    3   177807    50.007344467  4148  A   R 8719103 + 256 <- (8,1) 8719040
>   8,0    3   177808    50.007344779  4148  Q   R 8719103 + 256 [dd]
>   8,0    3   177809    50.007346636  4148  G   R 8719103 + 256 [dd]
>   8,0    3   177810    50.007347821  4148  P   N [dd]
>   8,0    3   177811    50.007348346  4148  I   R 8719103 + 256 [dd]
>   8,0    3   177812    50.007480827  4148  A   R 8719359 + 256 <- (8,1) 8719296
>   8,0    3   177813    50.007481187  4148  Q   R 8719359 + 256 [dd]
>   8,0    3   177814    50.007482669  4148  M   R 8719359 + 256 [dd]
>   8,0    3   177815    50.007483965  4148  U   N [dd] 1
>   8,0    3   177816    50.007485171  4148  D   R 8719103 + 512 [dd]
>   8,0    3   177817    50.009885672     0  C   R 8719103 + 512 [0]
>   8,0    3   177818    50.010077696  4148  A   R 8719615 + 256 <- (8,1) 8719552
>   8,0    3   177819    50.010078008  4148  Q   R 8719615 + 256 [dd]
>   8,0    3   177820    50.010079841  4148  G   R 8719615 + 256 [dd]
>   8,0    3   177821    50.010081227  4148  P   N [dd]
>   8,0    3   177822    50.010081560  4148  I   R 8719615 + 256 [dd]
>   8,0    3   177823    50.010208686  4148  A   R 8719871 + 256 <- (8,1) 8719808
>   8,0    3   177824    50.010209046  4148  Q   R 8719871 + 256 [dd]
>   8,0    3   177825    50.010210366  4148  M   R 8719871 + 256 [dd]
>   8,0    3   177826    50.010211686  4148  U   N [dd] 1
>   8,0    3   177827    50.010212916  4148  D   R 8719615 + 512 [dd]
>   8,0    3   177828    50.013880081     0  C   R 8719615 + 512 [0]
>   8,0    3   177829    50.014071235  4148  A   R 8720127 + 256 <- (8,1) 8720064
>   8,0    3   177830    50.014071544  4148  Q   R 8720127 + 256 [dd]
>   8,0    3   177831    50.014073332  4148  G   R 8720127 + 256 [dd]
>   8,0    3   177832    50.014074517  4148  P   N [dd]
>   8,0    3   177833    50.014075084  4148  I   R 8720127 + 256 [dd]
>   8,0    3   177834    50.014201763  4148  A   R 8720383 + 256 <- (8,1) 8720320
>   8,0    3   177835    50.014202123  4148  Q   R 8720383 + 256 [dd]
>   8,0    3   177836    50.014203608  4148  M   R 8720383 + 256 [dd]
>   8,0    3   177837    50.014204889  4148  U   N [dd] 1
>   8,0    3   177838    50.014206095  4148  D   R 8720127 + 512 [dd]
>   8,0    3   177839    50.017545281     0  C   R 8720127 + 512 [0]
>   8,0    3   177840    50.017741679  4148  A   R 8720639 + 256 <- (8,1) 8720576
>   8,0    3   177841    50.017742006  4148  Q   R 8720639 + 256 [dd]
>   8,0    3   177842    50.017743848  4148  G   R 8720639 + 256 [dd]
>   8,0    3   177843    50.017745318  4148  P   N [dd]
>   8,0    3   177844    50.017745672  4148  I   R 8720639 + 256 [dd]
>   8,0    3   177845    50.017876956  4148  A   R 8720895 + 256 <- (8,1) 8720832
>   8,0    3   177846    50.017877286  4148  Q   R 8720895 + 256 [dd]
>   8,0    3   177847    50.017878615  4148  M   R 8720895 + 256 [dd]
>   8,0    3   177848    50.017880082  4148  U   N [dd] 1
>   8,0    3   177849    50.017881339  4148  D   R 8720639 + 512 [dd]
>   8,0    3   177850    50.020674534     0  C   R 8720639 + 512 [0]
>   8,0    3   177851    50.020864689  4148  A   R 8721151 + 256 <- (8,1) 8721088
>   8,0    3   177852    50.020865007  4148  Q   R 8721151 + 256 [dd]
>   8,0    3   177853    50.020866900  4148  G   R 8721151 + 256 [dd]
>   8,0    3   177854    50.020868283  4148  P   N [dd]
>   8,0    3   177855    50.020868628  4148  I   R 8721151 + 256 [dd]
>   8,0    3   177856    50.020997302  4148  A   R 8721407 + 256 <- (8,1) 8721344
>   8,0    3   177857    50.020997662  4148  Q   R 8721407 + 256 [dd]
>   8,0    3   177858    50.020998976  4148  M   R 8721407 + 256 [dd]
>   8,0    3   177859    50.021000305  4148  U   N [dd] 1
>   8,0    3   177860    50.021001520  4148  D   R 8721151 + 512 [dd]
>   8,0    3   177861    50.024269136     0  C   R 8721151 + 512 [0]
>   8,0    3   177862    50.024460931  4148  A   R 8721663 + 256 <- (8,1) 8721600
>   8,0    3   177863    50.024461337  4148  Q   R 8721663 + 256 [dd]
>   8,0    3   177864    50.024463175  4148  G   R 8721663 + 256 [dd]
>   8,0    3   177865    50.024464537  4148  P   N [dd]
>   8,0    3   177866    50.024464871  4148  I   R 8721663 + 256 [dd]
>   8,0    3   177867    50.024597943  4148  A   R 8721919 + 256 <- (8,1) 8721856
>   8,0    3   177868    50.024598213  4148  Q   R 8721919 + 256 [dd]
>   8,0    3   177869    50.024599323  4148  M   R 8721919 + 256 [dd]
>   8,0    3   177870    50.024600751  4148  U   N [dd] 1
>   8,0    3   177871    50.024602104  4148  D   R 8721663 + 512 [dd]
>   8,0    3   177872    50.026966145     0  C   R 8721663 + 512 [0]
>   8,0    3   177873    50.027157245  4148  A   R 8722175 + 256 <- (8,1) 8722112
>   8,0    3   177874    50.027157563  4148  Q   R 8722175 + 256 [dd]
>   8,0    3   177875    50.027159351  4148  G   R 8722175 + 256 [dd]
>   8,0    3   177876    50.027160731  4148  P   N [dd]
>   8,0    3   177877    50.027161064  4148  I   R 8722175 + 256 [dd]
>   8,0    3   177878    50.027288745  4148  A   R 8722431 + 256 <- (8,1) 8722368
>   8,0    3   177879    50.027289105  4148  Q   R 8722431 + 256 [dd]
>   8,0    3   177880    50.027290206  4148  M   R 8722431 + 256 [dd]
>   8,0    3   177881    50.027291697  4148  U   N [dd] 1
>   8,0    3   177882    50.027293119  4148  D   R 8722175 + 512 [dd]
>   8,0    3   177883    50.030406105     0  C   R 8722175 + 512 [0]
>   8,0    3   177884    50.030600613  4148  A   R 8722687 + 256 <- (8,1) 8722624
>   8,0    3   177885    50.030601199  4148  Q   R 8722687 + 256 [dd]
>   8,0    3   177886    50.030603269  4148  G   R 8722687 + 256 [dd]
>   8,0    3   177887    50.030604463  4148  P   N [dd]
>   8,0    3   177888    50.030604799  4148  I   R 8722687 + 256 [dd]
>   8,0    3   177889    50.030731757  4148  A   R 8722943 + 256 <- (8,1) 8722880
>   8,0    3   177890    50.030732117  4148  Q   R 8722943 + 256 [dd]
>   8,0    3   177891    50.030733397  4148  M   R 8722943 + 256 [dd]
>   8,0    3   177892    50.030734882  4148  U   N [dd] 1
>   8,0    3   177893    50.030736109  4148  D   R 8722687 + 512 [dd]
>   8,0    3   177894    50.032916699     0  C   R 8722687 + 512 [0]
>   8,0    3   177895    50.033176618  4148  A   R 8723199 + 256 <- (8,1) 8723136
>   8,0    3   177896    50.033177218  4148  Q   R 8723199 + 256 [dd]
>   8,0    3   177897    50.033181433  4148  G   R 8723199 + 256 [dd]
>   8,0    3   177898    50.033184757  4148  P   N [dd]
>   8,0    3   177899    50.033185642  4148  I   R 8723199 + 256 [dd]
>   8,0    3   177900    50.033371264  4148  A   R 8723455 + 256 <- (8,1) 8723392
>   8,0    3   177901    50.033371717  4148  Q   R 8723455 + 256 [dd]
>   8,0    3   177902    50.033374015  4148  M   R 8723455 + 256 [dd]
>   8,0    3   177903    50.033376814  4148  U   N [dd] 1
>   8,0    3   177904    50.033380126  4148  D   R 8723199 + 512 [dd]
>   8,0    3   177905    50.036715133     0  C   R 8723199 + 512 [0]
>   8,0    3   177906    50.036971296  4148  A   R 8723711 + 256 <- (8,1) 8723648
>   8,0    3   177907    50.036972136  4148  Q   R 8723711 + 256 [dd]
>   8,0    3   177908    50.036975673  4148  G   R 8723711 + 256 [dd]
>   8,0    3   177909    50.036978277  4148  P   N [dd]
>   8,0    3   177910    50.036979450  4148  I   R 8723711 + 256 [dd]
>   8,0    3   177911    50.037162429  4148  A   R 8723967 + 256 <- (8,1) 8723904
>   8,0    3   177912    50.037162840  4148  Q   R 8723967 + 256 [dd]
>   8,0    3   177913    50.037164967  4148  M   R 8723967 + 256 [dd]
>   8,0    3   177914    50.037167223  4148  U   N [dd] 1
>   8,0    3   177915    50.037170001  4148  D   R 8723711 + 512 [dd]
>   8,0    3   177916    50.040521790     0  C   R 8723711 + 512 [0]
>   8,0    3   177917    50.040729738  4148  A   R 8724223 + 256 <- (8,1) 8724160
>   8,0    3   177918    50.040730200  4148  Q   R 8724223 + 256 [dd]
>   8,0    3   177919    50.040732060  4148  G   R 8724223 + 256 [dd]
>   8,0    3   177920    50.040733551  4148  P   N [dd]
>   8,0    3   177921    50.040734109  4148  I   R 8724223 + 256 [dd]
>   8,0    3   177922    50.040860173  4148  A   R 8724479 + 160 <- (8,1) 8724416
>   8,0    3   177923    50.040860536  4148  Q   R 8724479 + 160 [dd]
>   8,0    3   177924    50.040861517  4148  M   R 8724479 + 160 [dd]
>   8,0    3   177925    50.040872542  4148  A   R 1055943 + 8 <- (8,1) 1055880
>   8,0    3   177926    50.040872800  4148  Q   R 1055943 + 8 [dd]
>   8,0    3   177927    50.040874849  4148  G   R 1055943 + 8 [dd]
>   8,0    3   177928    50.040875485  4148  I   R 1055943 + 8 [dd]
>   8,0    3   177929    50.040877045  4148  U   N [dd] 2
>   8,0    3   177930    50.040878625  4148  D   R 8724223 + 416 [dd]
>   8,0    3   177931    50.040895335  4148  D   R 1055943 + 8 [dd]
>   8,0    3   177932    50.044383267     0  C   R 8724223 + 416 [0]
>   8,0    3   177933    50.044704725     0  C   R 1055943 + 8 [0]
>   8,0    3   177934    50.044749068  4148  A   R 8724639 + 96 <- (8,1) 8724576
>   8,0    3   177935    50.044749472  4148  Q   R 8724639 + 96 [dd]
>   8,0    3   177936    50.044752184  4148  G   R 8724639 + 96 [dd]
>   8,0    3   177937    50.044753552  4148  P   N [dd]
>   8,0    3   177938    50.044754032  4148  I   R 8724639 + 96 [dd]
>   8,0    3   177939    50.044896095  4148  A   R 8724735 + 256 <- (8,1) 8724672
>   8,0    3   177940    50.044896443  4148  Q   R 8724735 + 256 [dd]
>   8,0    3   177941    50.044897538  4148  M   R 8724735 + 256 [dd]
>   8,0    3   177942    50.044948546  4148  U   N [dd] 1
>   8,0    3   177943    50.044950001  4148  D   R 8724639 + 352 [dd]
>   8,0    3   177944    50.047150137     0  C   R 8724639 + 352 [0]
>   8,0    3   177945    50.047294824  4148  A   R 8724991 + 256 <- (8,1) 8724928
>   8,0    3   177946    50.047295142  4148  Q   R 8724991 + 256 [dd]
>   8,0    3   177947    50.047296978  4148  G   R 8724991 + 256 [dd]
>   8,0    3   177948    50.047298301  4148  P   N [dd]
>   8,0    3   177949    50.047298637  4148  I   R 8724991 + 256 [dd]
>   8,0    3   177950    50.047429027  4148  A   R 8725247 + 256 <- (8,1) 8725184
>   8,0    3   177951    50.047429387  4148  Q   R 8725247 + 256 [dd]
>   8,0    3   177952    50.047430479  4148  M   R 8725247 + 256 [dd]
>   8,0    3   177953    50.047431736  4148  U   N [dd] 1
>   8,0    3   177954    50.047432951  4148  D   R 8724991 + 512 [dd]
>   8,0    3   177955    50.050313976     0  C   R 8724991 + 512 [0]
>   8,0    3   177956    50.050507961  4148  A   R 8725503 + 256 <- (8,1) 8725440
>   8,0    3   177957    50.050508273  4148  Q   R 8725503 + 256 [dd]
>   8,0    3   177958    50.050510139  4148  G   R 8725503 + 256 [dd]
>   8,0    3   177959    50.050511522  4148  P   N [dd]
>   8,0    3   177960    50.050512062  4148  I   R 8725503 + 256 [dd]
>   8,0    3   177961    50.050645393  4148  A   R 8725759 + 256 <- (8,1) 8725696
>   8,0    3   177962    50.050645867  4148  Q   R 8725759 + 256 [dd]
>   8,0    3   177963    50.050647171  4148  M   R 8725759 + 256 [dd]
>   8,0    3   177964    50.050648593  4148  U   N [dd] 1
>   8,0    3   177965    50.050649985  4148  D   R 8725503 + 512 [dd]
>   8,0    3   177966    50.053380250     0  C   R 8725503 + 512 [0]
>   8,0    3   177967    50.053576324  4148  A   R 8726015 + 256 <- (8,1) 8725952
>   8,0    3   177968    50.053576615  4148  Q   R 8726015 + 256 [dd]
>   8,0    3   177969    50.053578994  4148  G   R 8726015 + 256 [dd]
>   8,0    3   177970    50.053580173  4148  P   N [dd]
>   8,0    3   177971    50.053580509  4148  I   R 8726015 + 256 [dd]
>   8,0    3   177972    50.053711503  4148  A   R 8726271 + 256 <- (8,1) 8726208
>   8,0    3   177973    50.053712001  4148  Q   R 8726271 + 256 [dd]
>   8,0    3   177974    50.053713332  4148  M   R 8726271 + 256 [dd]
>   8,0    3   177975    50.053714583  4148  U   N [dd] 1
>   8,0    3   177976    50.053715768  4148  D   R 8726015 + 512 [dd]
>   8,0    3   177977    50.056970395     0  C   R 8726015 + 512 [0]
>   8,0    3   177978    50.057161408  4148  A   R 8726527 + 256 <- (8,1) 8726464
>   8,0    3   177979    50.057161726  4148  Q   R 8726527 + 256 [dd]
>   8,0    3   177980    50.057163718  4148  G   R 8726527 + 256 [dd]
>   8,0    3   177981    50.057165098  4148  P   N [dd]
>   8,0    3   177982    50.057165431  4148  I   R 8726527 + 256 [dd]
>   8,0    3   177983    50.057294630  4148  A   R 8726783 + 256 <- (8,1) 8726720
>   8,0    3   177984    50.057294990  4148  Q   R 8726783 + 256 [dd]
>   8,0    3   177985    50.057296070  4148  M   R 8726783 + 256 [dd]
>   8,0    3   177986    50.057297402  4148  U   N [dd] 1
>   8,0    3   177987    50.057298899  4148  D   R 8726527 + 512 [dd]
>   8,0    3   177988    50.060326743     0  C   R 8726527 + 512 [0]
>   8,0    3   177989    50.060523768  4148  A   R 8727039 + 256 <- (8,1) 8726976
>   8,0    3   177990    50.060524095  4148  Q   R 8727039 + 256 [dd]
>   8,0    3   177991    50.060525910  4148  G   R 8727039 + 256 [dd]
>   8,0    3   177992    50.060527239  4148  P   N [dd]
>   8,0    3   177993    50.060527575  4148  I   R 8727039 + 256 [dd]
>   8,0    3   177994    50.060662280  4148  A   R 8727295 + 256 <- (8,1) 8727232
>   8,0    3   177995    50.060662778  4148  Q   R 8727295 + 256 [dd]
>   8,0    3   177996    50.060663993  4148  M   R 8727295 + 256 [dd]
>   8,0    3   177997    50.060665403  4148  U   N [dd] 1
>   8,0    3   177998    50.060666999  4148  D   R 8727039 + 512 [dd]
>   8,0    3   177999    50.063922341     0  C   R 8727039 + 512 [0]
>   8,0    3   178000    50.064113177  4148  A   R 8727551 + 256 <- (8,1) 8727488
>   8,0    3   178001    50.064113492  4148  Q   R 8727551 + 256 [dd]
>   8,0    3   178002    50.064115373  4148  G   R 8727551 + 256 [dd]
> 
> -2.6.30-rc6-patched
>   8,0    3   257297    50.000760847     0  C   R 9480703 + 256 [0]
>   8,0    3   257298    50.000944399  4139  A   R 9481215 + 256 <- (8,1) 9481152
>   8,0    3   257299    50.000944693  4139  Q   R 9481215 + 256 [dd]
>   8,0    3   257300    50.000946541  4139  G   R 9481215 + 256 [dd]
>   8,0    3   257301    50.000947954  4139  P   N [dd]
>   8,0    3   257302    50.000948368  4139  I   R 9481215 + 256 [dd]
>   8,0    3   257303    50.000948920  4139  U   N [dd] 2
>   8,0    3   257304    50.000950003  4139  D   R 9481215 + 256 [dd]
>   8,0    3   257305    50.000962541  4139  U   N [dd] 2
>   8,0    3   257306    50.003034240     0  C   R 9480959 + 256 [0]
>   8,0    3   257307    50.003076338     0  C   R 9481215 + 256 [0]
>   8,0    3   257308    50.003258111  4139  A   R 9481471 + 256 <- (8,1) 9481408
>   8,0    3   257309    50.003258402  4139  Q   R 9481471 + 256 [dd]
>   8,0    3   257310    50.003260190  4139  G   R 9481471 + 256 [dd]
>   8,0    3   257311    50.003261399  4139  P   N [dd]
>   8,0    3   257312    50.003261768  4139  I   R 9481471 + 256 [dd]
>   8,0    3   257313    50.003262335  4139  U   N [dd] 1
>   8,0    3   257314    50.003263406  4139  D   R 9481471 + 256 [dd]
>   8,0    3   257315    50.003430472  4139  A   R 9481727 + 256 <- (8,1) 9481664
>   8,0    3   257316    50.003430748  4139  Q   R 9481727 + 256 [dd]
>   8,0    3   257317    50.003433065  4139  G   R 9481727 + 256 [dd]
>   8,0    3   257318    50.003434343  4139  P   N [dd]
>   8,0    3   257319    50.003434658  4139  I   R 9481727 + 256 [dd]
>   8,0    3   257320    50.003435138  4139  U   N [dd] 2
>   8,0    3   257321    50.003436083  4139  D   R 9481727 + 256 [dd]
>   8,0    3   257322    50.003447795  4139  U   N [dd] 2
>   8,0    3   257323    50.004774693     0  C   R 9481471 + 256 [0]
>   8,0    3   257324    50.004959499  4139  A   R 9481983 + 256 <- (8,1) 9481920
>   8,0    3   257325    50.004959790  4139  Q   R 9481983 + 256 [dd]
>   8,0    3   257326    50.004961590  4139  G   R 9481983 + 256 [dd]
>   8,0    3   257327    50.004962793  4139  P   N [dd]
>   8,0    3   257328    50.004963153  4139  I   R 9481983 + 256 [dd]
>   8,0    3   257329    50.004964098  4139  U   N [dd] 2
>   8,0    3   257330    50.004965184  4139  D   R 9481983 + 256 [dd]
>   8,0    3   257331    50.004978967  4139  U   N [dd] 2
>   8,0    3   257332    50.006865854     0  C   R 9481727 + 256 [0]
>   8,0    3   257333    50.007052043  4139  A   R 9482239 + 256 <- (8,1) 9482176
>   8,0    3   257334    50.007052331  4139  Q   R 9482239 + 256 [dd]
>   8,0    3   257335    50.007054146  4139  G   R 9482239 + 256 [dd]
>   8,0    3   257336    50.007055355  4139  P   N [dd]
>   8,0    3   257337    50.007055724  4139  I   R 9482239 + 256 [dd]
>   8,0    3   257338    50.007056438  4139  U   N [dd] 2
>   8,0    3   257339    50.007057605  4139  D   R 9482239 + 256 [dd]
>   8,0    3   257340    50.007069963  4139  U   N [dd] 2
>   8,0    3   257341    50.008250294     0  C   R 9481983 + 256 [0]
>   8,0    3   257342    50.008431589  4139  A   R 9482495 + 256 <- (8,1) 9482432
>   8,0    3   257343    50.008431881  4139  Q   R 9482495 + 256 [dd]
>   8,0    3   257344    50.008433921  4139  G   R 9482495 + 256 [dd]
>   8,0    3   257345    50.008435097  4139  P   N [dd]
>   8,0    3   257346    50.008435466  4139  I   R 9482495 + 256 [dd]
>   8,0    3   257347    50.008436213  4139  U   N [dd] 2
>   8,0    3   257348    50.008437296  4139  D   R 9482495 + 256 [dd]
>   8,0    3   257349    50.008450034  4139  U   N [dd] 2
>   8,0    3   257350    50.010008843     0  C   R 9482239 + 256 [0]
>   8,0    3   257351    50.010135287  4139  C   R 9482495 + 256 [0]
>   8,0    3   257352    50.010226816  4139  A   R 9482751 + 256 <- (8,1) 9482688
>   8,0    3   257353    50.010227107  4139  Q   R 9482751 + 256 [dd]
>   8,0    3   257354    50.010229363  4139  G   R 9482751 + 256 [dd]
>   8,0    3   257355    50.010230728  4139  P   N [dd]
>   8,0    3   257356    50.010231097  4139  I   R 9482751 + 256 [dd]
>   8,0    3   257357    50.010231655  4139  U   N [dd] 1
>   8,0    3   257358    50.010232696  4139  D   R 9482751 + 256 [dd]
>   8,0    3   257359    50.010380946  4139  A   R 9483007 + 256 <- (8,1) 9482944
>   8,0    3   257360    50.010381264  4139  Q   R 9483007 + 256 [dd]
>   8,0    3   257361    50.010383358  4139  G   R 9483007 + 256 [dd]
>   8,0    3   257362    50.010384429  4139  P   N [dd]
>   8,0    3   257363    50.010384741  4139  I   R 9483007 + 256 [dd]
>   8,0    3   257364    50.010385395  4139  U   N [dd] 2
>   8,0    3   257365    50.010386364  4139  D   R 9483007 + 256 [dd]
>   8,0    3   257366    50.010397869  4139  U   N [dd] 2
>   8,0    3   257367    50.014210132     0  C   R 9482751 + 256 [0]
>   8,0    3   257368    50.014252938     0  C   R 9483007 + 256 [0]
>   8,0    3   257369    50.014430811  4139  A   R 9483263 + 256 <- (8,1) 9483200
>   8,0    3   257370    50.014431105  4139  Q   R 9483263 + 256 [dd]
>   8,0    3   257371    50.014433139  4139  G   R 9483263 + 256 [dd]
>   8,0    3   257372    50.014434520  4139  P   N [dd]
>   8,0    3   257373    50.014435110  4139  I   R 9483263 + 256 [dd]
>   8,0    3   257374    50.014435674  4139  U   N [dd] 1
>   8,0    3   257375    50.014436770  4139  D   R 9483263 + 256 [dd]
>   8,0    3   257376    50.014592117  4139  A   R 9483519 + 256 <- (8,1) 9483456
>   8,0    3   257377    50.014592573  4139  Q   R 9483519 + 256 [dd]
>   8,0    3   257378    50.014594391  4139  G   R 9483519 + 256 [dd]
>   8,0    3   257379    50.014595504  4139  P   N [dd]
>   8,0    3   257380    50.014595876  4139  I   R 9483519 + 256 [dd]
>   8,0    3   257381    50.014596366  4139  U   N [dd] 2
>   8,0    3   257382    50.014597368  4139  D   R 9483519 + 256 [dd]
>   8,0    3   257383    50.014609521  4139  U   N [dd] 2
>   8,0    3   257384    50.015937813     0  C   R 9483263 + 256 [0]
>   8,0    3   257385    50.016124825  4139  A   R 9483775 + 256 <- (8,1) 9483712
>   8,0    3   257386    50.016125116  4139  Q   R 9483775 + 256 [dd]
>   8,0    3   257387    50.016127162  4139  G   R 9483775 + 256 [dd]
>   8,0    3   257388    50.016128569  4139  P   N [dd]
>   8,0    3   257389    50.016128983  4139  I   R 9483775 + 256 [dd]
>   8,0    3   257390    50.016129538  4139  U   N [dd] 2
>   8,0    3   257391    50.016130627  4139  D   R 9483775 + 256 [dd]
>   8,0    3   257392    50.016143077  4139  U   N [dd] 2
>   8,0    3   257393    50.016925304     0  C   R 9483519 + 256 [0]
>   8,0    3   257394    50.017111307  4139  A   R 9484031 + 256 <- (8,1) 9483968
>   8,0    3   257395    50.017111598  4139  Q   R 9484031 + 256 [dd]
>   8,0    3   257396    50.017113410  4139  G   R 9484031 + 256 [dd]
>   8,0    3   257397    50.017114835  4139  P   N [dd]
>   8,0    3   257398    50.017115213  4139  I   R 9484031 + 256 [dd]
>   8,0    3   257399    50.017115765  4139  U   N [dd] 2
>   8,0    3   257400    50.017116839  4139  D   R 9484031 + 256 [dd]
>   8,0    3   257401    50.017129023  4139  U   N [dd] 2
>   8,0    3   257402    50.017396693     0  C   R 9483775 + 256 [0]
>   8,0    3   257403    50.017584595  4139  A   R 9484287 + 256 <- (8,1) 9484224
>   8,0    3   257404    50.017585018  4139  Q   R 9484287 + 256 [dd]
>   8,0    3   257405    50.017586866  4139  G   R 9484287 + 256 [dd]
>   8,0    3   257406    50.017587997  4139  P   N [dd]
>   8,0    3   257407    50.017588393  4139  I   R 9484287 + 256 [dd]
>   8,0    3   257408    50.017589105  4139  U   N [dd] 2
>   8,0    3   257409    50.017590173  4139  D   R 9484287 + 256 [dd]
>   8,0    3   257410    50.017602614  4139  U   N [dd] 2
>   8,0    3   257411    50.020578876     0  C   R 9484031 + 256 [0]
>   8,0    3   257412    50.020721857  4139  C   R 9484287 + 256 [0]
>   8,0    3   257413    50.020803183  4139  A   R 9484543 + 256 <- (8,1) 9484480
>   8,0    3   257414    50.020803507  4139  Q   R 9484543 + 256 [dd]
>   8,0    3   257415    50.020805256  4139  G   R 9484543 + 256 [dd]
>   8,0    3   257416    50.020806672  4139  P   N [dd]
>   8,0    3   257417    50.020807065  4139  I   R 9484543 + 256 [dd]
>   8,0    3   257418    50.020807668  4139  U   N [dd] 1
>   8,0    3   257419    50.020808733  4139  D   R 9484543 + 256 [dd]
>   8,0    3   257420    50.020957132  4139  A   R 9484799 + 256 <- (8,1) 9484736
>   8,0    3   257421    50.020957423  4139  Q   R 9484799 + 256 [dd]
>   8,0    3   257422    50.020959205  4139  G   R 9484799 + 256 [dd]
>   8,0    3   257423    50.020960276  4139  P   N [dd]
>   8,0    3   257424    50.020960594  4139  I   R 9484799 + 256 [dd]
>   8,0    3   257425    50.020961062  4139  U   N [dd] 2
>   8,0    3   257426    50.020961959  4139  D   R 9484799 + 256 [dd]
>   8,0    3   257427    50.020974191  4139  U   N [dd] 2
>   8,0    3   257428    50.023987847     0  C   R 9484543 + 256 [0]
>   8,0    3   257429    50.024093062  4139  C   R 9484799 + 256 [0]
>   8,0    3   257430    50.024207161  4139  A   R 9485055 + 256 <- (8,1) 9484992
>   8,0    3   257431    50.024207434  4139  Q   R 9485055 + 256 [dd]
>   8,0    3   257432    50.024209567  4139  G   R 9485055 + 256 [dd]
>   8,0    3   257433    50.024210728  4139  P   N [dd]
>   8,0    3   257434    50.024211097  4139  I   R 9485055 + 256 [dd]
>   8,0    3   257435    50.024211661  4139  U   N [dd] 1
>   8,0    3   257436    50.024212693  4139  D   R 9485055 + 256 [dd]
>   8,0    3   257437    50.024359266  4139  A   R 9485311 + 256 <- (8,1) 9485248
>   8,0    3   257438    50.024359584  4139  Q   R 9485311 + 256 [dd]
>   8,0    3   257439    50.024361720  4139  G   R 9485311 + 256 [dd]
>   8,0    3   257440    50.024362794  4139  P   N [dd]
>   8,0    3   257441    50.024363106  4139  I   R 9485311 + 256 [dd]
>   8,0    3   257442    50.024363760  4139  U   N [dd] 2
>   8,0    3   257443    50.024364759  4139  D   R 9485311 + 256 [dd]
>   8,0    3   257444    50.024376535  4139  U   N [dd] 2
>   8,0    3   257445    50.026532544     0  C   R 9485055 + 256 [0]
>   8,0    3   257446    50.026714236  4139  A   R 9485567 + 256 <- (8,1) 9485504
>   8,0    3   257447    50.026714524  4139  Q   R 9485567 + 256 [dd]
>   8,0    3   257448    50.026716354  4139  G   R 9485567 + 256 [dd]
>   8,0    3   257449    50.026717791  4139  P   N [dd]
>   8,0    3   257450    50.026718175  4139  I   R 9485567 + 256 [dd]
>   8,0    3   257451    50.026718778  4139  U   N [dd] 2
>   8,0    3   257452    50.026719876  4139  D   R 9485567 + 256 [dd]
>   8,0    3   257453    50.026736383  4139  U   N [dd] 2
>   8,0    3   257454    50.028531879     0  C   R 9485311 + 256 [0]
>   8,0    3   257455    50.028684347  4139  C   R 9485567 + 256 [0]
>   8,0    3   257456    50.028758787  4139  A   R 9485823 + 256 <- (8,1) 9485760
>   8,0    3   257457    50.028759069  4139  Q   R 9485823 + 256 [dd]
>   8,0    3   257458    50.028760884  4139  G   R 9485823 + 256 [dd]
>   8,0    3   257459    50.028762099  4139  P   N [dd]
>   8,0    3   257460    50.028762447  4139  I   R 9485823 + 256 [dd]
>   8,0    3   257461    50.028763038  4139  U   N [dd] 1
>   8,0    3   257462    50.028764268  4139  D   R 9485823 + 256 [dd]
>   8,0    3   257463    50.028909841  4139  A   R 9486079 + 256 <- (8,1) 9486016
>   8,0    3   257464    50.028910156  4139  Q   R 9486079 + 256 [dd]
>   8,0    3   257465    50.028911896  4139  G   R 9486079 + 256 [dd]
>   8,0    3   257466    50.028912964  4139  P   N [dd]
>   8,0    3   257467    50.028913270  4139  I   R 9486079 + 256 [dd]
>   8,0    3   257468    50.028913912  4139  U   N [dd] 2
>   8,0    3   257469    50.028914878  4139  D   R 9486079 + 256 [dd]
>   8,0    3   257470    50.028927497  4139  U   N [dd] 2
>   8,0    3   257471    50.031158357     0  C   R 9485823 + 256 [0]
>   8,0    3   257472    50.031292365  4139  C   R 9486079 + 256 [0]
>   8,0    3   257473    50.031369697  4139  A   R 9486335 + 160 <- (8,1) 9486272
>   8,0    3   257474    50.031369988  4139  Q   R 9486335 + 160 [dd]
>   8,0    3   257475    50.031371779  4139  G   R 9486335 + 160 [dd]
>   8,0    3   257476    50.031372850  4139  P   N [dd]
>   8,0    3   257477    50.031373198  4139  I   R 9486335 + 160 [dd]
>   8,0    3   257478    50.031384931  4139  A   R 1056639 + 8 <- (8,1) 1056576
>   8,0    3   257479    50.031385201  4139  Q   R 1056639 + 8 [dd]
>   8,0    3   257480    50.031388480  4139  G   R 1056639 + 8 [dd]
>   8,0    3   257481    50.031388904  4139  I   R 1056639 + 8 [dd]
>   8,0    3   257482    50.031390362  4139  U   N [dd] 2
>   8,0    3   257483    50.031391523  4139  D   R 9486335 + 160 [dd]
>   8,0    3   257484    50.031403403  4139  D   R 1056639 + 8 [dd]
>   8,0    3   257485    50.033630747     0  C   R 1056639 + 8 [0]
>   8,0    3   257486    50.033690300  4139  A   R 9486495 + 96 <- (8,1) 9486432
>   8,0    3   257487    50.033690810  4139  Q   R 9486495 + 96 [dd]
>   8,0    3   257488    50.033694581  4139  G   R 9486495 + 96 [dd]
>   8,0    3   257489    50.033696739  4139  P   N [dd]
>   8,0    3   257490    50.033697357  4139  I   R 9486495 + 96 [dd]
>   8,0    3   257491    50.033698611  4139  U   N [dd] 2
>   8,0    3   257492    50.033700945  4139  D   R 9486495 + 96 [dd]
>   8,0    3   257493    50.033727763  4139  C   R 9486335 + 160 [0]
>   8,0    3   257494    50.033996024  4139  A   R 9486591 + 256 <- (8,1) 9486528
>   8,0    3   257495    50.033996396  4139  Q   R 9486591 + 256 [dd]
>   8,0    3   257496    50.034000030  4139  G   R 9486591 + 256 [dd]
>   8,0    3   257497    50.034002268  4139  P   N [dd]
>   8,0    3   257498    50.034002820  4139  I   R 9486591 + 256 [dd]
>   8,0    3   257499    50.034003924  4139  U   N [dd] 2
>   8,0    3   257500    50.034006201  4139  D   R 9486591 + 256 [dd]
>   8,0    3   257501    50.034091438  4139  U   N [dd] 2
>   8,0    3   257502    50.034637372     0  C   R 9486495 + 96 [0]
>   8,0    3   257503    50.034841508  4139  A   R 9486847 + 256 <- (8,1) 9486784
>   8,0    3   257504    50.034842072  4139  Q   R 9486847 + 256 [dd]
>   8,0    3   257505    50.034846117  4139  G   R 9486847 + 256 [dd]
>   8,0    3   257506    50.034848676  4139  P   N [dd]
>   8,0    3   257507    50.034849384  4139  I   R 9486847 + 256 [dd]
>   8,0    3   257508    50.034850545  4139  U   N [dd] 2
>   8,0    3   257509    50.034852795  4139  D   R 9486847 + 256 [dd]
>   8,0    3   257510    50.034875503  4139  U   N [dd] 2
>   8,0    3   257511    50.035370009     0  C   R 9486591 + 256 [0]
>   8,0    3   257512    50.035622315  4139  A   R 9487103 + 256 <- (8,1) 9487040
>   8,0    3   257513    50.035622954  4139  Q   R 9487103 + 256 [dd]
>   8,0    3   257514    50.035627101  4139  G   R 9487103 + 256 [dd]
>   8,0    3   257515    50.035629510  4139  P   N [dd]
>   8,0    3   257516    50.035630143  4139  I   R 9487103 + 256 [dd]
>   8,0    3   257517    50.035631058  4139  U   N [dd] 2
>   8,0    3   257518    50.035632657  4139  D   R 9487103 + 256 [dd]
>   8,0    3   257519    50.035656358  4139  U   N [dd] 2
>   8,0    3   257520    50.036703329     0  C   R 9486847 + 256 [0]
>   8,0    3   257521    50.036963604  4139  A   R 9487359 + 256 <- (8,1) 9487296
>   8,0    3   257522    50.036964057  4139  Q   R 9487359 + 256 [dd]
>   8,0    3   257523    50.036967636  4139  G   R 9487359 + 256 [dd]
>   8,0    3   257524    50.036969710  4139  P   N [dd]
>   8,0    3   257525    50.036970586  4139  I   R 9487359 + 256 [dd]
>   8,0    3   257526    50.036971684  4139  U   N [dd] 2
>   8,0    3   257527    50.036973631  4139  D   R 9487359 + 256 [dd]
>   8,0    3   257528    50.036995034  4139  U   N [dd] 2
>   8,0    3   257529    50.038904428     0  C   R 9487103 + 256 [0]
>   8,0    3   257530    50.039161508  4139  A   R 9487615 + 256 <- (8,1) 9487552
>   8,0    3   257531    50.039161934  4139  Q   R 9487615 + 256 [dd]
>   8,0    3   257532    50.039165834  4139  G   R 9487615 + 256 [dd]
>   8,0    3   257533    50.039168561  4139  P   N [dd]
>   8,0    3   257534    50.039169353  4139  I   R 9487615 + 256 [dd]
>   8,0    3   257535    50.039170343  4139  U   N [dd] 2
>   8,0    3   257536    50.039171645  4139  D   R 9487615 + 256 [dd]
>   8,0    3   257537    50.039193195  4139  U   N [dd] 2
>   8,0    3   257538    50.040570003     0  C   R 9487359 + 256 [0]
>   8,0    3   257539    50.040842161  4139  A   R 9487871 + 256 <- (8,1) 9487808
>   8,0    3   257540    50.040842827  4139  Q   R 9487871 + 256 [dd]
>   8,0    3   257541    50.040846803  4139  G   R 9487871 + 256 [dd]
>   8,0    3   257542    50.040849902  4139  P   N [dd]
>   8,0    3   257543    50.040850715  4139  I   R 9487871 + 256 [dd]
>   8,0    3   257544    50.040851642  4139  U   N [dd] 2
>   8,0    3   257545    50.040853658  4139  D   R 9487871 + 256 [dd]
>   8,0    3   257546    50.040876270  4139  U   N [dd] 2
>   8,0    3   257547    50.042081391     0  C   R 9487615 + 256 [0]
>   8,0    3   257548    50.042215837  4139  C   R 9487871 + 256 [0]
>   8,0    3   257549    50.042316192  4139  A   R 9488127 + 256 <- (8,1) 9488064
>   8,0    3   257550    50.042316633  4139  Q   R 9488127 + 256 [dd]
>   8,0    3   257551    50.042319213  4139  G   R 9488127 + 256 [dd]
>   8,0    3   257552    50.042320803  4139  P   N [dd]
>   8,0    3   257553    50.042321412  4139  I   R 9488127 + 256 [dd]
>   8,0    3   257554    50.042322219  4139  U   N [dd] 1
>   8,0    3   257555    50.042323362  4139  D   R 9488127 + 256 [dd]
>   8,0    3   257556    50.042484350  4139  A   R 9488383 + 256 <- (8,1) 9488320
>   8,0    3   257557    50.042484602  4139  Q   R 9488383 + 256 [dd]
>   8,0    3   257558    50.042486744  4139  G   R 9488383 + 256 [dd]
>   8,0    3   257559    50.042487908  4139  P   N [dd]
>   8,0    3   257560    50.042488223  4139  I   R 9488383 + 256 [dd]
>   8,0    3   257561    50.042488754  4139  U   N [dd] 2
>   8,0    3   257562    50.042489927  4139  D   R 9488383 + 256 [dd]
>   8,0    3   257563    50.042502678  4139  U   N [dd] 2
>   8,0    3   257564    50.045166592     0  C   R 9488127 + 256 [0]
>   8,0    3   257565    50.045355163  4139  A   R 9488639 + 256 <- (8,1) 9488576
>   8,0    3   257566    50.045355493  4139  Q   R 9488639 + 256 [dd]
>   8,0    3   257567    50.045357497  4139  G   R 9488639 + 256 [dd]
>   8,0    3   257568    50.045358673  4139  P   N [dd]
>   8,0    3   257569    50.045359267  4139  I   R 9488639 + 256 [dd]
>   8,0    3   257570    50.045359831  4139  U   N [dd] 2
>   8,0    3   257571    50.045360911  4139  D   R 9488639 + 256 [dd]
>   8,0    3   257572    50.045373959  4139  U   N [dd] 2
>   8,0    3   257573    50.046450730     0  C   R 9488383 + 256 [0]
>   8,0    3   257574    50.046641639  4139  A   R 9488895 + 256 <- (8,1) 9488832
>   8,0    3   257575    50.046642086  4139  Q   R 9488895 + 256 [dd]
>   8,0    3   257576    50.046643937  4139  G   R 9488895 + 256 [dd]
>   8,0    3   257577    50.046645092  4139  P   N [dd]
>   8,0    3   257578    50.046645527  4139  I   R 9488895 + 256 [dd]
>   8,0    3   257579    50.046646244  4139  U   N [dd] 2
>   8,0    3   257580    50.046647327  4139  D   R 9488895 + 256 [dd]
>   8,0    3   257581    50.046660234  4139  U   N [dd] 2
>   8,0    3   257582    50.047826305     0  C   R 9488639 + 256 [0]
>   8,0    3   257583    50.048011468  4139  A   R 9489151 + 256 <- (8,1) 9489088
>   8,0    3   257584    50.048011762  4139  Q   R 9489151 + 256 [dd]
>   8,0    3   257585    50.048013793  4139  G   R 9489151 + 256 [dd]
>   8,0    3   257586    50.048014966  4139  P   N [dd]
>   8,0    3   257587    50.048015380  4139  I   R 9489151 + 256 [dd]
>   8,0    3   257588    50.048016112  4139  U   N [dd] 2
>   8,0    3   257589    50.048017202  4139  D   R 9489151 + 256 [dd]
>   8,0    3   257590    50.048029553  4139  U   N [dd] 2
>   8,0    3   257591    50.049319830     0  C   R 9488895 + 256 [0]
>   8,0    3   257592    50.049446089  4139  C   R 9489151 + 256 [0]
>   8,0    3   257593    50.049545199  4139  A   R 9489407 + 256 <- (8,1) 9489344
>   8,0    3   257594    50.049545628  4139  Q   R 9489407 + 256 [dd]
>   8,0    3   257595    50.049547512  4139  G   R 9489407 + 256 [dd]
>   8,0    3   257596    50.049548886  4139  P   N [dd]
>   8,0    3   257597    50.049549318  4139  I   R 9489407 + 256 [dd]
>   8,0    3   257598    50.049550047  4139  U   N [dd] 1
>   8,0    3   257599    50.049551241  4139  D   R 9489407 + 256 [dd]
>   8,0    3   257600    50.049699283  4139  A   R 9489663 + 256 <- (8,1) 9489600
>   8,0    3   257601    50.049699556  4139  Q   R 9489663 + 256 [dd]
>   8,0    3   257602    50.049701266  4139  G   R 9489663 + 256 [dd]
>   8,0    3   257603    50.049702310  4139  P   N [dd]
>   8,0    3   257604    50.049702656  4139  I   R 9489663 + 256 [dd]
>   8,0    3   257605    50.049703118  4139  U   N [dd] 2
>   8,0    3   257606    50.049704020  4139  D   R 9489663 + 256 [dd]
>   8,0    3   257607    50.049715940  4139  U   N [dd] 2
>   8,0    3   257608    50.052662150     0  C   R 9489407 + 256 [0]
>   8,0    3   257609    50.052853688  4139  A   R 9489919 + 256 <- (8,1) 9489856
>   8,0    3   257610    50.052853985  4139  Q   R 9489919 + 256 [dd]
>   8,0    3   257611    50.052855869  4139  G   R 9489919 + 256 [dd]
>   8,0    3   257612    50.052857057  4139  P   N [dd]
>   8,0    3   257613    50.052857423  4139  I   R 9489919 + 256 [dd]
>   8,0    3   257614    50.052858065  4139  U   N [dd] 2
>   8,0    3   257615    50.052859164  4139  D   R 9489919 + 256 [dd]
>   8,0    3   257616    50.052871806  4139  U   N [dd] 2
>   8,0    3   257617    50.053470795     0  C   R 9489663 + 256 [0]
>   8,0    3   257618    50.053661719  4139  A   R 9490175 + 256 <- (8,1) 9490112
>   8,0    3   257619    50.053662097  4139  Q   R 9490175 + 256 [dd]
>   8,0    3   257620    50.053663891  4139  G   R 9490175 + 256 [dd]
>   8,0    3   257621    50.053665034  4139  P   N [dd]
>   8,0    3   257622    50.053665436  4139  I   R 9490175 + 256 [dd]
>   8,0    3   257623    50.053665982  4139  U   N [dd] 2
>   8,0    3   257624    50.053667077  4139  D   R 9490175 + 256 [dd]
>   8,0    3   257625    50.053679732  4139  U   N [dd] 2
>   8,0    3   257626    50.055776383     0  C   R 9489919 + 256 [0]
>   8,0    3   257627    50.055915017  4139  C   R 9490175 + 256 [0]
>   8,0    3   257628    50.055997812  4139  A   R 9490431 + 256 <- (8,1) 9490368
>   8,0    3   257629    50.055998085  4139  Q   R 9490431 + 256 [dd]
>   8,0    3   257630    50.055999867  4139  G   R 9490431 + 256 [dd]
>   8,0    3   257631    50.056001049  4139  P   N [dd]
>   8,0    3   257632    50.056001451  4139  I   R 9490431 + 256 [dd]
>   8,0    3   257633    50.056002189  4139  U   N [dd] 1
>   8,0    3   257634    50.056003197  4139  D   R 9490431 + 256 [dd]
>   8,0    3   257635    50.056149977  4139  A   R 9490687 + 256 <- (8,1) 9490624
>   8,0    3   257636    50.056150279  4139  Q   R 9490687 + 256 [dd]
>   8,0    3   257637    50.056152047  4139  G   R 9490687 + 256 [dd]
>   8,0    3   257638    50.056153109  4139  P   N [dd]
>   8,0    3   257639    50.056153442  4139  I   R 9490687 + 256 [dd]
>   8,0    3   257640    50.056153904  4139  U   N [dd] 2
>   8,0    3   257641    50.056154852  4139  D   R 9490687 + 256 [dd]
>   8,0    3   257642    50.056166948  4139  U   N [dd] 2
>   8,0    3   257643    50.057600660     0  C   R 9490431 + 256 [0]
>   8,0    3   257644    50.057786753  4139  A   R 9490943 + 256 <- (8,1) 9490880
>   8,0    3   257645    50.057787050  4139  Q   R 9490943 + 256 [dd]
>   8,0    3   257646    50.057788865  4139  G   R 9490943 + 256 [dd]
>   8,0    3   257647    50.057790236  4139  P   N [dd]
>   8,0    3   257648    50.057790614  4139  I   R 9490943 + 256 [dd]
>   8,0    3   257649    50.057791169  4139  U   N [dd] 2
>   8,0    3   257650    50.057792246  4139  D   R 9490943 + 256 [dd]
>   8,0    3   257651    50.057804469  4139  U   N [dd] 2
>   8,0    3   257652    50.060322995     0  C   R 9490687 + 256 [0]
>   8,0    3   257653    50.060464005  4139  C   R 9490943 + 256 [0]
>   8,0    3   257654    50.060548216  4139  A   R 9491199 + 256 <- (8,1) 9491136
>   8,0    3   257655    50.060548696  4139  Q   R 9491199 + 256 [dd]
>   8,0    3   257656    50.060550922  4139  G   R 9491199 + 256 [dd]
>   8,0    3   257657    50.060552096  4139  P   N [dd]
>   8,0    3   257658    50.060552531  4139  I   R 9491199 + 256 [dd]
>   8,0    3   257659    50.060553101  4139  U   N [dd] 1
>   8,0    3   257660    50.060554100  4139  D   R 9491199 + 256 [dd]
>   8,0    3   257661    50.060701569  4139  A   R 9491455 + 256 <- (8,1) 9491392
>   8,0    3   257662    50.060701890  4139  Q   R 9491455 + 256 [dd]
>   8,0    3   257663    50.060703993  4139  G   R 9491455 + 256 [dd]
>   8,0    3   257664    50.060705070  4139  P   N [dd]
>   8,0    3   257665    50.060705385  4139  I   R 9491455 + 256 [dd]
>   8,0    3   257666    50.060706012  4139  U   N [dd] 2
>   8,0    3   257667    50.060706987  4139  D   R 9491455 + 256 [dd]
>   8,0    3   257668    50.060718784  4139  U   N [dd] 2
>   8,0    3   257669    50.062964966     0  C   R 9491199 + 256 [0]
>   8,0    3   257670    50.063102772  4139  C   R 9491455 + 256 [0]
>   8,0    3   257671    50.063182666  4139  A   R 9491711 + 256 <- (8,1) 9491648
>   8,0    3   257672    50.063182939  4139  Q   R 9491711 + 256 [dd]
>   8,0    3   257673    50.063184889  4139  G   R 9491711 + 256 [dd]
>   8,0    3   257674    50.063186074  4139  P   N [dd]
>   8,0    3   257675    50.063186440  4139  I   R 9491711 + 256 [dd]
>   8,0    3   257676    50.063187271  4139  U   N [dd] 1
>   8,0    3   257677    50.063188312  4139  D   R 9491711 + 256 [dd]
>   8,0    3   257678    50.063340467  4139  A   R 9491967 + 256 <- (8,1) 9491904
>   8,0    3   257679    50.063340749  4139  Q   R 9491967 + 256 [dd]
>   8,0    3   257680    50.063342529  4139  G   R 9491967 + 256 [dd]
>   8,0    3   257681    50.063343597  4139  P   N [dd]
>   8,0    3   257682    50.063343915  4139  I   R 9491967 + 256 [dd]
>   8,0    3   257683    50.063344374  4139  U   N [dd] 2
>   8,0    3   257684    50.063345313  4139  D   R 9491967 + 256 [dd]
>   8,0    3   257685    50.063357370  4139  U   N [dd] 2
>   8,0    3   257686    50.066605011     0  C   R 9491711 + 256 [0]
>   8,0    3   257687    50.066643587     0  C   R 9491967 + 256 [0]
>   8,0    3   257688    50.066821310  4139  A   R 9492223 + 256 <- (8,1) 9492160
>   8,0    3   257689    50.066821601  4139  Q   R 9492223 + 256 [dd]
>   8,0    3   257690    50.066823605  4139  G   R 9492223 + 256 [dd]
>   8,0    3   257691    50.066825063  4139  P   N [dd]
> 
> 
> 
> >
> >
> >> 
> >>                if (PageReadahead(page))
> >>                         page_cache_async_readahead()
> >>                 if (!PageUptodate(page))
> >>                                 goto page_not_up_to_date;
> >>                 //...
> >> page_not_up_to_date:
> >>                 lock_page_killable(page);
> >> 
> >> Therefore explicit unplugging can help.
> >> 
> >> Signed-off-by: Hisashi Hifumi <hifumi.hisashi@oss.ntt.co.jp>
> >> Acked-by: Wu Fengguang <fengguang.wu@intel.com> 
> >> 
> >> 
> >>  mm/readahead.c |   10 ++++++++++
> >>  1 file changed, 10 insertions(+)
> >> 
> >> --- linux.orig/mm/readahead.c
> >> +++ linux/mm/readahead.c
> >> @@ -490,5 +490,15 @@ page_cache_async_readahead(struct addres
> >>  
> >>  	/* do read-ahead */
> >>  	ondemand_readahead(mapping, ra, filp, true, offset, req_size);
> >> +
> >> +	/*
> >> +	* Normally the current page is !uptodate and lock_page() will be
> >> +	* immediately called to implicitly unplug the device. However this
> >> +	* is not always true for RAID conifgurations, where data arrives
> >> +	* not strictly in their submission order. In this case we need to
> >> +	* explicitly kick off the IO.
> >> +	*/
> >> +	if (PageUptodate(page))
> >> +		blk_run_backing_dev(mapping->backing_dev_info, NULL);
> >>  }
> >>  EXPORT_SYMBOL_GPL(page_cache_async_readahead); 
> >> 
> 



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Wed, May 27, 2009 at 11:06:37AM +0800, Hisashi Hifumi wrote:
> 
> At 11:57 09/05/27, Wu Fengguang wrote:
> >On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
> >> 
> >> At 11:36 09/05/27, Wu Fengguang wrote:
> >> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
> >> >>
> >> >> At 11:09 09/05/27, Wu Fengguang wrote:
> >> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >> >> >>
> >> >> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >> >> >
> >> >> >> >> > I tested above patch, and I got same performance number.
> >> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >> >> >>
> >> >> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> >> >> demands some research work.  The above check is to confirm if it's
> >> >> >> >> the PageUptodate() case that makes the difference. So why that case
> >> >> >> >> happens so frequently so as to impact the performance? Will it also
> >> >> >> >> happen in NFS?
> >> >> >> >>
> >> >> >> >> The problem is readahead IO pipeline is not running smoothly, which is
> >> >> >> >> undesirable and not well understood for now.
> >> >> >> >
> >> >> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >> >> >reduction in time for a linear read? I'd be surprised if the workload
> >> >> >>
> >> >> >> Hi Andrew.
> >> >> >> Yes, I tested this with dd.
> >> >> >>
> >> >> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >> >> >
> >> >> >> >Have you been able to reproduce this in your testing?
> >> >> >>
> >> >> >> Yes, this test on my environment is reproducible.
> >> >> >
> >> >> >Hisashi, does your environment have some special configurations?
> >> >>
> >> >> Hi.
> >> >> My testing environment is as follows:
> >> >> Hardware: HP DL580
> >> >> CPU:Xeon 3.2GHz *4 HT enabled
> >> >> Memory:8GB
> >> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> >> >
> >> >This is a big hardware RAID. What's the readahead size?
> >> >
> >> >The numbers look too small for a 7 disk RAID:
> >> >
> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
> >> >        >
> >> >        > -2.6.30-rc6
> >> >        > 1048576+0 records in
> >> >        > 1048576+0 records out
> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> >        >
> >> >        > -2.6.30-rc6-patched
> >> >        > 1048576+0 records in
> >> >        > 1048576+0 records out
> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> >
> >> >I'd suggest you to configure the array properly before coming back to
> >> >measuring the impact of this patch.
> >> 
> >> 
> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
> >this directory.
> >
> >I mean, you should get >300MB/s throughput with 7 disks, and you
> >should seek ways to achieve that before testing out this patch :-)
> 
> Throughput number of storage array is very from one product to another.
> On my hardware environment I think this number is valid and
> my patch is effective.

What's your readahead size? Is it large enough to cover the stripe width?

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 11:37 09/06/01, Wu Fengguang wrote:
>On Wed, May 27, 2009 at 11:06:37AM +0800, Hisashi Hifumi wrote:
>> 
>> At 11:57 09/05/27, Wu Fengguang wrote:
>> >On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
>> >> 
>> >> At 11:36 09/05/27, Wu Fengguang wrote:
>> >> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
>> >> >>
>> >> >> At 11:09 09/05/27, Wu Fengguang wrote:
>> >> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> >> >> >>
>> >> >> >> At 08:42 09/05/27, Andrew Morton wrote:
>> >> >> >> >On Fri, 22 May 2009 10:33:23 +0800
>> >> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> >> >> >
>> >> >> >> >> > I tested above patch, and I got same performance number.
>> >> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> >> >> >>
>> >> >> >> >> Thanks!  This is an interesting micro timing behavior that
>> >> >> >> >> demands some research work.  The above check is to confirm if it's
>> >> >> >> >> the PageUptodate() case that makes the difference. So why that case
>> >> >> >> >> happens so frequently so as to impact the performance? Will it also
>> >> >> >> >> happen in NFS?
>> >> >> >> >>
>> >> >> >> >> The problem is readahead IO pipeline is not running smoothly, 
>which is
>> >> >> >> >> undesirable and not well understood for now.
>> >> >> >> >
>> >> >> >> >The patch causes a remarkably large performance increase.  A 9%
>> >> >> >> >reduction in time for a linear read? I'd be surprised if the workload
>> >> >> >>
>> >> >> >> Hi Andrew.
>> >> >> >> Yes, I tested this with dd.
>> >> >> >>
>> >> >> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
>> >> >> >> >
>> >> >> >> >Have you been able to reproduce this in your testing?
>> >> >> >>
>> >> >> >> Yes, this test on my environment is reproducible.
>> >> >> >
>> >> >> >Hisashi, does your environment have some special configurations?
>> >> >>
>> >> >> Hi.
>> >> >> My testing environment is as follows:
>> >> >> Hardware: HP DL580
>> >> >> CPU:Xeon 3.2GHz *4 HT enabled
>> >> >> Memory:8GB
>> >> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>> >> >
>> >> >This is a big hardware RAID. What's the readahead size?
>> >> >
>> >> >The numbers look too small for a 7 disk RAID:
>> >> >
>> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
>> >> >        >
>> >> >        > -2.6.30-rc6
>> >> >        > 1048576+0 records in
>> >> >        > 1048576+0 records out
>> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> >> >        >
>> >> >        > -2.6.30-rc6-patched
>> >> >        > 1048576+0 records in
>> >> >        > 1048576+0 records out
>> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> >> >
>> >> >I'd suggest you to configure the array properly before coming back to
>> >> >measuring the impact of this patch.
>> >> 
>> >> 
>> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
>> >this directory.
>> >
>> >I mean, you should get >300MB/s throughput with 7 disks, and you
>> >should seek ways to achieve that before testing out this patch :-)
>> 
>> Throughput number of storage array is very from one product to another.
>> On my hardware environment I think this number is valid and
>> my patch is effective.
>
>What's your readahead size? Is it large enough to cover the stripe width?

Do you mean strage's readahead size?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Mon, Jun 01, 2009 at 10:51:56AM +0800, Hisashi Hifumi wrote:
> 
> At 11:37 09/06/01, Wu Fengguang wrote:
> >On Wed, May 27, 2009 at 11:06:37AM +0800, Hisashi Hifumi wrote:
> >> 
> >> At 11:57 09/05/27, Wu Fengguang wrote:
> >> >On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
> >> >> 
> >> >> At 11:36 09/05/27, Wu Fengguang wrote:
> >> >> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
> >> >> >>
> >> >> >> At 11:09 09/05/27, Wu Fengguang wrote:
> >> >> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >> >> >> >>
> >> >> >> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >> >> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >> >> >> >
> >> >> >> >> >> > I tested above patch, and I got same performance number.
> >> >> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >> >> >> >>
> >> >> >> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> >> >> >> demands some research work.  The above check is to confirm if it's
> >> >> >> >> >> the PageUptodate() case that makes the difference. So why that case
> >> >> >> >> >> happens so frequently so as to impact the performance? Will it also
> >> >> >> >> >> happen in NFS?
> >> >> >> >> >>
> >> >> >> >> >> The problem is readahead IO pipeline is not running smoothly, 
> >which is
> >> >> >> >> >> undesirable and not well understood for now.
> >> >> >> >> >
> >> >> >> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >> >> >> >reduction in time for a linear read? I'd be surprised if the workload
> >> >> >> >>
> >> >> >> >> Hi Andrew.
> >> >> >> >> Yes, I tested this with dd.
> >> >> >> >>
> >> >> >> >> >even consumed 9% of a CPU, so where on earth has the kernel gone to?
> >> >> >> >> >
> >> >> >> >> >Have you been able to reproduce this in your testing?
> >> >> >> >>
> >> >> >> >> Yes, this test on my environment is reproducible.
> >> >> >> >
> >> >> >> >Hisashi, does your environment have some special configurations?
> >> >> >>
> >> >> >> Hi.
> >> >> >> My testing environment is as follows:
> >> >> >> Hardware: HP DL580
> >> >> >> CPU:Xeon 3.2GHz *4 HT enabled
> >> >> >> Memory:8GB
> >> >> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> >> >> >
> >> >> >This is a big hardware RAID. What's the readahead size?
> >> >> >
> >> >> >The numbers look too small for a 7 disk RAID:
> >> >> >
> >> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
> >> >> >        >
> >> >> >        > -2.6.30-rc6
> >> >> >        > 1048576+0 records in
> >> >> >        > 1048576+0 records out
> >> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> >> >        >
> >> >> >        > -2.6.30-rc6-patched
> >> >> >        > 1048576+0 records in
> >> >> >        > 1048576+0 records out
> >> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> >> >
> >> >> >I'd suggest you to configure the array properly before coming back to
> >> >> >measuring the impact of this patch.
> >> >> 
> >> >> 
> >> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
> >> >this directory.
> >> >
> >> >I mean, you should get >300MB/s throughput with 7 disks, and you
> >> >should seek ways to achieve that before testing out this patch :-)
> >> 
> >> Throughput number of storage array is very from one product to another.
> >> On my hardware environment I think this number is valid and
> >> my patch is effective.
> >
> >What's your readahead size? Is it large enough to cover the stripe width?
> 
> Do you mean strage's readahead size?

What's strage? I mean if your RAID's block device file is /dev/sda, then

        blockdev --getra /dev/sda

will tell its readahead size in unit of 512 bytes.

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[KOSAKI Motohiro]

> > >> >I mean, you should get >300MB/s throughput with 7 disks, and you
> > >> >should seek ways to achieve that before testing out this patch :-)
> > >> 
> > >> Throughput number of storage array is very from one product to another.
> > >> On my hardware environment I think this number is valid and
> > >> my patch is effective.
> > >
> > >What's your readahead size? Is it large enough to cover the stripe width?
> > 
> > Do you mean strage's readahead size?
> 
> What's strage? I mean if your RAID's block device file is /dev/sda, then

I guess it's typo :-)
but I recommend he use sane test environment...


> 
>         blockdev --getra /dev/sda
> 
> will tell its readahead size in unit of 512 bytes.
> 
> Thanks,
> Fengguang
> 



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Hisashi Hifumi]

At 12:02 09/06/01, Wu Fengguang wrote:
>On Mon, Jun 01, 2009 at 10:51:56AM +0800, Hisashi Hifumi wrote:
>> 
>> At 11:37 09/06/01, Wu Fengguang wrote:
>> >On Wed, May 27, 2009 at 11:06:37AM +0800, Hisashi Hifumi wrote:
>> >> 
>> >> At 11:57 09/05/27, Wu Fengguang wrote:
>> >> >On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
>> >> >> 
>> >> >> At 11:36 09/05/27, Wu Fengguang wrote:
>> >> >> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
>> >> >> >>
>> >> >> >> At 11:09 09/05/27, Wu Fengguang wrote:
>> >> >> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
>> >> >> >> >>
>> >> >> >> >> At 08:42 09/05/27, Andrew Morton wrote:
>> >> >> >> >> >On Fri, 22 May 2009 10:33:23 +0800
>> >> >> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
>> >> >> >> >> >
>> >> >> >> >> >> > I tested above patch, and I got same performance number.
>> >> >> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks!  This is an interesting micro timing behavior that
>> >> >> >> >> >> demands some research work.  The above check is to confirm 
>if it's
>> >> >> >> >> >> the PageUptodate() case that makes the difference. So why 
>that case
>> >> >> >> >> >> happens so frequently so as to impact the performance? 
>Will it also
>> >> >> >> >> >> happen in NFS?
>> >> >> >> >> >>
>> >> >> >> >> >> The problem is readahead IO pipeline is not running smoothly, 
>> >which is
>> >> >> >> >> >> undesirable and not well understood for now.
>> >> >> >> >> >
>> >> >> >> >> >The patch causes a remarkably large performance increase.  A 9%
>> >> >> >> >> >reduction in time for a linear read? I'd be surprised if the 
>workload
>> >> >> >> >>
>> >> >> >> >> Hi Andrew.
>> >> >> >> >> Yes, I tested this with dd.
>> >> >> >> >>
>> >> >> >> >> >even consumed 9% of a CPU, so where on earth has the kernel 
>gone to?
>> >> >> >> >> >
>> >> >> >> >> >Have you been able to reproduce this in your testing?
>> >> >> >> >>
>> >> >> >> >> Yes, this test on my environment is reproducible.
>> >> >> >> >
>> >> >> >> >Hisashi, does your environment have some special configurations?
>> >> >> >>
>> >> >> >> Hi.
>> >> >> >> My testing environment is as follows:
>> >> >> >> Hardware: HP DL580
>> >> >> >> CPU:Xeon 3.2GHz *4 HT enabled
>> >> >> >> Memory:8GB
>> >> >> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
>> >> >> >
>> >> >> >This is a big hardware RAID. What's the readahead size?
>> >> >> >
>> >> >> >The numbers look too small for a 7 disk RAID:
>> >> >> >
>> >> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
>> >> >> >        >
>> >> >> >        > -2.6.30-rc6
>> >> >> >        > 1048576+0 records in
>> >> >> >        > 1048576+0 records out
>> >> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
>> >> >> >        >
>> >> >> >        > -2.6.30-rc6-patched
>> >> >> >        > 1048576+0 records in
>> >> >> >        > 1048576+0 records out
>> >> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
>> >> >> >
>> >> >> >I'd suggest you to configure the array properly before coming back to
>> >> >> >measuring the impact of this patch.
>> >> >> 
>> >> >> 
>> >> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
>> >> >this directory.
>> >> >
>> >> >I mean, you should get >300MB/s throughput with 7 disks, and you
>> >> >should seek ways to achieve that before testing out this patch :-)
>> >> 
>> >> Throughput number of storage array is very from one product to another.
>> >> On my hardware environment I think this number is valid and
>> >> my patch is effective.
>> >
>> >What's your readahead size? Is it large enough to cover the stripe width?
>> 
>> Do you mean strage's readahead size?
>
>What's strage? I mean if your RAID's block device file is /dev/sda, then
>
>        blockdev --getra /dev/sda
>
>will tell its readahead size in unit of 512 bytes.

256 sectors.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH] readahead:add blk_run_backing_dev

[Wu Fengguang]

On Mon, Jun 01, 2009 at 11:07:42AM +0800, Hisashi Hifumi wrote:
> 
> At 12:02 09/06/01, Wu Fengguang wrote:
> >On Mon, Jun 01, 2009 at 10:51:56AM +0800, Hisashi Hifumi wrote:
> >> 
> >> At 11:37 09/06/01, Wu Fengguang wrote:
> >> >On Wed, May 27, 2009 at 11:06:37AM +0800, Hisashi Hifumi wrote:
> >> >> 
> >> >> At 11:57 09/05/27, Wu Fengguang wrote:
> >> >> >On Wed, May 27, 2009 at 10:47:47AM +0800, Hisashi Hifumi wrote:
> >> >> >> 
> >> >> >> At 11:36 09/05/27, Wu Fengguang wrote:
> >> >> >> >On Wed, May 27, 2009 at 10:21:53AM +0800, Hisashi Hifumi wrote:
> >> >> >> >>
> >> >> >> >> At 11:09 09/05/27, Wu Fengguang wrote:
> >> >> >> >> >On Wed, May 27, 2009 at 08:25:04AM +0800, Hisashi Hifumi wrote:
> >> >> >> >> >>
> >> >> >> >> >> At 08:42 09/05/27, Andrew Morton wrote:
> >> >> >> >> >> >On Fri, 22 May 2009 10:33:23 +0800
> >> >> >> >> >> >Wu Fengguang <fengguang.wu@intel.com> wrote:
> >> >> >> >> >> >
> >> >> >> >> >> >> > I tested above patch, and I got same performance number.
> >> >> >> >> >> >> > I wonder why if (PageUptodate(page)) check is there...
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thanks!  This is an interesting micro timing behavior that
> >> >> >> >> >> >> demands some research work.  The above check is to confirm 
> >if it's
> >> >> >> >> >> >> the PageUptodate() case that makes the difference. So why 
> >that case
> >> >> >> >> >> >> happens so frequently so as to impact the performance? 
> >Will it also
> >> >> >> >> >> >> happen in NFS?
> >> >> >> >> >> >>
> >> >> >> >> >> >> The problem is readahead IO pipeline is not running smoothly, 
> >> >which is
> >> >> >> >> >> >> undesirable and not well understood for now.
> >> >> >> >> >> >
> >> >> >> >> >> >The patch causes a remarkably large performance increase.  A 9%
> >> >> >> >> >> >reduction in time for a linear read? I'd be surprised if the 
> >workload
> >> >> >> >> >>
> >> >> >> >> >> Hi Andrew.
> >> >> >> >> >> Yes, I tested this with dd.
> >> >> >> >> >>
> >> >> >> >> >> >even consumed 9% of a CPU, so where on earth has the kernel 
> >gone to?
> >> >> >> >> >> >
> >> >> >> >> >> >Have you been able to reproduce this in your testing?
> >> >> >> >> >>
> >> >> >> >> >> Yes, this test on my environment is reproducible.
> >> >> >> >> >
> >> >> >> >> >Hisashi, does your environment have some special configurations?
> >> >> >> >>
> >> >> >> >> Hi.
> >> >> >> >> My testing environment is as follows:
> >> >> >> >> Hardware: HP DL580
> >> >> >> >> CPU:Xeon 3.2GHz *4 HT enabled
> >> >> >> >> Memory:8GB
> >> >> >> >> Storage: Dothill SANNet2 FC (7Disks RAID-0 Array)
> >> >> >> >
> >> >> >> >This is a big hardware RAID. What's the readahead size?
> >> >> >> >
> >> >> >> >The numbers look too small for a 7 disk RAID:
> >> >> >> >
> >> >> >> >        > #dd if=testdir/testfile of=/dev/null bs=16384
> >> >> >> >        >
> >> >> >> >        > -2.6.30-rc6
> >> >> >> >        > 1048576+0 records in
> >> >> >> >        > 1048576+0 records out
> >> >> >> >        > 17179869184 bytes (17 GB) copied, 224.182 seconds, 76.6 MB/s
> >> >> >> >        >
> >> >> >> >        > -2.6.30-rc6-patched
> >> >> >> >        > 1048576+0 records in
> >> >> >> >        > 1048576+0 records out
> >> >> >> >        > 17179869184 bytes (17 GB) copied, 206.465 seconds, 83.2 MB/s
> >> >> >> >
> >> >> >> >I'd suggest you to configure the array properly before coming back to
> >> >> >> >measuring the impact of this patch.
> >> >> >> 
> >> >> >> 
> >> >> >> I created 16GB file to this disk array, and mounted to testdir, dd to 
> >> >> >this directory.
> >> >> >
> >> >> >I mean, you should get >300MB/s throughput with 7 disks, and you
> >> >> >should seek ways to achieve that before testing out this patch :-)
> >> >> 
> >> >> Throughput number of storage array is very from one product to another.
> >> >> On my hardware environment I think this number is valid and
> >> >> my patch is effective.
> >> >
> >> >What's your readahead size? Is it large enough to cover the stripe width?
> >> 
> >> Do you mean strage's readahead size?
> >
> >What's strage? I mean if your RAID's block device file is /dev/sda, then
> >
> >        blockdev --getra /dev/sda
> >
> >will tell its readahead size in unit of 512 bytes.
> 
> 256 sectors.

That's too small! Try this:

        blockdev --setra 8192 /dev/sda

Thanks,
Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>