[RFC] Accelerate dbench

[RFC] Accelerate dbench

[Daniel Phillips]

This simple patch to 2.4.8-pre4 accelerates dbench quite spectacularly
for me.  It also slows down my more realistic make+grep load, so it's
not generally a good thing, but... if we could detect the conditions
where this activation strategy is desirable, it could turn into a good
thing.

This doesn't address any burning issue at the moment, it's just for
comment.  Note that it effectively turns check_used_once into a no-op.

The effect of the patch is to always activate pages when they are
successfully looked up, not when first created.  This allows unused
readahead pages to be quickly reclaimed.  (Buffer pages could easily
be treated the same way but I didn't try it.)  Together with -pre4's
shorter IO queue (also probably not a good thing, as Linus pointed
out) this seems to be very friendly to dbench, for reasons I still
don't understand.

With this patch, I saw very large flucuations in the timings obtained
from dbench 12 on this 64 MB Vaio, but the times ranged from good to
excellent, varying by about 30%.  This is with identical running
conditions, starting from a clean reboot each time, and mkfs'ing the
test partition.  Go figure.  As always, the best dbench times were
associated with very unfair scheduling, with frequent bursts where a
single process makes rapid progress and the others appear to be
pushed aside.

--- ../2.4.7.clean/mm/filemap.c	Sat Aug  4 14:27:16 2001
+++ ./mm/filemap.c	Sat Aug  4 14:32:51 2001
@@ -307,9 +307,9 @@
 		if (page->index == offset)
 			break;
 	}
-	/* Mark the page referenced, kswapd will find it later. */
 	SetPageReferenced(page);
-
+	if (!PageActive(page))
+		activate_page(page);
 not_found:
 	return page;
 }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: [RFC] Accelerate dbench

[Rik van Riel]

On Sun, 5 Aug 2001, Daniel Phillips wrote:

> --- ../2.4.7.clean/mm/filemap.c	Sat Aug  4 14:27:16 2001
> +++ ./mm/filemap.c	Sat Aug  4 14:32:51 2001

> -	/* Mark the page referenced, kswapd will find it later. */
>  	SetPageReferenced(page);
> -
> +	if (!PageActive(page))
> +		activate_page(page);

I think this is wrong.

By doing this the page will end up at the far end
of the active list with the referenced bit already
set.

This doesn't allow us to distinguish between pages
which get accessed again after we put them on the
active list and pages which aren't.

Also, it effectively gives a double boost for this
one access...

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: [RFC] Accelerate dbench

[Daniel Phillips]

On Sunday 05 August 2001 02:02, Rik van Riel wrote:
> On Sun, 5 Aug 2001, Daniel Phillips wrote:
> > --- ../2.4.7.clean/mm/filemap.c	Sat Aug  4 14:27:16 2001
> > +++ ./mm/filemap.c	Sat Aug  4 14:32:51 2001
> >
> > -	/* Mark the page referenced, kswapd will find it later. */
> >  	SetPageReferenced(page);
> > -
> > +	if (!PageActive(page))
> > +		activate_page(page);
>
> I think this is wrong.
>
> By doing this the page will end up at the far end
> of the active list with the referenced bit already
> set.
>
> This doesn't allow us to distinguish between pages
> which get accessed again after we put them on the
> active list and pages which aren't.
>
> Also, it effectively gives a double boost for this
> one access...

How wrong could it be when it's turning in results like this:

  dbench 12, 2.4.8-pre4 vanilla
  12.76user 76.49system 6:20.56elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (426major+405minor)pagefaults 0swaps

  dbench 12, 2.4.8-pre4 with immediate activate in find_page
  16.81user 58.05system 3:13.92elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (1112major+632minor)pagefaults 0swaps

Almost twice as fast with the patch.  Mind you, it doesn't
turn in this spectacular result all the time, sometimes it's
more like:

  dbench 12, 2.4.8-pre4 with immediate activate in find_page
  15.81user 69.09system 4:40.60elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k
  0inputs+0outputs (991major+766minor)pagefaults 0swaps

which is still very good compared to the stock kernel and
its predecessors.  (Note the huge difference between timings
take under identical conditions, from clean reboots.)

But you're right, this fits our aging model better:

-  	SetPageReferenced(page);
+	if (PageActive(page))
+	  	SetPageReferenced(page);
+	else
+		activate_page(page);

So I'll try it...<time passes>...OK, it doesn't make a lot of
difference, results still range from "pretty good" to "really
great".  Not really suprising, I have this growing gut feeling
that we're not doing that well on the active page aging anyway,
and that random selection of candidates for trial on the
inactive queue would perform almost as well - which might be
worth testing.  Anyway, I'm putting this on the back burner for
now.  Interesting as it is, it's hardly a burning issue.

--
Daniel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: [RFC] Accelerate dbench

[Rik van Riel]

On Sun, 5 Aug 2001, Daniel Phillips wrote:

> -  	SetPageReferenced(page);
> +	if (PageActive(page))
> +	  	SetPageReferenced(page);
> +	else
> +		activate_page(page);
>
> So I'll try it...<time passes>...OK, it doesn't make a lot of
> difference, results still range from "pretty good" to "really
> great".  Not really suprising, I have this growing gut feeling
> that we're not doing that well on the active page aging anyway,

Yes, I have the feeling that exponential down aging wasn't
such a good idea in combination with the fact that most of
the access bits are "hidden" in page tables ...

> and that random selection of candidates for trial on the
> inactive queue would perform almost as well - which might be
> worth testing.  Anyway, I'm putting this on the back burner for
> now.  Interesting as it is, it's hardly a burning issue.

Well, we found that doing instant activation gives a huge
performance increase. That's one important point already ;)

regards,

Rik
--
IA64: a worthy successor to i860.

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: [RFC] Accelerate dbench

[Daniel Phillips]

On Sunday 05 August 2001 04:39, Rik van Riel wrote:
> On Sun, 5 Aug 2001, Daniel Phillips wrote:
> > -  	SetPageReferenced(page);
> > +	if (PageActive(page))
> > +	  	SetPageReferenced(page);
> > +	else
> > +		activate_page(page);
> >
> > So I'll try it...<time passes>...OK, it doesn't make a lot of
> > difference, results still range from "pretty good" to "really
> > great".  Not really suprising, I have this growing gut feeling
> > that we're not doing that well on the active page aging anyway,
>
> Yes, I have the feeling that exponential down aging wasn't
> such a good idea in combination with the fact that most of
> the access bits are "hidden" in page tables ...

Ignoring the hardware access bits in page_launder has to hurt.  I'm
playing with an idea now to get access to the hardware referenced
bit for swap pages on the inactive queue, needs more research.

> > and that random selection of candidates for trial on the
> > inactive queue would perform almost as well - which might be
> > worth testing.  Anyway, I'm putting this on the back burner for
> > now.  Interesting as it is, it's hardly a burning issue.
>
> Well, we found that doing instant activation gives a huge
> performance increase. That's one important point already ;)

*Note*: only for dbench, a bizarre load I don't fully understand.
For my more realistic make+grep load, this change slows it down as
I would expect.  But yes, it indicates that there may be more big
gains to get from replacement policy once we understand what the
dbench load really is and how to detect that kind of load
automatically.  Or maybe if it just results in a useful
adjustment available from userlond it would be worth the effort.

There is one disturbing possibility I haven't looked into: what if
dbench has a bug?  What if its not really doing all the IO its
supposed to on those really fast runs?

--
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: [RFC] Accelerate dbench

[Andrew Morton]

Daniel Phillips wrote:
> 
> How wrong could it be when it's turning in results like this:
> 
>   dbench 12, 2.4.8-pre4 vanilla
>   12.76user 76.49system 6:20.56elapsed 23%CPU (0avgtext+0avgdata 0maxresident)k
>   0inputs+0outputs (426major+405minor)pagefaults 0swaps

Oi!  Didn't Andrew say not to optimise for dbench?

We've had some interesting times with ext3 and dbench lately.
It all boils down to the fact that dbench deletes its own
working files inside the kupdate writeback interval.

This means that:

a) If dbench is running fast enough to delete its file within
   the writeback interval, it'll run even faster because it
   does less IO!  Non-linear behaviour there.

b) If a butterfly flaps its wing, and something triggers
   bdflush then your dbench throughput is demolished, because
   data which ordinarily is deleted before ever getting written
   out ends up hitting disk.

   It was discovered that with one particular workload ext2
   was not triggering bdflush but ext3 was.   Twiddling the
   bdflush nfract and nfract_sync parameters prevented this
   and our throughput went from something unmentionable up
   to 85% of ext2.  (actually, it was a teeny bit faster with
   80 clients - dunno why).

All this was with ext3 in data writeback mode, of course - the
other journalling modes write data out within 5 seconds anyway,
which is another reason why ext3 dbench numbers are unrepresentatively
lower than ext2 - we do about four times as much I/O!

So...  This artifact makes *gross* throughput differences, and
if your VM changes are somehow causing changed flush behaviour
then perhaps you won't see what you're looking for.

And note the positive feedback cycle: a slower dbench run will
result in more IO, which will result in a slower dbench run, 
which will....

For VM/fs tuning efforts I'd recommend that you consider hacking
dbench to not delete its files - just rename them or something.

(Not blaming dbench here - it is merely emulating netbench dopiness).

-
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/