Benchmarking Tool

Benchmarking Tool

[Abhishek Nayani]

Hi all,

	There was a discussion about the current benchmarking tools being
not suitable or sufficient for testing the performance of the Linux VMs.
I am interested in writing one and would like to have your opinions on
the matter. I would like to know what is missing in the current set of
tools (lmbench, dbench..) and what is required. 


					Bye,
						Abhi.
	
--------------------------------------------------------------------------------
Those who cannot remember the past are condemned to repeat it - George Santayana
--------------------------------------------------------------------------------
                          Home Page: http://www.abhi.tk
-----BEGIN GEEK CODE BLOCK------------------------------------------------------
GCS d+ s:- a-- C+++ UL P+ L+++ E- W++ N+ o K- w--- O-- M- V- PS PE Y PGP 
t+ 5 X+ R- tv+ b+++ DI+ D G e++ h! !r y- 
------END GEEK CODE BLOCK-------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: Benchmarking Tool

[Benjamin LaHaise]

On Wed, Jul 03, 2002 at 11:34:46AM +0530, Abhishek Nayani wrote:
> 	There was a discussion about the current benchmarking tools being
> not suitable or sufficient for testing the performance of the Linux VMs.
> I am interested in writing one and would like to have your opinions on
> the matter. I would like to know what is missing in the current set of
> tools (lmbench, dbench..) and what is required. 

dbench essentially produces a random number for "performance", as it 
is overly dependent on small changes in timing, available memory and 
execution patterns.  lmbench doesn't really test the vm at all.  What 
is needed are a series of tests that represent the kinds of loads 
that various Linux users run.  This includes:

	- software development
	- scientific apps
	- desktop
	- various server workloads

Most of the time results are presented, they tend to be lacking in some 
areas -- much of the high end work being done on >8GB systems tends not 
to rerun tests on small systems to make sure behaviour isn't badly 
affected.  Of the cases listed above, the desktop workload is probably 
the most badly represented with current tests.

>From a few of the discussions at OLS, it became apparent that more 
important than the actual benchmarking tools is the collection of the 
results.  Common tests, like compiling the kernel, are useful metrics 
for noticing changes in kernel behaviour if all other aspects of the 
test environment are held constant, but only if we have a set of data 
to compare the results against.

		-ben
-- 
"You will be reincarnated as a toad; and you will be much happier."
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: Benchmarking Tool

[Rik van Riel]

On Wed, 3 Jul 2002, Abhishek Nayani wrote:

> the matter. I would like to know what is missing in the current set of
> tools (lmbench, dbench..) and what is required.

Most of the current "VM tests" don't seem to have anything
like a working set.  This basically means that one of the
central and important parts of the VM - page replacement -
isn't getting tested AT ALL.

It might be interesting to have some "working set emulator"
where a program accesses N out of M MB of total memory a
lot and the rest a little, where N, M and the ratio between
the accesses are varied in such a way that the system is
confronted with various sizes of workload.

Of course, you could also go into multitasking such programs ;)

The way to display the result of this could be a graph, showing
the working set size on the X axis and the program "speed" on
the Y axis. It might also be useful to express the size of the
working set as a percentage of main memory.

This way you could show "VM X" runs well until the working set
reaches 50% of RAM size, while "VM Y" runs well until the working
set size reaches 70% of RAM.

This is just one example of things we could do, I'm sure there
are many more aspects of the VM subsystem for which we don't
have any benchmarks yet.

regards,

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: Benchmarking Tool

[Abhishek Nayani]

Hi,

On Thu, 04 Jul 2002, Rik van Riel wrote:
> On Wed, 3 Jul 2002, Abhishek Nayani wrote:
> 
> > the matter. I would like to know what is missing in the current set of
> > tools (lmbench, dbench..) and what is required.
> 
> Most of the current "VM tests" don't seem to have anything
> like a working set.  This basically means that one of the
> central and important parts of the VM - page replacement -
> isn't getting tested AT ALL.
> 
> It might be interesting to have some "working set emulator"
> where a program accesses N out of M MB of total memory a
> lot and the rest a little, where N, M and the ratio between
> the accesses are varied in such a way that the system is
> confronted with various sizes of workload.
> 

	We will start creating a list of variables needing to be
measured and any current solutions/ideas to test them.

	o Working Set
		Useful URLs:
		http://www.cs.inf.ethz.ch/CoPs/ECT/



	Please add to the list.

					Bye,
						Abhi.
	
--------------------------------------------------------------------------------
Those who cannot remember the past are condemned to repeat it - George Santayana
--------------------------------------------------------------------------------
                          Home Page: http://www.abhi.tk
-----BEGIN GEEK CODE BLOCK------------------------------------------------------
GCS d+ s:- a-- C+++ UL P+ L+++ E- W++ N+ o K- w--- O-- M- V- PS PE Y PGP 
t+ 5 X+ R- tv+ b+++ DI+ D G e++ h! !r y- 
------END GEEK CODE BLOCK-------------------------------------------------------
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/