From mboxrd@z Thu Jan 1 00:00:00 1970 Message-Id: <200412011828.iB1ISOr04501@mail.osdl.org> Subject: Re: Automated performance testing system was Re: Text form for STP tests In-Reply-To: Your message of "Mon, 29 Nov 2004 22:42:12 -0200." <20041130004212.GB2310@dmt.cyclades> Date: Wed, 01 Dec 2004 10:28:24 -0800 From: Cliff White Sender: owner-linux-mm@kvack.org Return-Path: To: Marcelo Tosatti Cc: linux-mm@kvack.org List-ID: > Linux-MM fellows, > > I've been talking to Cliff about the need for a set of benchmarks, > covering as many different workloads as possible, for developers to have a > better notion of impact on performance changes. > > Usually when one does a change which affects performance, he/she runs one > or two benchmarks with a limited amount of hardware configurations. > This is a very painful, boring and time consuming process, which can > result in misinterpretation and/or limited understading of the results > of such changes. > > It is important to automate such process, with a set of benchmarks > covering as wide as possible range of workloads, running on common > and most used hardware variations. > > OSDL's STP provides the base framework for this. > [ snip ] > bonnie++ > reaim (default, new_fserver, shared) > dbench_long > kernbench > tiobench > > Each of these running one the following combinations: > > 1CPU, 2CPU, 4CPU, 8CPU (4 variants). > > total memory, half memory, a quarter of total memory (3 variants). > > Thats 12 results for each benchmark." > The configuration files to do these tests are now written, and the humble robots are running this test series against linux-2.6.7 ( for history data ) There will need to be some adjustments - some of these tests will no doubt fail for reasons of script error or configuration ( i see already kernbench will have to be redunced for 1-cpu systems, as it runs > 13.5 hours :( ) And, the second part of the automation is already done, but needs input. I can aim this test battery at any kernel patch, where 'any kernel patch' is identified by a regexp. What kernels do you want this against? I've heard mention of 'baseline' - we call this baseline: /^(patch|linux)-\d+\.\d+\.\d+$/ ( starts with 'patch' or 'linux', then '-' followed by three decimals ) cliffw -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: aart@kvack.org