Re: [LSF/MM TOPIC] Test cases to choose for demonstrating mm features or fixing mm bugs

[Balbir Singh <bsingharora@gmail.com>]

On Mon, Jan 28, 2019 at 12:34:42PM +0100, Michal Hocko wrote:
> On Mon 28-01-19 22:20:33, Balbir Singh wrote:
> > Sending a patch to linux-mm today has become a complex task. One of the
> > reasons for the complexity is a lack of fundamental expectation of what
> > tests to run.
> > 
> > Mel Gorman has a set of tests [1], but there is no easy way to select
> > what tests to run. Some of them are proprietary (spec*), but others
> > have varying run times. A single line change may require hours or days
> > of testing, add to that complexity of configuration. It requires a lot
> > of tweaking and frequent test spawning to settle down on what to run,
> > what configuration to choose and benefit to show.
> > 
> > The proposal is to have a discussion on how to design a good sanity
> > test suite for the mm subsystem, which could potentially include
> > OOM test cases and known problem patterns with proposed changes
> 
> I am not sure I follow. So what is the problem you would like to solve.
> If tests are taking too long then there is a good reason for that most
> probably. Are you thinking of any specific tests which should be run or
> even included to MM tests or similar?

Let me elaborate, everytime I think I find something interesting, in terms
of something to develop/fix, I think of how to test the changes. I think
for well established code (such as reclaim) or even other features, it's hard
to find good test cases to run as a base to ensure that

1. There is good coverage of tests against the changes
2. The right test cases have been run from a performance perspective

The reason I brought up the time was not the time for a single test,
but all the tests cumulative in the absence of good guidance for
(1) and (2) above.

IOW, what guidance can we provide to patch writers and bug fixers in terms
of what testing to carry out? How do we avoid biases in results and
ensure consistency?

Balbir Singh.