Re: [Ksummit-discuss] [CORE TOPIC] Recruitment (Reviewers, Testers, Maintainers, Hobbyists)

[Theodore Ts'o <tytso@mit.edu>]

On Thu, Jul 09, 2015 at 01:50:49PM -0700, Guenter Roeck wrote:
> 
> Earlier it was discussed how to improve the recognition of reviewers.
> Your comments seems to suggest the opposite, and may actually discourage
> reviewers. Why should I review Linux kernel code if that is seen
> by some as me trying to "game" the system ?

So I were designing an initial system that automatically scored
reviewers, I'd be looking to see, from a holistic point of view, how
many reviews were zero-length:

Reviewed-by: John Q. Random <seventeen@random.org>

... and nothing else,

.... versus how many reviews had specific comments on various
different portions of the patch.  If possible, the automated system
would try to distinguish between comments that were just pointing out
whitespace issues (which would be a slight positive) with comments
that point out genuine design issues (this will be really hard to do
in an automated fashion, but a very sophisticated nueral network[1]
mgiht be able to hack it).

I might also try using some kind scheme that counted the number of
words in a review (stripping out lines of patch or commit description
that the review was a reply to), etc.  But of course, if it was public
knowledge that the system was just stripping out the original e-mail,
and then just doing a "wc -w", then people would game the system by
adding list of random words at the bottom of the review.

And, of course, I'd have the system give a huge negative score if a
commit that got a "LGTM" positive review caused a bug that required
the patch to be reverted.  *That* signal, at least, would be hard to
game, and would hopefuly encourage people to actually take time
reviewing a commit, and not blindly slapping a reviewed-by on a commit
they don't understand.

You see?  It's not that reviews in and of themselves are attempts to
game the system ---- just so long as they are genuine reviews.  If
there is evidence that the reviews are issued within seconds of the
original patch going out, with a Reviewed-by: line and nothing else,
what would *you* think about the quality of that review?

> That may be true for some people, but at the same time I think statements
> like the above might discourage people who just like cleaning up code for
> fun. There are several of those working on cleaning up the Linux kernel,
> and I truly appreciate their efforts.

Sure, but that's not the people who (in my opinion as a program
committee member) should be attendingt he Kernel Summit, where we want
people who are genuinely clueful about technical and policy issues,
and not people like (for example) Nick Krause.

Regards,

						- Ted

[1] http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html