[Ksummit-discuss] [CORE-TOPIC] Documentation

[Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

Hi,

as we talked about recruiting in the other thread I realized that one thing 
that might reduce the entrance barrier a bit (apart from tooling and flow) 
would be proper documentation.

I know documenation is usually not a developers most favorite task, but here 
are my findings anyway:



I had a quick look into our guideline (Documentation/kernel-doc-nano-
HOWTO.txt) and it states:

1)
> We definitely need kernel-doc formatted documentation for functions
> that are exported to loadable modules using EXPORT_SYMBOL.
This is definitely not the case at the moment :/ -- there are a lot of 
undocumented EXPORT_SYMBOL functions.

2)
> We also look to provide kernel-doc formatted documentation for
> functions externally visible to other kernel files (not marked
> "static").
Even less

3)
> We also recommend providing kernel-doc formatted documentation
> for private (file "static") routines, for consistency of kernel
> source code layout.  But this is lower priority and at the
> discretion of the MAINTAINER of that kernel source file.
Don't even talk about that :)


Imho new code should definitely have this documentation, especially for #1 and 
#2 -- but maybe even for #3.




So this leads me to following questions:
- How can we easily identify missing documentation? 
-- Maybe Julia can come up with some coccinelle magic?
-- Maybe even mark non-extractable documentation and convert it.
-- In the document it mentions scripts/basic/doproc.c checks for missing 
documentation, but this file does not exist anymore :/




- how can we (automatically) support documentation? 
-- Can / should we create a tool that annotates functions and data types with 
a template documentation (I know good editors do something similar, but is the 
proposed style in concordance with the kernel doc style? At least it looks 
quite a lot like doxygen?)

- How different is the kernel doc style to doxygen?


- We are constantly looking for tasks newcomers can take on -- writing 
appropriate documentation is usually not that hard on the one hand and on the 
other people will learn a lot about the code they are documenting.
-- maybe make it easier make use of Steven Rostedts proposed Kernel Patch 
submitting web form :P

- new checkpatch option for checking the existance of documentation? *ducks 
and runs away*


- and who would be in to join me in documenting everything :P


Thanks,
Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Julia Lawall]

> So this leads me to following questions:
> - How can we easily identify missing documentation?
> -- Maybe Julia can come up with some coccinelle magic?
> -- Maybe even mark non-extractable documentation and convert it.
> -- In the document it mentions scripts/basic/doproc.c checks for missing
> documentation, but this file does not exist anymore :/

Interesting idea.  I had not thought of this.  Coccinelle doesn't really
process comments, but one can always use grep.  So the idea would be to
find function definitions that don't have anything that looks like a
comment in the lines above (ie, the lines since the end of the previous
function definition).  One could furthermore rank the results by the
number of non-local calls to the function.  I guess there would be a lot
of reports, and it would be most productive to start with functions that
are commonly used,

julia

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Sun, 12 Jul 2015 19:15:32 -0400 (EDT)
Julia Lawall <julia.lawall@lip6.fr> escreveu:

> > So this leads me to following questions:
> > - How can we easily identify missing documentation?
> > -- Maybe Julia can come up with some coccinelle magic?
> > -- Maybe even mark non-extractable documentation and convert it.
> > -- In the document it mentions scripts/basic/doproc.c checks for missing
> > documentation, but this file does not exist anymore :/
> 
> Interesting idea.  I had not thought of this.  Coccinelle doesn't really
> process comments, but one can always use grep.  So the idea would be to
> find function definitions that don't have anything that looks like a
> comment in the lines above (ie, the lines since the end of the previous
> function definition).  One could furthermore rank the results by the
> number of non-local calls to the function.  I guess there would be a lot
> of reports, and it would be most productive to start with functions that
> are commonly used,

I guess it should first try to classify the type of the exported symbols,
in order to check either if they're used only internally inside a subsystem or
global wide, and report, by default, just the global wide undocumented ones,
sorted by number of occurrences.

Maybe the tool could be based on LXR[1] or use some output of it.

[1] http://free-electrons.com/pub/source/lxr-0.3.1-fe1.tar.bz2

Regards,
Mauro

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Alexey Dobriyan]

On Mon, Jul 13, 2015 at 1:38 AM, Peter Hüwe <PeterHuewe@gmx.de> wrote:
> as we talked about recruiting in the other thread I realized that one thing
> that might reduce the entrance barrier a bit (apart from tooling and flow)
> would be proper documentation.

Newbies should not write documentation because they, by definition,
have little knowledge of the code. It will lead to many, many useless
and redundant comments like below whose only purpose is stealing
vertical whitespace:

    /**
     * is_module_percpu_address - test whether address is from module
static percpu
     * @addr: address to test
     *
     * Test whether @addr belongs to module static percpu area.
     *
     * RETURNS:
     * %true if @addr is from module static percpu area
     */

    bool is_module_percpu_address(unsigned long addr)
    {

I have counter suggestion:
* remove kernel-doc official status, remove generating scripts
  because it doesn't work,
* trim redundant and semi-redundant kernel-doc comments like this:

    /**
     * module_refcount - return the refcount or -1 if unloading
     *
     * @mod:        the module we're checking
     *
     * Returns:
     *      -1 if the module is in the process of unloading
     *      otherwise the number of references in the kernel to the module
     */
    int module_refcount(struct module *mod)
    {

("return reference count or -1 if module is being unloaded" is enough here).

      Alexey

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Randy Dunlap]

On 07/13/15 02:47, Alexey Dobriyan wrote:
> On Mon, Jul 13, 2015 at 1:38 AM, Peter Hüwe <PeterHuewe@gmx.de> wrote:
>> as we talked about recruiting in the other thread I realized that one thing
>> that might reduce the entrance barrier a bit (apart from tooling and flow)
>> would be proper documentation.
> 
> Newbies should not write documentation because they, by definition,
> have little knowledge of the code. It will lead to many, many useless
> and redundant comments like below whose only purpose is stealing
> vertical whitespace:
> 
>     /**
>      * is_module_percpu_address - test whether address is from module
> static percpu
>      * @addr: address to test
>      *
>      * Test whether @addr belongs to module static percpu area.
>      *
>      * RETURNS:
>      * %true if @addr is from module static percpu area
>      */
> 
>     bool is_module_percpu_address(unsigned long addr)
>     {
> 
> I have counter suggestion:
> * remove kernel-doc official status, remove generating scripts
>   because it doesn't work,

I have no problem with that suggestion.  scripts/kernel-doc is fragile and
either needs lots of continuous help or can be discarded.

As I have said in the past, the main thing is that there should be decent
documentation for (new) interfaces.  It doesn't need to be kernel-doc.


> * trim redundant and semi-redundant kernel-doc comments like this:
> 
>     /**
>      * module_refcount - return the refcount or -1 if unloading
>      *
>      * @mod:        the module we're checking
>      *
>      * Returns:
>      *      -1 if the module is in the process of unloading
>      *      otherwise the number of references in the kernel to the module
>      */
>     int module_refcount(struct module *mod)
>     {
> 
> ("return reference count or -1 if module is being unloaded" is enough here).


-- 
~Randy

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Steven Rostedt]

On Mon, 13 Jul 2015 12:47:23 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:


> Newbies should not write documentation because they, by definition,
> have little knowledge of the code. It will lead to many, many useless
> and redundant comments like below whose only purpose is stealing
> vertical whitespace:

I agree that newbies shouldn't write documentation. But maybe what they
can do is to question what a function does. And perhaps poke the
maintainer (or author of said function) to write something that
explains that function (only for non-static functions).

Now, I say "newbies" but I would really mean experienced developers
that are new to a subsystem. We don't need silly questions. Something
more on the line of one experienced kernel developer reading some code
of the kernel they have no idea about, and if they can't figure out
what a function does, ask the question to the author. Perhaps we can
get better documentation of internal interfaces out of it.

There's been times where I look at other subsystems and wonder WTF a
function is suppose to do. Perhaps I need to start poking the author of
that code to explain it.

-- Steve

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1086 bytes --]

On Mon, 2015-07-13 at 12:37 -0400, Steven Rostedt wrote:
> There's been times where I look at other subsystems and wonder WTF a
> function is suppose to do. Perhaps I need to start poking the author 
> of that code to explain it.

That should never happen. It's OK if you just don't know how to do
something (or don't even know if it's possible), and you have to ask or
resort to documentation.

But you, Steven, should *never* find yourself looking at a function and
wondering WTF it does. If a function is complex enough that you need an
explanation, then either:

 1. It should have been made simpler, or 
 2. There should be an explanation *right* there in the C file.

Those should be caught in review. Too often, I think, we allow opaque
code to get merged.

I'm guilty of it myself. And I've also suffered from it. Sometimes,
with a few years in between, I have even inflicted this pain on
*myself* :)

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5691 bytes --]

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Steven Rostedt]

On Mon, 13 Jul 2015 17:46:46 +0100
David Woodhouse <dwmw2@infradead.org> wrote:

> I'm guilty of it myself. And I've also suffered from it. Sometimes,
> with a few years in between, I have even inflicted this pain on
> *myself* :)

Me too. I have to admit, that I've done the same with my own functions.
But I've become much better lately at documenting.

A lot of these functions are from years past. We should go back and try
to document them. Perhaps we may even find and fix a few bugs in doing
so.

I forgot who said it, but I remember reading a quote that stated "When
the code does not match the comments, they are probably both wrong".

-- Steve

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

Am Montag, 13. Juli 2015, 19:35:19 schrieb Steven Rostedt:

> A lot of these functions are from years past. We should go back and try
> to document them. Perhaps we may even find and fix a few bugs in doing
> so.
+100!

> 
> I forgot who said it, but I remember reading a quote that stated "When
> the code does not match the comments, they are probably both wrong".
+1000! :)

Of course that's a huge problem, even more with our fast paced development.

I think we should definitely spend more time to fix stuff/documentation we 
stumble upon and spent hours figuring out anyway.

Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

Am Montag, 13. Juli 2015, 18:46:46 schrieb David Woodhouse: 
>  1. It should have been made simpler, or
>  2. There should be an explanation *right* there in the C file.
> 
> Those should be caught in review.

But also we have tons of undocumented obscure legacy code which unfortunately 
won't get new review cycles.

Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Jason Cooper]

On Mon, Jul 13, 2015 at 12:37:50PM -0400, Steven Rostedt wrote:
> On Mon, 13 Jul 2015 12:47:23 +0300
> Alexey Dobriyan <adobriyan@gmail.com> wrote:
> 
> > Newbies should not write documentation because they, by definition,
> > have little knowledge of the code. It will lead to many, many useless
> > and redundant comments like below whose only purpose is stealing
> > vertical whitespace:
> 
> I agree that newbies shouldn't write documentation.

I semi-disagree.  Assuming you meant newbies here as you defined below.

Not that an exception should make the rule, but please take a look at
the Crypto API documentation (with userspace example code!) written by
Stephan Mueller:

  git log --author="smueller@chronox.de"

His first patch was in May 2014.

  541af946fe13 crypto: drbg - SP800-90A Deterministic Random Bit Generator

By Nov/Dec 2014 he had documented the entire crypto API.

  52744af3af97 crypto: doc - document uncovered member variables
  47ca5be9eb06 crypto: doc - HASH API documentation
  16e61030aecb crypto: doc - CIPHER API documentation
  58284f0d6c4a crypto: doc - BLKCIPHER API documentation
  fced7b02623e crypto: doc - AEAD API documentation
  f13ec330a787 crypto: doc - ABLKCIPHER API documentation
  0d7f488f0305 crypto: doc - cipher data structures
  968ab2910780 crypto: doc - SHASH API documentation
  90240ffb1277 crypto: doc - AHASH API documentation
  5d8c723f61f2 crypto: doc - hash data structures
  aa1b6fbcbeac crypto: doc - RNG API documentation
  e63b673f601d crypto: doc - userspace interface spec
  e9a44230dbca crypto: doc - compile crypto API spec
  7d12993ed890 crypto: doc - crypto API high level spec

Often the best person to write the docs for a newcomer to understand is
someone who was just recently a newcomer.

> But maybe what they
> can do is to question what a function does. And perhaps poke the
> maintainer (or author of said function) to write something that
> explains that function (only for non-static functions).

Submitting patches is often the *best* way to poke a maintainer. ;-)

I know I'm not the only one who would find it a lot easier to tweak an
80% correct documentation patch as opposed to explaining the function
call, how it fits in with the others, how it historically evolved that
way, etc.

> Now, I say "newbies" but I would really mean experienced developers
> that are new to a subsystem. We don't need silly questions. Something
> more on the line of one experienced kernel developer reading some code
> of the kernel they have no idea about, and if they can't figure out
> what a function does, ask the question to the author. Perhaps we can
> get better documentation of internal interfaces out of it.

thx,

Jason.

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Steven Rostedt]

On Mon, 13 Jul 2015 17:42:44 +0000
Jason Cooper <jason@lakedaemon.net> wrote:

> On Mon, Jul 13, 2015 at 12:37:50PM -0400, Steven Rostedt wrote:

> > I agree that newbies shouldn't write documentation.
> 
> I semi-disagree.  Assuming you meant newbies here as you defined below.

This really shows what the issue with new recruits is all about. Every
maintainer is different. The Linux kernel is the largest and fastest
open source project in the world and its amazing how much that everyone
does agree on. But every maintainer has a slightly different
perspective of how to do something, and this can be really aggravating
to a new comer. As it's been said lots of times, maintaining the Linux
kernel is much like herding cats. And this burden also weighs down on
new comers, as they try to figure out which type of cat they are
dealing with. And all cats just happen to be extremely annoying to deal
with ;-)

The best we can do is all to try to be patient with those that contact
us.

-- Steve

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Zefan Li]

On 2015/7/14 2:11, Steven Rostedt wrote:
> On Mon, 13 Jul 2015 17:42:44 +0000
> Jason Cooper <jason@lakedaemon.net> wrote:
> 
>> On Mon, Jul 13, 2015 at 12:37:50PM -0400, Steven Rostedt wrote:
> 
>>> I agree that newbies shouldn't write documentation.
>>
>> I semi-disagree.  Assuming you meant newbies here as you defined below.
> 
> This really shows what the issue with new recruits is all about. Every
> maintainer is different. The Linux kernel is the largest and fastest
> open source project in the world and its amazing how much that everyone
> does agree on. But every maintainer has a slightly different
> perspective of how to do something, and this can be really aggravating
> to a new comer. As it's been said lots of times, maintaining the Linux
> kernel is much like herding cats. And this burden also weighs down on
> new comers, as they try to figure out which type of cat they are
> dealing with. And all cats just happen to be extremely annoying to deal
> with ;-)
> 
> The best we can do is all to try to be patient with those that contact
> us.
> 

I think it's more often that we just ignore them rather than being impatient...

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

> > I agree that newbies shouldn't write documentation.
> 
> I semi-disagree.  Assuming you meant newbies here as you defined below.
> 
> Not that an exception should make the rule, but please take a look at
> the Crypto API documentation (with userspace example code!) written by
> Stephan Mueller:

Thanks Jason for this example -- of course that's an (exceptional) exception 
to the rule, I really like it anyway.
 
> Often the best person to write the docs for a newcomer to understand is
> someone who was just recently a newcomer.
:) definitely.

I mentor quite a lot of new colleagues at my company, and you cannot image how 
many 'blind spots' new colleagues can find -- even in 'perfect' 
documentation/tutorials.

Thanks
Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Laurent Pinchart]

On Monday 13 July 2015 17:42:44 Jason Cooper wrote:
> On Mon, Jul 13, 2015 at 12:37:50PM -0400, Steven Rostedt wrote:
> > On Mon, 13 Jul 2015 12:47:23 +0300 Alexey Dobriyan wrote:
> > > Newbies should not write documentation because they, by definition,
> > > have little knowledge of the code. It will lead to many, many useless
> > > and redundant comments like below whose only purpose is stealing
> > 
> > > vertical whitespace:
> > I agree that newbies shouldn't write documentation.
> 
> I semi-disagree.  Assuming you meant newbies here as you defined below.
> 
> Not that an exception should make the rule, but please take a look at
> the Crypto API documentation (with userspace example code!) written by
> Stephan Mueller:
> 
>   git log --author="smueller@chronox.de"
> 
> His first patch was in May 2014.
> 
>   541af946fe13 crypto: drbg - SP800-90A Deterministic Random Bit Generator
> 
> By Nov/Dec 2014 he had documented the entire crypto API.
> 
>   52744af3af97 crypto: doc - document uncovered member variables
>   47ca5be9eb06 crypto: doc - HASH API documentation
>   16e61030aecb crypto: doc - CIPHER API documentation
>   58284f0d6c4a crypto: doc - BLKCIPHER API documentation
>   fced7b02623e crypto: doc - AEAD API documentation
>   f13ec330a787 crypto: doc - ABLKCIPHER API documentation
>   0d7f488f0305 crypto: doc - cipher data structures
>   968ab2910780 crypto: doc - SHASH API documentation
>   90240ffb1277 crypto: doc - AHASH API documentation
>   5d8c723f61f2 crypto: doc - hash data structures
>   aa1b6fbcbeac crypto: doc - RNG API documentation
>   e63b673f601d crypto: doc - userspace interface spec
>   e9a44230dbca crypto: doc - compile crypto API spec
>   7d12993ed890 crypto: doc - crypto API high level spec
> 
> Often the best person to write the docs for a newcomer to understand is
> someone who was just recently a newcomer.

I agree. Not that I want to show off, but there's at least another example.

9cad9c95d7e8 Documentation: DocBook DRM framework documentation

> > But maybe what they
> > can do is to question what a function does. And perhaps poke the
> > maintainer (or author of said function) to write something that
> > explains that function (only for non-static functions).
> 
> Submitting patches is often the *best* way to poke a maintainer. ;-)
> 
> I know I'm not the only one who would find it a lot easier to tweak an
> 80% correct documentation patch as opposed to explaining the function
> call, how it fits in with the others, how it historically evolved that
> way, etc.
> 
> > Now, I say "newbies" but I would really mean experienced developers
> > that are new to a subsystem. We don't need silly questions. Something
> > more on the line of one experienced kernel developer reading some code
> > of the kernel they have no idea about, and if they can't figure out
> > what a function does, ask the question to the author. Perhaps we can
> > get better documentation of internal interfaces out of it.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Jonathan Corbet]

On Mon, 13 Jul 2015 12:47:23 +0300
Alexey Dobriyan <adobriyan@gmail.com> wrote:

> I have counter suggestion:
> * remove kernel-doc official status, remove generating scripts
>   because it doesn't work,

kerneldoc does work, after a fashion.  It works well enough that people
use it, and I hear about it quickly when somebody's comment change breaks
the docs build.  I don't think we can remove it in the absence of
something better.

Creating "something better" is actually on my list of things to do.  The
only problem is that I've had to buy a second 4TB drive to hold that list,
so I don't know when I'll have time to think about it.

jon

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

Am Montag, 13. Juli 2015, 19:45:09 schrieb Jonathan Corbet:
> On Mon, 13 Jul 2015 12:47:23 +0300
> 
> Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > I have counter suggestion:
> > * remove kernel-doc official status, remove generating scripts
> > 
> >   because it doesn't work,
> 
> kerneldoc does work, after a fashion.  It works well enough that people
> use it, and I hear about it quickly when somebody's comment change breaks
> the docs build.  I don't think we can remove it in the absence of
> something better.
Jonathan, do you keep an up-to-date copy of the generated files somewhere?
(I cannot get it to work out of the box on my machine)

> Creating "something better" is actually on my list of things to do.  The
> only problem is that I've had to buy a second 4TB drive to hold that list,
> so I don't know when I'll have time to think about it.
Where do you think are the major issues from your point of view? 

Would doxygen be "something better"? The kernel-doc style looks already quite 
similar.
Maybe the kernel should not reinvent everything :)




Thanks,
Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Jonathan Corbet]

On Mon, 13 Jul 2015 21:46:46 +0200
Peter Hüwe <PeterHuewe@gmx.de> wrote:

> > kerneldoc does work, after a fashion.  It works well enough that people
> > use it, and I hear about it quickly when somebody's comment change breaks
> > the docs build.  I don't think we can remove it in the absence of
> > something better.  
> Jonathan, do you keep an up-to-date copy of the generated files somewhere?
> (I cannot get it to work out of the box on my machine)

I don't, sorry.  That would probably be a good thing for me to do for a
number of reasons.

> > Creating "something better" is actually on my list of things to do.  The
> > only problem is that I've had to buy a second 4TB drive to hold that list,
> > so I don't know when I'll have time to think about it.  
> Where do you think are the major issues from your point of view? 

Well, as mentioned before, kerneldoc is fragile.  A little while back
somebody changed a struct to a union without updating the comments to
match; that was enough to break the entire document build process.  That
kind of thing happens a lot; I don't think it should have to be that way.

Beyond that, DocBook is a major pain.  I get a fair number of patches
along the lines of "that </para> should really have been ahead of that
</whatever>".  It's verbose, intimidating to newcomers, and causes more
worn-out shift keys than anything else.  We don't need something with the
complexity of DocBook; something closer to Markdown or ReST would do us
just fine and make the documentation much more accessible.

But making any such change is a big job, and convincing the community to
go with a change in tooling is probably a bigger job.  So I'm not rushing
into it.

> Would doxygen be "something better"? The kernel-doc style looks already quite 
> similar.
> Maybe the kernel should not reinvent everything :)

Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
still haven't done a big project in it.

But this is all just dreaming until somebody has the time to do a major
docs overhaul and sell it to everybody else.  Until then, I'm focused on
slowly improving what we have now...

jon

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Laurent Pinchart]

(CC'ing Daniel Vetter)

On Monday 13 July 2015 20:36:47 Jonathan Corbet wrote:
> On Mon, 13 Jul 2015 21:46:46 +0200 Peter Hüwe wrote:
> > > kerneldoc does work, after a fashion.  It works well enough that people
> > > use it, and I hear about it quickly when somebody's comment change
> > > breaks the docs build.  I don't think we can remove it in the absence of
> > > something better.
> > 
> > Jonathan, do you keep an up-to-date copy of the generated files somewhere?
> > (I cannot get it to work out of the box on my machine)
> 
> I don't, sorry.  That would probably be a good thing for me to do for a
> number of reasons.
> 
> > > Creating "something better" is actually on my list of things to do.  The
> > > only problem is that I've had to buy a second 4TB drive to hold that
> > > list, so I don't know when I'll have time to think about it.
> > 
> > Where do you think are the major issues from your point of view?
> 
> Well, as mentioned before, kerneldoc is fragile.  A little while back
> somebody changed a struct to a union without updating the comments to
> match; that was enough to break the entire document build process.  That
> kind of thing happens a lot; I don't think it should have to be that way.
> 
> Beyond that, DocBook is a major pain.  I get a fair number of patches
> along the lines of "that </para> should really have been ahead of that
> </whatever>".  It's verbose, intimidating to newcomers, and causes more
> worn-out shift keys than anything else.  We don't need something with the
> complexity of DocBook; something closer to Markdown or ReST would do us
> just fine and make the documentation much more accessible.
> 
> But making any such change is a big job, and convincing the community to
> go with a change in tooling is probably a bigger job.  So I'm not rushing
> into it.
> 
> > Would doxygen be "something better"? The kernel-doc style looks already
> > quite similar. Maybe the kernel should not reinvent everything :)
> 
> Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
> still haven't done a big project in it.
> 
> But this is all just dreaming until somebody has the time to do a major
> docs overhaul and sell it to everybody else.  Until then, I'm focused on
> slowly improving what we have now...

Daniel, I think I recall you mentioning some efforts going in that direction. 
Any update ?

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Daniel Vetter]

On Tue, Jul 14, 2015 at 10:40 AM, Laurent Pinchart
<laurent.pinchart@ideasonboard.com> wrote:
> (CC'ing Daniel Vetter)
>
> On Monday 13 July 2015 20:36:47 Jonathan Corbet wrote:
>> On Mon, 13 Jul 2015 21:46:46 +0200 Peter Hüwe wrote:
>> > > kerneldoc does work, after a fashion.  It works well enough that people
>> > > use it, and I hear about it quickly when somebody's comment change
>> > > breaks the docs build.  I don't think we can remove it in the absence of
>> > > something better.
>> >
>> > Jonathan, do you keep an up-to-date copy of the generated files somewhere?
>> > (I cannot get it to work out of the box on my machine)
>>
>> I don't, sorry.  That would probably be a good thing for me to do for a
>> number of reasons.
>>
>> > > Creating "something better" is actually on my list of things to do.  The
>> > > only problem is that I've had to buy a second 4TB drive to hold that
>> > > list, so I don't know when I'll have time to think about it.
>> >
>> > Where do you think are the major issues from your point of view?
>>
>> Well, as mentioned before, kerneldoc is fragile.  A little while back
>> somebody changed a struct to a union without updating the comments to
>> match; that was enough to break the entire document build process.  That
>> kind of thing happens a lot; I don't think it should have to be that way.
>>
>> Beyond that, DocBook is a major pain.  I get a fair number of patches
>> along the lines of "that </para> should really have been ahead of that
>> </whatever>".  It's verbose, intimidating to newcomers, and causes more
>> worn-out shift keys than anything else.  We don't need something with the
>> complexity of DocBook; something closer to Markdown or ReST would do us
>> just fine and make the documentation much more accessible.
>>
>> But making any such change is a big job, and convincing the community to
>> go with a change in tooling is probably a bigger job.  So I'm not rushing
>> into it.
>>
>> > Would doxygen be "something better"? The kernel-doc style looks already
>> > quite similar. Maybe the kernel should not reinvent everything :)
>>
>> Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
>> still haven't done a big project in it.
>>
>> But this is all just dreaming until somebody has the time to do a major
>> docs overhaul and sell it to everybody else.  Until then, I'm focused on
>> slowly improving what we have now...
>
> Daniel, I think I recall you mentioning some efforts going in that direction.
> Any update ?

We're contracting collabora (Daniel is working on it, cc'ed) to pimp
the kerneldoc toolchain for us:
- hyperlink functions/structures in the generated docbook
- add markdown support for kerneldoc paragraphs so that you can do
lists, basic formatting and tables

That's the basics I really need, whith that we can probably move all
the text sections from the drm docbook into the code. On top of that
there's a few wishlist items:
- allow to split up per-member sections for long structs (especially
useful for vtable structs where you want some longer text per each
function)
- asciiart or something (box model of drm planes/crtcs isn't the simplest thing)

We do use other stuff in userspace which is orders of magnitude better
than kerneldoc, but kerneldoc+docbook template has intertia. At least
in drm-land, and that's why I decided it's better to stick with it.

My longer-term goal is to fully document drm and i915 internals using
this, to help us bring new engineers up to speed faster. We have a
massive problem with training in that area in drm/i915. I'm also
trying to get a dedicated tech writer to keep all the drm docs written
by various people consistent. Plan is to do that once the kerneldoc
tooling is in more useful shape.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Mauro Carvalho Chehab]

Em Mon, 13 Jul 2015 20:36:47 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> > Would doxygen be "something better"? The kernel-doc style looks already quite 
> > similar.
> > Maybe the kernel should not reinvent everything :)

My experience is that, from time to time, people send us drivers written
by the manufacturer with Doxygen-compatible tags. Converting them to
kernel-doc is a huge task, not worthing the effort. I tried in the past
to request developers to "fix" the comments. The net result is either
to not have a final version of the driver submitted or to receive a
comments-stripped version of it, with is actually worse, as we loose
lots of otherwise good documentation. So, nowadays, I'm opting to keep
whatever comments are there at the code, provided that it is not using
C99 comments.

So, I'm a huge fan of extending the kernel-doc to accept all markup tags
that Doxygen supports.

> Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
> still haven't done a big project in it.

I never used Sphinx. Doxygen seems better on my eyes, because people
already sends us media drivers using Doxygen tags style. Also, as
mentioned, kernel-doc style is a subset. Not to mention that, in the
specific case of media,we're using Doxygen to document some libraries
packaged inside v4l-utils (with is the userspace counterpart of the 
Kernel subsystem). So, at least for me, Doxygen would be easier to use.

Regards,
Mauro

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Jonathan Corbet]

On Tue, 14 Jul 2015 09:43:42 -0300
Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:

> > Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
> > still haven't done a big project in it.  
> 
> I never used Sphinx. Doxygen seems better on my eyes, because people
> already sends us media drivers using Doxygen tags style. Also, as
> mentioned, kernel-doc style is a subset. Not to mention that, in the
> specific case of media,we're using Doxygen to document some libraries
> packaged inside v4l-utils (with is the userspace counterpart of the 
> Kernel subsystem). So, at least for me, Doxygen would be easier to use.

Just for clarity: any new documentation solution that required
reformatting all existing doc comments is clearly not going anywhere; even
I am not dumb enough to try to push something like that through.

In any case, this is all in the "would be nice to look at seriously" stage
over here.  I have no real idea if something like Sphinx would be a useful
and practical improvement or not; I'm certainly not agitating for a change
in that direction now.

jon

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Laurent Pinchart]

On Tuesday 14 July 2015 09:43:42 Mauro Carvalho Chehab wrote:
> Em Mon, 13 Jul 2015 20:36:47 -0600 Jonathan Corbet escreveu:
> > > Would doxygen be "something better"? The kernel-doc style looks already
> > > quite similar.
> > > Maybe the kernel should not reinvent everything :)
> 
> My experience is that, from time to time, people send us drivers written
> by the manufacturer with Doxygen-compatible tags. Converting them to
> kernel-doc is a huge task, not worthing the effort. I tried in the past
> to request developers to "fix" the comments. The net result is either
> to not have a final version of the driver submitted or to receive a
> comments-stripped version of it, with is actually worse, as we loose
> lots of otherwise good documentation. So, nowadays, I'm opting to keep
> whatever comments are there at the code, provided that it is not using
> C99 comments.
> 
> So, I'm a huge fan of extending the kernel-doc to accept all markup tags
> that Doxygen supports.
> 
> > Doxygen may be worth a look.  I'm personally fond of Sphinx, though I
> > still haven't done a big project in it.
> 
> I never used Sphinx. Doxygen seems better on my eyes, because people
> already sends us media drivers using Doxygen tags style. Also, as
> mentioned, kernel-doc style is a subset. Not to mention that, in the
> specific case of media,we're using Doxygen to document some libraries
> packaged inside v4l-utils (with is the userspace counterpart of the
> Kernel subsystem). So, at least for me, Doxygen would be easier to use.

We certainly need to accept documentation formats that are easy to use and 
write, but let's not forget that the biggest issue (in my opinion) is that we 
often don't get documentation at all. Cumbersome formats and tools might 
explain part of it, but that's not the whole story. How can we ensure that 
documentation gets written, merged and kept up to date ? What kind of 
incentives can we give ? The stick is easy to wave by not merging code changes 
that are not documented, but that can easily backfire.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Johannes Berg]

On Mon, 2015-07-13 at 21:46 +0200, Peter Hüwe wrote:
> Am Montag, 13. Juli 2015, 19:45:09 schrieb Jonathan Corbet:
> > On Mon, 13 Jul 2015 12:47:23 +0300
> > 
> > Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > I have counter suggestion:
> > > * remove kernel-doc official status, remove generating scripts
> > > 
> > >   because it doesn't work,
> > 
> > kerneldoc does work, after a fashion.  It works well enough that 
> > people
> > use it, and I hear about it quickly when somebody's comment change 
> > breaks
> > the docs build.  I don't think we can remove it in the absence of
> > something better.
> Jonathan, do you keep an up-to-date copy of the generated files 
> somewhere?
> (I cannot get it to work out of the box on my machine)
> 

kernel.org does, no?
https://www.kernel.org/doc/htmldocs/

johannes

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Peter Hüwe]

Am Montag, 13. Juli 2015, 11:47:23 schrieb Alexey Dobriyan:
> On Mon, Jul 13, 2015 at 1:38 AM, Peter Hüwe <PeterHuewe@gmx.de> wrote:
> > as we talked about recruiting in the other thread I realized that one
> > thing that might reduce the entrance barrier a bit (apart from tooling
> > and flow) would be proper documentation.
> 
> Newbies should not write documentation because they, by definition,
> have little knowledge of the code. It will lead to many, many useless
> and redundant comments like below whose only purpose is stealing
> vertical whitespace:

While I agree with your 'stupid documentation is stupid' argument, I tend to 
somewhat disagree about that newbies should not write documentation.

By newbies I do not mean someone new to programming - that of course makes no 
sense - but why shouldn't a experienced c-programmer who is currently figuring 
out how the kernel works not document stuff along the way?

I think that's much better than fixing whitespaces and long lines.

Peter

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Laurent Pinchart]

On Monday 13 July 2015 21:20:32 Peter Hüwe wrote:
> Am Montag, 13. Juli 2015, 11:47:23 schrieb Alexey Dobriyan:
> > On Mon, Jul 13, 2015 at 1:38 AM, Peter Hüwe <PeterHuewe@gmx.de> wrote:
> > > as we talked about recruiting in the other thread I realized that one
> > > thing that might reduce the entrance barrier a bit (apart from tooling
> > > and flow) would be proper documentation.
> > 
> > Newbies should not write documentation because they, by definition,
> > have little knowledge of the code. It will lead to many, many useless
> > and redundant comments like below whose only purpose is stealing
> 
> > vertical whitespace:
> While I agree with your 'stupid documentation is stupid' argument, I tend to
> somewhat disagree about that newbies should not write documentation.
> 
> By newbies I do not mean someone new to programming - that of course makes
> no sense - but why shouldn't a experienced c-programmer who is currently
> figuring out how the kernel works not document stuff along the way?

I fully agree, as I've gone through that process when I wrote my first DRM/KMS 
driver, and found it very valuable.

Writing documentation is a very good way to try and understand a subsystem for 
newcomers. It's time consuming though, which goes in the way of the tight 
schedules everybody seems to have. Furthermore employers are unfortunately not 
usually keen to fund documentation, especially when the schedule is tight.

On the other hand, I realized that writing down my findings while I was 
exploring the DRM subsystem was a great way to learn about it. It then took a 
bit of time to format my notes into proper documentation, but less than 
writing everything from scratch in one go. Taking notes also help structuring 
the documentation in a more natural way.

The picture wasn't all bright though. The documentation patch I submitted was 
of course not 100% complete, but the problem came later when the subsystem got 
rearchitectured without updating the documentation. Code changes were merged 
with kerneldoc comments to document functions, but the global DocBook 
architecture documentation was left untouched (except for integration of 
kerneldoc as API reference). The end result is that the DRM/KMS subsystem went 
from having a reasonable architecture documentation to being undocumented. 
Needless to say my motivation as a documentation writer for DRM/KMS took a big 
blow.

This being said, not everything is dark either. The V4L2 subsystem, for 
instance, does a pretty good job at keeping its documentation up-to-date. The 
maintainers and core developers always nack code changes that do not come with 
related documentation updates. This proved to be effective, but requires the 
rule to be strictly enforced. I wonder if this could scale to other 
subsystems, and what kind of other incentives we could create.

> I think that's much better than fixing whitespaces and long lines.

-- 
Regards,

Laurent Pinchart

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Lai Jiangshan]

On Mon, Jul 13, 2015 at 6:38 AM, Peter Hüwe <PeterHuewe@gmx.de> wrote:
>
> Hi,
>
> as we talked about recruiting in the other thread I realized that one thing
> that might reduce the entrance barrier a bit (apart from tooling and flow)
> would be proper documentation.
>
> I know documenation is usually not a developers most favorite task, but here
> are my findings anyway:

Hi, all

Comparing to the documents of functions, some other kinds of documents
might be more important but even seldom.

First, documents of why/what/how.
- Why the functionality/subsystem/structure/function/marco is needed?
- what is it
- How to implement it.

These documents are less and sometimes they are hidden in changelog.
why/what/how is suggested in changelog, so I often use git-blame
to fetch them, but it is inconvenient for normal-reviewers when it is
stored outside of kernel_src/Documentation/.

Secondly, documents of architecture/design.
- architecture overview
- architecture for control/data flow

These documents are less and sometimes they don't have proper picture
inside them.  Kernel developers like ascii pictures, but ascii pictures
can't provide enough information with decent density. and it is hard
to draw, to maintain... etc.

I once borrowed the two pictures in the page7/page9 of this:
http://www.linux-kongress.org/2010/slides/KVM-Architecture-LK2010.pdf
and gave 3 hours training to someones.  It helped me showed/remembered the
architecture, the interfaces between the components and the major
structures inside every components.  It would be useful if we also have
sufficient document of this kind in the kernel.

The documents of functions are also the documents of interfaces
between the components, they are important, so I also support for
Peter Hüwe.

And as noted before, changlog is also document for me, and IDEs(vim)
have the functionality to find the definition/usage, I also hope they
have the functionality to "find the changelog of this line".

Thanks,
Lai

>
>
>
> I had a quick look into our guideline (Documentation/kernel-doc-nano-
> HOWTO.txt) and it states:
>
> 1)
> > We definitely need kernel-doc formatted documentation for functions
> > that are exported to loadable modules using EXPORT_SYMBOL.
> This is definitely not the case at the moment :/ -- there are a lot of
> undocumented EXPORT_SYMBOL functions.
>
> 2)
> > We also look to provide kernel-doc formatted documentation for
> > functions externally visible to other kernel files (not marked
> > "static").
> Even less
>
> 3)
> > We also recommend providing kernel-doc formatted documentation
> > for private (file "static") routines, for consistency of kernel
> > source code layout.  But this is lower priority and at the
> > discretion of the MAINTAINER of that kernel source file.
> Don't even talk about that :)
>
>
> Imho new code should definitely have this documentation, especially for #1 and
> #2 -- but maybe even for #3.
>
>
>
>
> So this leads me to following questions:
> - How can we easily identify missing documentation?
> -- Maybe Julia can come up with some coccinelle magic?
> -- Maybe even mark non-extractable documentation and convert it.
> -- In the document it mentions scripts/basic/doproc.c checks for missing
> documentation, but this file does not exist anymore :/
>
>
>
>
> - how can we (automatically) support documentation?
> -- Can / should we create a tool that annotates functions and data types with
> a template documentation (I know good editors do something similar, but is the
> proposed style in concordance with the kernel doc style? At least it looks
> quite a lot like doxygen?)
>
> - How different is the kernel doc style to doxygen?
>
>
> - We are constantly looking for tasks newcomers can take on -- writing
> appropriate documentation is usually not that hard on the one hand and on the
> other people will learn a lot about the code they are documenting.
> -- maybe make it easier make use of Steven Rostedts proposed Kernel Patch
> submitting web form :P
>
> - new checkpatch option for checking the existance of documentation? *ducks
> and runs away*
>
>
> - and who would be in to join me in documenting everything :P
>
>
> Thanks,
> Peter
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [CORE-TOPIC] Documentation

[Jonathan Corbet]

On Mon, 13 Jul 2015 00:38:01 +0200
Peter Hüwe <PeterHuewe@gmx.de> wrote:

> - We are constantly looking for tasks newcomers can take on -- writing 
> appropriate documentation is usually not that hard on the one hand and on the 
> other people will learn a lot about the code they are documenting.

If you know people who can write coherently about kernel stuff — and
actually want to do so — please send them my way.  I can't find such
people, and I'm willing to pay for the work.  I don't hold out much hope
for getting newcomers to fill that gap.

jon