Re: [TECH TOPIC] Kernel documentation - update and future directions

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

On Tue, Sep 02, 2025 at 02:00:43PM +0100, Mark Brown wrote:
> On Tue, Sep 02, 2025 at 02:14:34PM +0200, Mauro Carvalho Chehab wrote:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> 
> > > > That takes about 90s for me.  
> 
> > > I wander why here is 3 times faster... disk cache? python version?
> > > faster ssd?
> 
> > > What python version are you using?
> 
> > Heh, after running twice or three times to avoid cache issues, I tested
> > running it on my machine with two different python versions:
> 
> > $ time python3.13 scripts/kernel-doc.py . --none
> 
> > real    0m31,660s
> > user    0m30,911s
> > sys     0m0,588s
> 
> For me it's fairly consistent with python 3.11.2, I get some variation
> depending on what else is going on with the system but nothing hugely
> surprising.  It should mostly be running from cache, the underlying disk
> is a reasonable SSD. 

Yeah, it sounds that the huge performance increment was indeed on 3.11:

$ time python3.6 ./scripts/kernel-doc --none .
real	1m17,854s
user	1m16,651s
sys	0m0,707s

$ time python3.9 ./scripts/kernel-doc --none .
real	1m0,193s
user	0m58,942s
sys	0m0,614s

$ time python3.10 ./scripts/kernel-doc --none .
real	0m55,376s
user	0m54,168s
sys	0m0,636s

$ time python3.11 ./scripts/kernel-doc --none .
real	0m34,878s
user	0m33,665s
sys	0m0,661s

$ time python3.12 ./scripts/kernel-doc --none .
real	0m34,590s
user	0m33,844s
sys	0m0,613s

$ time python3.13 ./scripts/kernel-doc --none .
real	0m31,751s
user	0m30,951s
sys	0m0,640s

==============  =============   =============   ================================
Python Version	Real Time (s)	User Time (s)	Performance Increase (Real Time)
==============  =============   =============   ================================
3.6		77.854 s	76.651 s	(baseline)
3.9		60.193 s	58.942 s	22.7% faster
3.10		55.376 s	54.168 s	28.9% faster
3.11		34.878 s	33.665 s	55.2% faster
3.12		34.590 s	33.844 s	55.6% faster
3.13		31.751 s	30.951 s	59.2% faster
==============  =============   =============   ================================

> The single core performance on this machine isn't
> amazing, it's more getting it's speed from having a bunch of cores.

As kernel-doc is currently single threaded, performance for a single
core is what matters most. I suspect that most of kernel-doc time is
spent handling regular expressions, specially when I/O is fast. 

-

To sum-up those discussions, I can propose a patchset for the next
merge window that would:

1. change kernel-doc exec to re-run using the latest available python
   version if version < 3.11, on a similar same way to what
   scripts-pre-install and scripts-build-wrapper does(*);

2. add a command line parameter for kernel-doc to make it pick only
   files that have a .. kernel-doc markup;

3. add a build logic to run it with make all, perhaps inside a Kconfig
   symbol like "config DOC_WARNINGS", disabled by default, but enabled
   with allyesconfig/allmodconfig.

4. with time, add more validations there, like checking for
   EXPORT_SYMBOL without documentation and other neat features.

This needs to be coordinated with some efforts to cleanup warnings,
to avoid having hundreds of new warnings at build time.

This way, even on LTS, we'll have fast kernel-doc check, and will likely
take lot less than 32 seconds, as it will only validate a small set of
files that are pointed by a kernel-doc markup notation inside
Documentation, specially if Python >= 3.11 is installed.

It should be noticed that can generate out of blue lots of new
warnings that are currently there by currently hidden, specially
if we add "-Wall" to the build target.

Comments?

(*) The logic there checks if python version is below a minimal
    version. If it is, it seeks for python 3.x exec files, picking
    the latest one if found, and re-running it with such version.

-- 
Thanks,
Mauro