[RFC 0/6] Raise the bar with regards to Python and Sphinx requirements

[RFC 0/6] Raise the bar with regards to Python and Sphinx requirements

[Mauro Carvalho Chehab]

This series comes after https://lore.kernel.org/linux-doc/87a5b96296.fsf@trenco.lwn.net/T/#t
It  increases the minimal requirements for Sphinx and Python.

Sphinx release dates:

	Release 2.4.0 (released Feb 09, 2020)
	Release 2.4.4 (released Mar 05, 2020) (current minimal requirement)
	Release 3.4.0 (released Dec 20, 2020)
	Release 3.4.3 (released Jan 08, 2021)

	(https://www.sphinx-doc.org/en/master/changes/index.html)

In terms of Python, we're currently at 3.5:

	Python	Release date 
	3.5	2015-09-13    (current minimal requirement)
	3.6	2016-12-23
	3.7 	2018-06-27
	3.8 	2019-10-14
	3.9 	2020-10-05
	3.10	2021-10-04

	(according with https://en.wikipedia.org/w/index.php?title=History_of_Python)

The new minimal requirements are now:
	- Sphinx 3.4.3;
	- Python 3.9

The new Sphinx minimal requirement allows dropping all backward-compatible code
we have at kernel-doc and at Sphinx extensions.

The new Python minimal requirement matches the current required level for
all scripts but one (*). The one that doesn't match is at tools/net/sunrpc/xdrgen.

Those matches a 4-years old toolchain, which sounds a reasonable period
of time, as Python/Sphinx aren't required for the Kernel build.

Mauro Carvalho Chehab (6):
  scripts/get_abi.py: make it backward-compatible with Python 3.6
  docs: extensions: don't use utf-8 syntax for descriptions
  docs: automarkup: drop legacy support
  scripts/kernel-doc: drop Sphinx version check
  docs: changes: update Sphinx minimal version to 3.4.3
  doc: changes: update Python minimal version

 Documentation/conf.py                       |   2 +-
 Documentation/process/changes.rst           |   4 +-
 Documentation/sphinx/automarkup.py          |  32 ++---
 Documentation/sphinx/cdomain.py             |   7 +-
 Documentation/sphinx/kernel_abi.py          |   6 +-
 Documentation/sphinx/kernel_feat.py         |   4 +-
 Documentation/sphinx/kernel_include.py      |   4 +-
 Documentation/sphinx/kerneldoc.py           |   5 -
 Documentation/sphinx/kfigure.py             |  10 +-
 Documentation/sphinx/load_config.py         |   2 +-
 Documentation/sphinx/maintainers_include.py |   4 +-
 Documentation/sphinx/rstFlatTable.py        |  10 +-
 scripts/get_abi.py                          |  16 ++-
 scripts/kernel-doc                          | 129 +++-----------------
 14 files changed, 58 insertions(+), 177 deletions(-)

-- 
2.48.1

[RFC 5/6] docs: changes: update Sphinx minimal version to 3.4.3

[Mauro Carvalho Chehab]

Doing that allows us to get rid of all backward-compatible code.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/conf.py             | 2 +-
 Documentation/process/changes.rst | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 0c2205d536b3..3dad1f90b098 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -47,7 +47,7 @@ from load_config import loadConfig
 # -- General configuration ------------------------------------------------
 
 # If your documentation needs a minimal Sphinx version, state it here.
-needs_sphinx = '2.4.4'
+needs_sphinx = '3.4.3'
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 82b5e378eebf..012d2b715c2a 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -58,7 +58,7 @@ mcelog                 0.6              mcelog --version
 iptables               1.4.2            iptables -V
 openssl & libcrypto    1.0.0            openssl version
 bc                     1.06.95          bc --version
-Sphinx\ [#f1]_         2.4.4            sphinx-build --version
+Sphinx\ [#f1]_         3.4.3            sphinx-build --version
 cpio                   any              cpio --version
 GNU tar                1.28             tar --version
 gtags (optional)       6.6.5            gtags --version
-- 
2.48.1

[RFC 6/6] doc: changes: update Python minimal version

[Mauro Carvalho Chehab]

The current minimal version doesn't match what we have currently
at the Kernel:

	$ vermin -v  $(git ls-files *.py)
	...
	Minimum required versions: 3.10
	Incompatible versions:     2

Those are the Python scripts requiring versions higher than current minimal (3.5):

        !2, 3.10     tools/net/sunrpc/xdrgen/generators/__init__.py
        !2, 3.10     tools/net/sunrpc/xdrgen/generators/program.py
        !2, 3.10     tools/net/sunrpc/xdrgen/subcmds/source.py
        !2, 3.10     tools/net/sunrpc/xdrgen/xdr_ast.py
        !2, 3.10     tools/power/cpupower/bindings/python/test_raw_pylibcpupower.py
        !2, 3.9      tools/testing/selftests/net/rds/test.py
        !2, 3.9      tools/net/ynl/ethtool.py
        !2, 3.9      tools/net/ynl/cli.py
        !2, 3.9      scripts/checktransupdate.py
        !2, 3.8      tools/testing/selftests/tc-testing/plugin-lib/nsPlugin.py
        !2, 3.8      tools/testing/selftests/hid/tests/base.py
        !2, 3.7      tools/testing/selftests/turbostat/smi_aperf_mperf.py
        !2, 3.7      tools/testing/selftests/turbostat/defcolumns.py
        !2, 3.7      tools/testing/selftests/turbostat/added_perf_counters.py
        !2, 3.7      tools/testing/selftests/hid/tests/conftest.py
        !2, 3.7      tools/testing/kunit/qemu_config.py
        !2, 3.7      tools/testing/kunit/kunit_tool_test.py
        !2, 3.7      tools/testing/kunit/kunit.py
        !2, 3.7      tools/testing/kunit/kunit_parser.py
        !2, 3.7      tools/testing/kunit/kunit_kernel.py
        !2, 3.7      tools/testing/kunit/kunit_json.py
        !2, 3.7      tools/testing/kunit/kunit_config.py
        !2, 3.7      tools/perf/scripts/python/gecko.py
        !2, 3.7      scripts/rust_is_available_test.py
        !2, 3.7      scripts/bpf_doc.py
        !2, 3.6      tools/writeback/wb_monitor.py
        !2, 3.6      tools/workqueue/wq_monitor.py
        !2, 3.6      tools/workqueue/wq_dump.py
        !2, 3.6      tools/usb/p9_fwd.py
        !2, 3.6      tools/tracing/rtla/sample/timerlat_load.py
        !2, 3.6      tools/testing/selftests/net/openvswitch/ovs-dpctl.py
        !2, 3.6      tools/testing/selftests/net/nl_netdev.py
        !2, 3.6      tools/testing/selftests/net/lib/py/ynl.py
        !2, 3.6      tools/testing/selftests/net/lib/py/utils.py
        !2, 3.6      tools/testing/selftests/net/lib/py/nsim.py
        !2, 3.6      tools/testing/selftests/net/lib/py/netns.py
        !2, 3.6      tools/testing/selftests/net/lib/py/ksft.py
        !2, 3.6      tools/testing/selftests/kselftest/ksft.py
        !2, 3.6      tools/testing/selftests/hid/tests/test_tablet.py
        !2, 3.6      tools/testing/selftests/hid/tests/test_sony.py
        !2, 3.6      tools/testing/selftests/hid/tests/test_multitouch.py
        !2, 3.6      tools/testing/selftests/hid/tests/test_mouse.py
        !2, 3.6      tools/testing/selftests/hid/tests/base_gamepad.py
        !2, 3.6      tools/testing/selftests/hid/tests/base_device.py
        !2, 3.6      tools/testing/selftests/drivers/net/stats.py
        !2, 3.6      tools/testing/selftests/drivers/net/shaper.py
        !2, 3.6      tools/testing/selftests/drivers/net/queues.py
        !2, 3.6      tools/testing/selftests/drivers/net/ping.py
        !2, 3.6      tools/testing/selftests/drivers/net/lib/py/remote_ssh.py
        !2, 3.6      tools/testing/selftests/drivers/net/lib/py/load.py
        !2, 3.6      tools/testing/selftests/drivers/net/lib/py/__init__.py
        !2, 3.6      tools/testing/selftests/drivers/net/lib/py/env.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/rss_ctx.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/pp_alloc_fail.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/nic_performance.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/nic_link_layer.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/lib/py/linkconfig.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/lib/py/__init__.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/devmem.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/devlink_port_split.py
        !2, 3.6      tools/testing/selftests/drivers/net/hw/csum.py
        !2, 3.6      tools/testing/selftests/devices/probe/test_discoverable_devices.py
        !2, 3.6      tools/testing/selftests/bpf/test_bpftool_synctypes.py
        !2, 3.6      tools/testing/selftests/bpf/generate_udp_fragments.py
        !2, 3.6      tools/testing/kunit/run_checks.py
        !2, 3.6      tools/testing/kunit/kunit_printer.py
        !2, 3.6      tools/sched_ext/scx_show_state.py
        !2, 3.6      tools/perf/tests/shell/lib/perf_metric_validation.py
        !2, 3.6      tools/perf/tests/shell/lib/perf_json_output_lint.py
        !2, 3.6      tools/perf/scripts/python/parallel-perf.py
        !2, 3.6      tools/perf/scripts/python/flamegraph.py
        !2, 3.6      tools/perf/scripts/python/arm-cs-trace-disasm.py
        !2, 3.6      tools/perf/pmu-events/models.py
        !2, 3.6      tools/perf/pmu-events/metric_test.py
        !2, 3.6      tools/perf/pmu-events/metric.py
        !2, 3.6      tools/perf/pmu-events/jevents.py
        !2, 3.6      tools/net/ynl/ynl-gen-rst.py
        !2, 3.6      tools/net/ynl/ynl-gen-c.py
        !2, 3.6      tools/net/ynl/lib/ynl.py
        !2, 3.6      tools/net/ynl/lib/nlspec.py
        !2, 3.6      tools/crypto/tcrypt/tcrypt_speed_compare.py
        !2, 3.6      tools/cgroup/iocost_monitor.py
        !2, 3.6      tools/cgroup/iocost_coef_gen.py
        !2, 3.6      scripts/make_fit.py
        !2, 3.6      scripts/macro_checker.py
        !2, 3.6      scripts/get_abi.py
        !2, 3.6      scripts/generate_rust_analyzer.py
        !2, 3.6      scripts/gdb/linux/timerlist.py
        !2, 3.6      scripts/gdb/linux/pgtable.py
        !2, 3.6      scripts/clang-tools/run-clang-tools.py
        !2, 3.6      Documentation/sphinx/automarkup.py

Even if we exclude tools/net/sunrpc/xdrgen/, the minimal version is
Python 3.9.

Update process/changes to reflect the current minimal version required to
run Python scripts outside tools.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/process/changes.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/process/changes.rst b/Documentation/process/changes.rst
index 012d2b715c2a..e12aa044e09a 100644
--- a/Documentation/process/changes.rst
+++ b/Documentation/process/changes.rst
@@ -63,7 +63,7 @@ cpio                   any              cpio --version
 GNU tar                1.28             tar --version
 gtags (optional)       6.6.5            gtags --version
 mkimage (optional)     2017.01          mkimage --version
-Python (optional)      3.5.x            python3 --version
+Python (optional)      3.9.x            python3 --version
 GNU AWK (optional)     5.1.0            gawk --version
 ====================== ===============  ========================================
 
-- 
2.48.1

Re: [RFC 0/6] Raise the bar with regards to Python and Sphinx requirements

[Donald Hunter]

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> This series comes after https://lore.kernel.org/linux-doc/87a5b96296.fsf@trenco.lwn.net/T/#t
> It  increases the minimal requirements for Sphinx and Python.
>
> Sphinx release dates:
>
> 	Release 2.4.0 (released Feb 09, 2020)
> 	Release 2.4.4 (released Mar 05, 2020) (current minimal requirement)
> 	Release 3.4.0 (released Dec 20, 2020)
> 	Release 3.4.3 (released Jan 08, 2021)
>
> 	(https://www.sphinx-doc.org/en/master/changes/index.html)

It's worth mentioning here that my fix for the C performance regression
landed in Sphinx 7.4.0. All versions from 3.0.0 to 7.3.x are much slower
for building the kernel docs. See #12162 here:

https://www.sphinx-doc.org/en/master/changes/7.4.html#id7

Re: [RFC 0/6] Raise the bar with regards to Python and Sphinx requirements

[Jonathan Corbet]

Donald Hunter <donald.hunter@gmail.com> writes:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>
>> This series comes after https://lore.kernel.org/linux-doc/87a5b96296.fsf@trenco.lwn.net/T/#t
>> It  increases the minimal requirements for Sphinx and Python.
>>
>> Sphinx release dates:
>>
>> 	Release 2.4.0 (released Feb 09, 2020)
>> 	Release 2.4.4 (released Mar 05, 2020) (current minimal requirement)
>> 	Release 3.4.0 (released Dec 20, 2020)
>> 	Release 3.4.3 (released Jan 08, 2021)
>>
>> 	(https://www.sphinx-doc.org/en/master/changes/index.html)
>
> It's worth mentioning here that my fix for the C performance regression
> landed in Sphinx 7.4.0. All versions from 3.0.0 to 7.3.x are much slower
> for building the kernel docs. See #12162 here:
>
> https://www.sphinx-doc.org/en/master/changes/7.4.html#id7

Indeed, we have noticed the speedup - much appreciated, thank you!

jon