OOMs on PS3 since kernel 6.9-rc4

[Damian Dudycz <damiandudycz@yahoo.com>]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

I'm running Gentoo on the PlayStation 3 console (PPC64BE CPU), using custom
firmware (OtherOS++) feature.

Upgrading from 6.6 to 6.10, I have noticed that OOM kills started during long
and intense processes, like compiling code or extracting a large archive.

The OOM usually occurs after about 10-20 minutes of for example
compiling the gentoo-kernel package.

This system has limited amount of RAM (256MB) and there's possibility to use
another 256MB of VRam as a fast swap device. Besides that, there's also
standard swap partition of 4GB enabled. I bisected with vanilla upstream sources
with the exception of some irrelevant patches mentioned at the end.

After bisecting, I have found that the issue first started to appear in commit
c0cd6f557b9090525d288806cccbc73440ac235a (build 6.9.0-rc4-test)
(titled: "page_alloc: fix freelist movement during block conversion").
https://github.com/torvalds/linux/commit/c0cd6f557b9090525d288806cccbc73440ac235a

Unfortunately, it doesn't revert cleanly on 6.11 so I couldn't test that.

# Files and directories:
- patches: contains patches applied to kernel when preparing a test build. These are working with version 6.9
- config: kernel config used
- bisect.txt: log from bisecting process
- dmesg.txt: log from dmesg after issue occurred
- c0cd6f557b9090525d288806cccbc73440ac235a.patch: diff from commit that introduced the issue
- proc - Collection of files from /proc, before, after and during test. („During" was always taken 5 minutes after test was started).
	- 6.9.0-rc3-test-dirty - working version, issue didn't happened.
	- 6.9.0-rc4-test-00116-gc0cd6f557b90-dirty - commit that introduced the issue, OOM occurred.
	- 6.11.0-test-dirty - newer version of kernel, OOM still occurred.

# Patches:
In order for kernel to work on the PS3 using OtherOS++, some patches are
required. I reduced the number of patches during testing, only to the ones
that are essential to boot correctly. The patches I have used are in "patches"
directory.

These are used mainly to enable linux to use disk regions that are
used for linux and I doubt they have any impact on the issue, but I'm adding them
in case this needs verification.

There are also 2 disabled patches related to page allocation, that I have left there,
but these were not used in tests, as they don't affect the result in this situation,
I'm leaving them just in case.

Mentioned logs and files are in attached tarball.


[-- Attachment #2: files-linux-6.9.0-rc4-test.tar.xz --]
[-- Type: application/x-xz, Size: 308736 bytes --]