Re: [LSF/MM/BPF TOPIC]mTHP reliable allocation and reclamation

[Yang Shi <shy828301@gmail.com>]

On Thu, May 9, 2024 at 7:22 PM Barry Song <21cnbao@gmail.com> wrote:
>
> Hi,
>
> I'd like to propose a session about the allocation and reclamation of
> mTHP. This is related to Yu Zhao's
> TAO[1] but not the same.
>
> OPPO has implemented mTHP-like large folios across thousands of
> genuine Android devices, utilizing
> ARM64 CONT-PTE. However, we've encountered challenges:
>
> - The allocation of mTHP isn't consistently reliable; even after
> prolonged use, obtaining large folios
>   remains uncertain.
>   As an instance, following a few hours of operation, the likelihood
> of successfully allocating large
>   folios on a phone may decrease to just 2%.
>
> - Mixing large and small folios in the same LRU list can lead to
> mutual blocking and unpredictable
>   latency during reclamation/allocation.

I'm also curious how much large folios can improve reclamation
efficiency. Having large folios is supposed to reduce the scan time
since there should be fewer folios on LRU. But IIRC I haven't seen too
much data or benchmark (particularly real life workloads) regarding
this.

>
>   For instance, if you require large folios, the LRU list's tail could
> be filled with small folios.
>   LRU(LF- large folio, SF- small folio):
>
>    LF - LF - LF -  SF - SF - SF - SF - SF - SF -SF - SF - SF - SF - SF - SF - SF
>
>  You might end up reclaiming many small folios yet still struggle to
> allocate large folios. Conversely,
>  the inverse scenario can occur when the LRU list's tail is populated
> with large folios.
>
>    SF - SF - SF -  LF - LF - LF - LF - LF - LF -LF - LF - LF - LF - LF - LF - LF
>
> In OPPO's products, we allocate dedicated pageblocks solely for large
> folios allocation, and we've
> fine-tuned the LRU mechanism to support dual LRU—one for small folios
> and another for large ones.
> Dedicated page blocks offer a fundamental guarantee of allocating
> large folios. Additionally, segregating
> small and large folios into two LRUs ensures that both can be
> efficiently reclaimed for their respective
> users' requests.  However, while the implementation may lack aesthetic
> appeal and is primarily tailored
> for product purposes, it isn't fully upstreamable.
>
> You can obtain the architectural diagram of OPPO's approach from link[2].
>
> Therefore, my plan is to present:
>
> - Introduce the architecture of OPPO's mTHP-like approach, which
> encompasses additional optimizations
>   we've made to address swap fragmentation issues and improve swap
> performance, such as dual-zRAM
>   and compression/decompression of large folios [3].
>
> - Present OPPO's method of utilizing dedicated page blocks and a
> dual-LRU system for mTHP.
>
> - Share our observations from employing Yu Zhao's TAO on Pixel 6 phones.
>
> - Discuss our future direction—are we leaning towards TAO or dedicated
> page blocks? If we opt for page
>   blocks, how do we plan to resolve the LRU issue?
>
> [1] https://lore.kernel.org/linux-mm/20240229183436.4110845-1-yuzhao@google.com/
> [2] https://github.com/21cnbao/mTHP/blob/main/largefoliosarch.png
> [3] https://lore.kernel.org/linux-mm/20240327214816.31191-1-21cnbao@gmail.com/
>
> Thanks,
> Barry
>