From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82FEFE7717B for ; Sat, 7 Dec 2024 08:03:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB74C8D0018; Sat, 7 Dec 2024 03:03:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A68538D0015; Sat, 7 Dec 2024 03:03:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9080A8D0018; Sat, 7 Dec 2024 03:03:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6BF898D0015 for ; Sat, 7 Dec 2024 03:03:32 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D426CA02BF for ; Sat, 7 Dec 2024 08:03:31 +0000 (UTC) X-FDA: 82867423254.02.6204323 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf20.hostedemail.com (Postfix) with ESMTP id CD18A1C0002 for ; Sat, 7 Dec 2024 08:03:09 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=i5vBAPmb; spf=pass (imf20.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733558591; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gDUJfhQEwThVj3l12vGRVX0Pvt/LPPYQM+5JRuWaSRM=; b=bUytRsTg4YEbvnIdtmjQq5CE0TAuGx30ky29RasYTM4Oghx7of8A1f9HYEgrJy7kkVpx/L 3niPu4k+Spbw5FSzrOGd1PYQGAS4LWLsdYneY+PrspirQU8OJVvS3dRdC9fOOVLE9+0Bm3 UpGt9+JlbEyxndI+SHflce5BGXM1kTI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733558591; a=rsa-sha256; cv=none; b=4NvAV1jQ30bLJYpVBsMtOD8pdVxsC8UilBFqj09RGULbzFJOTi/1e9dT/S19QPzVZFF8nF D3QGVdqouYuNzQwKUcEYs/9qNtvC3jdwlQ1lHYNJ9xsVVJ9UoY8QA8PebwOePumbmdRW1y /WK7sUMXnwaMNnGiyxXfa2iiXVDzp40= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=i5vBAPmb; spf=pass (imf20.hostedemail.com: domain of luxu.kernel@bytedance.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=luxu.kernel@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2eec9b3a1bbso2071497a91.3 for ; Sat, 07 Dec 2024 00:03:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733558607; x=1734163407; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=gDUJfhQEwThVj3l12vGRVX0Pvt/LPPYQM+5JRuWaSRM=; b=i5vBAPmb8lHzhFUgkVB1qutngv2s0/qQmF76nbFZVINYdACBxHQOJgG5bsoMtwYS7K nOwf6l4RsFgXRxhATr48bYaX4ceB9no3VlMi2W8FHjWC5GoJPovsdMjIzzc09X6gvJrq 3G2MEpxAPnWe0UFIvIapcr0bEZeZxP6tfs3IysRuXeE3vybSuMboXS/Vy7ArcocX2bIM /5ISSkjuMEl6ITJFlFRJwFt/VBI2xuJ2wAGFhpDbqzbsJ4WSrJn5u91rKIWW5SZGTKz/ 6/gHjBF4Iz3esCghdxcCf8VHH8EpYuhpWGPTrVtwLLIqBnwpXXTvnDJ0FsfMKuwT3ihE MA5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733558607; x=1734163407; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gDUJfhQEwThVj3l12vGRVX0Pvt/LPPYQM+5JRuWaSRM=; b=YCGr9ZbXhDAShmp1zeviKmuGlUBue7CUdBw1eQOTId9VEwsScpQkytB7ty5mPrdfvd 7M+INwkzU+nBy9EqdHJ5rS28ggGUJ7ZSS2K8miUIFrlTCoqh2MnXmuYZj8vRn4PukXMs pWHMh9MxGT8mcQJKsPH+sFTwdOmMJgIvYz9ZYQjYUdnsbhBfP7FIpHu2OJ9GycQsAP0G NO/Mp8Lc3H8NCAmnqvTW30ISb+xjybVLzXh40lqa8Mc7bQCk3Czr+KWvAQWRL8vXsKs7 D8M0010JXuAWiHqdn32n7wt2rq/9HWAKMKpMkw7Z+T7rbqnNKsQs6DhkifkvTWY3sY63 +Uxg== X-Forwarded-Encrypted: i=1; AJvYcCWy1EffF8ByVn5TLUKPblpJqDo0gBDAQwCvDg730smJbXuFgK0jXwW9OZg3mm1vxvNHBCOoI6+0lg==@kvack.org X-Gm-Message-State: AOJu0YwHgpC0XPyzBwooqH3ntjvTBKzIp3exSEbQCZ0YR79qSaxCtvWl PSK4MQaBGvlTWqCd6rCu5FVM+m60qcIj/HrROAM5A//sfLJ+TXNB601Mo6RKAa2HtrdtpCPKmkN Z0QHfEbSZ6JBNuhHb2Add9PuCDiC30lioKpBJWg== X-Gm-Gg: ASbGncsiDLG3cd1Cfldb5wCTvxBXjeFZV/x+pvk15vKd6vNNibJOVvsYiJ1vK54C+oY J3Opyq5Cfa+FxXBmDPaPXyIl11q+X4jCBmg== X-Google-Smtp-Source: AGHT+IHcu94k1jRETR3BaF5fuRy6SP6WomFWoYn2yTNFMTKyEMNckr0xNnbkNH/kStFnM2Y/znXINnSK9kfyVAXT61E= X-Received: by 2002:a17:90b:5292:b0:2ee:49c4:4a7c with SMTP id 98e67ed59e1d1-2ef6a6bdb76mr8824607a91.18.1733558607363; Sat, 07 Dec 2024 00:03:27 -0800 (PST) MIME-Version: 1.0 References: <20241205103729.14798-1-luxu.kernel@bytedance.com> <315752c5-6129-4c8b-bf8c-0cc26f0ad5c5@redhat.com> In-Reply-To: From: Xu Lu Date: Sat, 7 Dec 2024 16:03:16 +0800 Message-ID: Subject: Re: [External] Re: [RFC PATCH v2 00/21] riscv: Introduce 64K base page To: Pedro Falcato Cc: David Hildenbrand , Zi Yan , paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, ardb@kernel.org, anup@brainfault.org, atishp@atishpatra.org, xieyongji@bytedance.com, lihangjing@bytedance.com, punit.agrawal@bytedance.com, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Linux MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: CD18A1C0002 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: erhyhfwgwie9urrns9hwzrast5ce5arr X-HE-Tag: 1733558589-814831 X-HE-Meta: U2FsdGVkX190yVahVcdR1oy1NooHUkRK9H6u022B6xaoJ2NPgno2Po9hE0f0weXti3bd7uBsKeQWG5rXz3+YPojvjAiNzJHetago/6PtuwyisV/tHkSyY2zWgvs8TnGU43WFzQGmv1VE+7izQP4IC3XD8/+7+YFly9V8cuLw2m7UDpNQ38A/4QW8QGV038JRwZTgVGEu0J6KhwoQg6yCT90dM2U96l92YPkpFwg9IzDHsiL7jOHgsvyb2WekSvC6c+fnvjPoyaUMIzyUJqJhzlz1Y48OWSSsdE5RaO8bbhuav4cYP4Q0FW7dvtMuGGGiIkHHZrmeUiFn/y8uDDUWaFDJHRIzAdsrY+gk2YwQTAijavKPzEDUizRNCLz3k2gOG/PIOHTviQY88re4WpKN1ipKxC9+dfn/o8oFHk5qqR88mEagaKYHgHvBKk2MrGJFTThZ2X3gPeJoPfmqFDuPAEi0HZbrlepcVC31lNO98nVF6ZgY7lzgt0nSsQoUKeDy7NGzi+gJlhbDh2rOPGLhLSpyIlzAf5gPT2ajb3JpiTjHSbzx3JnDHkDTEHbt+G1oh7QUrEeoAFXJZMmeOYqj7y6fG/PmxqY6WXQ7rPLZxEJ2FbZVZrJYHmR9cjtOFmKpKiIAMf9GsY4ufY22Z2x7Hl+L1vhUiCvE3aRogJ7TdEwlQ4B9unwofrNnmAmyO8zNVQAikwQOsC1g9LKPzaTKHSQLvuyE5cy234mMW0ibC/W/tUcmyDwgz2k+4NYXCRCWsTVjEQTj29CjRQ7qxVkuhzHnUKdxLuRu+Qp/q1wKVJqA5SX9K7HR3Scu6U0S8VtVPaJFzZP7Bew4K1WBBkwAvAXWth/jUPj0PRS9TNPuDB3mQ0jnsKlj8lMMHlyfHbxK1M2uyIaizz+FWT4KD6xkWu3WzrZpggAIywUc4pVZiZfa3ybSmNovS1cXB2R7ckkL6eu9xTMltAicWb+70Z5 371WM4tZ Ji4AYayLnmXR0nG6zwUsZLo73lFmHVUQMsQXuMUO4iNeeQ5qvnw/kdQgJOov7jEyJJeaep4FM/2ndnwd7gy81z/JQdptMJ9t+/jaJPz768I6BqcFSzoLj8Lb9z5vXz+GBuIGUUrBW5Tmj768n/nGpePlj/pH2eS193bJTz5CxMA/NFznb4UmSAmNnhD5LcrkO5y0XjQRsqZUBq+bUH6zSDYXM9CxsyMyDD6fwRQcqLveF6uni3J6j9mI/vdtmSVg5YPKnrCoq2iAm0UfQKwNmkPQv5wB9nAWVG4dflL/vI6YbAj9ZBQLWR5RTMqdkSYgtk7Luzx1xZssl8ZGitH702dtOKriYF+o28snVlHEkFByaCtDR52QEfbQeWWdUHOc+Kvgk0QNBZqu5JNvFFk+5Z847Nj35QoXAB/oQ4lg4nx5cPSRSRkWHq+m7MYjsJeNX4IyIudfmg2AE9JJ2aEnhbh1fhA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Pedro, On Sat, Dec 7, 2024 at 2:49=E2=80=AFAM Pedro Falcato wrote: > > On Fri, Dec 6, 2024 at 1:42=E2=80=AFPM Xu Lu = wrote: > > > > Hi David, > > > > On Fri, Dec 6, 2024 at 6:13=E2=80=AFPM David Hildenbrand wrote: > > > > > > On 06.12.24 03:00, Zi Yan wrote: > > > > On 5 Dec 2024, at 5:37, Xu Lu wrote: > > > > > > > >> This patch series attempts to break through the limitation of MMU = and > > > >> supports larger base page on RISC-V, which only supports 4K page s= ize > > > >> now. The key idea is to always manage and allocate memory at a > > > >> granularity of 64K and use SVNAPOT to accelerate address translati= on. > > > >> This is the second version and the detailed introduction can be fo= und > > > >> in [1]. > > > >> > > > >> Changes from v1: > > > >> - Rebase on v6.12. > > > >> > > > >> - Adjust the page table entry shift to reduce page table memory us= age. > > > >> For example, in SV39, the traditional va behaves as: > > > >> > > > >> ---------------------------------------------- > > > >> | pgd index | pmd index | pte index | offset | > > > >> ---------------------------------------------- > > > >> | 38 30 | 29 21 | 20 12 | 11 0 | > > > >> ---------------------------------------------- > > > >> > > > >> When we choose 64K as basic software page, va now behaves as: > > > >> > > > >> ---------------------------------------------- > > > >> | pgd index | pmd index | pte index | offset | > > > >> ---------------------------------------------- > > > >> | 38 34 | 33 25 | 24 16 | 15 0 | > > > >> ---------------------------------------------- > > > >> > > > >> - Fix some bugs in v1. > > > >> > > > >> Thanks in advance for comments. > > > >> > > > >> [1] https://lwn.net/Articles/952722/ > > > > > > > > This looks very interesting. Can you cc me and linux-mm@kvack.org > > > > in the future? Thanks. > > > > > > > > Have you thought about doing it for ARM64 4KB as well? ARM64=E2=80= =99s contig PTE > > > > should have similar effect of RISC-V=E2=80=99s SVNAPOT, right? > > > > > > What is the real benefit over 4k + large folios/mTHP? > > > > > > 64K comes with the problem of internal fragmentation: for example, a > > > page table that only occupies 4k of memory suddenly consumes 64K; qui= te > > > a downside. > > > > The original idea comes from the performance benefits we achieved on > > the ARM 64K kernel. We run several real world applications on the ARM > > Ampere Altra platform and found these apps' performance based on the > > 64K page kernel is significantly higher than that on the 4K page > > kernel: > > For Redis, the throughput has increased by 250% and latency has > > decreased by 70%. > > For Mysql, the throughput has increased by 16.9% and latency has > > decreased by 14.5%. > > For our own newsql database, throughput has increased by 16.5% and > > latency has decreased by 13.8%. > > > > Also, we have compared the performance between 64K and 4k + large > > folios/mTHP on ARM Neoverse-N2. The result shows considerable > > performance improvement on 64K kernel for both speccpu and lmbench, > > even when 4K kernel enables THP and ARM64_CONTPTE: > > For speccpu benchmark, 64K kernel without any huge pages optimization > > can still achieve 4.17% higher score than 4K kernel with transparent > > huge pages as well as CONTPTE optimization. > > For lmbench, 64K kernel achieves 75.98% lower memory mapping > > latency(16MB) than 4K kernel with transparent huge pages and CONTPTE > > optimization, 84.34% higher map read open2close bandwidth(16MB), and > > 10.71% lower random load latency(16MB). > > Interestingly, sometimes kernel with transparent pages support have > > poorer performance for both 4K and 64K (for example, mmap read > > bandwidth bench). We assume this is due to the overhead of huge pages' > > combination and collapse. > > Also, if you check the full result, you will find that usually the > > larger the memory size used for testing is, the better the performance > > of 64k kernel is (compared to 4K kernel). Unless the memory size lies > > in a range where 4K kernel can apply 2MB huge pages while 64K kernel > > can't. > > In summary, for performance sensitive applications which require > > higher bandwidth and lower latency, sometimes 4K pages with huge pages > > may not be the best choice and 64k page can achieve better results. > > The test environment and result is attached. > > > > As RISC-V has no native 64K MMU support, we introduce a software > > implementation and accelerate it via Svnapot. Of course, there will be > > some extra overhead compared with native 64K MMU. Thus, we are also > > trying to persuade the RISC-V community to support the extension of > > native 64K MMU [1]. Please join us if you are interested. > > > > Ok, so you... didn't test this on riscv? And you're basing this > patchset off of a native 64KiB page size kernel being faster than 4KiB > + CONTPTE? I don't see how that makes sense? Sorry for the misleading. I didn't intend to use ARM data to support this patch, just to explain the idea source. We do prefer 64K MMU for the performance improvement it brought to real applications and benchmarks. And since RISC-V does not support it yet, we internally use this patch as a transitional solution for RISC-V. And if native 64k MMU is available, this patch can be canceled. The only usage of this patch I can think of then is to make the kernel support more page sizes than MMU, as long as Svnapot supports the corresponding size. We will try to release the performance data in the next version. There have been more issues with applications and OS adaptation:) So this version is still an RFC. > > /me is confused > > How many of these PAGE_SIZE wins are related to e.g userspace basing > its buffer sizes (or whatever) off of the system page size? Where > exactly are you gaining time versus the CONTPTE stuff? > I think MM in general would be better off if we were more transparent > with regard to CONTPTE and page sizes instead of hand waving with > "hardware page size !=3D software page size", which is such a *checks > notes* 4.4BSD idea... :) At the very least, this patchset seems to go > against all the work on better supporting large folios and CONTPTE. By the way, the core modification of this patch is turning pte structure to an array of 16 entries to map a 64K page and accelerating it via Svnapot. I think it is all about architectural pte and has little impact on pages or folios. Please remind me if anything is missed and I will try to fix it. > > -- > Pedro Thanks, Xu Lu