From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F2D5C4725D for ; Fri, 19 Jan 2024 09:27:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A5696B0072; Fri, 19 Jan 2024 04:27:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 955816B0074; Fri, 19 Jan 2024 04:27:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81D096B0075; Fri, 19 Jan 2024 04:27:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6D9E06B0072 for ; Fri, 19 Jan 2024 04:27:14 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2E375120147 for ; Fri, 19 Jan 2024 09:27:14 +0000 (UTC) X-FDA: 81695531988.24.C16AB34 Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) by imf02.hostedemail.com (Postfix) with ESMTP id 446648000E for ; Fri, 19 Jan 2024 09:27:12 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=m6wNPqs9; spf=pass (imf02.hostedemail.com: domain of nylon.chen@sifive.com designates 209.85.219.181 as permitted sender) smtp.mailfrom=nylon.chen@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705656432; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PX3oyvjTSeaEmMA7BmGcrPiugjRL8juVncN6NIp0Uhw=; b=wbS1LQN1rzh19Jf3cvtzf8IC2Lf5tXpoJsM9hRC8w6XU6OI1IN1DpZzbS9rCZKbLD6wvr3 y/6xd3BXv1oPUZvY5lcEcZHPRKTIh/ckbwMFhKdW5q/Nk43Eg2n72WLYFemJaziPZcqjHy QaWuNQb4cSjZlbi0nOxWnAvmL29sot4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=sifive.com header.s=google header.b=m6wNPqs9; spf=pass (imf02.hostedemail.com: domain of nylon.chen@sifive.com designates 209.85.219.181 as permitted sender) smtp.mailfrom=nylon.chen@sifive.com; dmarc=pass (policy=reject) header.from=sifive.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705656432; a=rsa-sha256; cv=none; b=qh5x6B0hy9xSSeShCn3AHtBDg7zleHp1Gcb3misSS/EOVcrsediTC3dCRHoaMb/g6U4egv aNBr8xQ+EQZ0YnvieR55ffQ9juOiQ3AsVsDyGwdVasF850kXXiImwLFDrL7pCj1+vmgDTC 913s/cxGD7NAnVUenBxwRt+/KsrHmSg= Received: by mail-yb1-f181.google.com with SMTP id 3f1490d57ef6-dc236729a2bso565914276.0 for ; Fri, 19 Jan 2024 01:27:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1705656431; x=1706261231; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=PX3oyvjTSeaEmMA7BmGcrPiugjRL8juVncN6NIp0Uhw=; b=m6wNPqs96GZqrvRd0t5GkO81aJb//mva4Wd65emBiHuC+VaQQA7BvABimU+G9n2sxR 8lDM9ivp2tEnztlOgM+KLpW18zRDGpep6yBDbbH/B68FxA2SxASbV0DHRtqtac8hYN+y PGeK3yVgSf2XWuvTmXHxg74utrUtjQi2ww1ACxbeDgQadVyQE6Ez0r2CGrgKQXRWx8U9 aVB8HVA/i8Y8arBlLiPjOxkPH9EAYgcm/03o36S3OkAf8DlhTJ3VBzFx1xHs7IHRBVBs 8PS9Y1A38E2mxK0GtqY79AQyf3ysPB7espRTinvV8s2gipbalnUUytUsxmjAqnTEYrLe xrfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705656431; x=1706261231; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PX3oyvjTSeaEmMA7BmGcrPiugjRL8juVncN6NIp0Uhw=; b=Qg+nDFQyo7HRmZB81G33B3OGIR13ttdT1RVvktEO2S4J/6fm0Efv3LKj/HNjcsR1Ye 6hQHTkRVk4cEo8Gy3D1aOT3/Wg9DHAGa5uERsQ1cDHeWq+csq2eZLgYz5oqtUXsIYge4 bXHXHWhy6n1CO927xW4f7yUeDsoasSTfUkJmOh1Sy5vf2o+W8vwCOlcx32KJG/F0GwB7 2+WBI22KfLX360CDSdBJRjCkk7sbv1mMsG+R7VjxsA2x9+3A++fBB1FGaSr2WlH5xhP6 fSxNUBCHe3J/GhqxXXFaq8ngigfIGYKYQ1beP6MSK+MVvaP30Baql8jnRcNb7kLzUaBu ND3g== X-Gm-Message-State: AOJu0YwGjuLjpHJdrsJfNRkGddBRpZ1KImMiK7IanVx5KaBWsElNJQBk Vu0AlzLgwV1Z7xaDmCGgOl1yL4daw4V+FlHoOLIBjgL+Gs2mpNZ9EUM/jY/XBMRBM3pliQsDwKC q++rsk5T9qhBJgCc0P2gbAHKIhuPAQZ/uRF1YWA== X-Google-Smtp-Source: AGHT+IFnB4ZjPfFgAynVnG8PcS3bLndBbDG8SKTtMgiDYr+D8XJYREVe4SCkVUjmR+ve+2tw/4wEl6cj595gux0wucE= X-Received: by 2002:a25:1003:0:b0:dc2:65e7:b616 with SMTP id 3-20020a251003000000b00dc265e7b616mr463094ybq.74.1705656431151; Fri, 19 Jan 2024 01:27:11 -0800 (PST) MIME-Version: 1.0 References: <20230316131711.1284451-1-alexghiti@rivosinc.com> <20240118082346.GB31078@hsinchu15> In-Reply-To: From: Nylon Chen Date: Fri, 19 Jan 2024 17:26:59 +0800 Message-ID: Subject: Re: Fwd: [PATCH v8 0/4] riscv: Use PUD/P4D/PGD pages for the linear mapping To: Alexandre Ghiti Cc: alex@ghiti.fr, apatel@ventanamicro.com, catalin.marinas@arm.com, will@kernel.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, robh+dt@kernel.org, frowand.list@gmail.com, rppt@kernel.org, akpm@linux-foundation.org, anup@brainfault.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, linux-mm@kvack.org, zong.li@sifive.com, nylon7717@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 446648000E X-Rspam-User: X-Stat-Signature: jyj844cqkfgrbauxf57nm95f7ufisngm X-Rspamd-Server: rspam01 X-HE-Tag: 1705656432-153489 X-HE-Meta: U2FsdGVkX19ha4NqBoKyZvFxWPEEEvOBfgRbJX2VDw1zLVnPJ0EeqDQ/+0Qvr8TFEOe0dbhzvu8xBw8iYN76XBDjyANzwBLLL0Cq82O2Uwoh1bbqw0CsaYsZ5+f0cE8Ea/etmt8grtqUYTaGjUk8p5MyUr8z4+6rGpkDVU9KAB2M4Ylx+zAGzbf7Ko0lFpezx56BJk/7B1km40XMdlNVjR51OZ29LFFBWPSw+RqP/cus9ZRRZ+kXKBBTmy5yygE358bdHjQ7oVcp73/L8cSzEQ3q7+KyZbZRayc7rUuPrbTfH6fScLtAtXrog7O+L+cOhMzlMIDzqH/MsvID8e35Pm/iONjLdhh36lBqtlgUxaDge54wnqq/IuhADiHDLeLLqYYBuB2aRCalIqpIxZUyd6FBiIfFZQeXw2whBTDymEPDnSbkKtLTqtZOZGosVt1fwGC11HJLMO1QfDnhTej5R6q9pANEezUyanXiyBgpXjU0hkATY+RNmzqxQ7RYgv9RQR+dZFMDlQ2LJHpEoSsmtK8iotLJMFPrlLMqd1HXAu7CP/Zd/LN14W7D92tHfrhp3fKy1Be1fMjoAhTwZgq3q11AvG9u2wA2Q8F48t0UOhbRjGTjzf8hl2JBNxjuOPv+kITjwbDkR5bLfhFn++qNxwAN+r9/8V9fjhXgag3Wvauiyv9JnuBW6tFvjxPGGa+EVtKrpTMwErVUlvLpEFcg5jygw4lSf1eqHGKJJUx2parT7bSHbD4dXTErxs+j92mt2XuXidowI8dckNgZ4eUc2eKvjt3enjfxNojHl9mCEolR6YK07Ts3BocY5YRv7RNT1qHhnOix65rrikZ7VEJ/n+EeAF8XV9lGz9bXtMxO9kjzxgXeP8cBrzgp2B94hgd/3xJZZ3UB3Vs5g2MDbgr9jEvx72eM1vXyEWhR0qsaGQnl4Zl8OLH/28Y4DKKQ8Xe8I2o1ut1q0TzJA+wadL+ mBc6G2tp qZoxSrmFYbYJPQ47Z0GyEMWgk77446B6+k7voqmQEo8JrP0rENCbrJhkKYRHCbbIaExN4u8cG59RyM/cdINKiFBkUwBPiIVSQQA8YomclCFoEOKj544zJayX2PRPu0YULz5X6G8hMJ9bADA7xHLwYNsHjQtj/xpl7N2UhyI30X0+VC+L9CDGzAgGw+cSM5k8ClyjNCDmIjz9yxMprcft75Bny8Km1H3c+0g8FYeBKvl4U88JcSNpplkBmjc03JPsxpxdh+VgLDctoBT8ZysAi9BWDyAknLwZu22eNFYk5ghUEY/0a5VQawBFM1DCbxjT+7dIgPo7AnGol8Ntfsa+GkrFtuYahc705riNqpRWMzeLoP8FivkmxjL11RQCXFMvmaZzlAR4/sGdofvgpp7sHcYmf2u9GHt8y30Wdk3vjRoZGmfTx+9Virbq1qYp6WG85F+ZbMnqIKmsh8Dhu/hju1v8HN/+dlxIBzuv8bYkBNbWMuEcRHrgPCAI7iw2mJbMFSapbHUkATGgfv1Juxpq9MDdmP9pw1xQz4/OhiZAuoBnXAedVsGVS+cRj4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Alexandre Ghiti =E6=96=BC 2024=E5=B9=B41=E6=9C=881= 8=E6=97=A5 =E9=80=B1=E5=9B=9B =E4=B8=8B=E5=8D=889:01=E5=AF=AB=E9=81=93=EF= =BC=9A > > Hi Nylon, Hi Alexandre, thanks for your feedback, > > On Thu, Jan 18, 2024 at 9:23=E2=80=AFAM Nylon Chen wrote: > > > > > On 3/23/23 15:55, Anup Patel wrote: > > > > On Thu, Mar 23, 2023 at 6:24=E2=80=AFPM Alexandre Ghiti wrote: > > > >> Hi Anup, > > > >> > > > >> On Thu, Mar 23, 2023 at 1:18=E2=80=AFPM Anup Patel wrote: > > > >>> Hi Alex, > > > >>> > > > >>> On Thu, Mar 16, 2023 at 6:48=E2=80=AFPM Alexandre Ghiti wrote: > > > >>>> This patchset intends to improve tlb utilization by using hugepa= ges for > > > >>>> the linear mapping. > > > >>>> > > > >>>> As reported by Anup in v6, when STRICT_KERNEL_RWX is enabled, we= must > > > >>>> take care of isolating the kernel text and rodata so that they a= re not > > > >>>> mapped with a PUD mapping which would then assign wrong permissi= ons to > > > >>>> the whole region: it is achieved by introducing a new memblock A= PI. > > > >>>> > > > >>>> Another patch makes use of this new API in arm64 which used some= sort of > > > >>>> hack to solve this issue: it was built/boot tested successfully. > > > >>>> > > > >>>> base-commit-tag: v6.3-rc1 > > > >>>> > > > >>>> v8: > > > >>>> - Fix rv32, as reported by Anup > > > >>>> - Do not modify memblock_isolate_range and fixes comment, as sug= gested by Mike > > > >>>> - Use the new memblock API for crash kernel too in arm64, as sug= gested by Andrew > > > >>>> - Fix arm64 double mapping (which to me did not work in v7), but= ends up not > > > >>>> being pretty at all, will wait for comments from arm64 review= ers, but > > > >>>> this patch can easily be dropped if they do not want it. > > > >>>> > > > >>>> v7: > > > >>>> - Fix Anup bug report by introducing memblock_isolate_memory whi= ch > > > >>>> allows us to split the memblock mappings and then avoid to ma= p the > > > >>>> the PUD which contains the kernel as read only > > > >>>> - Add a patch to arm64 to use this newly introduced API > > > >>>> > > > >>>> v6: > > > >>>> - quiet LLVM warning by casting phys_ram_base into an unsigned l= ong > > > >>>> > > > >>>> v5: > > > >>>> - Fix nommu builds by getting rid of riscv_pfn_base in patch 1, = thanks > > > >>>> Conor > > > >>>> - Add RB from Andrew > > > >>>> > > > >>>> v4: > > > >>>> - Rebase on top of v6.2-rc3, as noted by Conor > > > >>>> - Add Acked-by Rob > > > >>>> > > > >>>> v3: > > > >>>> - Change the comment about initrd_start VA conversion so that it= fits > > > >>>> ARM64 and RISCV64 (and others in the future if needed), as su= ggested > > > >>>> by Rob > > > >>>> > > > >>>> v2: > > > >>>> - Add a comment on why RISCV64 does not need to set initrd_start= /end that > > > >>>> early in the boot process, as asked by Rob > > > >>>> > > > >>>> Alexandre Ghiti (4): > > > >>>> riscv: Get rid of riscv_pfn_base variable > > > >>>> mm: Introduce memblock_isolate_memory > > > >>>> arm64: Make use of memblock_isolate_memory for the linear map= ping > > > >>>> riscv: Use PUD/P4D/PGD pages for the linear mapping > > > >>> Kernel boot fine on RV64 but there is a failure which is still no= t > > > >>> addressed. You can see this failure as following message in > > > >>> kernel boot log: > > > >>> 0.000000] Failed to add a System RAM resource at 80200000 > > > >> Hmmm I don't get that in any of my test configs, would you mind > > > >> sharing yours and your qemu command line? > > > > Try alexghiti_test branch at > > > > https://github.com/avpatel/linux.git > > > > > > > > I am building the kernel using defconfig and my rootfs is > > > > based on busybox. > > > > > > > > My QEMU command is: > > > > qemu-system-riscv64 -M virt -m 512M -nographic -bios > > > > opensbi/build/platform/generic/firmware/fw_dynamic.bin -kernel > > > > ./build-riscv64/arch/riscv/boot/Image -append "root=3D/dev/ram rw > > > > console=3DttyS0 earlycon" -initrd ./rootfs_riscv64.img -smp 4 > > > > > > > > > So splitting memblock.memory is the culprit, it "confuses" the resour= ces > > > addition and I can only find hacky ways to fix that... > > Hi Alexandre, > > > > We encountered the same error as Anup. After adding your patch > > (3335068f87217ea59d08f462187dc856652eea15), we will not encounter the > > error again. > > > > What I have observed so far is > > > > - before your patch > > When merging consecutive memblocks, if the memblock types are different= , > > they will be merged into reserved > > - after your patch > > When consecutive memblocks are merged, if the memblock types are > > different, they will be merged into memory. > > > > Such a result will cause the memory location of OpenSBI to be changed > > from reserved to memory. Will this have any side effects? > > I guess it will end up in the memory pool and pages from openSBI > region will be allocated, so we should see very quickly bad stuff > happening (either PMP violation or M-mode ecall never > returning/trapping/etc). > > But I don't observe the same thing, I always see the openSBI region > being reserved: > > reserved[0x0] [0x0000000080000000-0x000000008007ffff], > 0x0000000000080000 bytes flags: 0x0 > > Can you elaborate a bit more about "When consecutive memblocks are > merged, if the memblock types are different, they will be merged into > memory"? Where/when does this merge happen? Can you give me a config > file and a kernel revision so that I can take a look? Ok, If you want to reproduce the same results you just need to modify OpenS= BI [ lib/sbi/sbi_domain.c ] +#define TEST_SIZE 0x200000 - (scratch->fw_size - scratch->fw_rw_offset= ), + (TEST_SIZE - scratch->fw_rw_offset), In addition, you can insert checks in the kernel merged function [ mm/memblock.c ] static void __init_memblock memblock_merge_regions(struct memblock_type *ty= pe) while (i < type->cnt - 1) { ... /* move forward from next + 1, index of which is i + 2 */ memmove(next, next + 1, (type->cnt - (i + 2)) * sizeof(*nex= t)); type->cnt--; } + pr_info("Merged memblock_type: cnt =3D %lu, max =3D %lu, total_size =3D 0x%llx\n",type->cnt, type->max, type->total_size); + for (i =3D 0; i < type->cnt; i++) { + const char *region_type =3D memblock_is_memory(type->regions[i].base) ? "memory" : "reserve"; + pr_info("Region %d: base =3D 0x%llx, size =3D 0x%llx, type =3D %s\n", i, type->regions[i].base, type->regions[i].size, region_type); + } } This is kernel boot log - before your patch ... [ 0.000000] OF: fdt: Reserving memory: base =3D 0x80000000, size =3D 0x2= 00000 [ 0.000000] Merged memblock_type: cnt =3D 4, max =3D 128, total_size =3D= 0x1628501 [ 0.000000] Region 0: base =3D 0x80000000, size =3D 0x1600000, type =3D = reserve ... - after your patch ... [ 0.000000] OF: fdt: Reserving memory: base =3D 0x80000000, size =3D 0x2= 00000 [ 0.000000] Merged memblock_type: cnt =3D 4, max =3D 128, total_size =3D= 0x180c42e [ 0.000000] Region 0: base =3D 0x80000000, size =3D 0x1800000, type =3D = memory ... [ 0.000000] Failed to add a system RAM resource at 80200000 ... > > Thanks, > > Alex > > > > > > > So given that the arm64 patch with the new API is not pretty and that > > > the simplest solution is to re-merge the memblock regions afterwards > > > (which is done by memblock_clear_nomap), I'll drop the new API and th= e > > > arm64 patch to use the nomap API like arm64: I'll take advantage of t= hat > > > to clean setup_vm_final which I have wanted to do for a long time. > > > > > > @Mike Thanks for you reviews! > > > > > > @Anup Thanks for all your bug reports on this patchset, I have to > > > improve my test flow (it is in the work :)). > > > > > > > > > > Regards, > > > > Anup > > > > > > > >> Thanks > > > >> > > > >>> Regards, > > > >>> Anup > > > >>> > > > >>>> arch/arm64/mm/mmu.c | 25 +++++++++++------ > > > >>>> arch/riscv/include/asm/page.h | 19 +++++++++++-- > > > >>>> arch/riscv/mm/init.c | 53 +++++++++++++++++++++++++++= +------- > > > >>>> arch/riscv/mm/physaddr.c | 16 +++++++++++ > > > >>>> drivers/of/fdt.c | 11 ++++---- > > > >>>> include/linux/memblock.h | 1 + > > > >>>> mm/memblock.c | 20 +++++++++++++ > > > >>>> 7 files changed, 119 insertions(+), 26 deletions(-) > > > >>>> > > > >>>> -- > > > >>>> 2.37.2 > > > >>>> > > > > _______________________________________________ > > > > linux-riscv mailing list > > > > linux-riscv@lists.infradead.org > > > > http://lists.infradead.org/mailman/listinfo/linux-riscv > > > > > > _______________________________________________ > > > linux-riscv mailing list > > > linux-riscv@lists.infradead.org > > > http://lists.infradead.org/mailman/listinfo/linux-riscv