From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FDB6C02187 for ; Mon, 20 Jan 2025 00:38:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 346296B0085; Sun, 19 Jan 2025 19:38:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F6A16B0088; Sun, 19 Jan 2025 19:38:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E6156B0089; Sun, 19 Jan 2025 19:38:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F2A216B0085 for ; Sun, 19 Jan 2025 19:38:16 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7AD4B160DDA for ; Mon, 20 Jan 2025 00:38:16 +0000 (UTC) X-FDA: 83025968592.16.91F28B8 Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) by imf09.hostedemail.com (Postfix) with ESMTP id 9E01B140004 for ; Mon, 20 Jan 2025 00:38:14 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=C11oGJ0r; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737333494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8winaI/8gSI85nYGrjAnA40gamWekRdjzIs0phxqs24=; b=4ZYFRs5YuOd8HyjyjkaHNwWJs2n7RefAdcKxGFM7Q+YSlG/z6dtz1oZCSFca+SHTutQvwR s+KMJw6Z547gdhDdDAgUIhDMKNwhErQD53vx7p3/EJcGe3GupA4lmWyV6pBPp3pYxT1hE1 gM5yNeY0TXF3i6azrCgSzI+E3OgKkoQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737333494; a=rsa-sha256; cv=none; b=vx2wHBIS9yzoiRbdR5Xgmt8nwShouyKr8TyebLZlg9fUl/ppQCt6nEmM4PXuF2U3cApZER P8iF2V+q96ejLGkLMkUGMqfIrjfa4DzCISmTJ+SalqIfxzUow/IqxAG1OZ2Dsat0i2RtKV FnAkOp0zjYGRsEDpQfG+oet5ow4Xeqg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=C11oGJ0r; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-vs1-f49.google.com with SMTP id ada2fe7eead31-4b10dd44c8bso1137104137.3 for ; Sun, 19 Jan 2025 16:38:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1737333494; x=1737938294; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8winaI/8gSI85nYGrjAnA40gamWekRdjzIs0phxqs24=; b=C11oGJ0rMcoaK3LqnGQV4BkRabXZ9F63NfuuiZeTlsZpzrEnlebNSPBBm2snsgwiiI Z+SNtfai7IK61kMVieSC6T9zBcLKkUvb/XCGkS4FAXnwJY+/Czdf+zkYdU9KI1HbHP7D lycqPp1wOYxiBOVBo/vUJHQ2KJKxTzHqlgmPBNBIH0+cnldVnJ/UhuwsV99vspeyGTwE bZFwnJOXbW0Q7lTLJu8TMbfhz6VN1udPoIcBWP7KcxBDZHfo1sYoPDEg8Mierv5SCLEt pMmRAcvU6nITju5OSG+xy0tuvnHwti0yZXj9Lx9/oeHwepUaO4KnT4pPFAscG1FyM+hZ uIaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737333494; x=1737938294; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8winaI/8gSI85nYGrjAnA40gamWekRdjzIs0phxqs24=; b=ckM0MUyXk5i+dAVqNxMUkNNoTerUcH4ihOFEIiW7gddd6zBx8XG4PSJY3vdTt1Xf3C +eWxzAoA8N3O1pHakLivzuIbLyv+vS+KwuTOH0acYgsDR/FURH2RC+Gh7JA6juFMiS3Y wPtguz/QZDZYcsUXD3PpSdO5haeP9KOQqfxk4RDgZbEfKBKQzt3+C1A4Dt8PS9WCk0zE m/y2B1Dq/9STeDKnHzusxk4B8dAJnfp8aZ6LXhzLIRfwQGfaVbITWj0dMFUI3CjOjDSu iSP5B/fHkSVgxBGCY7U7gBd4zLIFsMjjq6pMZXXmvRGUB4yzKqPEdDP72nGDFohAZsyQ O1lg== X-Forwarded-Encrypted: i=1; AJvYcCXGjsVzZttg9EDsHenwGTa9vKSamiAFvqt7XaNookbXdfE7RXzBHCLPWthy5TqOyJu2Y7t1pmLEbA==@kvack.org X-Gm-Message-State: AOJu0YzMBqaAsCNEQ9bfAopPrhbEzZuXf7tjbpi8WVl6DQnHLNOEarcO QQrLspcH+myuHUvLQtQBIG541yYJ51Y+qqc1ulnkUSmqHFIIJy6htxMpIt0GMawG37t0Ac1isWF hB/HEKjEp1wKSCoAsNaifftmcWgvw9aNrEEQ= X-Gm-Gg: ASbGnctPrUbk/VYVtClaE4xrs0nsPTPX5yTtMMfuX8hDdXg5WE2vM/CTUcv+ceDbJbN b4A5Sfw0ubnPmAK4OqbH+42cBTyAdOuaJXCdJ/1RTL+h6Q4ORFoA7gRQp4KHK8Ieixbg91ZwM1Q j/dGqr+0b6kQ== X-Google-Smtp-Source: AGHT+IGOJ/IjtxUi8Ng8beZLX3zAULMYAxFF0iJEG/Ikx1fyXId6Gw1t0GvdlNm7bs+lWX0sPgR7NQiJHuSZ5KdTprA= X-Received: by 2002:a05:6102:3f45:b0:4b1:1565:f4f1 with SMTP id ada2fe7eead31-4b690b6518emr9703266137.3.1737333493703; Sun, 19 Jan 2025 16:38:13 -0800 (PST) MIME-Version: 1.0 References: <52839BED-606F-4BE0-AFC3-5632299C0070@nvidia.com> In-Reply-To: <52839BED-606F-4BE0-AFC3-5632299C0070@nvidia.com> From: Barry Song <21cnbao@gmail.com> Date: Mon, 20 Jan 2025 13:38:02 +1300 X-Gm-Features: AbW1kvZMhqUF79Rg-eiVO-3Wu4cePPuzDTVLsgViu-Pvsb1OLPLDR_ZbGJxQWGM Message-ID: Subject: Re: mm: CMA reservations require 32MiB alignment in 16KiB page size kernels instead of 8MiB in 4KiB page size kernel. To: Zi Yan Cc: Juan Yescas , linux-mm@kvack.org, muchun.song@linux.dev, rppt@kernel.org, david@redhat.com, osalvador@suse.de, akpm@linux-foundation.org, lorenzo.stoakes@oracle.com, Jann Horn , Liam.Howlett@oracle.com, minchan@kernel.org, jaewon31.kim@samsung.com, charante@codeaurora.org, Suren Baghdasaryan , Kalesh Singh , "T.J. Mercier" , Isaac Manjarres , iamjoonsoo.kim@lge.com, quic_charante@quicinc.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: xdyzzohp768itkwt71rf6caho9wwjm5q X-Rspam-User: X-Rspamd-Queue-Id: 9E01B140004 X-Rspamd-Server: rspam03 X-HE-Tag: 1737333494-63303 X-HE-Meta: U2FsdGVkX19yNoUxFab9ySdhgVV0sX08xJgO/84tOBGzlWGFr6T3P8xHiIiv3MzyvpGv26Yt7EIHYsi7igsWP189vg1wpQqTnVDDRFobO6ZAYcuhH9Bf1bxp4fy/UhpxVotoshMRr3CwcXIoQraE0xUDyK73xhr5urR1l+vBQ5CoSDbYP7St2q0W7+zFTtCaH9hSxP+L9vOB/IXkD4HEAHDmarkgSe+xvTKUIxHWDwcoZ/tU91DTg0+QP8URSS5/kGcDIuSYbCVsjrgTzr6IveZOGkCE8JdcBCwV4XBwtQLy7k/HxqZSsutT4ZDTcK9MhTiHPVeHq+TgAoA6eSOcEYQv7bJWM3mJW+77ygmwibXZqiqy/MJSdjJLmLDwC4pramc5dRbnSsOYB+IUzpOO0lpr5gFKuv6h6UlKz/Z3JZGdr6c2Ctt1edgNQ17o/A5RSATQi72BxSAUjE3zlJb3YeV2lU2eODy4ya3WWZnfpmg+HKX9ZjSUZCKHN/TvIqn7JPHD/3x1clrtrTbvVg0wKwo7rZiNVvbN/8BB+8c3hQ8n2ZPXpPysdv1LpMrNsITxfCig43PMrvdFbtk3CtrEwCef2j3NNf/UYIoWcGz+DHhacJXFvp4UanXy3yGx1Xy23XYucCwq9ErbcfQ13uSgY8PoKAo83wJOeFirNV4gps0ZdMdvxwMwE/BDz3X0DfXKPUI11qEuiYCFOWNaPq2RL7XvvpwzFOiMX8tGHk1NKiUjKPbE+Ad0WQvZBqwkJTAmP/PJPbnRQ7cbmjACiKUWZ83FwUcqxcQIKFZ8iL+MuZ2Jc7f32IV8kqDVPaSt9dlqMqJ0AzwfEngDs189vjtuNinlBKs/4VfXPFLzLAbYLhOq+L0uz7+xanv5CiWI05qAfFu4JIWm+aobchQg58YTpyZhaoxtXbfN7xXss2aVPk8LjSrijMj91M+3rSIJ+pL3xwfJb8DcCkribjgOrxc 7lz6lj77 +ARRJj9i8/pwLHOx32xlZwq9CvwWtGiexNlOecYTgs2x4jhkaT8j1cy4q8IIQtpDy5V6+OKOXVuuGDMhkAsX4iyxBjaFGpZ3eLuqbs+Ys3FZW+4a3DuynYo0u2peRQ14VIwJSPJEX6oCoBCQpqK8ja/l/H2ueSOsbndZxWJP6l1OJvXUplVnnuKG+PI+zHTjfiLtSR1e8xpqywFFpryM3sEncn4ci02vVyIj+AvJ+3pP1FWlZ4R9m5CHlGRf2AFm6uiWo4sk9WEYdk+S+Htw/4SuBOBR/HMVOdXx6I274VK3fiPRiUC8sQvagjjH5GJYHXwWfTLRDbgs1sEHrpm1HPf4kNghFHAixteI35OkmaVHZlndX35z8w5DQQeUkF+KnkqiZbAubD4VQ4w5mSSz29cNYFdZw9CEv/cq+3HY14qkLsjpu17soIROy9cat/12jtjXm X-Bogosity: Ham, tests=bogofilter, spamicity=0.001609, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jan 20, 2025 at 1:26=E2=80=AFPM Zi Yan wrote: > > On 19 Jan 2025, at 19:17, Barry Song wrote: > > > On Sat, Jan 18, 2025 at 12:00=E2=80=AFPM Juan Yescas wrote: > >> > >> + iamjoonsoo.kim@lge.com > >> + quic_charante@quicinc.com > >> > >> On Fri, Jan 17, 2025 at 2:52=E2=80=AFPM Juan Yescas wrote: > >>> > >>> +Suren Baghdasaryan > >>> +Kalesh Singh > >>> +T.J. Mercier > >>> +Isaac Manjarres > >>> > >>> On Fri, Jan 17, 2025 at 2:51=E2=80=AFPM Juan Yescas wrote: > >>>> > >>>> Hi Linux memory team > >>>> > >>>> When the drivers reserve CMA memory in 16KiB kernels, the minimum > >>>> alignment is 32 MiB as per CMA_MIN_ALIGNMENT_BYTES. However, in 4KiB > >>>> kernels, the CMA alignment is 4MiB. > >>>> > >>>> This is forcing the drivers to reserve more memory in 16KiB kernels, > >>>> even if they only require 4MiB or 8MiB. > >>>> > >>>> reserved-memory { > >>>> #address-cells =3D <2>; > >>>> #size-cells =3D <2>; > >>>> ranges; > >>>> tpu_cma_reserve: tpu_cma_reserve { > >>>> compatible =3D "shared-dma-pool"; > >>>> reusable; > >>>> size =3D <0x0 0x2000000>; /* 32 MiB */ > >>>> } > >>>> > >>>> One workaround to continue using 4MiB alignment is: > >>>> > >>>> - Disable CONFIG_TRANSPARENT_HUGEPAGE so the buddy allocator does NO= T > >>>> have to allocate huge pages (32 MiB in 16KiB page sizes) > >>>> - Set ARCH_FORCE_MAX_ORDER for ARM64_16K_PAGES to "8", instead of > >>>> "11", so CMA_MIN_ALIGNMENT_BYTES is equals to 4 MiB > >>>> > >>>> config ARCH_FORCE_MAX_ORDER > >>>> int > >>>> default "13" if ARM64_64K_PAGES > >>>> default "8" if ARM64_16K_PAGES > >>>> default "10" > >>>> > >>>> #define MAX_PAGE_ORDER CONFIG_ARCH_FORCE_MAX_ORDER // 8 > >>>> #define pageblock_order MAX_PAGE_ORDER // 8 > >>>> #define pageblock_nr_pages (1UL << pageblock_order) // 256 > >>>> #define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages // 256 > >>>> #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES= ) > >>>> // 16384 * 256 =3D 4194304 =3D 4 MiB > >>>> > >>>> After compiling the kernel with this changes, the kernel boots witho= ut > >>>> warnings and the memory is reserved: > >>>> > >>>> [ 0.000000] Reserved memory: created CMA memory pool at > >>>> 0x000000007f800000, size 8 MiB > >>>> [ 0.000000] OF: reserved mem: initialized node tpu_cma_reserve, > >>>> compatible id shared-dma-pool > >>>> [ 0.000000] OF: reserved mem: > >>>> 0x000000007f800000..0x000000007fffffff (8192 KiB) map reusable > >>>> tpu_cma_reserve > >>>> > >>>> # uname -a > >>>> Linux buildroot 6.12.9-dirty > >>>> # zcat /proc/config.gz | grep ARM64_16K > >>>> CONFIG_ARM64_16K_PAGES=3Dy > >>>> # zcat /proc/config.gz | grep TRANSPARENT_HUGE > >>>> CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=3Dy > >>>> # CONFIG_TRANSPARENT_HUGEPAGE is not set > >>>> # cat /proc/pagetypeinfo > >>>> Page block order: 8 > >>>> Pages per block: 256 > >>>> > >>>> Free pages count per migrate type at order 0 1 2 > >>>> 3 4 5 6 7 8 > >>>> Node 0, zone DMA, type Unmovable 1 1 13 > >>>> 6 5 2 0 0 1 > >>>> Node 0, zone DMA, type Movable 9 16 19 > >>>> 13 13 5 2 0 182 > >>>> Node 0, zone DMA, type Reclaimable 0 1 0 > >>>> 1 1 0 0 1 0 > >>>> Node 0, zone DMA, type HighAtomic 0 0 0 > >>>> 0 0 0 0 0 0 > >>>> Node 0, zone DMA, type CMA 1 0 0 > >>>> 0 0 0 0 0 49 > >>>> Node 0, zone DMA, type Isolate 0 0 0 > >>>> 0 0 0 0 0 0 > >>>> Number of blocks type Unmovable Movable Reclaimable > >>>> HighAtomic CMA Isolate > >>>> Node 0, zone DMA 6 199 1 > >>>> 0 50 0 > >>>> > >>>> > >>>> However, with this workaround, we can't use transparent huge pages. > > > > I don=E2=80=99t think this is accurate. You can still use mTHP with a s= ize > > equal to or smaller than 4MiB, > > right? > > > > By the way, what specific regression have you observed when reserving > > a larger size like > > 32MB? > > For CMA, the over-reserved memory is still available to the system for > > movable folios. 28MiB > > The fallbacks table does not have MIGRATE_CMA as a fallback for any > migratetype. How can it be used for movable folios? Am I missing somethin= g? The whole purpose of CMA is to allow the memory reserved for a device's dma_alloc_coherent or other contiguous memory needs to be freely used by movable allocations when the device doesn't require it. When the device's DMA needs the memory, the movable folios can be migrated to make it available for the device. /* Must be called after current_gfp_context() which can change gfp_mask */ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, unsigned int alloc_flags) { #ifdef CONFIG_CMA if (gfp_migratetype(gfp_mask) =3D=3D MIGRATE_MOVABLE) alloc_flags |=3D ALLOC_CMA; #endif return alloc_flags; } So there=E2=80=99s no waste here. cma can be used by normal buddy. > > > doesn=E2=80=99t seem significant enough to cause a noticeable regressio= n, does it? > > -- > Best Regards, > Yan, Zi Thanks Barry