From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3BF9CA9EB7 for ; Mon, 21 Oct 2019 14:16:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C2F9206C2 for ; Mon, 21 Oct 2019 14:16:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lca.pw header.i=@lca.pw header.b="kCntxG+F" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C2F9206C2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=lca.pw Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1BDC76B0006; Mon, 21 Oct 2019 10:16:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 148EB6B0007; Mon, 21 Oct 2019 10:16:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29BB6B0008; Mon, 21 Oct 2019 10:16:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0079.hostedemail.com [216.40.44.79]) by kanga.kvack.org (Postfix) with ESMTP id CAF876B0006 for ; Mon, 21 Oct 2019 10:16:00 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id 5E6018249980 for ; Mon, 21 Oct 2019 14:16:00 +0000 (UTC) X-FDA: 76067990880.04.chair19_341fd8246084f X-HE-Tag: chair19_341fd8246084f X-Filterd-Recvd-Size: 10952 Received: from mail-qt1-f193.google.com (mail-qt1-f193.google.com [209.85.160.193]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Mon, 21 Oct 2019 14:15:58 +0000 (UTC) Received: by mail-qt1-f193.google.com with SMTP id u22so21187889qtq.13 for ; Mon, 21 Oct 2019 07:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=lca.pw; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=SR4XTutzmthc5d7HZlzvb9fPYpgDWiu5nZW7fLVerx8=; b=kCntxG+F1zGFLNKi29nV21oIr+oWcnNbXh3PUmsWE3G90Dm4KkfQgx0khoIx807KF/ Fo1lr/49orQnWqqI6pseXkRQc0bPFDWZ4Yn+lX0tanl/fsf2aafO+az+wHWPEGEQcreU 2jcEJTTZ93QcKji+Li9QyTYXDDmqIaVZJnOo+mss3C7COKX9swvfSC1+h5uEyv+bEs8X mzsdRItRKrn4AXS/tfSqpQmM9EA9WwUIVFffPwRHxfv8Ie6G5IWxpCNIQdCoEK0tTg9V 0CvjLyXqcJIrNKdc3CDyG4fpx7bjYKzknXwgdyjwPaqpQz+K8iJcsEP7e0bNtwRXWPxF 9PnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=SR4XTutzmthc5d7HZlzvb9fPYpgDWiu5nZW7fLVerx8=; b=VSIXm35zYZFztUYAkk2yzhbJVQxX2suh0VUC7NgvUVLveOo0k9lnJpvE0wuC9t1zHN tbioQhlNmHkYgp5xtekL6+QD9fI90yZak7S5/Olp+AdS+PwxK2vgfQHYr+Ja9sG0a3wM CYa7G9gbL7EJh/VziZYXglLrCqW95iYQeOHJ6sLxAtBf3Nh01NnWp0sLVhbFCoHc0dQq 27gyRSplxcQDZ/LYSKxh/gA75pDARtFsln85qhGWVVaLR91YZmMcbsaRFgCJmevzhbgx e91odzHsmeiGXLDA14P4DigKiiG9C0Z0coQWfYrMRdBVWHSclvKn3lFGTDRuS6N03hMs RiKA== X-Gm-Message-State: APjAAAWblR9WbKeL4qO/l2TAXN4iRrcvme1C8F9MxOIPhtQ+hIClEcSa o7Z8k6p+G7B5fkOwiJDIMt++Bw== X-Google-Smtp-Source: APXvYqx0A0iC7aDo3GjKGWsrUQLEpvuI2joDEdM2ghjsafGIxZjhcGWwTQP5TnIKoz1WBOGKbTUiiA== X-Received: by 2002:a0c:c10d:: with SMTP id f13mr23889157qvh.88.1571667357222; Mon, 21 Oct 2019 07:15:57 -0700 (PDT) Received: from [192.168.1.153] (pool-71-184-117-43.bstnma.fios.verizon.net. [71.184.117.43]) by smtp.gmail.com with ESMTPSA id h184sm8536827qkf.89.2019.10.21.07.15.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 07:15:56 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: [PATCH v6 3/4] arm64: use both ZONE_DMA and ZONE_DMA32 From: Qian Cai In-Reply-To: <20190911182546.17094-4-nsaenzjulienne@suse.de> Date: Mon, 21 Oct 2019 10:15:55 -0400 Cc: catalin.marinas@arm.com, Christoph Hellwig , wahrenst@gmx.net, marc.zyngier@arm.com, Rob Herring , f.fainelli@gmail.com, will@kernel.org, linux-mm@kvack.org, robin.murphy@arm.com, linux-kernel@vger.kernel.org, mbrugger@suse.com, linux-rpi-kernel@lists.infradead.org, phill@raspberrypi.org, linux-arm-kernel@lists.infradead.org, m.szyprowski@samsung.com Content-Transfer-Encoding: quoted-printable Message-Id: <3956E54B-6C7A-4C47-B9B6-75F556EFD3F5@lca.pw> References: <20190911182546.17094-1-nsaenzjulienne@suse.de> <20190911182546.17094-4-nsaenzjulienne@suse.de> To: Nicolas Saenz Julienne X-Mailer: Apple Mail (2.3445.104.11) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Sep 11, 2019, at 2:25 PM, Nicolas Saenz Julienne = wrote: >=20 > So far all arm64 devices have supported 32 bit DMA masks for their > peripherals. This is not true anymore for the Raspberry Pi 4 as most = of > it's peripherals can only address the first GB of memory on a total of > up to 4 GB. >=20 > This goes against ZONE_DMA32's intent, as it's expected for ZONE_DMA32 > to be addressable with a 32 bit mask. So it was decided to = re-introduce > ZONE_DMA in arm64. >=20 > ZONE_DMA will contain the lower 1G of memory, which is currently the > memory area addressable by any peripheral on an arm64 device. > ZONE_DMA32 will contain the rest of the 32 bit addressable memory. >=20 > Signed-off-by: Nicolas Saenz Julienne > Reviewed-by: Catalin Marinas >=20 > --- >=20 > I kept the Reviewed-by as the last bug solution was proposed by = Catalin >=20 > Changes in v6: > - Fixed bug in max_zone_phys >=20 > Changes in v5: > - Fixed swiotlb initialization >=20 > Changes in v4: > - Fixed issue when NUMA=3Dn and ZONE_DMA=3Dn > - Merged two max_zone_dma*_phys() functions >=20 > Changes in v3: > - Used fixed size ZONE_DMA > - Fix check befor swiotlb_init() >=20 > Changes in v2: > - Update comment to reflect new zones split > - ZONE_DMA will never be left empty >=20 > arch/arm64/Kconfig | 4 +++ > arch/arm64/include/asm/page.h | 2 ++ > arch/arm64/mm/init.c | 54 +++++++++++++++++++++++++---------- > 3 files changed, 45 insertions(+), 15 deletions(-) >=20 > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index 6b6362b83004..2dbe0165bd15 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -267,6 +267,10 @@ config GENERIC_CSUM > config GENERIC_CALIBRATE_DELAY > def_bool y >=20 > +config ZONE_DMA > + bool "Support DMA zone" if EXPERT > + default y > + > config ZONE_DMA32 > bool "Support DMA32 zone" if EXPERT > default y > diff --git a/arch/arm64/include/asm/page.h = b/arch/arm64/include/asm/page.h > index d39ddb258a04..7b8c98830101 100644 > --- a/arch/arm64/include/asm/page.h > +++ b/arch/arm64/include/asm/page.h > @@ -38,4 +38,6 @@ extern int pfn_valid(unsigned long); >=20 > #include >=20 > +#define ARCH_ZONE_DMA_BITS 30 > + > #endif > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c > index 8e9bc64c5878..44f07fdf7a59 100644 > --- a/arch/arm64/mm/init.c > +++ b/arch/arm64/mm/init.c > @@ -56,6 +56,13 @@ EXPORT_SYMBOL(physvirt_offset); > struct page *vmemmap __ro_after_init; > EXPORT_SYMBOL(vmemmap); >=20 > +/* > + * We create both ZONE_DMA and ZONE_DMA32. ZONE_DMA covers the first = 1G of > + * memory as some devices, namely the Raspberry Pi 4, have = peripherals with > + * this limited view of the memory. ZONE_DMA32 will cover the rest of = the 32 > + * bit addressable memory area. > + */ > +phys_addr_t arm64_dma_phys_limit __ro_after_init; > phys_addr_t arm64_dma32_phys_limit __ro_after_init; >=20 > #ifdef CONFIG_KEXEC_CORE > @@ -169,15 +176,16 @@ static void __init reserve_elfcorehdr(void) > { > } > #endif /* CONFIG_CRASH_DUMP */ > + > /* > - * Return the maximum physical address for ZONE_DMA32 = (DMA_BIT_MASK(32)). It > - * currently assumes that for memory starting above 4G, 32-bit = devices will > - * use a DMA offset. > + * Return the maximum physical address for a zone with a given = address size > + * limit. It currently assumes that for memory starting above 4G, = 32-bit > + * devices will use a DMA offset. > */ > -static phys_addr_t __init max_zone_dma32_phys(void) > +static phys_addr_t __init max_zone_phys(unsigned int zone_bits) > { > - phys_addr_t offset =3D memblock_start_of_DRAM() & = GENMASK_ULL(63, 32); > - return min(offset + (1ULL << 32), memblock_end_of_DRAM()); > + phys_addr_t offset =3D memblock_start_of_DRAM() & = GENMASK_ULL(63, zone_bits); > + return min(offset + (1ULL << zone_bits), = memblock_end_of_DRAM()); > } >=20 > #ifdef CONFIG_NUMA > @@ -186,6 +194,9 @@ static void __init zone_sizes_init(unsigned long = min, unsigned long max) > { > unsigned long max_zone_pfns[MAX_NR_ZONES] =3D {0}; >=20 > +#ifdef CONFIG_ZONE_DMA > + max_zone_pfns[ZONE_DMA] =3D PFN_DOWN(arm64_dma_phys_limit); > +#endif > #ifdef CONFIG_ZONE_DMA32 > max_zone_pfns[ZONE_DMA32] =3D PFN_DOWN(arm64_dma32_phys_limit); > #endif > @@ -201,13 +212,18 @@ static void __init zone_sizes_init(unsigned long = min, unsigned long max) > struct memblock_region *reg; > unsigned long zone_size[MAX_NR_ZONES], zhole_size[MAX_NR_ZONES]; > unsigned long max_dma32 =3D min; > + unsigned long max_dma =3D min; >=20 > memset(zone_size, 0, sizeof(zone_size)); >=20 > - /* 4GB maximum for 32-bit only capable devices */ > +#ifdef CONFIG_ZONE_DMA > + max_dma =3D PFN_DOWN(arm64_dma_phys_limit); > + zone_size[ZONE_DMA] =3D max_dma - min; > + max_dma32 =3D max_dma; > +#endif > #ifdef CONFIG_ZONE_DMA32 > max_dma32 =3D PFN_DOWN(arm64_dma32_phys_limit); > - zone_size[ZONE_DMA32] =3D max_dma32 - min; > + zone_size[ZONE_DMA32] =3D max_dma32 - max_dma; > #endif > zone_size[ZONE_NORMAL] =3D max - max_dma32; >=20 > @@ -219,11 +235,17 @@ static void __init zone_sizes_init(unsigned long = min, unsigned long max) >=20 > if (start >=3D max) > continue; > - > +#ifdef CONFIG_ZONE_DMA > + if (start < max_dma) { > + unsigned long dma_end =3D min_not_zero(end, = max_dma); > + zhole_size[ZONE_DMA] -=3D dma_end - start; > + } > +#endif > #ifdef CONFIG_ZONE_DMA32 > if (start < max_dma32) { > - unsigned long dma_end =3D min(end, max_dma32); > - zhole_size[ZONE_DMA32] -=3D dma_end - start; > + unsigned long dma32_end =3D min(end, max_dma32); > + unsigned long dma32_start =3D max(start, = max_dma); > + zhole_size[ZONE_DMA32] -=3D dma32_end - = dma32_start; > } > #endif > if (end > max_dma32) { > @@ -418,9 +440,11 @@ void __init arm64_memblock_init(void) >=20 > early_init_fdt_scan_reserved_mem(); >=20 > - /* 4GB maximum for 32-bit only capable devices */ > + if (IS_ENABLED(CONFIG_ZONE_DMA)) > + arm64_dma_phys_limit =3D = max_zone_phys(ARCH_ZONE_DMA_BITS); > + > if (IS_ENABLED(CONFIG_ZONE_DMA32)) > - arm64_dma32_phys_limit =3D max_zone_dma32_phys(); > + arm64_dma32_phys_limit =3D max_zone_phys(32); > else > arm64_dma32_phys_limit =3D PHYS_MASK + 1; >=20 > @@ -430,7 +454,7 @@ void __init arm64_memblock_init(void) >=20 > high_memory =3D __va(memblock_end_of_DRAM() - 1) + 1; >=20 > - dma_contiguous_reserve(arm64_dma32_phys_limit); > + dma_contiguous_reserve(arm64_dma_phys_limit ? : = arm64_dma32_phys_limit); > } >=20 > void __init bootmem_init(void) > @@ -534,7 +558,7 @@ static void __init free_unused_memmap(void) > void __init mem_init(void) > { > if (swiotlb_force =3D=3D SWIOTLB_FORCE || > - max_pfn > (arm64_dma32_phys_limit >> PAGE_SHIFT)) > + max_pfn > PFN_DOWN(arm64_dma_phys_limit ? : = arm64_dma32_phys_limit)) > swiotlb_init(1); > else > swiotlb_force =3D SWIOTLB_NO_FORCE; > --=20 > 2.23.0 >=20 >=20 With ZONE_DMA=3Dy, this config will fail to reserve 512M CMA on a = server, https://raw.githubusercontent.com/cailca/linux-mm/master/arm64.config CONFIG_DMA_CMA=3Dy CONFIG_CMA_SIZE_MBYTES=3D64 CONFIG_CMA_SIZE_SEL_MBYTES=3Dy CONFIG_CMA_ALIGNMENT=3D8 CONFIG_CMA=3Dy CONFIG_CMA_DEBUGFS=3Dy CONFIG_CMA_AREAS=3D7 Is this expected?=