From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D38FC54EBD for ; Mon, 9 Jan 2023 13:33:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4846900003; Mon, 9 Jan 2023 08:33:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F7528E0001; Mon, 9 Jan 2023 08:33:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BE50900003; Mon, 9 Jan 2023 08:33:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 809518E0001 for ; Mon, 9 Jan 2023 08:33:35 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5C54F40A31 for ; Mon, 9 Jan 2023 13:33:35 +0000 (UTC) X-FDA: 80335352790.29.1957242 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf13.hostedemail.com (Postfix) with ESMTP id 69AD620012 for ; Mon, 9 Jan 2023 13:33:33 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=iWEnEjc1; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673271213; a=rsa-sha256; cv=none; b=O8hcWLvtSCVmXnK3TEh/ZRZNusYUTMxJdklGQPDaY0Uv4FjzZJqnJRhl4rC/RxoFEhPecd Zc7cU0ptjbV8q61qq6jYgXYYGUttsv9ipj0hpguxb+ttzBukTiRPGb1SOJT4EGfJUgETdp h1bEw/+Iq5zw6eWNhiovlWtN8l5k1u8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=iWEnEjc1; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673271213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j+1tNsr5M28f7D0N4IYIB7Row4Ut41WVuSdmguXmCpc=; b=e0raWHK0AVuJRfVqpc/GgrBlmdZNfyhUi8HXhOJ7KdIzCAIspjq4AiJClR0ZfymDdylFuD TQfxUkseoYQCfEtmZuAYxKoleUhNltAML3HEJpbAX7NqsLGnbqaumbS2c/zDk9yC7iE7k6 8ZFocLgUBVByKOyIpYLvty1dBU3E4G4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id AE7A9B80DD4; Mon, 9 Jan 2023 13:33:31 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BBDA9C433EF; Mon, 9 Jan 2023 13:33:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1673271210; bh=0Ilxi1rMtgAs305FaWmpiguznC+wkJwOtG9vo1iisvo=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=iWEnEjc1iN3SVDaTHmDISxS07zcZdzOyHfMFbHBIkG28rGbuAKo7QUuGT3/uOM3Gk z8c6mBdGha3cufG4JciS8Dg3dwEmEb+3QoUVh6VmtQoo7N85CeXO2nBerrGGs4uul8 X3lWIjeC2w8PLvXkyDD4NEVPn73yjwsdraGBzIvMPjyxn/BRqZHku6kaosRvdA1NEa YliCXTwh74+axsNgb8Xo08PUlmy/OWdaT9Urik5Nig6wNPuyECA55GCNRf78UraCJg +BirGO5zsowURkAi3tHC3ekgykhb9PyStZx2stU8MjSTfovHmAj5ap4jERKeWqxu2n tJ18hSkGkVnoA== Date: Mon, 9 Jan 2023 15:33:15 +0200 From: Mike Rapoport To: Lorenzo Stoakes Cc: Jonathan Corbet , Andrew Morton , David Hildenbrand , Johannes Weiner , "Matthew Wilcox (Oracle)" , Mel Gorman , Michal Hocko , Vlastimil Babka , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 2/2] docs/mm: Physical Memory: add structure, introduction and nodes description Message-ID: References: <20230101094523.1522109-1-rppt@kernel.org> <20230101094523.1522109-3-rppt@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 69AD620012 X-Stat-Signature: deu9a5cf6oemt8hjsxi5qgdibhfw9gkz X-HE-Tag: 1673271213-915686 X-HE-Meta: U2FsdGVkX1+hqQvguN6WOehtJtR7fd+nX4qa0KqZCclB5aqplRPMkym9JUDFGN6EICUgozEiCBCt1ETd5SMXIjHzAK3kqfWQ7VE9RuAYMzHIw41m8/C2VWL+Kp8awFPgQOwkx1kYeCgJVYhh/end9XNRVi1yNUtJWltXb77900BeOeFUNsa3MdvlXurN075jriIDSKHVU4ypNtctwMn2UOHOsfV2ArSaOSqB/cXseWmdttdK7IhQ0IOB1DxVxaGJKQBn4RVb1A3EfQ9yP7wVAHY8pT+DYX1Ffng/tdSF+CEGLuAujsbGEL4rAc0ZQLEfeSv6lXasKZd3J7DRyTSKLLSGGmpy3hR+3lTvq33eomNwyeDVBO06VoBPVZMIA3dDjbkqxImwMDHbW9acdQ862Zy54eoN2GNFsQWEnpsq7KvhwM4XrfFkgWR4rd2ib8ykLKzjykermhsNKuWzOKtg+EiJKEetTjdEurAELRyqsOOA7YlFUrlDLTmsPAbBgjAXjIPuu1VRJCXiosTNuUt00rygh1GGYEFLVG3a37DgCjj5Hjjc5e8oIEn0xRVpWCEdrHsYx6j5Nellhz6NAxRvFQgnn7HsSm9+bTrYo/Q//nt81PzKMsIGni+vmp524EW8j+AoM0Pms5ioxKpSif66zbX77EfyZpJxEJRT1cJAAm6CkEOJrIyQnfApS86sb1O1gkjuiZrt/2KYIzHtRV+hbINh6UYmD7oxXKO4lK5Bc1GX1Pl7lrVWwPkX+0UCvIp30ys69H7Jb+5I79tISQhHpGB39LQs9AvrR7JhcLyyagizP+4RL/VemjxeZeZdKZhin86yjFFaFkxv4pTgbjRjyLDkA8dP0YF4/PkqkEzBmewwH1RZvf3eFJGHFbJbWcSvWpTwZBSU3ZNAF1CHoUZGcaMTr924HYv/jY2vMuPcxT2WIcpoysSb3CwlqRZO+cUI2NlM3voXPVWV09E+pMy kksDG7Ez lODkxxxDYSq6Vf/fX/tAlVKskOyr85mz7sBLf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jan 06, 2023 at 10:32:46PM +0000, Lorenzo Stoakes wrote: > On Sun, Jan 01, 2023 at 11:45:23AM +0200, Mike Rapoport wrote: > > From: "Mike Rapoport (IBM)" > > > > Signed-off-by: Mike Rapoport (IBM) > > --- > > Documentation/mm/physical_memory.rst | 322 +++++++++++++++++++++++++++ > > 1 file changed, 322 insertions(+) > > > > diff --git a/Documentation/mm/physical_memory.rst b/Documentation/mm/physical_memory.rst > > index 2ab7b8c1c863..fcf52f1db16b 100644 > > --- a/Documentation/mm/physical_memory.rst > > +++ b/Documentation/mm/physical_memory.rst > > @@ -3,3 +3,325 @@ > > =============== > > Physical Memory > > =============== > > + > > +Linux is available for a wide range of architectures so there is a need for an > > +architecture-independent abstraction to represent the physical memory. This > > +chapter describes the structures used to manage physical memory in a running > > +system. > > + > > +The first principal concept prevalent in the memory management is > > +`Non-Uniform Memory Access (NUMA) > > +`_. > > +With multi-core and multi-socket machines, memory may be arranged into banks > > +that incur a different cost to access depending on the “distance” from the > > +processor. For example, there might be a bank of memory assigned to each CPU or > > +a bank of memory very suitable for DMA near peripheral devices. > > Absolutely wonderfully written. Thanks to Mel :) > Perhaps put a sub-heading for NUMA here? I consider all this text as an high level overview and I'd prefer to keep it as a single piece. > An aside, but I think it'd be a good idea to mention base pages, folios and > folio order pretty early on as they get touched as concepts all over the place > in physical memory (but perhaps can wait for other contribs!) The plan is to have "Pages" section Really Soon :) > > + > > +Each bank is called a node and the concept is represented under Linux by a > > +``struct pglist_data`` even if the architecture is UMA. This structure is > > +always referenced to by it's typedef ``pg_data_t``. A pg_data_t structure > > +for a particular node can be referenced by ``NODE_DATA(nid)`` macro where > > +``nid`` is the ID of that node. > > + > > +For NUMA architectures, the node structures are allocated by the architecture > > +specific code early during boot. Usually, these structures are allocated > > +locally on the memory bank they represent. For UMA architectures, only one > > +static pg_data_t structure called ``contig_page_data`` is used. Nodes will > > +be discussed further in Section :ref:`Nodes ` > > + > > +Each node may be divided up into a number of blocks called zones which > > +represent ranges within memory. These ranges are usually determined by > > +architectural constraints for accessing the physical memory. A zone is > > +described by a ``struct zone_struct``, typedeffed to ``zone_t`` and each zone > > +has one of the types described below. > > I don't think it's quite right to say 'may' be divided up into zones, as they > absolutely will be so (and the entire phsyical memory allocator hinges on being > zoned, even if trivially in UMA/single zone cases). Not necessarily. ZONE_DMA or ZONE_NORMAL may span the entire memory. > Also it's struct zone right, not zone_struct/zone_t? Right, thanks. > I think it's important to clarify that a given zone does not map to a single > struct zone, rather that a struct zone (contained within a pg_data_t object's > array node_zones[]) represents only the portion of the zone that resides in this > node. > > It's fiddly because when I talk about a zone like this I am referring to one of > the 'classifications' of zones you mention below, e.g. ZONE_DMA, ZONE_DMA32, > etc. but you might also want to refer to a zone as being equivalent to a struct > zone object. > > I think the clearest thing however is to use the term zone to refer to each of > the ZONE_xxx types, e.g. 'this memory is located in ZONE_NORMAL' and to clarify > that one zone can span different individual struct zones (and thus nodes). > > I know it's tricky because you and others have rightly pointed out that my own > explanation of this is confusing, and it is something I intend to rejig a bit > myself! The term 'zone' is indeed somewhat ambiguous, I'll try to come up with more clear version. > > + > > +`ZONE_DMA` and `ZONE_DMA32` > > + represent memory suitable for DMA by peripheral devices that cannot > > + access all of the addressable memory. Depending on the architecture, > > + either of these zone types or even they both can be disabled at build > > + time using ``CONFIG_ZONE_DMA`` and ``CONFIG_ZONE_DMA32`` configuration > > + options. Some 64-bit platforms may need both zones as they support > > + peripherals with different DMA addressing limitations. > > It might be worth pointing out ZONE_DMA spans an incredibly little range that > probably won't matter for any peripherals this side of the cretaceous period, On RPi4 ZONE_DMA spans 1G, which is quite some part of the memory ;-) > > + > > +`ZONE_NORMAL` > > + is for normal memory that can be accessed by the kernel all the time. DMA > > + operations can be performed on pages in this zone if the DMA devices support > > + transfers to all addressable memory. ZONE_NORMAL is always enabled. > > + > > Might be worth saying 'this is where memory ends up if not otherwise in another > zone'. This may not be the case on !x86. > > +`ZONE_HIGHMEM` > > + is the part of the physical memory that is not covered by a permanent mapping > > + in the kernel page tables. The memory in this zone is only accessible to the > > + kernel using temporary mappings. This zone is available only some 32-bit > > + architectures and is enabled with ``CONFIG_HIGHMEM``. > > + > > I comment here only to say 'wow I am so glad I chose to only focus on 64-bit so > I could side-step all the awkward discussion of high pages' :) > > > +The relation between node and zone extents is determined by the physical memory > > +map reported by the firmware, architectural constraints for memory addressing > > +and certain parameters in the kernel command line. > > Perhaps worth mentioning device tree here? Though perhaps encapsulated in the > 'firmware' reference. It is :) > > +Node structure > > +-------------- > > + > > +The struct pglist_data is declared in `include/linux/mmzone.h > > +`_. > > +Here we briefly describe fields of this structure: > > Perhaps worth saying 'The node structure' just to reiterate. Not sure I follow, can you phrase the entire sentence? > > + > > +General > > +~~~~~~~ > > + > > +`node_zones` > > + The zones for this node. Not all of the zones may be populated, but it is > > + the full list. It is referenced by this node's node_zonelists as well as > > + other node's node_zonelists. > > Perhaps worth describing what zonelists (and equally zonerefs) are here or > above, and that this is the canonical place where zones reside. Maybe reference > populated_zone() and for_each_populated_zone() in reference to the fact that not > all here may be populated? I'd prefer to start simple and than add more content on top. > > + > > +`node_zonelists` The list of all zones in all nodes. This list defines the > > + order of zones that allocations are preferred from. The `node_zonelists` is > > + set up by build_zonelists() in mm/page_alloc.c during the initialization of > > + core memory management structures. > > + > > +`nr_zones` > > + Number of populated zones in this node. > > + > > +`node_mem_map` > > + For UMA systems that use FLATMEM memory model the 0's node (and the only) > > + `node_mem_map` is array of struct pages representing each physical frame. > > + > > +`node_page_ext` > > + For UMA systems that use FLATMEM memory model the 0's (and the only) node > > + `node_mem_map` is array of extensions of struct pages. Available only in the > > + kernels built with ``CONFIG_PAGE_EXTENTION`` enabled. > > + > > +`node_start_pfn` > > + The page frame number of the starting page frame in this node. > > + > > +`node_present_pages` > > + Total number of physical pages present in this node. > > + > > +`node_spanned_pages` > > + Total size of physical page range, including holes. > > + > > I think it'd be useful to discuss briefly the meaning of managed, spanned and > present pages in the context of zones. This will be a part of the Zones section. > Cheers, Lorenzo -- Sincerely yours, Mike.