From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D118C77B75 for ; Fri, 21 Apr 2023 04:22:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ABE34900003; Fri, 21 Apr 2023 00:22:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A6E02900002; Fri, 21 Apr 2023 00:22:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9358F900003; Fri, 21 Apr 2023 00:22:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 83B70900002 for ; Fri, 21 Apr 2023 00:22:49 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 374D98088F for ; Fri, 21 Apr 2023 04:22:49 +0000 (UTC) X-FDA: 80704102458.05.6034092 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf06.hostedemail.com (Postfix) with ESMTP id ACE5E18000E for ; Fri, 21 Apr 2023 04:22:45 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Sur9A/RX"; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682050966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8Qj8FSeZQgYCmGrz07ib0K0ctgvWVfY735gobePNH08=; b=66bImJlCpcQKNAw3YOon+f1kjTyUHcDgNeL0DuNjwj+nFPYoEK4SaPgXoRZeLvtQQ0ORrM pTaI2OX7uCt8dRLM/ga9Lihm4KGp9qip820YcW4K+JUVYJU7iFhfGefNCJ1wxmBN2Z3Cl3 X14K08dBYRowxeGhBEqeNBVSBo93trU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="Sur9A/RX"; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682050966; a=rsa-sha256; cv=none; b=NRhSVFnPGIqGMeRY6VBTWjPVCDYL09mxbIsKvUW959G4kuoXMMa4GfU9eSMB+uFO8K3N0E tKDYuMGo+CGM5EVf5I44PzofgXl+PqT+3aUNCcwyQACr29mwvOJUvcirIcleywzla1QHYi 2itYUbGbm1RzQDz7jZOsEx2Xg6RFZ+c= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682050965; x=1713586965; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=H9iJKcVIMytaSDCm2vBtv7QGVs8Ip4teKb26JClgYJM=; b=Sur9A/RXHvqJim5KN2dchUR9r2N6ZATPuRlZttqH3iuuF2o6dUTuTSpa mTSWxzP7ZmcPFjqHqF/+GsiDkm2yRFLFLl9xvSusJmGNA8n2tCWNcC/7q BibvYk/yrh6xI3i6buICWLUgYjUHWrDY3TU6e2suvdD0AqYb36PHf/Gzp DG7FW25eWNUofwbRPvaVzQ+Iofzzv2UeyEkPUYT10BNI180wfuUsTSZ0K o/hLLIK/rpU1fcHeVEsA9hlIYvBwLRNUNkDlHJGaFPdX5Tv8MdOziCnsb Y3uOPgW+EiBlgDnkXIqA5GyBVSS0PVLaIw/gb1wA4mCl1pckxMXxa+CNC Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10686"; a="408849272" X-IronPort-AV: E=Sophos;i="5.99,214,1677571200"; d="scan'208";a="408849272" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 21:22:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10686"; a="1021794073" X-IronPort-AV: E=Sophos;i="5.99,214,1677571200"; d="scan'208";a="1021794073" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Apr 2023 21:22:37 -0700 From: "Huang, Ying" To: Baolin Wang Cc: David Hildenbrand , , , , , , Subject: Re: [PATCH] mm/page_alloc: consider pfn holes after pfn_valid() in __pageblock_pfn_to_page() References: <62e231a8f2e50c04dcadc7a0cfaa6dea5ce1ec05.1681296022.git.baolin.wang@linux.alibaba.com> <94bfa3cc-674e-25b0-e7e2-d74c970acef7@redhat.com> <87cz3zt3u6.fsf@yhuang6-desk2.ccr.corp.intel.com> <52dfdd2e-9c99-eac4-233e-59919a24323e@linux.alibaba.com> Date: Fri, 21 Apr 2023 12:21:28 +0800 In-Reply-To: <52dfdd2e-9c99-eac4-233e-59919a24323e@linux.alibaba.com> (Baolin Wang's message of "Thu, 20 Apr 2023 17:11:45 +0800") Message-ID: <874jp9uapj.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: arqu4p4ko95tryjwjryqz9o6937jedas X-Rspamd-Queue-Id: ACE5E18000E X-HE-Tag: 1682050965-354046 X-HE-Meta: U2FsdGVkX18KPLNolEdMmrJFpUnW83euAGUOX6MOc1WeAM55kt/qn55O1XDS1yOr0a+tmGZ+aT7Vi3UTZwNEhnJo9ChiX4e4hLA8UZzaHnn6v3Zcr6QCpjN047GqOcHLN/ILo4jOGSUmgVEngGR5aEDk4OIHxmANP10ppTpm9agN4eq6lAfTnZwTn6+L2QfidvpTKZbYHmyFufJ9/GmCLLf4sUNPV3uQ4bz52IsiPOr/oaiAoZGnsqrLvmQ3h+Snnn1sRCTtMLc+vxs18f0qu6q+4+3c6ip7gGt6d3rid/THE8aqljL96ZbCVHwk5ki2wU0b3EzSyd/NGQhyg2DQ8l5fXPGDSgHocRpWarSg0rEYn6V+SVxtsQ6BUHp90vGbwT4SN0rIIjVIE33aIrDSSQ1uIYldm+QK0KlLJVCTRSVyrejVN/SKqjdfbGDXKh2JHdG2twaHRftaGOh8i5pNIGcjCxVPDOmLw1LWg3eHf2EL9oiWIlRSzKaPa/wDtFnuy2rfOz3PgzqY0ouKv/teNI7oPQp0Sv6OavWmDPhui9PT71fphwwedAtYtK5r9GjgASvNOBSI9ifa9SESVZ+8Vh3rnhwEsmMXKsGtuz9mtc2UYZr/OEYfLQ5QCdwJ0eAphU6nMJ2wkq1WILpkpp5A6G47b0P8hQ5e10AEZG+ZR2tfzzR0+x3DtUeKUKPbaABg8OFBgYMGreEC3dgOIQKH9/TXVnKEfbsgQj7GMWCKmmXmqwkKwkKRdyPfm8v5Nloq6D5Wjgk0+vGT0EnLqaOPv7ZW178dDM+K8Kpa3zZFeL9vJ9wf1ttP8aHuYV1oRd0t4Rjz22ISZBRAqTnqSkFzkWGrqXfYP8421zO3+xbZfvhEcVJibZ7amCl2tdnXh1zqi7fJgJra67L6ZDJKNBynizqXv73OCkhZpBg7MbGDotTZyhTNh4N7CBzNC3UQuJbzCQn6b7iv/LjixSPR8gb 0pLobAx0 HDzttpGUA1UasUHaggjFxaSoBnyD6GPgfSMoy5WezFNv4fWhhQfVgfS54bVk9l0nu5t8xK+j8vCD2iSlxbwA8sd4/6pGvyiygAFTlJHlhRI4aSwGzGmi2gi93ohIxIiP8PijslAbYKO/0Dp5JVrImA1T4VNSCf8pJmhiuJJbjW3hU7ku+HfiJ8iLC4f8IjHZgpJr5875Q/jWvdj2eO6r//iAfqeEKq3J+tasK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Baolin Wang writes: > On 4/20/2023 3:22 PM, Huang, Ying wrote: >> Baolin Wang writes: >> >>> On 4/12/2023 7:25 PM, David Hildenbrand wrote: >>>> On 12.04.23 12:45, Baolin Wang wrote: >>>>> Now the __pageblock_pfn_to_page() is used by set_zone_contiguous(), >>>>> which checks whether the given zone contains holes, and uses pfn_valid() >>>>> to check if the end pfn is valid. However pfn_valid() can not make sure >>>>> the end pfn is not a hole if the size of a pageblock is larger than the >>>>> size of a sub-mem_section, since the struct page getting by pfn_to_page() >>>>> may represent a hole or an unusable page frame, which may cause incorrect >>>>> zone contiguous is set. >>>>> >>>>> Though another user of pageblock_pfn_to_page() in compaction seems work >>>>> well now, it is better to avoid scanning or touching these offline pfns. >>>>> So like commit 2d070eab2e82 ("mm: consider zone which is not fully >>>>> populated to have holes"), we should also use pfn_to_online_page() for >>>>> the end pfn to make sure it is a valid pfn with usable page frame. >>>>> Meanwhile the pfn_valid() for end pfn can be dropped now. >>>>> >>>>> Moreover we've already used pfn_to_online_page() for start pfn to make >>>>> sure it is online and valid, so the pfn_valid() for the start pfn is >>>>> unnecessary, drop it. >>>> pageblocks are supposed to fall into a single memory section, so in >>>> mos > cases, if the start is online, so is the end. >>> >>> Yes, the granularity of memory hotplug is a mem_section. >>> >>> However, suppose the pageblock order is MAX_ORDER-1, and the size of a >>> sub-section is 2M, that means a pageblock will fall into 2 sub >>> mem-section, and if there is a hole in the zone, that means the 2nd >>> sub mem-section can be invalid without setting subsection_map bitmap. >>> >>> So the start is online can make sure the end pfn of a pageblock is >>> online, but a valid start pfn can not make sure the end pfn is valid >>> in the bitmap of ms->usage->subsection_map. >> arch_add_memory >> add_pages >> __add_pages >> sparse_add_section /* set subsection_map */ >> arch_add_memory() is only called by add_memory_resource() and >> pagemap_range() (called add_pages() too). In add_memory_resource(), >> check_hotplug_memory_range() will enforce a strict hotplug range >> alignment requirement (128 MB on x86_64). pagemap_range() are used for >> ZONE_DEVICE only. That is, for normal memory, hotplug granularity is >> much larger than 2MB. >> IIUC, the situation you mentioned above is impossible. Or do I miss >> something? > > Thanks for your input. Your example is correct, but this is not the > case I want to describe. My case is not about the memory hotplug, > instead about the early memory holes when initialzing the memory. Let > me try to describe explicity: > > First suppose the pageblock order is MAX_ORDER-1, and see below memory > layout as an example: > > [ 0.000000] Zone ranges: > [ 0.000000] DMA [mem 0x0000000040000000-0x00000000ffffffff] > [ 0.000000] DMA32 empty > [ 0.000000] Normal [mem 0x0000000100000000-0x0000001fa7ffffff] > [ 0.000000] Movable zone start for each node > [ 0.000000] Early memory node ranges > [ 0.000000] node 0: [mem 0x0000000040000000-0x0000001fa3c7ffff] > [ 0.000000] node 0: [mem 0x0000001fa3c80000-0x0000001fa3ffffff] > [ 0.000000] node 0: [mem 0x0000001fa4000000-0x0000001fa402ffff] > [ 0.000000] node 0: [mem 0x0000001fa4030000-0x0000001fa40effff] > [ 0.000000] node 0: [mem 0x0000001fa40f0000-0x0000001fa73cffff] > [ 0.000000] node 0: [mem 0x0000001fa73d0000-0x0000001fa745ffff] > [ 0.000000] node 0: [mem 0x0000001fa7460000-0x0000001fa746ffff] > [ 0.000000] node 0: [mem 0x0000001fa7470000-0x0000001fa758ffff] > [ 0.000000] node 0: [mem 0x0000001fa7590000-0x0000001fa7dfffff] > > Focus on the last memory range, and there is a hole for the range [mem > 0x0000001fa7590000-0x0000001fa7dfffff]. That means the last pageblock > will contain the range from 0x1fa7c00000 to 0x1fa7ffffff, since the > pageblock must be 4M aligned. And in this page block, these pfns will > fall into 2 sub-section (the sub-section size is 2M aligned). > > So, the 1st sub-section (indicates pfn range: 0x1fa7c00000 - > 0x1fa7dfffff ) in this pageblock is valid by > free_area_init()--->subsection_map_init(), but the 2nd sub-section > (indicates pfn range: 0x1fa7e00000 - 0x1fa7ffffff ) in this pageblock > is not valid. > > The problem is, if we just check the pageblock start of the hole pfn > (such as 0x1fa7dfffff) to make sure the hole pfn (0x1fa7dfffff) is > also valid, which is NOT correct. So that is what I mean "the start is > online can make sure the end pfn of a pageblock is online, but a valid > start pfn can not make sure the end pfn is valid in the bitmap of > ms->usage->subsection_map." > > Hope I make it clear. Does that make sense to you? Thanks. Thanks for your detailed description. You are right, it's possible that the second subsection of a pageblock is a hole. It's good to remove unnecessary pfn_valid(start_pfn) check in your original patch. But it appears unnecessary to replace pfn_valid(end_pfn) with pfn_to_online_page(end_pfn). Yes, it's possible that there's a hole in a page block. But it appears that this will not break anything. Per my understanding, even if we had fixed this one, there may be other smaller memory holes in a pageblock represented as reserved pages. Best Regards, Huang, Ying [snip]