From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7535C61DA4 for ; Tue, 14 Feb 2023 10:33:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EA0C6B0075; Tue, 14 Feb 2023 05:33:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 473986B0078; Tue, 14 Feb 2023 05:33:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ECBA6B007B; Tue, 14 Feb 2023 05:33:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1A32F6B0075 for ; Tue, 14 Feb 2023 05:33:22 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id BD1FE140DC1 for ; Tue, 14 Feb 2023 10:33:21 +0000 (UTC) X-FDA: 80465535402.08.EBCA61F Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf30.hostedemail.com (Postfix) with ESMTP id 00A6380021 for ; Tue, 14 Feb 2023 10:33:18 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=1iLeX6Z5; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676370799; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0Wj9/hXlrWGWDF+Thm8m1kU7n9xJTrILBDDgYL6C5Gk=; b=o5I7alVcCR+mfFstu0X73faGxk4+FkGzUvxAuThzs45KrzfdEc/6r40N+oonQGQ94scPBu bOwctAP2axr0ZLAx1O8xxYLipGjoysklO0VwiJzURn+R77I+mKeE2Vm15ONQ1tBrrCADx4 IB7j9SKH8BTq2g/5fCaxsMW5R/xonLw= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=1iLeX6Z5; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676370799; a=rsa-sha256; cv=none; b=HXKN/ItOnycTfEkRlWgdZfKLHRY06B6OKOv5JZaU9dQ47qpT+W+UmGqlmubpKyW3tqIiQe /t84dGQvYl1Avq+hgJagYQ6LX1WmW+JDQywzSYxQ5XCh3oxtvOj7YvDWySqq35plLZFz6E LZeGmKSUNaECNzz/i6g45Z1l2KXZD5s= Received: by mail-pj1-f46.google.com with SMTP id mg23so8111954pjb.0 for ; Tue, 14 Feb 2023 02:33:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; t=1676370797; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=0Wj9/hXlrWGWDF+Thm8m1kU7n9xJTrILBDDgYL6C5Gk=; b=1iLeX6Z5JlY1Z2cv15w0DRaIv/H28PpgSE04jrxfWnCOXPXIWgyNfB9NpmpY749eWh QO7wHnI+RDcoqW/+DNfKxsSuNB26POEijSbrR3t/SQJq+CF8J2cdHX6kYorrFNC6r7S5 tH/k4tyx2xbJ4iw49IO2N5lyOUBnd5JfxbYWmSkKR4kbPOu/3QH+pQLCc6IkXT9w4JN9 2kIdGskjbKSLJ6lu9eoa00qgwHUSCHtqpmzmnjEx+1EZnk9HYgSFShAlPGgY1FjV5L+y MbZgI58nKIbS3tv9k1vNRATp9T5PdXqnWxOaPdJiIpFhQVfU/yqYlsK6xiQoIIcs7Hsx UkFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1676370797; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0Wj9/hXlrWGWDF+Thm8m1kU7n9xJTrILBDDgYL6C5Gk=; b=uxFxiJm9oC2nJFe03BtShr72xOmp4Z6jAtK9sd9E8rzTQ7RgdEn1PbfqeEeHj6baSb B2/usaNHpEtIw2w/HKZ6XPrAebck4RK/Rs8fHpuJakLiy+3xsi89Xr41ufOenuJwxrTg haikk7jwWO6QsNXyNKS7U4rClBeThztVasp7pJFcTGcSUyedqgXlY+vGDr7lzanjMkbq QOgxY1VNeaDrGvdlZN8ecfl2QH2YfJnKeRpzVxM3GvlLyXHNgfitP1Aa5efggM2x1Voj 7chjDKLmhOveHGp/CfOuzILVkd0OdH4+Dp+JoH9YNuyQrWLZKvpzgM/7SlGeBqgKCx3v Zdeg== X-Gm-Message-State: AO0yUKW3eF0YVaCZEHvpvZYCoQ8Lm7urdINKT2HO4yB5AUgWkuiNDP6N CMiwkSfhAprxZbb1lwwhtphNxg== X-Google-Smtp-Source: AK7set+V+Y3xne1KgmezUg3Mf/Wh/rtyAH1xliVaYolYlfqKFN1pEqhpyKXovnTXS6IePQzxRbhHdw== X-Received: by 2002:a05:6a21:6d97:b0:bc:6c4f:308a with SMTP id wl23-20020a056a216d9700b000bc6c4f308amr2507380pzb.0.1676370797654; Tue, 14 Feb 2023 02:33:17 -0800 (PST) Received: from [10.200.11.190] ([139.177.225.228]) by smtp.gmail.com with ESMTPSA id i196-20020a636dcd000000b004fb171df68fsm2166259pgc.7.2023.02.14.02.33.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 14 Feb 2023 02:33:17 -0800 (PST) Message-ID: <4a9fba3c-3924-9d5f-7b42-522a183d1f2e@bytedance.com> Date: Tue, 14 Feb 2023 18:33:11 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.7.2 Subject: Re: [PATCH] mm: page_alloc: don't allocate page from memoryless nodes Content-Language: en-US To: Mike Rapoport , Vlastimil Babka Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Teng Hu , Matthew Wilcox , Mel Gorman , Oscar Salvador References: <20230212110305.93670-1-zhengqi.arch@bytedance.com> <2484666e-e78e-549d-e075-b2c39d460d71@suse.cz> From: Qi Zheng In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 00A6380021 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 9tyqdtd3g9xpijz5yewo1ewrfwd4dq1y X-HE-Tag: 1676370798-640837 X-HE-Meta: U2FsdGVkX1/a1JJNtsOxwM9D0Cc0pv783U3rFrfs610cEeMTECKgYTvO0bdV0UQ7quVP3l8yx8CjjW4PXYJLjT7MFOo7YlW5zP5LnK8K+Wh2bX3l5Zj58kIT3FyOygzAGiR/BbEvhiat6+vCGR1R+DofBL3C7kBYutGmoyY14rvJguYCERlXWvYnyQASZVpStj5R5+H8GOE5uDDHLp0TLeYELDQlmbac9GOvuZVpbB9SLRuWdCQgf4Z/Y3SSRdGlU3R8doTmETW/OhGmdu1+V3EKffqhXjC+GLOyflr+BhVRPpr5PHSc6tUKAd/PtxP92DP1zU+MX0bwYDuvzp/nXgbDxGK5DVfieGvAXLBeI3sU/wAvra1qmGfis3kanJPsqEsbHXgShoge5GNhZoWD80VvRww+8exfNhndRjRGLpxReSyK+F9n3syL+B+34nmJDv+7+BbITU5a1ECKme1vWkhbB59Pd4nFIpJS7n2sF3FcCZyudfbAK6fuGSVcmOLrGef/cRFpZiOijI36TPCQsFzLlDxus9OaQqLHE0bXbxCDkrUZO1XYA/7IFINqZHbk+Y2GkcXOG+zcKpreJWKGvy6hFRea4XrQ/+H3pGaciFB7E1Ec3HIggbBFpqKrV7WJBrvL5uib6MBJdtf32cRPPcEr6oLsVuvEL2io3QYoiB+8R79UOFvBBagiUsBwkyytZUpTp9NWpkdPepZj8eZGnBv/TBesKOSKFBC07Xu7ELNiK6ngnQEnSiN7wn2xrB9j0ttTyhnEbTeToZd/kvvmc0UV/vbPyVwUaDwqyJ013KPV9Y/WfLVTZwBOAR+nbAya98k0wad0Jb28p0qV2xstl5x1ommyMspB4qBZ5ykqi/PbT8+PiesbfYjkMvTII7nM7vwfDfnrEG6cmYKcavtqHUuezc9xpbRvFZCikGtBsHEWVQAqcGl7pDNDyvlJSKSXTE/je7pKwWEzKRVmnKv +MGe4MH0 Zgveohsm5GmumhnEWd3xCSY5DAheqAvBCbOdphDywmHzkLOboqT5yW8fhXJ1cO5eD6GavvRli1qGAch3EmWa4jpaSPIznKQKoNIRRR/Ya8Iygw9GKxx5iXlSig/mdgEr0SdODvtOz42/XTu9dlPNke8OYyHHV/LfpDUSjvLJCPjkSNfJBv90jlJeLLF9EPu7cQ1ruFJJr1fXziba0uxvA9mbiQGPwmQFPikH5jkEOZKsHdYVqFL/T5BKwMmUB0JHsqfp9qRKRvw5ksdq5cU9wQrUmLMPCoU5c7B2e1JJ6LHnT0CoHdtzsSHEwlLkQvAlZAojeDP/gC7g882YXcTYb16OK6v6Vb+agD6CcaNRT5ZFukVsmrgTyuE2lJOmyI4Gkxq7sXDa95DGMQ1ISsBpdH832dYK1vr7LwrFiVnoZGRRZsTvaMNYk6KFeCGnEWWR+Jeq+srurx9zx8GflnTVlreU80Bz12wEoMWHo1mGSTB625AL1kahYeOaqVJ5DPinJb1NY2lPsiU+3xTmx8vTfzaMnBvfaryTTVKB/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2023/2/14 17:10, Mike Rapoport wrote: > On Mon, Feb 13, 2023 at 09:47:43AM +0100, Vlastimil Babka wrote: >> On 2/12/23 12:03, Qi Zheng wrote: >>> In x86, numa_register_memblks() is only interested in >>> those nodes which have enough memory, so it skips over >>> all nodes with memory below NODE_MIN_SIZE (treated as >>> a memoryless node). Later on, we will initialize these >>> memoryless nodes (allocate pgdat in free_area_init() >>> and build zonelist etc), and will online these nodes >>> in init_cpu_to_node() and init_gi_nodes(). >>> >>> After boot, these memoryless nodes are in N_ONLINE >>> state but not in N_MEMORY state. But we can still allocate >>> pages from these memoryless nodes. >>> >>> In SLUB, we only process nodes in the N_MEMORY state, >>> such as allocating their struct kmem_cache_node. So if >>> we allocate a page from the memoryless node above to >>> SLUB, the struct kmem_cache_node of the node corresponding >>> to this page is NULL, which will cause panic. >>> >>> For example, if we use qemu to start a two numa node kernel, >>> one of the nodes has 2M memory (less than NODE_MIN_SIZE), >>> and the other node has 2G, then we will encounter the >>> following panic: >>> >>> [ 0.149844] BUG: kernel NULL pointer dereference, address: 0000000000000000 >>> [ 0.150783] #PF: supervisor write access in kernel mode >>> [ 0.151488] #PF: error_code(0x0002) - not-present page >>> <...> >>> [ 0.156056] RIP: 0010:_raw_spin_lock_irqsave+0x22/0x40 >>> <...> >>> [ 0.169781] Call Trace: >>> [ 0.170159] >>> [ 0.170448] deactivate_slab+0x187/0x3c0 >>> [ 0.171031] ? bootstrap+0x1b/0x10e >>> [ 0.171559] ? preempt_count_sub+0x9/0xa0 >>> [ 0.172145] ? kmem_cache_alloc+0x12c/0x440 >>> [ 0.172735] ? bootstrap+0x1b/0x10e >>> [ 0.173236] bootstrap+0x6b/0x10e >>> [ 0.173720] kmem_cache_init+0x10a/0x188 >>> [ 0.174240] start_kernel+0x415/0x6ac >>> [ 0.174738] secondary_startup_64_no_verify+0xe0/0xeb >>> [ 0.175417] >>> [ 0.175713] Modules linked in: >>> [ 0.176117] CR2: 0000000000000000 >>> >>> In addition, we can also encountered this panic in the actual >>> production environment. We set up a 2c2g container with two >>> numa nodes, and then reserved 128M for kdump, and then we >>> can encountered the above panic in the kdump kernel. >>> >>> To fix it, we can filter memoryless nodes when allocating >>> pages. >>> >>> Signed-off-by: Qi Zheng >>> Reported-by: Teng Hu >> >> Well AFAIK the key mechanism to only allocate from "good" nodes is the >> zonelist, we shouldn't need to start putting extra checks like this. So it >> seems to me that the code building the zonelists should take the >> NODE_MIN_SIZE constraint in mind. > > Why just not drop the memory for nodes with size < NODE_MIN_SIZE from > memblock at the first place? In this way, it seems that no pages of size < NODE_MIN_SIZE nodes will be released to buddy, so the pages of these nodes will not be allocated, and the above-mentioned panic will be avoided. But these nodes will still build zonelists for itself, which seems unnecessary? > Then we won't need runtime checks at all. > >>> --- >>> mm/page_alloc.c | 5 +++++ >>> 1 file changed, 5 insertions(+) >>> >>> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >>> index 588555754601..b9cce56f4e21 100644 >>> --- a/mm/page_alloc.c >>> +++ b/mm/page_alloc.c >>> @@ -4188,6 +4188,11 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, >>> (alloc_flags & ALLOC_CPUSET) && >>> !__cpuset_zone_allowed(zone, gfp_mask)) >>> continue; >>> + >>> + /* Don't allocate page from memoryless nodes. */ >>> + if (!node_state((zone_to_nid(zone)), N_MEMORY)) >>> + continue; >>> + >>> /* >>> * When allocating a page cache page for writing, we >>> * want to get it from a node that is within its dirty >> > -- Thanks, Qi