From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FC4CDB47E for ; Wed, 18 Oct 2023 20:55:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7099A80024; Wed, 18 Oct 2023 16:55:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B98F8D0016; Wed, 18 Oct 2023 16:55:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 55A4180024; Wed, 18 Oct 2023 16:55:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 42E6D8D0016 for ; Wed, 18 Oct 2023 16:55:10 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 186521CBDAD for ; Wed, 18 Oct 2023 20:55:10 +0000 (UTC) X-FDA: 81359787180.15.3851754 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by imf21.hostedemail.com (Postfix) with ESMTP id 424501C0015 for ; Wed, 18 Oct 2023 20:55:08 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LcYfYJCY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of ndesaulniers@google.com designates 209.85.221.46 as permitted sender) smtp.mailfrom=ndesaulniers@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697662508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aM7Lcg58eaICcI7n9hqCMKVzpyeJKHwBAKrGqarbnys=; b=ygwMYgWXd0MX1HDBHy5LoeZFx2UhVoZflIG+/dH+XAiFNOxHgfFTKJJfCJk2A1B0JyZEFM 9Ac/2IxwJOwM8CSC6itjaFniSMoQG5YNQf8O5v+8Z+FBVStpJN4Y87nEyox/Frjkoxsise WRENjxlLliH6igPeznvYgkF6VCZ0EFM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LcYfYJCY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of ndesaulniers@google.com designates 209.85.221.46 as permitted sender) smtp.mailfrom=ndesaulniers@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697662508; a=rsa-sha256; cv=none; b=ACSP3D99lZn8MvMfuFM4P8sDattCrnjEW2+QtF3JdfdSPgcToIv4h+Z0ZsjbAbnaX7dy9e a8alIS6y/cDHVY1S9O/eQF1jjsjZI0+HjFiOdPmbibTL0BVIeLny3Lw6ORS7GeY3lx/oHK 84Zzx6sQ+lAmV7ImhNvoUz2u2m3e1vk= Received: by mail-wr1-f46.google.com with SMTP id ffacd0b85a97d-32db8924201so2744110f8f.1 for ; Wed, 18 Oct 2023 13:55:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697662507; x=1698267307; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=aM7Lcg58eaICcI7n9hqCMKVzpyeJKHwBAKrGqarbnys=; b=LcYfYJCYKeLQtQ41iR8aQi1lHKFDR49pfFOhyEuvtLL4VRr4XOXrJtnxNe8iyogPOY kwHIUoIH/LVH9E/VeB565X9pily5EbIEWvR3KKL9hO3zy+LwP0flzRolrRGtc5AXlG7M rf0HfRED9Z3Sdd+uaEcnA2TFf+ETyBCaYlBvEBuh9kCt1DMGdtOIix5n7a4MUrmfRJye 5/nUYpYGtWdxxmmKFyknBWP89g0IH7uPIWN7HK8bf5/XEZAqG8DFh5306HIGszzb6dbS YPtXVWlqEGp0ihkfmRE+Ll01zHcSKAq3jcW90NOpTxtLbrhNUVpc/r3kNesu7A7kj0NZ m52Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697662507; x=1698267307; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aM7Lcg58eaICcI7n9hqCMKVzpyeJKHwBAKrGqarbnys=; b=FtCdo2E7xaJk/16W+FfZzLQPJf507rTxu4Un06s+cdWVcLyXxcQWR7tT5VPumq6oTA VxNY2YgA29k7lFKaDbQLQLb57SfwJXK5klwby6l2h1J1VhZ638NlOHF+Heyw6feabUyl uPfEXL0IphAOknnBh2AaNcvH9OJ5xjVQ2kjHtxZ3dXIsJ5K9I5fSbjgmk45umyuj6ou5 7/WCl9HrOnz4vjLa9yJMRu5RzMWJ3SvMcMdFn7039xjmYGcFPZglB+YqC4v8G5GSRLyb hnsB0NoFVJvAww/XvEWmkd3prtZt17rJ8ueRb6NobG3Ax2lcqfXGKqQyr1EZCk9D9Yeq 24Rg== X-Gm-Message-State: AOJu0YyTfdpQj8J54QQmhnog9lvuE/1Ek8GJ+hnEUSdVC4B0GPkN7WOq ZMzYNhUC0RTk4imfmVHCczLE8I+acdZN4ci2bsta4g== X-Google-Smtp-Source: AGHT+IGMqUrVfV+dz/gW9YaHJqOGThqNoUvNg3dX8yRlXQiXmYJ84aNlFY5GFAPOcqrTuilhkUJIA1J5zqNQf0x+Iq8= X-Received: by 2002:a05:6000:45:b0:32d:96a7:9551 with SMTP id k5-20020a056000004500b0032d96a79551mr137837wrx.36.1697662506602; Wed, 18 Oct 2023 13:55:06 -0700 (PDT) MIME-Version: 1.0 References: <20231009145605.2150897-1-usama.arif@bytedance.com> <20231010012345.GA108129@monkey> <20231012000327.GA1855399@dev-arch.thelio-3990X> <20231012145318.GA5127@monkey> <20231013001203.GA3812@monkey> <20231014000450.GA253713@monkey> In-Reply-To: <20231014000450.GA253713@monkey> From: Nick Desaulniers Date: Wed, 18 Oct 2023 13:54:52 -0700 Message-ID: Subject: Re: [PATCH] mm: hugetlb: Only prep and add allocated folios for non-gigantic pages To: Mike Kravetz Cc: Nathan Chancellor , Usama Arif , linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, muchun.song@linux.dev, songmuchun@bytedance.com, fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Konrad Dybcio , llvm@lists.linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 424501C0015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: q75zkh878c3iqmw91eayxgk4gb3gwwiy X-HE-Tag: 1697662508-276714 X-HE-Meta: U2FsdGVkX1/si88nVkdtnNIhAuKEeE2vHwW40LaGkI+iJAJQNL2oIQOwMOfhzy4Y+aFY97hEUIO8rGg6AXVNHAqKnRMOfKr74uPhgb+oy8IPmBsb3BjXWLgNjpx8aL8UY/7iH9htncNS6hOut1/RVrl8cBiPPZ/2y1bnsgB3KCi0lu940B9xseWiA0TdcCilO6tm3QdE1OCO1pszeX8qNyF0WbonsRsEaI0K7bff4jkmA/QEeLahnBFvP9nt/CpYoW4rpEEmysNzflP+dK2qCXvYQ3UXFHmRAqwA2RFFRB3qQ4nZxjmMRp4OzEtaONd7xtl7nGk8bMOzK2hbn7EgbZWLPHpLzfQVKCv5cEcYZwERGNS8SOKaNLGEvARAoJAX8nCojFYRmxWdl0Vrefc6WXZFLZmu6TctjnIeTNcwETVZNY9gs1DrYDp+imStObIgk4ap2XyP+Bm0S2Jn2VSyRXlKPv/a0rWAul7bPWSXx/95QAI8gDyOglaNGDz80MjLi7LlofRBgvTwRI5y6pcjDdVqxEircc+2k+V0ntYGeenpWaeDNqqZoZj1EpEYrIUau2+WP1DbZCSx1OpJuS/5bm61XEUnYi52971JaFHgcnWAWZx9cSqrkYSR+aF5uc5DPjbDzCevOQotlCnxHkzBKPtKBdbuoXNzAhzVH6KEtdT8hFK6F/lY/YTOn2Q2Bh4AM2Ug/Ey4SjGmeA1zHpw/PsQcdesjqJ86ETqnxuNFaWG0qaqJHcT85+lVj9ZaaMyeZqTASwTH8Qznots7SOAcfya41BU3drfUp4PBY1p9ckYTCt02cO6o337iolqKUr5tb+QnvPn4LB1XJiqoDU9UeCUDOaCnsQpNsnNR0k2mXcxI/iOaHR6Wf8GHyGhv9YvP268n0WJ0ogtFlmMR5HE/IQ3VRexPXUKyxv//4jVrvug5qeR/suvQnRm4RnWvYSND8Z6sbP5CRNR3W6Iire/ Y6nDhX5B Be05T2exiJOwDu5T+kejtYwwjDRO47ZEvEMZjGqHi0+INofg28IUG7tYr0dZn0fv95RKQoni1m5tDQIpqgCumcamgRH3eim9J2zrtxF4xp3CrUkYRRJWe0yUiueSySGPM9atsiUt9L9lFK1S+Al0FzpPZko4OTw33Kgmhfl+gST5usTMrywBJBA9kHbjz0cr4z6CVHa3tSEjWdW/a5yRUAp/tLH3cHtIv92XZpM5wtdeeQcjtLfszv17EuQU8ILoYDXX2LVuhpaCSCdjQN+jV4UhzqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Oct 13, 2023 at 5:05=E2=80=AFPM Mike Kravetz wrote: > > On 10/12/23 17:12, Mike Kravetz wrote: > > On 10/12/23 07:53, Mike Kravetz wrote: > > > On 10/11/23 17:03, Nathan Chancellor wrote: > > > > On Mon, Oct 09, 2023 at 06:23:45PM -0700, Mike Kravetz wrote: > > > > > On 10/09/23 15:56, Usama Arif wrote: > > > > > > Thank you Nathan! That is very helpful. > > > > > > I will use this information to try and recreate. If I can recreate, = I > > > should be able to get to root cause. > > > > I could easily recreate the issue using the provided instructions. Fir= st > > thing I did was add a few printk's to check/verify state. The beginnin= g > > of gather_bootmem_prealloc looked like this: > > Hi Nathan, > > This is looking more and more like a Clang issue to me. I did a little > more problem isolation today. Here is what I did: > > - Check out commit "hugetlb: restructure pool allocations" in linux-next > - Fix the known issue with early disable/enable IRQs via locking by > applying: > > commit 266789498210dff6cf9a14b64fa3a5cb2fcc5858 > Author: Mike Kravetz > Date: Fri Oct 13 13:14:15 2023 -0700 > > fix prep_and_add_allocated_folios locking > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index c843506654f8..d8ab2d9b391b 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -2246,15 +2246,16 @@ static struct folio *alloc_fresh_hugetlb_folio(st= ruct hstate *h, > static void prep_and_add_allocated_folios(struct hstate *h, > struct list_head *folio_list) > { > + unsigned long flags; > struct folio *folio, *tmp_f; > > /* Add all new pool pages to free lists in one lock cycle */ > - spin_lock_irq(&hugetlb_lock); > + spin_lock_irqsave(&hugetlb_lock, flags); > list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { > __prep_account_new_huge_page(h, folio_nid(folio)); > enqueue_hugetlb_folio(h, folio); > } > - spin_unlock_irq(&hugetlb_lock); > + spin_unlock_irqrestore(&hugetlb_lock, flags); > } > > /* > > - Add the following code which would only trigger a BUG if we were to > traverse an empty list; which should NEVER happen. > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index d8ab2d9b391b..be234831b33f 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -3294,11 +3294,21 @@ static void __init gather_bootmem_prealloc(void) > LIST_HEAD(folio_list); > struct huge_bootmem_page *m; > struct hstate *h, *prev_h =3D NULL; > + bool empty; > + > + empty =3D list_empty(&huge_boot_pages); > + if (empty) > + printk("gather_bootmem_prealloc: huge_boot_pages list emp= ty\n"); > > list_for_each_entry(m, &huge_boot_pages, list) { > struct page *page =3D virt_to_page(m); > struct folio *folio =3D (void *)page; > > + if (empty) { > + printk(" Traversing an empty list as if not em= pty!!!\n"); > + BUG(); > + } > + > h =3D m->hstate; > /* > * It is possible to have multiple huge page sizes (hstat= es) > > - As you have experienced, this will BUG if built with LLVM 17.0.2 and > CONFIG_INIT_STACK_NONE > > - It will NOT BUG if built with LLVM 13.0.1 but will BUG if built with > LLVM llvm-14.0.6-x86_64 and later. > > As mentioned in the previous email, the generated code for loop entry > looks wrong to my untrained eyes. Can you or someone on the llvm team > take a look? I think you need to initialize h, otherwise what value is passed to prep_and_add_bootmem_folios if the loop is not run because the list is empty. The compiler sees `h` is only given a value in the loop, so the loop must be run. That's obviously hazardous, but the compiler assumes there's no UB. At least that's my limited understanding looking at the IR diff Nathan got me in https://github.com/ClangBuiltLinux/linux/issues/1946. --=20 Thanks, ~Nick Desaulniers