From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28B6BC3ABDA for ; Wed, 14 May 2025 23:44:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F7958D0006; Wed, 14 May 2025 19:43:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 682378D0001; Wed, 14 May 2025 19:43:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B0888D0006; Wed, 14 May 2025 19:43:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2AD728D0001 for ; Wed, 14 May 2025 19:43:33 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 60C9681217 for ; Wed, 14 May 2025 23:43:34 +0000 (UTC) X-FDA: 83443142748.13.5D9AB97 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 967551C000A for ; Wed, 14 May 2025 23:43:32 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L4AUOQET; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3oyolaAsKCNQ02A4HB4OJD66EE6B4.2ECB8DKN-CCAL02A.EH6@flex--ackerleytng.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3oyolaAsKCNQ02A4HB4OJD66EE6B4.2ECB8DKN-CCAL02A.EH6@flex--ackerleytng.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747266212; a=rsa-sha256; cv=none; b=Q1BR5Utl38rkeQ6DmQRPH1fiLi7Y0MJznV1sJT993PEXKmOC1AQx42gaQdJX8FoOLjVYVj bVIh1lPYBHsaCot0sXhbOWYUrgMecKOxkS4i6PltupEgtzf9zPWR0aRMe6d0804COK6j6U VrFLERzy4xn5vFjB9OySb4WlF4OTj2I= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=L4AUOQET; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3oyolaAsKCNQ02A4HB4OJD66EE6B4.2ECB8DKN-CCAL02A.EH6@flex--ackerleytng.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3oyolaAsKCNQ02A4HB4OJD66EE6B4.2ECB8DKN-CCAL02A.EH6@flex--ackerleytng.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747266212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IVT2MIxWnWM/XLaHWjsnvPvOi1XED19jnd4mUMbhvto=; b=3qLAyTdZVbCLRyKkQnWwxwusQ5eCRfY9hhEHXoDzXhEiVPMNAfpOUE6xEFDg34ZK2+BikH xO5yBJad4gd+htOjNdD9E3lNkqVWHa2cZ+Ls0CvsOvuA6L3Ct1f/kGs8AEqJOR95u8LFsQ ls/a8KJKZF8TlreSRnJ/4ij6+zqwx2Q= Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-7395d07a3dcso276197b3a.3 for ; Wed, 14 May 2025 16:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1747266211; x=1747871011; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IVT2MIxWnWM/XLaHWjsnvPvOi1XED19jnd4mUMbhvto=; b=L4AUOQETyTKRrfcnexEseRtcjYKdDhuzg0uzjkcfLYYtjx0AsfsJEcwAaIRBsjUhHc kegnyMkPUp7Dq0e6KgeShVHLpWf40MqEK2+JVHSJPj4U3WUixRABjwVbGij9iKdOibul IL2ir4leL6RzefHEhkZFV0sOOaIUBta8uvBx2ysTmPdu3Qm0lwNcoS59qisE13Dd9RFO Uon76NpvDs1AULOuP30jLUVZzYPfH5ea2ot5kubXJZZGXUqHo6/dmwxxYoG+WofuHIf+ 9BnRjOsG2Y/B9P4UToju1J5tkMWhSyoMPJlavp0joaR+h9dFU09BNDCOt0bf5sdn27YE 13rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747266211; x=1747871011; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IVT2MIxWnWM/XLaHWjsnvPvOi1XED19jnd4mUMbhvto=; b=mWWIYgvxDufGVqkd5frmd+FGVLeGerEehR9AJ0NfxDXIcmoWpg4zPDwX4J/xOS8HFM cNFZymFYEOJI63D7k/vzT4tNJLWlemNCYSULMpn2gNhxZxzD0eBrw+MIuGRwqG/f8Jkh UwwYdTwIhnEF31+dlXbdPcMgDImKUK+ZpPyq07JhYey0T8LPhUvmF5JgOX3DWfN+V4Sn 42/rkq2VxpEJo8tmEXjZWiXR+utbM34D5eNSt3K5T0jtVAu5KFYhAgbU/aEen/KDx40b 6CT7cofTK0vrBisepB3jhHXvnem7wdYBaRdYt5OvtLh1nn7LYj58ycvXkWyDrwlUvUaN 502g== X-Forwarded-Encrypted: i=1; AJvYcCVrOKtSQcB/02oT/kJr/NJC7QrzCWF/MycIrqHXQedRMukN4IKYFRIxMjLZIVy0PjJhCtVR+5TuHg==@kvack.org X-Gm-Message-State: AOJu0YwgNBcoRmKcZyVdUQgVE25txZZyUYC198So5q4rrS5edr7d3ekI YNiRKtNUSHgHfVZ7OEiZpjdsbtPmTMfgtqFLdkuEeE4I8cbhRAyGjIK5g1ULdYiM9E7HKZb9vAY KbkEnY8u9/nGvpys/ssTltA== X-Google-Smtp-Source: AGHT+IGlIyq6CGoOQHt7EoQf0Q/6z4roPUZNjH46k/lCxiZd8Az5X5i/aF2lsphu6KPl7WGlY+LsQbU/H8QuAY5yUQ== X-Received: from pgar21.prod.google.com ([2002:a05:6a02:2e95:b0:b1f:dcda:276e]) (user=ackerleytng job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:700f:b0:1f5:6f61:a0ac with SMTP id adf61e73a8af0-215ff0970a6mr6985221637.5.1747266211396; Wed, 14 May 2025 16:43:31 -0700 (PDT) Date: Wed, 14 May 2025 16:42:01 -0700 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.49.0.1045.g170613ef41-goog Message-ID: <1f64e3c7f04fc725f4da4d57de1ea040b7a56952.1747264138.git.ackerleytng@google.com> Subject: [RFC PATCH v2 22/51] mm: hugetlb: Refactor hugetlb allocation functions From: Ackerley Tng To: kvm@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-fsdevel@vger.kernel.org Cc: ackerleytng@google.com, aik@amd.com, ajones@ventanamicro.com, akpm@linux-foundation.org, amoorthy@google.com, anthony.yznaga@oracle.com, anup@brainfault.org, aou@eecs.berkeley.edu, bfoster@redhat.com, binbin.wu@linux.intel.com, brauner@kernel.org, catalin.marinas@arm.com, chao.p.peng@intel.com, chenhuacai@kernel.org, dave.hansen@intel.com, david@redhat.com, dmatlack@google.com, dwmw@amazon.co.uk, erdemaktas@google.com, fan.du@intel.com, fvdl@google.com, graf@amazon.com, haibo1.xu@intel.com, hch@infradead.org, hughd@google.com, ira.weiny@intel.com, isaku.yamahata@intel.com, jack@suse.cz, james.morse@arm.com, jarkko@kernel.org, jgg@ziepe.ca, jgowans@amazon.com, jhubbard@nvidia.com, jroedel@suse.de, jthoughton@google.com, jun.miao@intel.com, kai.huang@intel.com, keirf@google.com, kent.overstreet@linux.dev, kirill.shutemov@intel.com, liam.merwick@oracle.com, maciej.wieczor-retman@intel.com, mail@maciej.szmigiero.name, maz@kernel.org, mic@digikod.net, michael.roth@amd.com, mpe@ellerman.id.au, muchun.song@linux.dev, nikunj@amd.com, nsaenz@amazon.es, oliver.upton@linux.dev, palmer@dabbelt.com, pankaj.gupta@amd.com, paul.walmsley@sifive.com, pbonzini@redhat.com, pdurrant@amazon.co.uk, peterx@redhat.com, pgonda@google.com, pvorel@suse.cz, qperret@google.com, quic_cvanscha@quicinc.com, quic_eberman@quicinc.com, quic_mnalajal@quicinc.com, quic_pderrin@quicinc.com, quic_pheragu@quicinc.com, quic_svaddagi@quicinc.com, quic_tsoni@quicinc.com, richard.weiyang@gmail.com, rick.p.edgecombe@intel.com, rientjes@google.com, roypat@amazon.co.uk, rppt@kernel.org, seanjc@google.com, shuah@kernel.org, steven.price@arm.com, steven.sistare@oracle.com, suzuki.poulose@arm.com, tabba@google.com, thomas.lendacky@amd.com, usama.arif@bytedance.com, vannapurve@google.com, vbabka@suse.cz, viro@zeniv.linux.org.uk, vkuznets@redhat.com, wei.w.wang@intel.com, will@kernel.org, willy@infradead.org, xiaoyao.li@intel.com, yan.y.zhao@intel.com, yilun.xu@intel.com, yuzenghui@huawei.com, zhiquan1.li@intel.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 967551C000A X-Rspam-User: X-Stat-Signature: spmni87kc4fwog7b4rh87jpit6jddc7w X-HE-Tag: 1747266212-680576 X-HE-Meta: U2FsdGVkX19kemmjBFT1SH+8myytUK0e67+sTd0kZdMXt42mowvLOUnEsCgvBQQxbq/UtP2RmmdKhsfaNdnT5Hb4zI7BiwLgfGvU6QYJC8vlRqcex7cH1XNe1GZlEmnBXO3tEtb30M118ijgsTfiQfWdTp2gUbHvSJTLkJhQJW0IBTO3nPkn3wZN18VSXa/eOWEuCdRFGfhs7H79fgcyCq4uWSvDHKgeeZq/7biuHvzkfGqayvyV8VdguDeEOU90abDsdpeoAgeNvmxEcG484yRr3goFPf32iqNrSBGyxWwmaocKCv9jaPhua8CvaLYEd0YnKLjl1FwA01wtsUpTTsywbAdzuYPjTTAs9uV0k7cl8Dr4dWp+MY5Q++SxGvAZNAuPDJcPbeu3CLPRjAUjjqOzRkpcehrbtC9ccyAwKHpN8oXiz7sT44TPKKaMmDs02VjKZvX0J57m1yGBRcH7kvFpbs5QyLN22T1RxaJh1A77voQb68ZXEl2dzp+vZ5xLXk/Wixi5A43r0bbwmxSP8QhE+tZe3XHFLDl32wMcyn1TilV9cFgED5FPOQD5OEEDZGEgCaqQxEoFve6ANO3j3z5TZZDbDcdOwpw/n9M9GxxG0i5qEKw0ukUa+foMKDl09YV8cideeoDz6QdQJgawhx+LBkbqI41P6rEaslGGNzgmAbReAMnKGMcYiujaEUM3IAqZHd2ZKaPP+5nargru/WItmD1p9w7JZRfF+n99KkKmrNYj/fu9aGlLP93kz1ON9duSEoqemOhXZkuOm3Oo4iTXg/Sk6qKIt2DFMBe+0M+B0EX3xXzUna0UpDDAtYi7PozXQp1NwmIqVDb2BHKsApiXEwQIBADPsV/QJZHhNdxWAflhTb1OnVe+mzqL6e5uSnkRHaGXkPrXIHHNhsQL59Jb5GGdWLeQ/IzFdZiaAvfhdfOgBx8BtmFTlpzQj027QijEvQjK4g7XcJFWnmD VQ8Z8w/U e/f5OAmn0/m23dLZyFveVpQIbd6DEXuocefFxkUICXXrOSGFyA9siu8OPGdWOuIzrDp6N7sd41cWYvmpRH9o3KEE/P6bssoF5BX1w0AnvcNvQiKhQ63SXGYhV4dzRRugEd9/EtadxkO03ZU0igWDiKIbvtiwVrrS7i8wPDaKBjG1H5VyrWvL/JAP0wciZXukTwcQVlTA488KTw6ElDSXZkTlJ7Fy+qpbRWCpW+KvdlZDkx/YjoRBNDJrGpxxbQfYJ4IiIwskphxTceEvL4n7siEA1vqF3fwIJufl/cXoEUGUMPx4dB2VPrSlh1NXICYTquXb1kBbAlPBwfoKwLSauNWs/eVMeI8mPVTt3C8ln2XG0j2eg2ZyAVeQf24ftcxgKAoHV3PmR0AGil0qm914grLxucqrqmdMKhl68lVpEro4SfvFH9nB32xHSHFKiDX+90NQzUZkY1+1bktz2OTlnhuP9NI4MvFkrVoF52fJ0Ut+tM4uWhI2h1JYedtW5ykyNB8z6tfg71NLK6U264zgb92D1V3k9jv8g3Fyaxf9UiRxk+zIDUGcRPYF6H87PkaM/sweHd4Djcc7rUbk9t/b/WEGTOjYRC5C8NmhzCYkJgiB2wkOOYob+lOmc9quwZHmojo+pUZK1c2TBtb4FoYj7yIbFGQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor dequeue_hugetlb_folio() and alloc_surplus_hugetlb_folio() to take mpol, nid and nodemask. This decouples allocation of a folio from a vma. Signed-off-by: Ackerley Tng Change-Id: I890fb46fe8c6349383d8cf89befc68a4994eb416 --- mm/hugetlb.c | 64 ++++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 34 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5cc261b90e39..29d1a3fb10df 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1364,34 +1364,22 @@ static unsigned long available_huge_pages(struct hstate *h) return h->free_huge_pages - h->resv_huge_pages; } -static struct folio *dequeue_hugetlb_folio(struct hstate *h, - struct vm_area_struct *vma, - unsigned long address) +static struct folio *dequeue_hugetlb_folio(struct hstate *h, gfp_t gfp_mask, + struct mempolicy *mpol, + int nid, nodemask_t *nodemask) { struct folio *folio = NULL; - struct mempolicy *mpol; - gfp_t gfp_mask; - nodemask_t *nodemask; - pgoff_t ilx; - int nid; - - gfp_mask = htlb_alloc_mask(h); - mpol = get_vma_policy(vma, address, h->order, &ilx); - nid = policy_node_nodemask(mpol, gfp_mask, ilx, &nodemask); if (mpol_is_preferred_many(mpol)) { - folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, - nid, nodemask); + folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, nid, nodemask); /* Fallback to all nodes if page==NULL */ nodemask = NULL; } if (!folio) - folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, - nid, nodemask); + folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, nid, nodemask); - mpol_cond_put(mpol); return folio; } @@ -2312,21 +2300,14 @@ static struct folio *alloc_migrate_hugetlb_folio(struct hstate *h, gfp_t gfp_mas } /* - * Use the VMA's mpolicy to allocate a huge page from the buddy. + * Allocate a huge page from the buddy allocator given memory policy and node information. */ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, - struct vm_area_struct *vma, - unsigned long addr) + gfp_t gfp_mask, + struct mempolicy *mpol, + int nid, nodemask_t *nodemask) { struct folio *folio = NULL; - struct mempolicy *mpol; - gfp_t gfp_mask = htlb_alloc_mask(h); - int nid; - nodemask_t *nodemask; - pgoff_t ilx; - - mpol = get_vma_policy(vma, addr, h->order, &ilx); - nid = policy_node_nodemask(mpol, gfp_mask, ilx, &nodemask); if (mpol_is_preferred_many(mpol)) { gfp_t gfp = gfp_mask & ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL); @@ -2339,7 +2320,7 @@ static struct folio *alloc_surplus_hugetlb_folio(struct hstate *h, if (!folio) folio = alloc_surplus_hugetlb_folio_nodemask(h, gfp_mask, nid, nodemask); - mpol_cond_put(mpol); + return folio; } @@ -2993,6 +2974,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, int ret, idx; struct hugetlb_cgroup *h_cg = NULL; gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL; + struct mempolicy *mpol; + nodemask_t *nodemask; + gfp_t gfp_mask; + pgoff_t ilx; + int nid; idx = hstate_index(h); @@ -3032,7 +3018,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, subpool_reservation_exists = npages_req == 0; } - reservation_exists = vma_reservation_exists || subpool_reservation_exists; /* @@ -3048,21 +3033,30 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, goto out_subpool_put; } + mpol = get_vma_policy(vma, addr, h->order, &ilx); + ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg); - if (ret) + if (ret) { + mpol_cond_put(mpol); goto out_uncharge_cgroup_reservation; + } + + gfp_mask = htlb_alloc_mask(h); + nid = policy_node_nodemask(mpol, gfp_mask, ilx, &nodemask); spin_lock_irq(&hugetlb_lock); folio = NULL; if (reservation_exists || available_huge_pages(h)) - folio = dequeue_hugetlb_folio(h, vma, addr); + folio = dequeue_hugetlb_folio(h, gfp_mask, mpol, nid, nodemask); if (!folio) { spin_unlock_irq(&hugetlb_lock); - folio = alloc_surplus_hugetlb_folio(h, vma, addr); - if (!folio) + folio = alloc_surplus_hugetlb_folio(h, gfp_mask, mpol, nid, nodemask); + if (!folio) { + mpol_cond_put(mpol); goto out_uncharge_cgroup; + } spin_lock_irq(&hugetlb_lock); list_add(&folio->lru, &h->hugepage_activelist); folio_ref_unfreeze(folio, 1); @@ -3087,6 +3081,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, spin_unlock_irq(&hugetlb_lock); + mpol_cond_put(mpol); + hugetlb_set_folio_subpool(folio, spool); /* If vma accounting wasn't bypassed earlier, follow up with commit. */ -- 2.49.0.1045.g170613ef41-goog