From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 153BACFB424 for ; Sun, 6 Oct 2024 14:37:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E37C6B0275; Sun, 6 Oct 2024 10:37:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 993456B0276; Sun, 6 Oct 2024 10:37:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 882316B0277; Sun, 6 Oct 2024 10:37:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 654AD6B0275 for ; Sun, 6 Oct 2024 10:37:39 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0E6F6C1184 for ; Sun, 6 Oct 2024 14:37:39 +0000 (UTC) X-FDA: 82643431038.16.BF4E08F Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) by imf13.hostedemail.com (Postfix) with ESMTP id 2B92A2000E for ; Sun, 6 Oct 2024 14:37:36 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T1Crwn6J; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728225281; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ND5Svx26eAZVcwpez7MW0bsvA1vYACvoD5M7KxqAkR0=; b=ToQMUS+Tb0mGyVBUjJ0Kn/w+eCBi9wmBUbc4L5rKKz5p9yuHFYgAuSuYzdeTTJOIw7l+jQ 0V0WtKqu4Ty36CLovNkkG4NB45Ub+rDTLaM+2FgWD0QPhx9ZiuZ2+Uj2bUtcsX4GdwjSBs +FfWq6sbrYjLnXWr3N3HVeyGT9W7fY8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=T1Crwn6J; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728225281; a=rsa-sha256; cv=none; b=IJpTmayIuhguhi8GrQ9qpiSCOfCQD+Hwr4kGkRCci9AWxX9x9Zrkp+3z5VGQVacxQ1JLh6 pYFFPgaM8P+3b+CAQ9VRNvJZp6gWs5A0iC1ruvh1QlogGfl+bamjvGV24WRIIFRZoxmlro zoXdIhc6Ca4R90oOVtosEMQZijRdGhk= Received: by mail-lf1-f52.google.com with SMTP id 2adb3069b0e04-5398a26b64fso3442486e87.3 for ; Sun, 06 Oct 2024 07:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728225455; x=1728830255; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ND5Svx26eAZVcwpez7MW0bsvA1vYACvoD5M7KxqAkR0=; b=T1Crwn6J/ZGdw4w+TCaCNAF6VQVDxQLcLxdgtqwob7Ih+qjLQmcrmsHAJ7QNdfnsNx Q5hPsQ7BCpF1UbHvMmWhgfp7mAKivBr/6zZCB9xbKKKEug4rZulwUuHzAN37Wf0PT89q 53E7qZId5Mpa4e9SdcfkvenjUILOPOR84X30z7/deSjSEX+Hq8KjgVUrHwH5/bGm/7St puP9F3ervJu9g01LDLgWCKemWJ3CqbiFqSmiNzmaMR5pSVxGoQq9aCYlaOC404A4ksZl s94od/zlfGiisOcv6XKj89Yh7SVaOda4zl9fu3gWmlHjuxNt2ovOXX0j6Bbs3bz6OkCM LbQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728225455; x=1728830255; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ND5Svx26eAZVcwpez7MW0bsvA1vYACvoD5M7KxqAkR0=; b=DDf7ZqWuJczmvPCPyTO0zzRn7IjzcDXqNuVXf3YK0+uZprgIiYjNL/iYB3GGu27qTf xbVRDs+0o3OaLIA0kvXRIHS1vBK0gXDgGal9e1vuRRoL8badEbbhY6mISc5gdTPiWy/r soI94f5pzhu5FXDKJa6ryVs60ynyfT69XPlt6O8ZIUGz6n1LfK1cezTLRQ78YIRCK6fZ 5TuJ+0Rrx7BrS4cte7XtSMOCfP0kss60CeSPKCUEl86vIdNk+hDbq8rHTWewDRdwRmOH nZ+V0mg4vpltXEUAQiwNB9XqmHuoQ3G2ZBQhFn/g+ITGEhZbcsHQrzDTTIb+C7LTlEA7 ls9w== X-Forwarded-Encrypted: i=1; AJvYcCV0bGNvXc9LacYrGCGM5t+crm/Vp3nplYAmYGNcy1LZrTK4tV8PdCQErOaj+tLSY/POK0xeT6/vJQ==@kvack.org X-Gm-Message-State: AOJu0Yyot3fzm4R6QVPmb2HMrp0LDDLNWrJQlcgonI7wFsjVrVQHO/mP HObzAwjGNNgOkxaqDn+Arb33I0FJ72lIRu+of5oeEaYVnCB3HkabspBV0yiMJfVM+DH2VyAvyqN WTXwtWsldEUH29S2TsOau7g/lMik= X-Google-Smtp-Source: AGHT+IHpGKIOejf5iRK2qFu64BjRL/QI8CawWgIdyDY+8/fGVSi35n4BU6U1tnTTr0Qo/rjJiqLbZzf9VgtOLcmclhs= X-Received: by 2002:a05:6512:3b8e:b0:537:a855:7d6f with SMTP id 2adb3069b0e04-539ab891f52mr4358542e87.34.1728225454894; Sun, 06 Oct 2024 07:37:34 -0700 (PDT) MIME-Version: 1.0 References: <20241001-strict_numa-v3-1-ee31405056ee@gentwo.org> In-Reply-To: <20241001-strict_numa-v3-1-ee31405056ee@gentwo.org> From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Date: Sun, 6 Oct 2024 23:37:22 +0900 Message-ID: Subject: Re: [PATCH v3] SLUB: Add support for per object memory policies To: cl@gentwo.org Cc: Vlastimil Babka , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Yang Shi , Christoph Lameter , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Huang Shijie Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2B92A2000E X-Stat-Signature: c8oh15ssnpqfjwgfxidgmggcugi1wrgj X-Rspam-User: X-HE-Tag: 1728225456-611528 X-HE-Meta: U2FsdGVkX18agQ8WfuJUH4IYGnQTqweiT8AFqJX+OONxYCy1ySn1r4l4qIzAIHPYwDbU1odXyMdQG1aycCF40QJCDxBMz3RFZf5rjPPt8OfTFHJH7AtofiHIk9p1lZV4TsPWwS6BlqAbqtWTqf+GOUfThlrnKqpbNML0Qf1zlLoqwkKqKlX6QBK4lFGffkrU2hvIWS9/K2s+h0xa6iet73hXxm8OWebn+9STq2Oem0guerafMI+P9SJWQeI+bq8GY52MZq5jKI7u73srTgEifr8lkiA/YhLMwDoOyiEDwQMifLAxEvdSO9Lzn3IOiOgqf3KgpVsJp8gpeR0fAOiu+ubOH9cWTaVpnubXRNOLFg979wq6V+MNZjrWCZcmqhrQYBHBhSvd86mNvhPIrN9VRzjKjiRbP/p0aewI/FTTYllDXp3QFHSqkLWbq02lKMoQtkw0vxTuEPrZtm364QsDhh1Quti6N4/n7udM4oM0oazRH0vBoxf0Tj6j7g4LQ/Ave6K0PY60i/qyj0WJG5AyaDH49yk4XgxZjfJAx1AopspobCXAgFwJp98zuMKT2oZGICshNZR1Mst+clzh+9jX7MPTQbOKRNmxS6nO5QxvCiUMPg2Qkvx91j1C1hiiJ+uIZS9Wt9OvvEa9M7+PDou51HMyWqrc7mobVFdA0PTPNeMn1q6O8wEHFMjB4GtqJmcbQdkL3LYBRfRe39DNPGR2Pb8o/Zxja/ZtJoJTa5Vg5E/elZAJBPPTzwVhGjhyE+7w3wu1rmP1C0bBO+d8wauWcHS7KJT/9BZN9mAaMn+vv7ZiGgyV5BZMXKZTjPg+gtc3Ist9YXHrIX2si2JpqTJLx30samndm8NpuSYe5f+HIVQX8kJCTpkxSnQhtVqqldQQJXb0w3eJiJLflLQGuAK1wQctXNA0DhtxHFtZUhQUTE21X2Mb5rmKskA+y+YbGt06DZ5fmWrsq9mE9zzZ+aD voAWCb2c L/mh7L+5dQ466k9vbp05EO3n1gqqQ7lTEh7oywhm/GICa3J6JqY1WVUTQZoqzAiolC5mpkphTWdEhRax1yPlvaeGfdql3hcz0dg+yDoEbf66oP5DaMM8iJdO7XbWqJjkYLetplAfWlq5Nw+4V5te/aSOfuryJbAuUBjhddmQaaUNuz2dXOKVQsC1X+FloeWi+wEpSprcU6cD6j6l0gUlSkOU7hjc4fjOWZ0GC3Ef6I6Cwljb8oWuhUaxi11gZ2Ek9oxDunuFT1ns5iJ21hTvYYzr5/AK0sjREWmRVhvtEg8nrsPETVilmzzC8CC9V3FS+d3NoFgOUrV8dEbUyXEWFYFsB7t7g+YHuzxpUIh2nno1AnlF2Jko0IZk7bi9bBf0Fl+OP5rYgG1N63wJaiu+H2hlAC+wBHtJieReBD+chu9HF8/Mw45Vx7DtPA5kFVBY0gry/6hP+qZOzK5BGlpP9ag8DSmvVyVz9RqriLZYAiSrrF21aLUouz/rKVg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Oct 2, 2024 at 4:08=E2=80=AFAM Christoph Lameter via B4 Relay wrote: > > From: Christoph Lameter > > The old SLAB allocator used to support memory policies on a per > allocation bases. In SLUB the memory policies are applied on a > per page frame / folio bases. Doing so avoids having to check memory > policies in critical code paths for kmalloc and friends. > > This worked on general well on Intel/AMD/PowerPC because the > interconnect technology is mature and can minimize the latencies > through intelligent caching even if a small object is not > placed optimally. > > However, on ARM we have an emergence of new NUMA interconnect > technology based more on embedded devices. Caching of remote content > can currently be ineffective using the standard building blocks / mes= h > available on that platform. Such architectures benefit if each slab > object is individually placed according to memory policies > and other restrictions. > > This patch adds another kernel parameter > > slab_strict_numa > > If that is set then a static branch is activated that will cause > the hotpaths of the allocator to evaluate the current memory > allocation policy. Each object will be properly placed by > paying the price of extra processing and SLUB will no longer > defer to the page allocator to apply memory policies at the > folio level. > > This patch improves performance of memcached running > on Ampere Altra 2P system (ARM Neoverse N1 processor) > by 3.6% due to accurate placement of small kernel objects. > > Tested-by: Huang Shijie > Signed-off-by: Christoph Lameter (Ampere) > --- > Changes in v3: > - Make the static key a static in slub.c > - Use pr_warn / pr_info instead of printk > - Link to v2: https://lore.kernel.org/r/20240906-strict_numa-v2-1-f104e6d= e6d1e@gentwo.org > > Changes in v2: > - Fix various issues > - Testing > --- > mm/slub.c | 42 ++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 42 insertions(+) > > diff --git a/mm/slub.c b/mm/slub.c > index 21f71cb6cc06..7ae94f79740d 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -218,6 +218,10 @@ DEFINE_STATIC_KEY_FALSE(slub_debug_enabled); > #endif > #endif /* CONFIG_SLUB_DEBUG */ > > +#ifdef CONFIG_NUMA > +static DEFINE_STATIC_KEY_FALSE(strict_numa); > +#endif > + > /* Structure holding parameters for get_partial() call chain */ > struct partial_context { > gfp_t flags; > @@ -3957,6 +3961,28 @@ static __always_inline void *__slab_alloc_node(str= uct kmem_cache *s, > object =3D c->freelist; > slab =3D c->slab; > > +#ifdef CONFIG_NUMA > + if (static_branch_unlikely(&strict_numa) && > + node =3D=3D NUMA_NO_NODE) { > + > + struct mempolicy *mpol =3D current->mempolicy; > + > + if (mpol) { > + /* > + * Special BIND rule support. If existing slab > + * is in permitted set then do not redirect > + * to a particular node. > + * Otherwise we apply the memory policy to get > + * the node we need to allocate on. > + */ > + if (mpol->mode !=3D MPOL_BIND || !slab || > + !node_isset(slab_nid(slab), mpol-= >nodes)) > + > + node =3D mempolicy_slab_node(); > + } Is it intentional to allow the local node only (via mempolicy_slab_node()) in interrupt contexts? > + } > +#endif > + > if (!USE_LOCKLESS_FAST_PATH() || > unlikely(!object || !slab || !node_match(slab, node))) { > object =3D __slab_alloc(s, gfpflags, node, addr, c, orig_= size); > @@ -5601,6 +5627,22 @@ static int __init setup_slub_min_objects(char *str= ) > __setup("slab_min_objects=3D", setup_slub_min_objects); > __setup_param("slub_min_objects=3D", slub_min_objects, setup_slub_min_ob= jects, 0); > > +#ifdef CONFIG_NUMA > +static int __init setup_slab_strict_numa(char *str) > +{ > + if (nr_node_ids > 1) { > + static_branch_enable(&strict_numa); > + pr_info("SLUB: Strict NUMA enabled.\n"); > + } else > + pr_warn("slab_strict_numa parameter set on non NUMA syste= m.\n"); nit: this statement should be enclosed within braces per coding style guide= line. Otherwise everything looks good to me (including the document amended). Best, Hyeonggon