From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1666C4167B for ; Wed, 29 Nov 2023 00:46:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F21EC8D0018; Tue, 28 Nov 2023 19:46:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED2B48D0001; Tue, 28 Nov 2023 19:46:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DC32A8D0018; Tue, 28 Nov 2023 19:46:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CDED08D0001 for ; Tue, 28 Nov 2023 19:46:53 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9C08FC03E2 for ; Wed, 29 Nov 2023 00:46:53 +0000 (UTC) X-FDA: 81509151906.21.9B21DFA Received: from mail-vk1-f171.google.com (mail-vk1-f171.google.com [209.85.221.171]) by imf29.hostedemail.com (Postfix) with ESMTP id D5492120008 for ; Wed, 29 Nov 2023 00:46:51 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LK5+iV8c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.221.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701218811; a=rsa-sha256; cv=none; b=onfwoa4lcfNPGP3kdIVviVE9hdpUR5ldyycMr9A5D0GD5msZQfxCWtu0ythol/KrbN53yG PN17ilaWp1nLllVwSE79m1bdRHWzlQgh4EMqSG6+iZBTlaTJEDF0HXPT5kygpE3cdHKZQZ IhGDRv4mbYtJj+HthftyvD4nfrtVp4w= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LK5+iV8c; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.221.171 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701218811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+oDh1aIguEdHpwRt07bECjWeSQzO3Fxi6hNx6EgBsZw=; b=G2mVOam5ZrLKUfmzdH5jE7wYI4KuuS1FGERJtGGCeJUZVijTaLdY1AhXkG4EqRcmt/D3I5 mQhbVjzCcCMy5eMtPuolN1kV3SkGtNI2F4LVA7XY87eXLndwV7Hz2SfuKCU3skcpl0KHi3 +L1h1NrQOIn45gFC8CQnrAVu18w5/T8= Received: by mail-vk1-f171.google.com with SMTP id 71dfb90a1353d-4acf9dd3d35so1964079e0c.0 for ; Tue, 28 Nov 2023 16:46:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701218811; x=1701823611; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=+oDh1aIguEdHpwRt07bECjWeSQzO3Fxi6hNx6EgBsZw=; b=LK5+iV8civZZ4hKheMoau96RlBWO9jB+bz0IgCJEuSMB+gAgyhuMoHR3T8dWY34MXl If9P7GPGfzc5a4qqB6Fp7gRcLGzd28pXJoAz9k5Sve3Gknfry4QMF6VkpM1ZikVwwh8q 5g+ns1qfk/hXbEHQVwqYJycHHi/3JelWbGfEJ8oGx9dO4hU6U1sjn87WRNhW98XUQty5 97tN0A9WQ328Bz19QarFlh6+w7cUbDzIhN/3KGQmrHEuzYs2u515s1kWR9lQzxC8YgR0 /B7v/wkRZDhou2NWlAy/mWvKXxDdY6bC2jD2SayvwpT7WyR/0IthgBXdPrmiwwzu9zFl ItWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701218811; x=1701823611; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+oDh1aIguEdHpwRt07bECjWeSQzO3Fxi6hNx6EgBsZw=; b=G3+yL8SatpHeBvzisBhkfMTXPMGZU1RbUFxigxkFPjhbY8iGmxn3BqqZNLnl3X1chf 8j1c2UUmzgY1e4woBfzk7nNbNcaeerSAZvkQsQdAVqwpjBKQ4Fi+dn6Ov+X+nTUq+3NJ U7xdAe84JREWBqnorzBdySWJOrgP4tCWupPDOVzEbzRTg3tMFBDKGTYKg/XeZfxxn7S4 k8fEin7KhdbL8U3eTIKijdVvqIW1nwNIeyJMcu7essw++AiLZ+8gmPFg/MjpKAzbbOGZ xqNBkX0o5BsqNNOMjaPFloQYnIG/gRCgrnjSJGaf21a74FpxXBCedGgmMvrY/mgkvyAY hzcA== X-Gm-Message-State: AOJu0YzVTcKws9qt81+0e3j/hHMy4xPJKoIEochzdoJUfb+Rz0/QDIwz PenHGOLpyosiZ5Sog3DWm6puY+7r1iIFevqNTRU= X-Google-Smtp-Source: AGHT+IEk8yTd8eI4EdeyKZW6reXCMwZOddixQRa28CFP3qyfYrswVX53unzj5ErXKXftiuXmRwb92oAWzv7HWxQq7w0= X-Received: by 2002:a1f:f8cf:0:b0:4b2:777a:a860 with SMTP id w198-20020a1ff8cf000000b004b2777aa860mr3478556vkh.13.1701218810686; Tue, 28 Nov 2023 16:46:50 -0800 (PST) MIME-Version: 1.0 References: <20230810163627.6206-9-vbabka@suse.cz> <20230810163627.6206-11-vbabka@suse.cz> <077e8e97-e88f-0b8e-2788-4031458be090@suse.cz> In-Reply-To: <077e8e97-e88f-0b8e-2788-4031458be090@suse.cz> From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Date: Wed, 29 Nov 2023 09:46:38 +0900 Message-ID: Subject: Re: [RFC v2 2/7] mm, slub: add opt-in slub_percpu_array To: Vlastimil Babka Cc: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D5492120008 X-Stat-Signature: 4hft6dcfzdz9ierxoot5uk9g84ya45ct X-HE-Tag: 1701218811-901192 X-HE-Meta: U2FsdGVkX1/IdODS4jg83TszojDf2Tqvsrh8wHkjGIDlzqr5rFVZGXzxEbv2nQ1yWp8KM/L3u0aAwsoN2KUj2KcqwXRta+SMPmdciQ3ttw5pzzQSs8t0eN8rpPTrbqgg5MSDvm3BC+QwmnNhFNB2Fp4klwOnA2TU4DUNbBOaM6eoLs4CNQzV0ntqZhr/s2KEjq01b6QR19IAg7RxiSu977SpiQNNWlNeZLYuOIiSQ8l1BOXWTjskXwvad9DM0RA+BufqoAwsljNWQfgzbDZAnBbEkCpnwWeeQPe/jUc+enH9HvQqBR0I+bg2PTXad+dYpz1HxwT7MmLIeoKEQv+RbJS52HACUa0WBEwxCAdhBw2yVmxt8SZmAtAQr7WR8UdE4P5eCSc9Jb0KJTonbend6LmB2XsADNPcvkIGBUOTg2wQ0cThsH+J3NDlxRubmzULX0EYRoC8OycbYBHeMzuYRLaGnmoI8y6B7VEICQnkBysr2S5j5SyiPK04gvliGxlSyOtkUMGI1D3Tj5h0Ko0X2wQPFu/xlKN0ikRkZRIZHA6e80AzWZNo3ct4CJcpX4LkOcsrfaA5Ygz1bO7BF/theD5ZjDn1PuWWQ+Bk0k23aROFFwae9vJleNViZKNylYpQRbC/Y4AdCZK8t1ZkHqpD6j6DN1BM/LdylQahzY8/gsLt44a7tCa50H2xNutrRy133Hp1WZxDpphs7GYkl/OZK4D9bcIcj6VmE4qwlWDSkP20ZDmBKOP6xvXgu8BCZu5E6/r6AMaNE54LRhqUyDywlNs1ujKkj+IMfpXAQAXxAQDI2jYZWoZtR1qJh7u+PEq0Nlbj6QMk7UiDZFelMs0cfaG7ax/DjwVNsFuKMbLnuuuej9gOL3DHbwd/tIwrxiK0HtwU8cuNkROb+dVIbiFt7aJIHSUuMVw3YhzUzXy0OJb2mcBj0JsPl8BkxMShsGrQTngLFuTmF9Bo8XqURLi /G+/mq8u Ju9Bn8qY81Pj99YsX3UkJKsa+4PprmB8TjD3AtntOp7iE2uzWJZSGmifWUEKWF17XjAX0fXQLgVfsZ5JZkIH6hVudQ9P2bmiSWSUiipHMY3vVkOnlCj9y1+NVDVVk+rC8O2Yi8GOERI1gCmkorG9zJOX3I8zrlTOx1641Me1HV/06GtKDZwVT7qCYUK32JoOe7+lpgP/ybt5hNUBraVVzIsPVIHtbcaZsknAg8mlMRS9PSA6v0y8S4Cw4I36RmuX+WAJanPoaHuOPjzRjMX0PITTg0MpQFHStOzUIjAbdJ0HqBeyHaddYrF1jMLEzhVXLM9hVI0jgNKlwaR7wQ6lDEa6G/BIe9CXQbwUK0y9+iQN+7q11ZmBoNU9UG4rcGJscvTahws3D2mQMXpw90yCU0Kh6ZfnVE3NWzkT4eoFo5epSUIsJ6kFhwmLujg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Nov 29, 2023 at 2:37=E2=80=AFAM Vlastimil Babka wr= ote: > > On 8/21/23 16:57, Hyeonggon Yoo wrote: > > Hi, > > > > On Fri, Aug 11, 2023 at 1:36=E2=80=AFAM Vlastimil Babka wrote: > > Oops, looks like I forgot reply, sorry (preparing v3 now). It's fine, you were busy removing SLAB :) thanks for replying. > > > >> /* > >> * Inlined fastpath so that allocation functions (kmalloc, kmem_cache= _alloc) > >> * have the fastpath folded into their functions. So no function call > >> @@ -3465,7 +3564,11 @@ static __fastpath_inline void *slab_alloc_node(= struct kmem_cache *s, struct list > >> if (unlikely(object)) > >> goto out; > >> > >> - object =3D __slab_alloc_node(s, gfpflags, node, addr, orig_siz= e); > >> + if (s->cpu_array) > >> + object =3D alloc_from_pca(s); > >> + > >> + if (!object) > >> + object =3D __slab_alloc_node(s, gfpflags, node, addr, = orig_size); > >> > >> maybe_wipe_obj_freeptr(s, object); > >> init =3D slab_want_init_on_alloc(gfpflags, s); > >> @@ -3715,6 +3818,34 @@ static void __slab_free(struct kmem_cache *s, s= truct slab *slab, > >> discard_slab(s, slab); > >> } > > > >> #ifndef CONFIG_SLUB_TINY > >> /* > >> * Fastpath with forced inlining to produce a kfree and kmem_cache_fr= ee that > >> @@ -3740,6 +3871,11 @@ static __always_inline void do_slab_free(struct= kmem_cache *s, > >> unsigned long tid; > >> void **freelist; > >> > >> + if (s->cpu_array && cnt =3D=3D 1) { > >> + if (free_to_pca(s, head)) > >> + return; > >> + } > >> + > >> redo: > >> /* > >> * Determine the currently cpus per cpu slab. > >> @@ -3793,6 +3929,11 @@ static void do_slab_free(struct kmem_cache *s, > >> { > >> void *tail_obj =3D tail ? : head; > >> > >> + if (s->cpu_array && cnt =3D=3D 1) { > >> + if (free_to_pca(s, head)) > >> + return; > >> + } > >> + > >> __slab_free(s, slab, head, tail_obj, cnt, addr); > >> } > >> #endif /* CONFIG_SLUB_TINY */ > > > > Is this functionality needed for SLUB_TINY? > > Due to the prefill semantics, I think it has to be be even in TINY, or we > risk running out of memory reserves. Also later I want to investigate > extending this approach for supporting allocations in very constrained > contexts (NMI) so e.g. bpf doesn't have to reimplement the slab allocator= , > and that would also not be good to limit to !SLUB_TINY. I've got the point, thanks for the explanation! > >> @@ -4060,6 +4201,45 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s,= gfp_t flags, size_t size, > >> } > >> EXPORT_SYMBOL(kmem_cache_alloc_bulk); > >> > >> +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned in= t count, > >> + gfp_t gfp) > >> +{ > >> + struct slub_percpu_array *pca; > >> + void *objects[32]; > >> + unsigned int used; > >> + unsigned int allocated; > >> + > >> + if (!s->cpu_array) > >> + return -EINVAL; > >> + > >> + /* racy but we don't care */ > >> + pca =3D raw_cpu_ptr(s->cpu_array); > >> + > >> + used =3D READ_ONCE(pca->used); > > > > Hmm for the prefill to be meaningful, > > remote allocation should be possible, right? > > Remote in what sense? TL;DR) What I wanted to ask was: "How pre-filling a number of objects works when the pre-filled objects are not shared between CPUs" IIUC the prefill is opportunistically filling the array so (hopefully) expecting there are some objects filled in it. Let's say CPU X calls kmem_cache_prefill_percpu_array(32) and all 32 objects are filled into CPU X's array. But if CPU Y can't allocate from CPU X's array (which I referred to as "remote allocation"), the semantics differ from the maple tree's perspective because preallocated objects were shared between CPUs before, but now it's not? Thanks! -- Hyeonggon