From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F5AAEE4993 for ; Mon, 21 Aug 2023 14:57:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C5DF940009; Mon, 21 Aug 2023 10:57:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 176898E0006; Mon, 21 Aug 2023 10:57:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03E64940009; Mon, 21 Aug 2023 10:57:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EA5BE8E0006 for ; Mon, 21 Aug 2023 10:57:32 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C153A160456 for ; Mon, 21 Aug 2023 14:57:32 +0000 (UTC) X-FDA: 81148415544.04.A430ABD Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) by imf02.hostedemail.com (Postfix) with ESMTP id ED87A8000A for ; Mon, 21 Aug 2023 14:57:29 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=YvmsvfdR; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692629850; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nLIL9tDKiB9E84SMDyirarsxHFqYSz9ChQ6c4+Yu91Y=; b=utWzzzuu7pW+i4WjlX+uJfjn/OtfjXkk9qr6XV1Fn9+xTGs5FAK/mTd6opXoyjWtsDm9yw fCGS+8CyfXa+6Cm3Z3nSn9GCY2k0GORy80ZvtYc59M+yA1MzUsDpzVtiQsSAnZ9xIb3hqZ QAKBs5vYtUTexbkTbBIwb/SS9H46sIg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=YvmsvfdR; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf02.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692629850; a=rsa-sha256; cv=none; b=zLOvj3rJtgqx4BHtjTEei07PHuPPv5Z6fpX3DJgfdUr++emIV7GWaMzHsRRUTEenk3dACd PTTC9bRPQMjEex0+urCjoX9e4kadGLB2pI+RMgYTh1E8jqS7sapbwZXCuGrOnbROp6cHO+ c4PaoF3DUvprT/8UkyZ2fPjwzvFJiL8= Received: by mail-vs1-f49.google.com with SMTP id ada2fe7eead31-44c07c2b89aso1209591137.2 for ; Mon, 21 Aug 2023 07:57:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692629849; x=1693234649; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nLIL9tDKiB9E84SMDyirarsxHFqYSz9ChQ6c4+Yu91Y=; b=YvmsvfdR9S4nMYWDtVjmdw2/IaQx4w9dNZhQ9XCUfgjEtxcIZgsLknT55WGetIrGoI K2gC+cFFigORLGXORJFHUcQlY8i1B/8M6rlKEi30cImS7bGTT8ol4RWFFRUKP609qB/E bGaT0+8I6wc14aMrol0cp2CIPntQXvIpef60KM5m2dFVcohrFkXPCW3zE7D5CqXWSRj7 rye9HkCp9yIVIZ0WW+aMYwHKYpRsZEcRHcVyvPaNWQfvxN1x+0fp2mLDtp01c0St+/PP TP/RAdazsK/kwaSVoG2xtkf9upCpEF1MakU8Qq5hMaeZQa3CxfZZu7zb3NYQ6JLdTCmw aDwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692629849; x=1693234649; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nLIL9tDKiB9E84SMDyirarsxHFqYSz9ChQ6c4+Yu91Y=; b=HEcXjMSIAfIFZPamE5ZwWlVNV3784Ku8WskOfYYqm4LVuzmTk/K2Lin7KEQw/D5T2u JPPd8YUFr5p134dm8dI/Af4m0j5n7rNPHaBf3PHzVMwd0psR7lRN8DJ70vhcxFq4GLqF EmSrueGknl3T3f1zgnBeNyXNdRSSHJOd6xA4odY67C3n3EbqgBUnHrAlWGN744FVKhfL syKHizJ+Mhl+hOTokhkZGW+JVk+Q+oRjC688ZJFEE4PF0wVONldZkedWjI73BVGxAnWB Ntm7adofAWh3z4DJNatwqWZKmcYwkK96UJ78pS4pkMuSyqrj41VSqOCq08VSVOWP1Qus FbOw== X-Gm-Message-State: AOJu0YwpYq3dEDHLzda+pgAHFU0je8lnh2Okb7oMYs7PuH0xfV53pyM7 avCJaHYziimrt2Wkb4cl9nKctAj5RhoY8lkhQUs= X-Google-Smtp-Source: AGHT+IHzSJSubGnf2sksfrV5HKflo5QELCalkpYWSUIb/VAiNxthIV8NoWrXoMop6ham8TxSGrL70EExZwi0sFvobAk= X-Received: by 2002:a67:e915:0:b0:444:c7fa:1aad with SMTP id c21-20020a67e915000000b00444c7fa1aadmr5929948vso.17.1692629848931; Mon, 21 Aug 2023 07:57:28 -0700 (PDT) MIME-Version: 1.0 References: <20230810163627.6206-9-vbabka@suse.cz> <20230810163627.6206-11-vbabka@suse.cz> In-Reply-To: <20230810163627.6206-11-vbabka@suse.cz> From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Date: Mon, 21 Aug 2023 23:57:16 +0900 Message-ID: Subject: Re: [RFC v2 2/7] mm, slub: add opt-in slub_percpu_array To: Vlastimil Babka Cc: "Liam R. Howlett" , Matthew Wilcox , Suren Baghdasaryan , Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, patches@lists.linux.dev Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: ED87A8000A X-Stat-Signature: ux33opcxphxhz18kst5qah1hfbco7px8 X-Rspam-User: X-HE-Tag: 1692629849-308076 X-HE-Meta: U2FsdGVkX182lz793lUXzOOPvJdJFTgdrYilLZ6xEZ8xxgToSLExXyE4p8pGs/sKbnoyh1+8XCP7viCQLGV/gSMPIxqhjld39Gtcbs58V8r2hZQOwSZ+zwG2MEAkPj0W8xQQ9h3RMnsJ+pxTONXg9QqjpgidEdx6inSCVEItxxRVjmUsoZhAcicX8sBXA6+E+lo/6HSSG+6FpMIKFcC1+0ws/8C1vK7HPj4rwc66dPGa2P7cm83uvg8yONtmheJbJnONVY9iIzh0hKLzCUgkdyDcq57Ewpbe9orlJZ3SRvl4qaTNAZZ8+/M8FQMxjCOFie5Xmbt18KF3l+7L0DkMr71P2Z1rNQayS1AFt54JVFF/F2/v/ELLzLfPvDn6YGMSqFf5xL51Q8oVf5dTOgNuS2lqz6g7JBfMEb1S59TfcYEtXxbGVRheBR62pQsZH/zdmkX7pCVPZ54I7hWy3d1I351iV562NsA06Zz1fEOhGHqVwwcycCA+XUxinC7RzzADmEksuC/naag5ru5iv+gdRoqh4+znjscv+xdOiMvQ/stZpIxXkAWaOE980y/SacrOGbkRGq1p8FTSv/G9uUBOh/atuVWv6FmDLjSmQtQoOT5acgNCN7wFk29wQIlKAO54JAm1Hf7lPYxrjJO5ehBIrENrsuSn8ttvA5Tp8T+9NrCU7tJmUodIgqp1qmdLfA17vgcEjjgfY2SkHXTKBj3ifHP2JBv3nPa/BdQ/fIKE7HJcxANwlvLs7w8qx/V4vviZb7JdzByC/lRt2QXYhULWoGx9C2lHAfw0YuWCtYZ5VgS/rBZMKuFEWTZ+8RJo5Zu4HnTFiZ5lnsw9h2t5LP1FwR26eqz1ltiumzSuzlTRhF0PhW7qwxVbTCnbmE6pl/fLCHgsTPqqExOkw41h3sd47xvxEjB3DRM0YZI4MZG2JZ5n/WBixGCHr1xGuKC1I06Dk6F5HMBs6UC2O8npYMO 1KEoHIR3 esRrW43ivoSwXR1JuJdhiPBbOrgiJX17IIku5JbUHAurscF8G3XI4UcMcJizI0Aq1y/lTk8LStzT2WOtDaft6Ac7hcbhn8pnr8/bbqOoafXVrk/E1F5Z8BOLVZZIZiXpoyCQxNTVe7JO7Bn/hWxQH82tnixYEoS2rO17yuQxbkdBP0ukkEeiby6UIxtqnD0cS4hdVaKDlukBjtoW0JgwJCIiQW/J15tvobJaU2+REq40ehzlkHWr8SeHcrw63SP44iWTiNBQohyxU1Er2e2t0juUA0HiiLmMOvNapYWf+/FlgPQvxCcwICdeVSGPZp9s8EDVlJQ39rWuGT1q2kWjzovWlcvWd4jKvp42SCMri9wIKSLKQ4CV3GIH8SQ0/1GigAglT/rt4xlQlIoh79CAVJ1pDpfcXFVikiwMcOOLDbgChYzzr8uugmnSTGw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, On Fri, Aug 11, 2023 at 1:36=E2=80=AFAM Vlastimil Babka wr= ote: > > kmem_cache_setup_percpu_array() will allocate a per-cpu array for > caching alloc/free objects of given size for the cache. The cache > has to be created with SLAB_NO_MERGE flag. > > The array is filled by freeing. When empty for alloc or full for > freeing, it's simply bypassed by the operation, there's currently no > batch freeing/allocations. > > The locking is copied from the page allocator's pcplists, based on > embedded spin locks. Interrupts are not disabled, only preemption (cpu > migration on RT). Trylock is attempted to avoid deadlock due to > an intnerrupt, trylock failure means the array is bypassed. nit: s/intnerrupt/interrupt/ > /* > * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_al= loc) > * have the fastpath folded into their functions. So no function call > @@ -3465,7 +3564,11 @@ static __fastpath_inline void *slab_alloc_node(str= uct kmem_cache *s, struct list > if (unlikely(object)) > goto out; > > - object =3D __slab_alloc_node(s, gfpflags, node, addr, orig_size); > + if (s->cpu_array) > + object =3D alloc_from_pca(s); > + > + if (!object) > + object =3D __slab_alloc_node(s, gfpflags, node, addr, ori= g_size); > > maybe_wipe_obj_freeptr(s, object); > init =3D slab_want_init_on_alloc(gfpflags, s); > @@ -3715,6 +3818,34 @@ static void __slab_free(struct kmem_cache *s, stru= ct slab *slab, > discard_slab(s, slab); > } > #ifndef CONFIG_SLUB_TINY > /* > * Fastpath with forced inlining to produce a kfree and kmem_cache_free = that > @@ -3740,6 +3871,11 @@ static __always_inline void do_slab_free(struct km= em_cache *s, > unsigned long tid; > void **freelist; > > + if (s->cpu_array && cnt =3D=3D 1) { > + if (free_to_pca(s, head)) > + return; > + } > + > redo: > /* > * Determine the currently cpus per cpu slab. > @@ -3793,6 +3929,11 @@ static void do_slab_free(struct kmem_cache *s, > { > void *tail_obj =3D tail ? : head; > > + if (s->cpu_array && cnt =3D=3D 1) { > + if (free_to_pca(s, head)) > + return; > + } > + > __slab_free(s, slab, head, tail_obj, cnt, addr); > } > #endif /* CONFIG_SLUB_TINY */ Is this functionality needed for SLUB_TINY? > @@ -4060,6 +4201,45 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gf= p_t flags, size_t size, > } > EXPORT_SYMBOL(kmem_cache_alloc_bulk); > > +int kmem_cache_prefill_percpu_array(struct kmem_cache *s, unsigned int c= ount, > + gfp_t gfp) > +{ > + struct slub_percpu_array *pca; > + void *objects[32]; > + unsigned int used; > + unsigned int allocated; > + > + if (!s->cpu_array) > + return -EINVAL; > + > + /* racy but we don't care */ > + pca =3D raw_cpu_ptr(s->cpu_array); > + > + used =3D READ_ONCE(pca->used); Hmm for the prefill to be meaningful, remote allocation should be possible, right? Otherwise it only prefills for the CPU that requested it. > + if (used >=3D count) > + return 0; > + > + if (pca->count < count) > + return -EINVAL; > + > + count -=3D used; > + > + /* TODO fix later */ > + if (count > 32) > + count =3D 32; > + > + for (int i =3D 0; i < count; i++) > + objects[i] =3D NULL; > + allocated =3D kmem_cache_alloc_bulk(s, gfp, count, &objects[0]); > + > + for (int i =3D 0; i < count; i++) { > + if (objects[i]) { > + kmem_cache_free(s, objects[i]); > + } > + } nit: why not for (int i =3D 0; i < allocated; i++) { kmem_cache_free(s, objects[i]); } and skip objects[i] =3D NULL > + return allocated; > +} And a question: Does SLUB still need to maintain per-cpu partial slab lists even when an opt-in percpu array is used?