From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B43E1C004C0 for ; Sat, 21 Oct 2023 14:44:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E80068E0003; Sat, 21 Oct 2023 10:44:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E090A8D0008; Sat, 21 Oct 2023 10:44:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C810A8E0003; Sat, 21 Oct 2023 10:44:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B5FC18D0008 for ; Sat, 21 Oct 2023 10:44:14 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7DECD1CAB12 for ; Sat, 21 Oct 2023 14:44:14 +0000 (UTC) X-FDA: 81369738828.10.31718EA Received: from out-198.mta1.migadu.com (out-198.mta1.migadu.com [95.215.58.198]) by imf26.hostedemail.com (Postfix) with ESMTP id B5EAE140004 for ; Sat, 21 Oct 2023 14:44:12 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=VwdHbGN6; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.198 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697899453; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fc6duZ/q+apDqgoP4cLzy3p5t8cD8/eKarQEAhD6cwA=; b=Ffw4eZXWqazZfogKGK6WXuixx33Lk4XcACunVBopv9rhF8keubGOIWg8I94/WheQ7+yfIm DAp9oLy2HUpqjHRhdUWpVw0Fz1l6QWYNVP7cPb/D+dbJlIW1V0z6Ct6uXhesjwvw9Puph4 RQGUB3vztt7TvsfsLpyxlY2XxYn6NL4= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=VwdHbGN6; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf26.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.198 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697899453; a=rsa-sha256; cv=none; b=MylPzDrjuMuu+iAN3Dtx1LvOX7OraThMpARztu5rZtwap7c5ho9iF1XXRwuwK6v0Y1uCE9 ZimYhBTUErw/CrJHoZ1K0S76bCe5xOTHsvHWTfmIkDpqJdrVMAB5R7m50EelOZ2wxcM1OP mJYWbE/gVVJkU9N7wECOnTtOn3VUKME= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1697899451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Fc6duZ/q+apDqgoP4cLzy3p5t8cD8/eKarQEAhD6cwA=; b=VwdHbGN6iqPqOPPmjkoCZCqeya5Wrnfy29DfCh17oso4mTQaIrjGQ/bBj7cuImP6Ak0ZDt 74Ji4pLY0KWggdSHOhw8II53OTlr2K3tbgg2NViY04lO63dYbYrl8s/FsQ+pKx4DbOjdEG yYcnIac3YU/gH6s55G+958BKYZfu70Y= From: chengming.zhou@linux.dev To: cl@linux.com, penberg@kernel.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, willy@infradead.org, pcc@google.com, tytso@mit.edu, maz@kernel.org, ruansy.fnst@fujitsu.com, vishal.moola@gmail.com, lrh2000@pku.edu.cn, hughd@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v2 3/6] slub: Don't freeze slabs for cpu partial Date: Sat, 21 Oct 2023 14:43:14 +0000 Message-Id: <20231021144317.3400916-4-chengming.zhou@linux.dev> In-Reply-To: <20231021144317.3400916-1-chengming.zhou@linux.dev> References: <20231021144317.3400916-1-chengming.zhou@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: B5EAE140004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: rwjrsc6hm14wbca1s56ymz98bthmjzyr X-HE-Tag: 1697899452-839369 X-HE-Meta: U2FsdGVkX1/2hZ5cMHBvEgxI49UjK+v1UioyNDwDj3Eux97d35cBg9eeFprTWp3eOjqF4Dmyh80CI/VPWb0u0ahuhbW/Jyz+D2sAvOdVe2IUgGED0Lwj0/N9bKRjDPg3vaGLdibT0GMm3iFKhEDAkpa7Lj7oueB+DzNiwo7TvcKs6nD8CaY7jLWs1DjyGj8BFvBhDFy93lirOeUT89/yyTFFKktekC1uet9spE3Z5yJr+QWtnJhlShVZoirS/ZTvceMwfCR8qvzc7tCRSIcb9sD9cMhNXica+fzUY3PbL3nsHAeaodhqw2uPW/LD2ztWPgZqYplXH0MoRuijl4mCGZMLqnUrtx8lNIKynHsjXsb6eLA9cw8otwvsXy1YZKn7VulD9DfsOQWL3p/M3IgocfISwCks62w7oEwZUoK7Wl77fPoebVj7qHHIae3HohTu3aNyA/BS4gubBbJCEo0/mJ2rYBmOcj46H2yKsrjojgL5wF/B5U+61OSvRsZINc8/4kcoRfDyVhgh21i+GBMopNPo57DmkL0r973EVM7+X4cLNlMPDlhd6aQzUsYnsalEq61FU6G6eNib9XYYzUN0RLVHHf9EGI3nV/m/GNWS0z7zubHOPpCCCFUpSJrNSzQHRNRURs65DKas6PRSTIzKYWAEAaluxi0+CxfR+gftV9Ee1ERa6fX0oAnOPFrNAvM9FfnsAKiJiU0QgcqFEmpj0QKvZGdnC8FmxNVqtaoTR6n4/MLyfWGssR1j/Uf20fYrjkqS9+RAdjuT17h7FzJRB8qNWJsPQV+9mdkhH56e3KKcjZbkcvBA1T50cEc8nzRteRvb7lo48HQhPUuKohyRfCLwZONImrmdXtEt7rwXRoje35RJmip5tbCG3ZBo7ZralmaAVFimyv3HRsjgb81VivlBCFtlfs9JysT0pUKZ6X0Hpbho7OmKMTy22DYtTS9nd/i8xFp2yvluQPETWKA tHsauTLX Z6PpCTiPp/t2u+yp729sF4uMv312KAVmGiXF/67f9ai2Djxu9FZ10hHN9VP06Myl23gj44di323btgdWlEOAXvCFwPSaKmg6bPpb4NjSKDWcvrWWXfy+wjd0COVP4uVz8hYOBYvp7xhyd2V6oGAREG881tjrKGm5yT7YnEmcKaJs6cf9z9WPMxUUWX81ZGKGQJ3gnX4VCDp1Mwnk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chengming Zhou Now we will freeze slabs when moving them out of node partial list to cpu partial list, this method needs two cmpxchg_double operations: 1. freeze slab (acquire_slab()) under the node list_lock 2. get_freelist() when pick used in ___slab_alloc() Actually we don't need to freeze when moving slabs out of node partial list, we can delay freeze to use slab freelist in ___slab_alloc(), so we can save one cmpxchg_double(). And there are other good points: 1. The moving of slabs between node partial list and cpu partial list becomes simpler, since we don't need to freeze or unfreeze at all. 2. The node list_lock contention would be less, since we only need to freeze one slab under the node list_lock. (In fact, we can first move slabs out of node partial list, don't need to freeze any slab at all, so the contention on slab won't transfer to the node list_lock contention.) We can achieve this because there is no concurrent path would manipulate the partial slab list except the __slab_free() path, which is serialized now. Note this patch just change the parts of moving the partial slabs for easy code review, we will fix other parts in the following patches. Specifically this patch change three paths: 1. get partial slab from node: get_partial_node() 2. put partial slab to node: __unfreeze_partials() 3. cache partail slab on cpu when __slab_free() Signed-off-by: Chengming Zhou --- mm/slub.c | 63 +++++++++++++++++-------------------------------------- 1 file changed, 19 insertions(+), 44 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index adeff8df85ec..61ee82ea21b6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2277,7 +2277,9 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, struct slab *slab, *slab2; void *object = NULL; unsigned long flags; +#ifdef CONFIG_SLUB_CPU_PARTIAL unsigned int partial_slabs = 0; +#endif /* * Racy check. If we mistakenly see no partial slabs then we @@ -2303,20 +2305,22 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, continue; } - t = acquire_slab(s, n, slab, object == NULL); - if (!t) - break; - if (!object) { - *pc->slab = slab; - stat(s, ALLOC_FROM_PARTIAL); - object = t; - } else { - put_cpu_partial(s, slab, 0); - stat(s, CPU_PARTIAL_NODE); - partial_slabs++; + t = acquire_slab(s, n, slab, object == NULL); + if (t) { + *pc->slab = slab; + stat(s, ALLOC_FROM_PARTIAL); + object = t; + continue; + } } + #ifdef CONFIG_SLUB_CPU_PARTIAL + remove_partial(n, slab); + put_cpu_partial(s, slab, 0); + stat(s, CPU_PARTIAL_NODE); + partial_slabs++; + if (!kmem_cache_has_cpu_partial(s) || partial_slabs > s->cpu_partial_slabs / 2) break; @@ -2606,9 +2610,6 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) unsigned long flags = 0; while (partial_slab) { - struct slab new; - struct slab old; - slab = partial_slab; partial_slab = slab->next; @@ -2621,23 +2622,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct slab *partial_slab) spin_lock_irqsave(&n->list_lock, flags); } - do { - - old.freelist = slab->freelist; - old.counters = slab->counters; - VM_BUG_ON(!old.frozen); - - new.counters = old.counters; - new.freelist = old.freelist; - - new.frozen = 0; - - } while (!__slab_update_freelist(s, slab, - old.freelist, old.counters, - new.freelist, new.counters, - "unfreezing slab")); - - if (unlikely(!new.inuse && n->nr_partial >= s->min_partial)) { + if (unlikely(!slab->inuse && n->nr_partial >= s->min_partial)) { slab->next = slab_to_discard; slab_to_discard = slab; } else { @@ -3634,18 +3619,8 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, was_frozen = new.frozen; new.inuse -= cnt; if ((!new.inuse || !prior) && !was_frozen) { - - if (kmem_cache_has_cpu_partial(s) && !prior) { - - /* - * Slab was on no list before and will be - * partially empty - * We can defer the list move and instead - * freeze it. - */ - new.frozen = 1; - - } else { /* Needs to be taken off a list */ + /* Needs to be taken off a list */ + if (!kmem_cache_has_cpu_partial(s) || prior) { n = get_node(s, slab_nid(slab)); /* @@ -3675,7 +3650,7 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, * activity can be necessary. */ stat(s, FREE_FROZEN); - } else if (new.frozen) { + } else if (kmem_cache_has_cpu_partial(s) && !prior) { /* * If we just froze the slab then put it onto the * per cpu partial list. -- 2.20.1