From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CBE1C4725D for ; Fri, 19 Jan 2024 03:53:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C1BF6B0078; Thu, 18 Jan 2024 22:53:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9716D6B007B; Thu, 18 Jan 2024 22:53:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8393A6B007E; Thu, 18 Jan 2024 22:53:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7479E6B0078 for ; Thu, 18 Jan 2024 22:53:22 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 458DE1201CC for ; Fri, 19 Jan 2024 03:53:22 +0000 (UTC) X-FDA: 81694690644.24.9FA1F6F Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) by imf04.hostedemail.com (Postfix) with ESMTP id E69A240003 for ; Fri, 19 Jan 2024 03:53:19 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jjAi4S2j; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.161.53 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705636400; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GxoT1U8gk6NApziuwmbCdtrtydUDrNAn08O/sRw2V60=; b=Npuw7lrZea9afz/yljsvr62G3X/q3tULjs+3SAutfOCMRbV56ZZwObqU6ByVxCLw5ou+Rx SpIjfY1xpuOdc7WaaTQzqxaoiSZrQp9bVMWxqiKDknlTqkSIHxCU98UCZhj2n4AuHgYenD 6xYrAkClkrU7nwOaGgnFFU0HddvJsPI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=jjAi4S2j; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf04.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.161.53 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705636400; a=rsa-sha256; cv=none; b=ukVj0f0uWq07sGQqWZoTGd3Yslv6WFnKRb8g31WZ6mGp2ScmBYiuVHA7LIKz25Pc6uoHhp SjbCmXDXfxbSLv614DM8HH7GeDljCW0EYjFmv+aaoJ/rXmHzGSNlMUGmdR7xft60iyyYGR woXN+RnZ8/FbQvdvdqXyOQ5/koZrYiQ= Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-598bcccca79so169237eaf.2 for ; Thu, 18 Jan 2024 19:53:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1705636399; x=1706241199; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=GxoT1U8gk6NApziuwmbCdtrtydUDrNAn08O/sRw2V60=; b=jjAi4S2jnFrzqy/+QBf34uovcvCS8w2y8ffGUfR8hjd/WwXR4yrI2WDcKjJ/OhLSFf rz9p5TXIHYVehttga3mW6i10MIdhzE8c7DVvUg3F6e9cEnHsmwQeAB8NvkGF47IULHwC 5BTIICxic7PtPt/9uJlzGMxn2R75txWchmvTu9M79B88lT9wR2cznLv5Asbwca5KH0QC v1NArWsInL5jSTY9eyZoAD3TnqeaHKvwBUkBEkS0KrS0xXsWMVA3ZjR5NyTuhmPsq9YQ 1md3eOiCOj60XgJIFngMRKYAbwzmuP09Q4FYkEqhq5dAYy9VFJW2rlddfpFXOQfPCfbr UKrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705636399; x=1706241199; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GxoT1U8gk6NApziuwmbCdtrtydUDrNAn08O/sRw2V60=; b=SPtLAHYvOQohPFe3004qmCdciSoRTLdvHlCtXWI1OwjvDk5B1sxHJGmkN9Zr6tMD8I 1Ni+BPjvuIpTr7EF3ZLrDqK90UwpnEUrE4jTOy9H/PKISyB4YK1guyuzxs37f2aDIz0M BmyPJIwaAZ/jAZLCeGvGaqlDy4NPO0ACsFYQq17VKNfBuqxtEgsDeq6GJeiol1f6mTNO SeQ8TFAixKaZKMdqW0m1I4STL+aff70oiUcJfdKOlOXjZAnAM5k2WSNqklqpxK+Op4Rm DmF+mkKFsaO7mpPJtHg18tO/3erjKbA6nJIruRntAU+ET7Pmwa5WlA9xSkkUg2F+On9U 6/2w== X-Gm-Message-State: AOJu0Yxo9hHqIGPQPYrxteiEqj+25F+KnAdP/qCkAHyAW7eekHADK+Mj LMHfn15prdRIRrNKScOkkKmiL/SzyEIXK+jP6o4ihfVEL0iJQoNL6CrPi4EJUyA= X-Google-Smtp-Source: AGHT+IHN2GN07QZW5OBpL6BeZIde3k93KSIQdDeCP2Iuv+t5ZG/8NoyDRjDH3OU+/gccq12hDgNxcA== X-Received: by 2002:a05:6359:b87:b0:174:c9a0:cf99 with SMTP id gf7-20020a0563590b8700b00174c9a0cf99mr1694951rwb.57.1705636398851; Thu, 18 Jan 2024 19:53:18 -0800 (PST) Received: from [10.254.224.1] ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id pd5-20020a17090b1dc500b0028c5585fb41sm2705415pjb.45.2024.01.18.19.53.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 18 Jan 2024 19:53:18 -0800 (PST) Message-ID: Date: Fri, 19 Jan 2024 11:53:12 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/3] mm/slub: directly load freelist from cpu partial slab in the likely case Content-Language: en-US To: "Christoph Lameter (Ampere)" Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com>, Joonsoo Kim , Vlastimil Babka , Pekka Enberg , Andrew Morton , Roman Gushchin , David Rientjes , linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20240117-slab-misc-v1-0-fd1c49ccbe70@bytedance.com> <20240117-slab-misc-v1-1-fd1c49ccbe70@bytedance.com> <76641777-1918-2b29-b6aa-bda9b5467aa3@gentwo.org> <412b8618-0941-4d9d-85df-ee480695e7f7@bytedance.com> From: Chengming Zhou In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: E69A240003 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: q96um9g9ue3j45gbt83frdzmb8io3nrb X-HE-Tag: 1705636399-637060 X-HE-Meta: U2FsdGVkX1+SB17mxDBkI/q/BOklbZwgHqcGCOJWnt/lWZPLM7VGqtRF1tvwKPA4DO+GjvnBkRq1sUrVQI1PjP2a1TkHfO4+FT2Thcxyh8NZWC835EJm7yR/4ZR4vK3qUeZLWNGdB6r6vNgedKS97glnb8QXJ30OEKN0mpvVhkI+xbAM6XOOncBwfRn49KET5qOU0LknbUT4IOEBiJX4xUc4OZncTUHVZmFoqhuHjSLigRIu8BEZ8N0f9//+K68VVS4nDdFnCTdwbpXrhQeeU+DScZYM8ZqX2nbvBT0lkEY56CYeikOlT6meLM9uAkFOSXeKpLLUpOyAGTlAFOm2/AlAO7jgBWn++vXOLvh4NhobqmSa8gXsJcpep5e6ERWfl2+/1pgIESixbLsiQ9IoEEXOrrDQBEamoqcGQqucPW+GREMiyIzp7MTsXOBRBzvogKwfAvQ2v/JAL91qWq3IwpUxR3exuT97Ld7S2VVu2v7Rp7DF1BGyoUkvSZukZM6Ilc12UxNQRZk+z0Z4T9MqZGUPYJjigccUy3lS6ckOc9I8OwYCAmUzRYj7cAsDh2UzbmQ1wrgXNv70zpEvWsXO2WmoYn5odKKf15s4XcioEL196cq+7MeS8z0l0ZN6vmFtFmLp6q+SWDa06JeXNIUEDlHOY4dak/o4LEAWo5JQhN8dJjOj9+0YvENZ7Hc87mAQECEWZYKwT9LCKXd/tMGvJtMxHSmRaa91IXK4J7jROQueArJ1Ea5ySrdviaOEExN7Dncw4qq35F41cUo54/eR5YZSNYaIcTzbtKlfRgbvThCxBZuMXxjsnYgckZ079o0ATGsaiXvWCIFVjh/NGJT6WmQZ1NKA2E+werc9p1GR+h54b/lNSe9DNKZNsSAwPFATdin5n0KHmxy/AMP+t2yJjiunSggwx1OndP102HMttCH+DfLAa50usVpd0JPI33M20IokUvuobahPOSpe9Yt Lz9gFq52 IpQzpD97TgAfigJ89FKgOg1Vxruy4jVXoulzHh3hrmto67L/x+ir/A0Ug+pDGwD6oD4edcoeC9S+p5QFgUj18XumE4J5z4GccAzqgA3OOAhL/egOcdofBooog2cHw6Md+NrTIs0RV4CY1uY1RF8txRdADXbTDLhhU6YREZO+lXNVx48nisin2bk4m2sp8IjuBqDoFjVegIt8RC+C1214o3VJdFqgyQyLsiOWpqnl9mA7yy9IoL7TEelX5fG7oev9MgywuHw157UhB7tCORgT6uF7jaPuO0F9R6ce6vgswIvHs5H0IbeDgiULcybSPrGuUMQQvG71h8YFKrUbOhof1pHD8kd4I9u6OljEKluvi89Ur+jXoQZEOFHI05pond+urKQCqpScFYZMZ07bqHMbWUH9O5dA0gbpfUY7dPoraMyu/zqF7f+WYU7cRHDPLLZJKwgoXqKPE4Ln0vBk1zxawrG75N2Z48Dnu5Lii X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/1/19 06:14, Christoph Lameter (Ampere) wrote: > On Thu, 18 Jan 2024, Chengming Zhou wrote: > >> So get_freelist() has two cases to handle: cpu slab and cpu partial list slab. >> The latter is NOT frozen, so need to remove "VM_BUG_ON(!new.frozen)" from it. > > Right so keep the check if it is the former? > Ok, I get it. Maybe like this: diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..7fa9dbc2e938 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3313,7 +3313,7 @@ __update_cpu_freelist_fast(struct kmem_cache *s, * * If this function returns NULL then the slab has been unfrozen. */ -static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) +static inline void *get_freelist(struct kmem_cache *s, struct slab *slab, int frozen) { struct slab new; unsigned long counters; @@ -3326,7 +3326,7 @@ static inline void *get_freelist(struct kmem_cache *s, struct slab *slab) counters = slab->counters; new.counters = counters; - VM_BUG_ON(!new.frozen); + VM_BUG_ON(frozen && !new.frozen); new.inuse = slab->objects; new.frozen = freelist != NULL; @@ -3440,7 +3440,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto load_freelist; - freelist = get_freelist(s, slab); + freelist = get_freelist(s, slab, 1); if (!freelist) { c->slab = NULL; @@ -3498,18 +3498,19 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, slab = slub_percpu_partial(c); slub_set_percpu_partial(c, slab); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); - stat(s, CPU_PARTIAL_ALLOC); - if (unlikely(!node_match(slab, node) || - !pfmemalloc_match(slab, gfpflags))) { - slab->next = NULL; - __put_partials(s, slab); - continue; + if (likely(node_match(slab, node) && + pfmemalloc_match(slab, gfpflags))) { + c->slab = slab; + freelist = get_freelist(s, slab, 0); + stat(s, CPU_PARTIAL_ALLOC); + goto load_freelist; } - freelist = freeze_slab(s, slab); - goto retry_load_slab; + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + + slab->next = NULL; + __put_partials(s, slab); } #endif