From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1338AC83F1A for ; Thu, 10 Jul 2025 19:21:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9500E6B008C; Thu, 10 Jul 2025 15:21:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FF4B6B0093; Thu, 10 Jul 2025 15:21:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EE1A6B0095; Thu, 10 Jul 2025 15:21:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6CDB86B008C for ; Thu, 10 Jul 2025 15:21:53 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 06553160E5E for ; Thu, 10 Jul 2025 19:21:53 +0000 (UTC) X-FDA: 83649324906.13.202B221 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by imf27.hostedemail.com (Postfix) with ESMTP id 1530540005 for ; Thu, 10 Jul 2025 19:21:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IsxBvXvO; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752175311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qUiBFyM0Yn00R3t8T16IwBYSTuDffC72F3UZOtwQCEE=; b=F/cOGbAWxzThuZik1qki2ryIztF/bjk+vizurkUFgNMadqyyS7QyptVHvOnc6qaT5+X9TW 5TCXqI22FTH7C5gPrilL9KrWxvTANbFQmxhEYG/xS5xO8yrboKAOwnCzsXuJ1Yb6QLhk8C IjObmbpIlHhT94j+Q7oaC+goDmBlk3I= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IsxBvXvO; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752175311; a=rsa-sha256; cv=none; b=I8PHGoIV8yVoHfOMDclCbGd/jzsFAV40Q+IgKGFNRj/mhHfCUGjtSOCvk2jj1aTE2yZ7RL 3uMr9oz72zCJsVCezh876yzw747INsVbjr2BVzLnNQ52kTHvZzgALZUacf3DBvzZjcUo9B WhcrhaqfjCfi/qRrNy9VTDhsIeYoLU4= Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-453634d8609so8571665e9.3 for ; Thu, 10 Jul 2025 12:21:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752175309; x=1752780109; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=qUiBFyM0Yn00R3t8T16IwBYSTuDffC72F3UZOtwQCEE=; b=IsxBvXvOBSrImKmmiySFWb0TyYBzAxsxjoL4dkKUXsTc4wiJhyxnR0rEMvKBITwaul FnqNVE6KYr2k6xMER9yDQiBrQmgv6OvYw3iq8z1dVKlu/JF8q9lzMhNnzDWsHaPBsmMf YS0eTQ2otiHKHCSlLOMFdLZtK3NRlPrAttpnYYsmyrL/jko+lNqXFezTr/0Nb09wBR+g fyNj6uv4gDhFNDCv+nntje4+ZeBkKF4Zb0VUSESBkg/M6L94Xr0HVU+q3s39NcFxK1Ij 115Ka9Em6iO4eGYU7VPRThm7lvDEJyC2UScmEUmEGc6fIuYHk15oLPRlNNat0TMzsOfE SEpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752175309; x=1752780109; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qUiBFyM0Yn00R3t8T16IwBYSTuDffC72F3UZOtwQCEE=; b=BTFGL9RnGjO3YubeuuiVaPDlEgxfH+6GItJP+ALe8dg7CGlbqy3RYjQislbAi7ckT0 maojyAnbw6WL2OFW39odF/CaqZO/p6/U2G5nlcMNOnvU94PW9ElOf9fj623JVj9Hv2G7 76FDdsgk75g4utF3CIbjQGL+F09znD58X6Ju1qomLFUnX524OAYgZhqByi/4aaSA80II tB7hRMkpYdyR4yoD7gnUwI2kgV2mBfmrlOLarEMwJ0Eex7H6XPDzaq1tEa9fHDy/tFM3 kiEnE0LLqtPwsVmDJBeWJKoF/oUqE9mWSvo8eiWcLff1URH5FGEAHdzr0juWi/66Xq4R LTOA== X-Forwarded-Encrypted: i=1; AJvYcCWEkyQLSRqcXuDA/qnotSGUlo5/11lhs1Zq6CepG3mwFVxuqrP4UBaZaX2HvShPPsqDcv7F5e6mHA==@kvack.org X-Gm-Message-State: AOJu0YxUtKpJPJu/lvT/3wQZvXrWivqP1WeY52K8uXIYau/LseZ5bAfO dyNXOnz/ZOLsEXPXfivRqgKuTCmZr3eolrboM5A1ddEYjH1b72dD/rJ2wQchG2ZG64lsqPRQtDB xU6TKfM9kQbPwMjtO61cktNWLh5D+1uI= X-Gm-Gg: ASbGncsPWG3m3bVd7bwoxHwKjfOyBNPPgWtsJ1qpGelWz7TgdcDT4fuaGanJnKj2zBR IikVahhXkUX1dRDFF2Tl63aX0H35ykCB+CNKRQoNnOWJG8/4mYz8KhGZ21OAT/ii0uaDfWlHj2Z vOuB7YkAdXIGb7YkYiHMEignywC4fecyixoMbjpk544WEfzEtbudPjm1dALaIs7EjR2y4xUZUPy dTqTtUoauE= X-Google-Smtp-Source: AGHT+IE8J4fA8ruUMSURSR3iBDBJ6W96CyD0YH06p+VMUk8sYNYqP6QxPlwCGMKXkjHfQ0nc4Oo18etF+PbrbO1oViw= X-Received: by 2002:a05:600c:3b82:b0:43c:ec4c:25b4 with SMTP id 5b1f17b1804b1-454ec131ce1mr1027385e9.10.1752175309353; Thu, 10 Jul 2025 12:21:49 -0700 (PDT) MIME-Version: 1.0 References: <20250709015303.8107-1-alexei.starovoitov@gmail.com> <20250709015303.8107-7-alexei.starovoitov@gmail.com> <683189c3-934e-4398-b970-34584ac70a69@suse.cz> In-Reply-To: <683189c3-934e-4398-b970-34584ac70a69@suse.cz> From: Alexei Starovoitov Date: Thu, 10 Jul 2025 12:21:38 -0700 X-Gm-Features: Ac12FXxYKFCHpSKbbo5umlXWioHW1RzkGHfC30HxTA9RgRLit_mjyvQAYY3z4dY Message-ID: Subject: Re: [PATCH v2 6/6] slab: Introduce kmalloc_nolock() and kfree_nolock(). To: Vlastimil Babka Cc: bpf , linux-mm , Harry Yoo , Shakeel Butt , Michal Hocko , Sebastian Sewior , Andrii Nakryiko , Kumar Kartikeya Dwivedi , Andrew Morton , Peter Zijlstra , Steven Rostedt , Johannes Weiner Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1530540005 X-Stat-Signature: 9jq3stquzi3fratzicxxokyucdb89h87 X-Rspam-User: X-HE-Tag: 1752175310-488264 X-HE-Meta: U2FsdGVkX19ltPNt8TuYdcrO2/n/KSa047UUuHTSQISxMhLQC3+vBlGuAAj6HX8jP83nmAhfE432uzuJiJ1jcstw5PabrO0tgHcAKRYG3PCdGVjgcZKhvKZYK2TMSHhCTIeOpwCZ6Qazup3KK/fwp2HlZ7S4M1E38CM+w2bl6iWk/1N89MVaGjk50iBIGECe9NQQaZ5iCIEUgj4hnMNKGsqC4wG1wKE1lqCqhnagD/KeHgSW83Nha5dE9ZavYOkLJZCvo1yFD2xqixTW42AVIXBRUAlxw2Lq/ZgkPR75sMYIze5oDmI1LyyFvLn0PZ3exRizN/xJSm8ZjWqJ9o27i/R4NwqrrZbDQidYFtlfj/Z/bmjduDCtDVFTXxI7uqf7EjXa84rCrZFpVB0IUhJUZt0VS6EFHACQH/GlyXRZ6kg+FPRDYKWIKI4CGeDIzKr4pRpM1XmIVKmQlXmGvy7iksPjfRln0cKHOL0aY/uPnOgJKsxsUqqdToCxlcNNsBRmRiVnHWirA8eYDlX8K6XVrd93qZz06HV5BIfx2mUXP8EUjWfiaKzrUtmrGg+59fuJpkaQNj3nzgcsasw776TpMu3WwlVjuYvWdV/1XvON6TamfSA7Yawoj8kocLnxe/TM8J1VToyctk9USjuhnwgxgjp+EV6mvL34C7hsK22b0SGIc0rzR3AuVa4Wubz0tZpd6QJgv2zT5uVBXrasGYHxQl1sV2WAp8dYxFAk4PKlWvP+9TGijnaSgQfnjqGCmAmglESmosDb+mo9DInWhYDCWr+R/GCV3pXJt6uHf/whnHfTQk8am4PgmtYtGVhY+ORGZRW5ndZy8nGGAwjih2bpqjYp1fCtuZev2SOQkGC6sxLKOUqw/MreNggrhA8mkZ6DS2AzgrsVnMQxFgu31z1FzHj6uoG930pO8bZvmiTGv500Iy58J7GHfCRW51phlEFcUdRe0YzFVjGRCFDQoCa OHEAFkv8 3g9CQnQrjHhbkilC+6Gtnsb4vVwhFum3RykzPcLc74pxJn9r8VLBlFsZN8qAmATD6xr6r+kRMltNmev7pzUgj6IncKi4VQu21mM76uQvHjuaqnFaW8JaOVMFz0492iqgLuOsiqBrbXqReyBQqO+plHcn/iwBLKguAWD+CTpVL0HRYw/xPTWSx91yGm9e2HrEEbLUq4cBlt/C5zEt5Bhx6FbjzZsBsnhdBDnExYNAIXB7U+NpGVl4uHWGIxiP7PeoDuBSTdFvAas1fm4plaX6Jva7nyMLw7vJy1fdytgQomCo0ApndqLCyh6y15wgL62L3NpetkK3oLIDjWb7iPzoOxhX1G8IGJyHhOPOkn5mOa2OrWozB3rwLB05DcJAG9zcFhAlsdoUv/1cMRDC/0Ke8SKWZ4iK7LQVMBTMbpo98nBdawccE9OT0dYT63Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jul 10, 2025 at 2:36=E2=80=AFAM Vlastimil Babka wr= ote: > > + if (unlikely(!allow_spin)) { > > + folio =3D (struct folio *)alloc_frozen_pages_nolock(0/* _= _GFP_COMP is implied */, > > + node, o= rder); > > + } else if (node =3D=3D NUMA_NO_NODE) > > folio =3D (struct folio *)alloc_frozen_pages(flags, order= ); > > else > > folio =3D (struct folio *)__alloc_frozen_pages(flags, ord= er, node, NULL); > > Nit: should use { } either for everything or nothing (seems your new bran= ch > would work without them) leftover from v1. will fix. > > stat(s, ALLOC_NODE_MISMATCH); > > @@ -3730,7 +3762,7 @@ static void *___slab_alloc(struct kmem_cache *s, = gfp_t gfpflags, int node, > > * PFMEMALLOC but right now, we are losing the pfmemalloc > > * information when the page leaves the per-cpu allocator > > */ > > - if (unlikely(!pfmemalloc_match(slab, gfpflags))) > > + if (unlikely(!pfmemalloc_match(slab, gfpflags) && allow_spin)) > > goto deactivate_slab; > > > > /* must check again c->slab in case we got preempted and it chang= ed */ > > @@ -3803,7 +3835,12 @@ static void *___slab_alloc(struct kmem_cache *s,= gfp_t gfpflags, int node, > > slub_set_percpu_partial(c, slab); > > > > if (likely(node_match(slab, node) && > > - pfmemalloc_match(slab, gfpflags))) { > > + pfmemalloc_match(slab, gfpflags)) || > > + /* > > + * Reentrant slub cannot take locks necessary > > + * for __put_partials(), hence downgrade to any node > > + */ > > + !allow_spin) { > > Uh this seems rather ugly, I'd move the comment above everything. Also it= 's > not "downgrade" as when you assign NUMA_NO_NODE earlier, I'd say "ignore = the > preference". > Note that it would be bad to ignore with __GFP_THISNODE but then it's not > allowed for kmalloc_nolock() so that's fine. Yes. All correct. Will reword. > > @@ -3911,6 +3953,12 @@ static void *___slab_alloc(struct kmem_cache *s,= gfp_t gfpflags, int node, > > void *flush_freelist =3D c->freelist; > > struct slab *flush_slab =3D c->slab; > > > > + if (unlikely(!allow_spin)) > > + /* > > + * Reentrant slub cannot take locks > > + * necessary for deactivate_slab() > > + */ > > + return NULL; > > Hm but this is leaking the slab we allocated and have in the "slab" > variable, we need to free it back in that case. > > > c->slab =3D NULL; > > c->freelist =3D NULL; > > c->tid =3D next_tid(c->tid); > > > @@ -4593,10 +4792,31 @@ static __always_inline void do_slab_free(struct= kmem_cache *s, > > barrier(); > > > > if (unlikely(slab !=3D c->slab)) { > > - __slab_free(s, slab, head, tail, cnt, addr); > > + /* cnt =3D=3D 0 signals that it's called from kfree_noloc= k() */ > > + if (unlikely(!cnt)) { > > + /* > > + * __slab_free() can locklessly cmpxchg16 into a = slab, > > + * but then it might need to take spin_lock or lo= cal_lock > > + * in put_cpu_partial() for further processing. > > + * Avoid the complexity and simply add to a defer= red list. > > + */ > > + defer_free(head); > > + } else { > > + __slab_free(s, slab, head, tail, cnt, addr); > > + } > > return; > > } > > > > + if (unlikely(!cnt)) { > > + if ((in_nmi() || !USE_LOCKLESS_FAST_PATH()) && > > + local_lock_is_locked(&s->cpu_slab->lock)) { > > + defer_free(head); > > + return; > > + } > > + cnt =3D 1; > > Hmm we might end up doing a "goto redo" later and then do the wrong thing= above? Great catch. Will fix. That's two serious bugs. That's my penalty for reviewing other people code 99% of the time and little time to code myself.