From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E97CBC0219D for ; Thu, 13 Feb 2025 03:36:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F881280002; Wed, 12 Feb 2025 22:36:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A8A9280001; Wed, 12 Feb 2025 22:36:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 348F0280002; Wed, 12 Feb 2025 22:36:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 14FBA280001 for ; Wed, 12 Feb 2025 22:36:03 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 90AE5160289 for ; Thu, 13 Feb 2025 03:36:02 +0000 (UTC) X-FDA: 83113507764.21.1BDA00A Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf05.hostedemail.com (Postfix) with ESMTP id C3007100004 for ; Thu, 13 Feb 2025 03:36:00 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=de9K1Aq7; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739417760; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=/F/5Y1tCE2vrCT95xLgYeFjLhYdDffTJn3bYNGuc4gw=; b=yUoQSvBVnMkLxPNku2UiNM52dWF+PKshmzIsDaDW5w2A93COK2SSoKkwzdkBcAGzBaUljv xJqnmGOlQBJ3QelN2reFXD+X4K49geVPADRo2I3CNZMyC/2PjJm2hY9uTuPOXMnP5wHwiF kaC9MiFFHmVLQdXzfMZFHAAMDwIVwlo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=de9K1Aq7; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739417760; a=rsa-sha256; cv=none; b=61d+0Rh7tllc2K37H2VYNR5MPvv6cazsHIKbKdmEoQGBon4/cV5rJKAxK5Nkd0quIdwbk1 Y/0CedQ8vMWZsTSaCUPmVbwa50ChNoOA6DLV5ZudcExr5sunBYI26VgdtCFNrJEtrD9FD0 I3NyKpz9hcUnc8KnMCCZiHvmGCGZKZQ= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21f49bd087cso5578815ad.0 for ; Wed, 12 Feb 2025 19:36:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739417760; x=1740022560; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=/F/5Y1tCE2vrCT95xLgYeFjLhYdDffTJn3bYNGuc4gw=; b=de9K1Aq7ccq5vxJmHk+k2x39144iLAsJEn30/9ZZisjcFHmtED5jlcskAxnaIes9UC nSDTmOHjFj1mWI7SWKztFBw7lVl7r8QvKYfEdBxNRY0sjBrEe2b8V1IX+tX0wFkxn8W+ T5hRgYJJtUv7Gz5mCv3jJVMuYaq5IKcTn9pZzQJZGbm7LFUD2dvouGIW/8oIBZoCnV3z Vx9zQlsyPDhC9TJDNg4a0XDjPIpA6PQgYuMuCTgYO8aXxqTHaEtRnoRQ9w0qKPwapV78 Jd2lSp6sfLONijjiH78dBJwu/Zz+W4o3bPSQ7ur7Vb/7kuwsv4eES5Jn3p0jNQFweO9Q G3vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739417760; x=1740022560; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/F/5Y1tCE2vrCT95xLgYeFjLhYdDffTJn3bYNGuc4gw=; b=liA56gHV8W7y918tLx8NLFHeswNqtGegsKVFM+hc7mDcmwENZ/qi2gndhxfDEhbnn5 0oiRt0Fm4VV3WZtpinJYZkdC3697JFukANK+xF75+0+dgPbgDEh7PyyYvDRv3fRTF1Hr 1kUGj2zR2GGA6Sb5ydAQvGfGMsuBzzvb/EWxL0lPm4Pl/7www7H8M6NH4x/OyydoO14K vmlD3xWsrKMXe/ZYqtZdtVXYjnLzWgXyYTf72adevbyoU9gB056WhGar8b2DEa4tNDgC abrQzJO/jGMiZeGTf74u0U82nmXvELDL57qc3LDZJIrEJe0KFaTn9JJCb8SB4pIvhUJy 3FIA== X-Forwarded-Encrypted: i=1; AJvYcCVs0SaxAejh7z9Anb1CqSBw3ZtWixrYzjGCwsajnr0j45hkYJcQ2FEdL8XDB/UaWWPE3mqxUdr3nQ==@kvack.org X-Gm-Message-State: AOJu0YyD9RO2QO9V0tVXU8zObs8Qe4KY5ygZ1cFZ5+8d9m/hBYwXia+l MuPcMvLv97uT0D2vlKHCQzEbsUlgtzgp9L5jSOa0vZvfbNSmgLSd X-Gm-Gg: ASbGncvN4gZDvcIGazEktBFofNLmtSFBxeftqbBeafo1lxZTo0pCAxtGRxqeGxRUVvG yFsev5V1fdlP0vf4jDWwiqKelJvJSGmm/fC04Y47QLGwP/mId48In/bcTQOgYDQNFoFtA/D8wyP 7MIWBKZcc+YHu4Oi01iqAi/zFvhXdwQTETEXzetGxPywaLavvWIzXETlT1RpBb9i18y2qxcDEQj JVG9qVh3ujSODqvyftHXwrNlH8gXE9zjOw+UX+w6ThyT8PdivvzfKr8RdHmEKFCeJPXRKNbTZf7 mbSEYsxSiOTpq8vjDiu+0sc30yZu8fsYXVeqiywJ8SBXWIcoNw== X-Google-Smtp-Source: AGHT+IFVW3XUkxi8+x0+pwzPDKfCmXT8Cctzlltk41rn5tvKeWIrEe7ywpTu70J0Ezwq3Nwd3nXcLA== X-Received: by 2002:a05:6a00:35cf:b0:72f:f764:6269 with SMTP id d2e1a72fcca58-7323c139fa8mr2430260b3a.12.1739417759448; Wed, 12 Feb 2025 19:35:59 -0800 (PST) Received: from macbookpro.lan ([2603:3023:16e:5000:8af:ecd2:44cd:8027]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-7324256afabsm225850b3a.58.2025.02.12.19.35.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 12 Feb 2025 19:35:59 -0800 (PST) From: Alexei Starovoitov To: bpf@vger.kernel.org Cc: andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, vbabka@suse.cz, bigeasy@linutronix.de, rostedt@goodmis.org, houtao1@huawei.com, hannes@cmpxchg.org, shakeel.butt@linux.dev, mhocko@suse.com, willy@infradead.org, tglx@linutronix.de, jannh@google.com, tj@kernel.org, linux-mm@kvack.org, kernel-team@fb.com Subject: [PATCH bpf-next v8 0/6] bpf, mm: Introduce try_alloc_pages() Date: Wed, 12 Feb 2025 19:35:50 -0800 Message-Id: <20250213033556.9534-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C3007100004 X-Stat-Signature: cona4buywj59tjypzc71bghcgeey95qj X-HE-Tag: 1739417760-816748 X-HE-Meta: U2FsdGVkX1/2xt53Nr7b3kF1tLLOhH3ijBFtEbNsASkAbF0nOZyF4VRdV0jmRncW9JpDU1iMbPLrab1zxjHvan4p3kTFdacMta7MFQsYnqvEU3tsCZ8EgmHCnhp6OYX/c3dq2mLelBUYk9mvKwWNEr6YzG+rASE6zWQmmJN59tp24sGnacjJsZ7W/WZidENmVGic3hqgandpHJYEoRbQesUHETNvLNunUyQdFIG/QJAAnhaljJfbRuGivhSBtJTgnQMc6gsN8/mtnrceqGb7pNTfTw05MlUPjPABpK7N9Um5Wu3Za/n+QQVICKUuN0/xX/6JVX/0vruOj2zswlGFQGA0UoA0w3RQe8r5M4/77sPS3kL2imb3EVifIO2Z7QJVz0J0H4xU9QwtTqhaCb41k6hu36WSaQ5GN3wgkpbV4iyBepoQKvy9LW8YXuwDe+zHPVgxY6liNKp3uS4xyyqwc/NoB3DRkr+ZaY1wO1dlk2qZfw+7fsRdUCV6YqapMO2HLJZhYjmIJw9Ewel3eCThbYlI+L+coe9y+SlGVQog025iPZBGh/4jG5SPpFZoTDtQX7P77Hsu2bzoIM1P38ymsrIFzpnmwxPkhr5ufojnHu1g11MsITVruALm/yBx0DombCv6a6axFGEcNU8Q/jnyVyqtmuAyJzBJDvAlZj9+2pm3DtKbkLdKnymRQcWcp/1OmVzYUmVajcEUndPI529skpsJv/jyvpZGPlqwqmtbzAJvPxOIXYUTw8gys7lnVmwh9R/T8U6WCOjEtw7m0+YsA23y1va+nouIzJgIvZu2n5t3ol7MJ3OSjMT7wNE8bft2luG1Q2a8ooQ2VV1ecpSAI5BS1NxRWllZiMIRlMrUkGUWAdeNckczttXMIREzMxPOVRk898MmhaeB6mTKp18oxQm9R3YdoWecFYyJESek3PJjOKwxb1G4vrE507uGMJclJx2IG0i2ZiqU09pjxxD /Dj4NAys 0OmCzJOr9H21dHx/OzTUA5MzjXXbL8ONENhuexBn2YjkS5ClSvdEQ/tfy5Tq+LNtLeGH149k2VfB6QDTHwpLDZczfVAlZbID84w6BuVfYd65K3qGzbP1qIx4MlbtX10arDyN0mwulwQ5w2y/TaP5vjyJyVOtWzk3UDkJbvYkc6aVGVZ8suQuUhTbsqgGp7IIdYyiYlN2qnaibobgnIfJi+QERHDelBjS33RrP159799jj/fR11WqE6OBt02VthPKU3TFME9aw19tdoqS6lfK1sSQC5cDU6jIH5EXhnq4uO8TtCFuRtLtUYCrIwSILWKjW5aZEGF8vuW6y5HxFO+DpITGhjDUInr9CUcUX3+76Ag7+fDE0/D/U78Tl8fSzhnukNCJm7RuLoVMfBfaCqUbdjQOqqN3OA+k5N5LcpprfjTISbEYHCc9ADsIj9FLAm0esCDy2kIfHpz68IFU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov Hi All, The main motivation is to make alloc page and slab reentrant and remove bpf_mem_alloc. v7->v8: - rebase: s/free_unref_page/free_frozen_page/ v6->v7: - Took Sebastian's patch for localtry_lock_t as-is with minor addition of local_trylock_acquire() for proper LOCKDEP. Kept his authorship. - Adjusted patch 4 to use it. The rest is unchanged. v6: https://lore.kernel.org/bpf/20250124035655.78899-1-alexei.starovoitov@gmail.com/ v5->v6: - Addressed comments from Sebastian, Vlastimil - New approach for local_lock_t in patch 3. Instead of unconditionally increasing local_lock_t size to 4 bytes introduce local_trylock_t and use _Generic() tricks to manipulate active field. - Address stackdepot reentrance issues. alloc part in patch 1 and free part in patch 2. - Inlined mem_cgroup_cancel_charge() in patch 4 since this helper is being removed. - Added Acks. - Dropped failslab, kfence, kmemleak patch. - Improved bpf_map_alloc_pages() in patch 6 a bit to demo intended usage. It will be refactored further. - Considered using __GFP_COMP in try_alloc_pages to simplify free_pages_nolock a bit, but then decided to make it work for all types of pages, since free_pages_nolock() is used by stackdepot and currently it's using non-compound order 2. I felt it's best to leave it as-is and make free_pages_nolock() support all pages. v5: https://lore.kernel.org/all/20250115021746.34691-1-alexei.starovoitov@gmail.com/ v4->v5: - Fixed patch 1 and 4 commit logs and comments per Michal suggestions. Added Acks. - Added patch 6 to make failslab, kfence, kmemleak complaint with trylock mode. It's a prerequisite for reentrant slab patches. v4: https://lore.kernel.org/bpf/20250114021922.92609-1-alexei.starovoitov@gmail.com/ v3->v4: Addressed feedback from Michal and Shakeel: - GFP_TRYLOCK flag is gone. gfpflags_allow_spinning() is used instead. - Improved comments and commit logs. v3: https://lore.kernel.org/bpf/20241218030720.1602449-1-alexei.starovoitov@gmail.com/ v2->v3: To address the issues spotted by Sebastian, Vlastimil, Steven: - Made GFP_TRYLOCK internal to mm/internal.h try_alloc_pages() and free_pages_nolock() are the only interfaces. - Since spin_trylock() is not safe in RT from hard IRQ and NMI disable such usage in lock_trylock and in try_alloc_pages(). In such case free_pages_nolock() falls back to llist right away. - Process trylock_free_pages llist when preemptible. - Check for things like unaccepted memory and order <= 3 early. - Don't call into __alloc_pages_slowpath() at all. - Inspired by Vlastimil's struct local_tryirq_lock adopted it in local_lock_t. Extra 4 bytes in !RT in local_lock_t shouldn't affect any of the current local_lock_t users. This is patch 3. - Tested with bpf selftests in RT and !RT and realized how much more work is necessary on bpf side to play nice with RT. The urgency of this work got higher. The alternative is to convert bpf bits left and right to bpf_mem_alloc. v2: https://lore.kernel.org/bpf/20241210023936.46871-1-alexei.starovoitov@gmail.com/ v1->v2: - fixed buggy try_alloc_pages_noprof() in PREEMPT_RT. Thanks Peter. - optimize all paths by doing spin_trylock_irqsave() first and only then check for gfp_flags & __GFP_TRYLOCK. Then spin_lock_irqsave() if it's a regular mode. So new gfp flag will not add performance overhead. - patches 2-5 are new. They introduce lockless and/or trylock free_pages_nolock() and memcg support. So it's in usable shape for bpf in patch 6. v1: https://lore.kernel.org/bpf/20241116014854.55141-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (5): mm, bpf: Introduce try_alloc_pages() for opportunistic page allocation mm, bpf: Introduce free_pages_nolock() memcg: Use trylock to access memcg stock_lock. mm, bpf: Use memcg in try_alloc_pages(). bpf: Use try_alloc_pages() to allocate pages for bpf needs. Sebastian Andrzej Siewior (1): locking/local_lock: Introduce localtry_lock_t include/linux/bpf.h | 2 +- include/linux/gfp.h | 23 ++++ include/linux/local_lock.h | 59 ++++++++ include/linux/local_lock_internal.h | 123 +++++++++++++++++ include/linux/mm_types.h | 4 + include/linux/mmzone.h | 3 + kernel/bpf/arena.c | 5 +- kernel/bpf/syscall.c | 23 +++- lib/stackdepot.c | 10 +- mm/internal.h | 1 + mm/memcontrol.c | 52 +++++--- mm/page_alloc.c | 200 ++++++++++++++++++++++++++-- mm/page_owner.c | 8 +- 13 files changed, 472 insertions(+), 41 deletions(-) -- 2.43.5