From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2321C83F27 for ; Tue, 15 Jul 2025 21:34:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 469C88D0003; Tue, 15 Jul 2025 17:34:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41A918D0001; Tue, 15 Jul 2025 17:34:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3092E8D0003; Tue, 15 Jul 2025 17:34:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 21E978D0001 for ; Tue, 15 Jul 2025 17:34:16 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B072D8012D for ; Tue, 15 Jul 2025 21:34:15 +0000 (UTC) X-FDA: 83667802470.28.9F87E3F Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf17.hostedemail.com (Postfix) with ESMTP id 6828F4000F for ; Tue, 15 Jul 2025 21:34:13 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j6YdK8Mi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.222.176 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752615253; a=rsa-sha256; cv=none; b=sWtg2qa31KhfcjFFbMPnA1aYHVR54OCKgJzNwh5Gp4Bm5so6gXv7hHqd5ml+axxE6Kq16f cfDd79CXX8bUKzbzRUhxBpX90xtoB3GvUx38TGy/uOY7yGACkGNb0Ibp7O+FDWpMnGsB22 HxKKmxLJqF2gb4KWpGIMrLY2LEX6goE= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=j6YdK8Mi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of boqun.feng@gmail.com designates 209.85.222.176 as permitted sender) smtp.mailfrom=boqun.feng@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752615253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uF1fdpy8mFKFOKUM4AMZPjAjVNIzorDz52lRWvMF5DY=; b=opP5PIr5fzI44Eav3FFjbIoYRIj/KZR2+5ICyb4bqwjczA/ywDQx4xBv3tBfG2o96pDyap oHCoD3axhNp8nFYvxiSYi755Oi11zZuBuy58qdQJtWkKRMzTvn36EXkxgpJk2k75IsT7Cb FD21vwGQ/DqwnIXY9zZgHUoylUJKcl0= Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7e32c9577ddso237528085a.0 for ; Tue, 15 Jul 2025 14:34:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752615252; x=1753220052; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:from:to:cc:subject:date :message-id:reply-to; bh=uF1fdpy8mFKFOKUM4AMZPjAjVNIzorDz52lRWvMF5DY=; b=j6YdK8MiMxtYxntPkMZS1KdSlJzBxY3JPcU5yHNNAKam6l9cQ4mTkBfXTkXDzZx70Q m9tX9v8BJWvJ/KsrmvkJpu/XgRh64MrzlJv3+6uJNaBXwTIv4Tagqx1JIdqG/R0QlOaU 0ypSEryPdKD4u5Va4gO2jKaL1K3Q7jLP16/En5/ohhSffg+xHclt1BZIHwdWhhRvZoVj 2kUPEpjL0xPpHtt8lmI74efBXMfP0DMXxT90YrE8Lm092pJp9YrMONp4Phn7nG38uRSz nuGVuq7aDJXaww03OCsB+vj8FS+mm5Du96Mdmsx9ogodLHDJX3HhHYsc6rtCize2zTbv 1YPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752615252; x=1753220052; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:feedback-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uF1fdpy8mFKFOKUM4AMZPjAjVNIzorDz52lRWvMF5DY=; b=gx7dVnl1NBVC6uv2j3bgYfzfVckAfWbaZowjMToo5rPkWKDtuCGfspDAwdMa8qv5qe JgEzJZ6v1ZiwndZYu03kppQI+8TjrP/0IZK6FMfD0dJlCuaRAdELkxA/Y6SAmcXKSNL0 sNr3aU1jlckqaariE3h3g0KsoM+iNholxp6I3a8N+RhcLplfbCHhdhplFZnTiUXXc3gf 6g5mJnyjXb7HZHOwGS5c6iEwtTWlSxRdaHpoR1oy59P3zhXCM1t9Ekb3l8IPLhbjpM6G S4CP9+P2/Mr/TaorvRXs7QeiTEXoI+M6KTFlfb9vG60rjrrNuqnmKipV5XuzHndqyTNz HM7g== X-Forwarded-Encrypted: i=1; AJvYcCXBEbNjL/iA3luQkxpjofA0J2xfvbJ10AoDkJBFJyUQ6Dwtocv0uCcyB0PcQo68ySagpPbWYvfpsQ==@kvack.org X-Gm-Message-State: AOJu0YxiunnLmjbNFxo2h7b0U9tN3GY+JY4tIInOjy40ge/4B8jGXHtL tOvsS++/6vZpahWf0XCZ5fVpMbia3a1YGErN0k8qw4zYS2QkuQxc/LZk X-Gm-Gg: ASbGncsr5cVIB/uNvEAD8CwYnXb2RfX2RsqTdVsSUiGd66A91mbi71b8E2lJSvOlBxD HFlZIgrr9NkvN+1fGXoMQLyt/qMWVvNmklDc4Qya5HO08XwNXJfK9264TtGsYOUjPBWdE+wbOp9 dVSboy4HGZLIlIOUY+Q16vkzmGOEifcBnsVqFtdffI9yCxm9Y8Nsmf7NDcB4fAtBrNVMMz4wCuE 9S9WMCOOSsBNrWdqgu++Koy/RrnACmPlcfKHFSZQU6pqsV0SyN1TDHMwqp0JT0O72d2AQDxVhTR 9M6jZgmiFAfuoFrFfjC0KN80llUH17/v1ouJE7pnaPIZ9V9zQu0B2qiude4efjFhXBGI5xuUT8h 5ucAHF5znPLvRvmF9hSHR5UDFOuTaTUBfV/20znZnT7ZaMdoyH8iHBtazzYEvq5bioYYsM/Kiq0 Oox4W2WXisPjwL X-Google-Smtp-Source: AGHT+IHyYZXhhEuifJ0c3cYQNpKDwiC8JPDhu4ErrCxN1+8gG6b28IWoO1mhbKzSZxg7zJiuQs+61g== X-Received: by 2002:a05:620a:3916:b0:7db:52b9:2060 with SMTP id af79cd13be357-7e3435eb952mr83379185a.36.1752615252221; Tue, 15 Jul 2025 14:34:12 -0700 (PDT) Received: from fauth-a1-smtp.messagingengine.com (fauth-a1-smtp.messagingengine.com. [103.168.172.200]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7e26996873esm306092985a.64.2025.07.15.14.34.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jul 2025 14:34:11 -0700 (PDT) Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfauth.phl.internal (Postfix) with ESMTP id 2F942F40066; Tue, 15 Jul 2025 17:34:11 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Tue, 15 Jul 2025 17:34:11 -0400 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdefgdehheelvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpeeuohhquhhnucfh vghnghcuoegsohhquhhnrdhfvghnghesghhmrghilhdrtghomheqnecuggftrfgrthhtvg hrnhepffekudfhlefhvdetudehtedvkeffgfeuffehtdfhueefgfeileeghfetkedvlefh necuffhomhgrihhnpehophgvnhhgrhhouhhprdhorhhgpdgsohhothhlihhnrdgtohhmpd hruhhsthdqlhgrnhhgrdhorhhgpdhgihhthhhusgdrtghomhdpkhgvrhhnvghlrdhorhhg necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepsghoqh hunhdomhgvshhmthhprghuthhhphgvrhhsohhnrghlihhthidqieelvdeghedtieegqddu jeejkeehheehvddqsghoqhhunhdrfhgvnhhgpeepghhmrghilhdrtghomhesfhhigihmvg drnhgrmhgvpdhnsggprhgtphhtthhopedukedpmhhouggvpehsmhhtphhouhhtpdhrtghp thhtoheplhhoshhsihhnsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehlvghvhihmih httghhvghllhdtsehgmhgrihhlrdgtohhmpdhrtghpthhtohepohhjvggurgeskhgvrhhn vghlrdhorhhgpdhrtghpthhtoheprghlvgigrdhgrgihnhhorhesghhmrghilhdrtghomh dprhgtphhtthhopehgrghrhiesghgrrhihghhuohdrnhgvthdprhgtphhtthhopegsjhho rhhnfegpghhhsehprhhothhonhhmrghilhdrtghomhdprhgtphhtthhopegrrdhhihhnug gsohhrgheskhgvrhhnvghlrdhorhhgpdhrtghpthhtoheprghlihgtvghrhihhlhesghho ohhglhgvrdgtohhmpdhrtghpthhtohepthhmghhrohhsshesuhhmihgthhdrvgguuh X-ME-Proxy: Feedback-ID: iad51458e:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 15 Jul 2025 17:34:10 -0400 (EDT) Date: Tue, 15 Jul 2025 14:34:09 -0700 From: Boqun Feng To: Benno Lossin Cc: Mitchell Levy , Miguel Ojeda , Alex Gaynor , Gary Guo , =?iso-8859-1?Q?Bj=F6rn?= Roy Baron , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 3/5] rust: percpu: add a rust per-CPU variable test Message-ID: References: <20250712-rust-percpu-v2-0-826f2567521b@gmail.com> <20250712-rust-percpu-v2-3-826f2567521b@gmail.com> <68762e19.170a0220.33e203.a0b7@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 6828F4000F X-Stat-Signature: 7ptdn7grxhf9dxngsxw6yrrfsnnoigha X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1752615253-273839 X-HE-Meta: U2FsdGVkX18nkHzqGdknPx1oNGb3Oz5BlgV5y/IJj76MDi13v5TGHgC/FoHu4th4iUIGgnmKT21/K+QLmF/vYcvQLRwK1cPk46N0RsO3sFcFcfDDkuDkJt54EuuXOI0vSldsojQ+I1z5zOzFhqJcpz0pUKj0ZDz7OiKuOv86d+CyWugXgP845zbhFvNuPbrZzYPWdr7ZbPmKNuo1dY3aK+8oOyk0W87MWeEHzQCZAkD4eTOhfFhgCcQWiLA48nG/Gf0NwN4HaMa6U06I2PFqfO0NZLNT09KekXadURAEzNvt1gzFKDVLCmxsj1o+mrDtfyW+wMvcBYO50VbG14RnTIcDj9ox0tQE7wuUcONeVaHIpLI0MGoeghSJM0VtxdTl1Sff24OfVdp4QkYAaHqOFHYp3Ibdeg0CQZL6cTpwuoNYlWvNysYd061/37ut4DOtpPu0z/XdSQijBbrXNsNfXAtbqnyb4yHWemmO1AqVVLqXWT+4TJe715cLp7tuBqOrtb1MYEAXlJB+QpZ7ovCGzx5ocb4MNiAjRYQwb9QVZcx2jDiZd13iW6BqV1oGlvbAjUaivtmV20J7SK5RD29q5Lc0ZsYV3YQt75JKyX6XFyVxYfCL/ODx0ZjmwVggNVLOOB8Ap7TUMqzrp3/6AjMprwvi7rSJjtzdYFXtng9WXJYjqUy7pZmqnF/3HUph6wjFhLenOhyHoYWmRi0Ww4TWY8INYQ+gMeqXE98gpaX+g4LXMgjaB4t3HZ7v48SXO6Nim0QeikWnPKW96Eo4FG8GiITJTgE2JbmlUshSJBkuwuQafx+vMwEpQIzJGLWAeSXlkV2QwoJVTlU8boXzMnxOp8jirG8B5NeogwUbS+Ca1FlrjuxWKjD2KIELXahxFcXdy1hcZl7xa6cg0k5EaIIxWFqNVIuWKN0kD0I0zUGvgJi2tZbKZ5K+S392kXwzmyJD8aG5fHtUgfDxU03ozeH aH7SUqw/ ZhJzNBCSKng61fejTlePsQ7p45zqaDmTHnVPty9kFKq/q6pPE9CtbnmKjdq/FzlnhaAHkyK0pAsYULYb4trKHRlD/D4Lkp5pzn2Rl3ux+oLwZHeQh3xtO5e32jQTO+ZQ8w+yrIEfLz+XPz/agUXqPtBPE/KO1LFvBu9/GC10xCtye6iIkOVU1DU6BrTYSi60JTgTpcO6iQ+Kp943M4uFI26RERXY5JUnivK0/plNERO55ZYlcsdYIQoHq0CV1d4w2XCled7pi2iim/BHUSbCFItQ+7LrsKuqJISSQVIZyoE7uh3LQqlDNXYhfyyBGMz+ZMNbtbv85qLfd9Fj1oQJfzj4R6Ccz7igWmYEAG6zi2qk93q1Hso2e5yji4Ce1zCryCQ8lFDx3zyUL5Y71YBzTcyXf97WqPCgouWf4h4VCLKGYCp7p4qd0jHoFblpWuWNzVrp5kTqoin11YHSnJxmUwbnPSvkhCa52zDIIqRH+U6vlRvhCazi2Jhn8Lt15U0boCh5w3kWYhjbw+r+0jDm7IgjzJnb4avm+gUEl/150dsxuSLPhFUWTK7US1NYHJ1wzAAlm+1Eys7UNutlCLGAF10V1/iYcZHXGGV12Q31l7oIsxz6+XLBXmV30SsAKGJoSYOEkTW0o93OKdyVOt37oK8hkBFAkQzVCyBXxO8Ca8FcL5zSPwxhCWjoyW01HHivY12EneOnTUB65oXwyJW/hQ/eETsO2KRw6OJcFrwUgXn8Bq4LZ+Hz73Nc4yw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jul 15, 2025 at 07:44:01PM +0200, Benno Lossin wrote: [...] > >> > > >> > First of all, `thread_local!` has to be implemented by some sys-specific > >> > unsafe mechanism, right? For example on unix, I think it's using > >> > pthread_key_t: > >> > > >> > https://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_key_create.html > >> > > >> > what we are implementing (or wrapping) is the very basic unsafe > >> > mechanism for percpu here. Surely we can explore the design for a safe > >> > API, but the unsafe mechanism is probably necessary to look into at > >> > first. > >> > >> But this is intended to be used by drivers, right? If so, then we should > > > > Not necessarily only for drivers, we can also use it for implementing > > other safe abstraction (e.g. hazard pointers, percpu counters etc) > > That's fair, but then it should be `pub(crate)`. > Fine by me, but please see below. > >> do our usual due diligence and work out a safe abstraction. Only fall > >> back to unsafe if it isn't possible. > >> > > > > All I'm saying is instead of figuring out a safe abstraction at first, > > we should probably focus on identifying how to implement it and which > > part is really unsafe and the safety requirement for that. > > Yeah. But then we should do that before merging :) > Well, who's talknig about merging? ;-) I thought we just began reviewing here ;-) > >> I'm not familiar with percpu, but from the name I assumed that it's > >> "just a variable for each cpu" so similar to `thread_local!`, but it's > >> bound to the specific cpu instead of the thread. > >> > >> That in my mind should be rather easy to support in Rust at least with > >> the thread_local-style API. You just need to ensure that no reference > >> can escape the cpu, so we can make it `!Send` & `!Sync` + rely on klint > > > > Not really, in kernel, we have plenty of use cases that we read the > > other CPU's percpu variables. For example, each CPU keeps it's own > > counter and we sum them other in another CPU. > > But then you need some sort of synchronization? > Right, but the synchronization can exist either in the percpu operations themselves or outside the percpu operations. Some cases, the data types are small enough to fit in atomic data types, and operations are just load/store/cmpxchg etc, then operations on the current cpu and remote read will be naturally synchronized. Sometimes extra synchronization is needed. Keyword find all these cases are `per_cpu_ptr()`: https://elixir.bootlin.com/linux/v6.15.6/A/ident/per_cpu_ptr > > If we would like to model it conceptually, it's more like an array > > that's index by CpuId to me. > > Gotcha, but this model is missing the access control/synchronization. So > I'm not so sure how useful it is. > > (I think I asked this somewhere else, but the number of CPUs doesn't > change, right?) > In terms of percpu variable, yes. A percpu variable is even available for an offline CPU. > >> to detect context switches. > >> > >> >> >> has: > >> >> >> > >> >> >> https://doc.rust-lang.org/std/macro.thread_local.html > >> >> >> > >> >> >> So in this example you would store a `Cell` instead. > >> >> >> > >> >> >> I'm not familiar with per CPU variables, but if you're usually storing > >> >> >> `Copy` types, then this is much better wrt not having unsafe code > >> >> >> everywhere. > >> >> >> > >> >> >> If one also often stores `!Copy` types, then we might be able to get > >> >> >> away with `RefCell`, but that's a small runtime overhead -- which is > >> >> >> probably bad given that per cpu variables are most likely used for > >> >> >> performance reasons? In that case the user might just need to store > >> >> >> `UnsafeCell` and use unsafe regardless. (or we invent something > >> > > >> > This sounds reasonable to me. > >> > > >> >> >> specifically for that case, eg tokens that are statically known to be > >> >> >> unique etc) > >> >> > > >> >> > I'm open to including a specialization for `T: Copy` in a similar vein > >> >> > to what I have here for numeric types. Off the top of my head, that > >> >> > shouldn't require any user-facing `unsafe`. But yes, I believe there is > >> >> > a significant amount of interest in having `!Copy` per-CPU variables. > >> >> > (At least, I'm interested in having them around for experimenting with > >> >> > using Rust for HV drivers.) > >> >> > >> >> What kinds of types would you like to store? Allocations? Just integers > >> >> in bigger structs? Mutexes? > >> >> > >> > > >> > In the VMBus driver, there is a percpu work_struct. > >> > >> Do you have a link? Or better yet a Rust struct description of what you > >> think it will look like :) > >> > > > > Not Rust code yet, but here is the corresponding C code: > > > > https://github.com/Rust-for-Linux/linux/blob/rust-next/drivers/hv/vmbus_drv.c#L1396 > > Thanks! > > > But please note that we are not solely developing the abstraction for > > this usage, but more for generally understand how to wrap percpu > > functionality similar to the usage in C. > > Well, I have to start somewhere for looking at the use-cases :) > > If you have more, just let me see. (probably won't have enough time to > look at them now, but maybe in a couple weeks) > If you have time, feel free to take a look at hazard pointers; https://lore.kernel.org/lkml/20240917143402.930114-1-boqun.feng@gmail.com/ https://lore.kernel.org/lkml/20250625031101.12555-1-boqun.feng@gmail.com/ You can also take a look at existing usage of percpu, e.g. SRCU uses to track how many readers are active: https://elixir.bootlin.com/linux/v6.15.6/source/kernel/rcu/srcutree.c#L577 [...] Regards, Boqun