[PATCH v2] rust: page: add byte-wise atomic memory copy methods

[PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

When copying data from buffers that are mapped to user space, it is
impossible to guarantee absence of concurrent memory operations on those
buffers. Copying data to/from `Page` from/to these buffers would be
undefined behavior if no special considerations are made.

Add methods on `Page` to read and write the contents using byte-wise atomic
operations.

Also improve clarity by specifying additional requirements on
`read_raw`/`write_raw` methods regarding concurrent operations on involved
buffers.

Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
---
Changes in v2:
- Rewrite patch with byte-wise atomic operations as foundation of operation.
- Update subject and commit message.
- Link to v1: https://lore.kernel.org/r/20260130-page-volatile-io-v1-1-19f3d3e8f265@kernel.org
---
 rust/kernel/page.rs        | 65 ++++++++++++++++++++++++++++++++++++++++++++++
 rust/kernel/sync/atomic.rs | 32 +++++++++++++++++++++++
 2 files changed, 97 insertions(+)

diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
index 432fc0297d4a8..febe9621adee6 100644
--- a/rust/kernel/page.rs
+++ b/rust/kernel/page.rs
@@ -7,6 +7,7 @@
     bindings,
     error::code::*,
     error::Result,
+    ffi::c_void,
     uaccess::UserSliceReader,
 };
 use core::{
@@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
     /// # Safety
     ///
     /// * Callers must ensure that `dst` is valid for writing `len` bytes.
+    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
+    ///   destination memory region.
     /// * Callers must ensure that this call does not race with a write to the same page that
     ///   overlaps with this read.
     pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
@@ -274,6 +277,34 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
         })
     }
 
+    /// Maps the page and reads from it into the given IO memory region using byte-wise atomic
+    /// memory operations.
+    ///
+    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+    /// outside of the page, then this call returns [`EINVAL`].
+    ///
+    /// # Safety
+    /// Callers must ensure that:
+    ///
+    /// - `dst` is valid for writes for `len` bytes for the duration of the call.
+    /// - For the duration of the call, other accesses to the area described by `dst` and `len`,
+    ///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
+    ///   function. Note that if all other accesses are atomic, then this safety requirement is
+    ///   trivially fulfilled.
+    ///
+    /// [`LKMM`]: srctree/tools/memory-model
+    pub unsafe fn read_bytewise_atomic(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
+        self.with_pointer_into_page(offset, len, move |src| {
+            // SAFETY: If `with_pointer_into_page` calls into this closure, then
+            // it has performed a bounds check and guarantees that `src` is
+            // valid for `len` bytes.
+            //
+            // There caller guarantees that there is no data race at the source.
+            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
+            Ok(())
+        })
+    }
+
     /// Maps the page and writes into it from the given buffer.
     ///
     /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
@@ -282,6 +313,7 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
     /// # Safety
     ///
     /// * Callers must ensure that `src` is valid for reading `len` bytes.
+    /// * Callers must ensure that there are no concurrent writes to the source memory region.
     /// * Callers must ensure that this call does not race with a read or write to the same page
     ///   that overlaps with this write.
     pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
@@ -295,6 +327,39 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
         })
     }
 
+    /// Maps the page and writes into it from the given IO memory region using byte-wise atomic
+    /// memory operations.
+    ///
+    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
+    /// outside of the page, then this call returns [`EINVAL`].
+    ///
+    /// # Safety
+    ///
+    /// Callers must ensure that:
+    ///
+    /// - `src` is valid for reads for `len` bytes for the duration of the call.
+    /// - For the duration of the call, other accesses to the area described by `src` and `len`,
+    ///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
+    ///   function. Note that if all other accesses are atomic, then this safety requirement is
+    ///   trivially fulfilled.
+    ///
+    /// [`LKMM`]: srctree/tools/memory-model
+    pub unsafe fn write_bytewise_atomic(
+        &self,
+        src: *const u8,
+        offset: usize,
+        len: usize,
+    ) -> Result {
+        self.with_pointer_into_page(offset, len, move |dst| {
+            // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
+            // bounds check and guarantees that `dst` is valid for `len` bytes.
+            //
+            // There caller guarantees that there is no data race at the destination.
+            unsafe { bindings::memcpy_fromio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
+            Ok(())
+        })
+    }
+
     /// Maps the page and zeroes the given slice.
     ///
     /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
diff --git a/rust/kernel/sync/atomic.rs b/rust/kernel/sync/atomic.rs
index 4aebeacb961a2..8ab20126a88cf 100644
--- a/rust/kernel/sync/atomic.rs
+++ b/rust/kernel/sync/atomic.rs
@@ -560,3 +560,35 @@ pub fn fetch_add<Rhs, Ordering: ordering::Ordering>(&self, v: Rhs, _: Ordering)
         unsafe { from_repr(ret) }
     }
 }
+
+/// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
+///
+/// This copy operation is volatile.
+///
+/// # Safety
+///
+/// Callers must ensure that:
+///
+/// - `src` is valid for reads for `len` bytes for the duration of the call.
+/// - `dst` is valid for writes for `len` bytes for the duration of the call.
+/// - For the duration of the call, other accesses to the areas described by `src`, `dst` and `len`,
+///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
+///   function. Note that if all other accesses are atomic, then this safety requirement is
+///   trivially fulfilled.
+///
+/// [`LKMM`]: srctree/tools/memory-model
+pub unsafe fn atomic_per_byte_memcpy(src: *const u8, dst: *mut u8, len: usize) {
+    // SAFETY: By the safety requirements of this function, the following operation will not:
+    //  - Trap.
+    //  - Invalidate any reference invariants.
+    //  - Race with any operation by the Rust AM, as `bindings::memcpy` is a byte-wise atomic
+    //    operation and all operations by the Rust AM to the involved memory areas use byte-wise
+    //    atomic semantics.
+    unsafe {
+        bindings::memcpy(
+            dst.cast::<kernel::ffi::c_void>(),
+            src.cast::<kernel::ffi::c_void>(),
+            len,
+        )
+    };
+}

---
base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
change-id: 20260130-page-volatile-io-05ff595507d3

Best regards,
-- 
Andreas Hindborg <a.hindborg@kernel.org>

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Boqun Feng]

On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
> When copying data from buffers that are mapped to user space, it is
> impossible to guarantee absence of concurrent memory operations on those
> buffers. Copying data to/from `Page` from/to these buffers would be
> undefined behavior if no special considerations are made.
> 
> Add methods on `Page` to read and write the contents using byte-wise atomic
> operations.
> 

Thank you, but in this patch we still have "the given IO memory" and use
memcpy_{from,to}io() as the implementation, is that intended?

Regards,
Boqun

> Also improve clarity by specifying additional requirements on
> `read_raw`/`write_raw` methods regarding concurrent operations on involved
> buffers.
> 
> Signed-off-by: Andreas Hindborg <a.hindborg@kernel.org>
> ---
> Changes in v2:
> - Rewrite patch with byte-wise atomic operations as foundation of operation.
> - Update subject and commit message.
> - Link to v1: https://lore.kernel.org/r/20260130-page-volatile-io-v1-1-19f3d3e8f265@kernel.org
> ---
>  rust/kernel/page.rs        | 65 ++++++++++++++++++++++++++++++++++++++++++++++
>  rust/kernel/sync/atomic.rs | 32 +++++++++++++++++++++++
>  2 files changed, 97 insertions(+)
> 
> diff --git a/rust/kernel/page.rs b/rust/kernel/page.rs
> index 432fc0297d4a8..febe9621adee6 100644
> --- a/rust/kernel/page.rs
> +++ b/rust/kernel/page.rs
> @@ -7,6 +7,7 @@
>      bindings,
>      error::code::*,
>      error::Result,
> +    ffi::c_void,
>      uaccess::UserSliceReader,
>  };
>  use core::{
> @@ -260,6 +261,8 @@ fn with_pointer_into_page<T>(
>      /// # Safety
>      ///
>      /// * Callers must ensure that `dst` is valid for writing `len` bytes.
> +    /// * Callers must ensure that there are no other concurrent reads or writes to/from the
> +    ///   destination memory region.
>      /// * Callers must ensure that this call does not race with a write to the same page that
>      ///   overlaps with this read.
>      pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
> @@ -274,6 +277,34 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>          })
>      }
>  
> +    /// Maps the page and reads from it into the given IO memory region using byte-wise atomic
> +    /// memory operations.
> +    ///
> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> +    /// outside of the page, then this call returns [`EINVAL`].
> +    ///
> +    /// # Safety
> +    /// Callers must ensure that:
> +    ///
> +    /// - `dst` is valid for writes for `len` bytes for the duration of the call.
> +    /// - For the duration of the call, other accesses to the area described by `dst` and `len`,
> +    ///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
> +    ///   function. Note that if all other accesses are atomic, then this safety requirement is
> +    ///   trivially fulfilled.
> +    ///
> +    /// [`LKMM`]: srctree/tools/memory-model
> +    pub unsafe fn read_bytewise_atomic(&self, dst: *mut u8, offset: usize, len: usize) -> Result {
> +        self.with_pointer_into_page(offset, len, move |src| {
> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then
> +            // it has performed a bounds check and guarantees that `src` is
> +            // valid for `len` bytes.
> +            //
> +            // There caller guarantees that there is no data race at the source.
> +            unsafe { bindings::memcpy_toio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
> +            Ok(())
> +        })
> +    }
> +
>      /// Maps the page and writes into it from the given buffer.
>      ///
>      /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> @@ -282,6 +313,7 @@ pub unsafe fn read_raw(&self, dst: *mut u8, offset: usize, len: usize) -> Result
>      /// # Safety
>      ///
>      /// * Callers must ensure that `src` is valid for reading `len` bytes.
> +    /// * Callers must ensure that there are no concurrent writes to the source memory region.
>      /// * Callers must ensure that this call does not race with a read or write to the same page
>      ///   that overlaps with this write.
>      pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Result {
> @@ -295,6 +327,39 @@ pub unsafe fn write_raw(&self, src: *const u8, offset: usize, len: usize) -> Res
>          })
>      }
>  
> +    /// Maps the page and writes into it from the given IO memory region using byte-wise atomic
> +    /// memory operations.
> +    ///
> +    /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> +    /// outside of the page, then this call returns [`EINVAL`].
> +    ///
> +    /// # Safety
> +    ///
> +    /// Callers must ensure that:
> +    ///
> +    /// - `src` is valid for reads for `len` bytes for the duration of the call.
> +    /// - For the duration of the call, other accesses to the area described by `src` and `len`,
> +    ///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
> +    ///   function. Note that if all other accesses are atomic, then this safety requirement is
> +    ///   trivially fulfilled.
> +    ///
> +    /// [`LKMM`]: srctree/tools/memory-model
> +    pub unsafe fn write_bytewise_atomic(
> +        &self,
> +        src: *const u8,
> +        offset: usize,
> +        len: usize,
> +    ) -> Result {
> +        self.with_pointer_into_page(offset, len, move |dst| {
> +            // SAFETY: If `with_pointer_into_page` calls into this closure, then it has performed a
> +            // bounds check and guarantees that `dst` is valid for `len` bytes.
> +            //
> +            // There caller guarantees that there is no data race at the destination.
> +            unsafe { bindings::memcpy_fromio(dst.cast::<c_void>(), src.cast::<c_void>(), len) };
> +            Ok(())
> +        })
> +    }
> +
>      /// Maps the page and zeroes the given slice.
>      ///
>      /// This method will perform bounds checks on the page offset. If `offset .. offset+len` goes
> diff --git a/rust/kernel/sync/atomic.rs b/rust/kernel/sync/atomic.rs
> index 4aebeacb961a2..8ab20126a88cf 100644
> --- a/rust/kernel/sync/atomic.rs
> +++ b/rust/kernel/sync/atomic.rs
> @@ -560,3 +560,35 @@ pub fn fetch_add<Rhs, Ordering: ordering::Ordering>(&self, v: Rhs, _: Ordering)
>          unsafe { from_repr(ret) }
>      }
>  }
> +
> +/// Copy `len` bytes from `src` to `dst` using byte-wise atomic operations.
> +///
> +/// This copy operation is volatile.
> +///
> +/// # Safety
> +///
> +/// Callers must ensure that:
> +///
> +/// - `src` is valid for reads for `len` bytes for the duration of the call.
> +/// - `dst` is valid for writes for `len` bytes for the duration of the call.
> +/// - For the duration of the call, other accesses to the areas described by `src`, `dst` and `len`,
> +///   must not cause data races (defined by [`LKMM`]) against atomic operations executed by this
> +///   function. Note that if all other accesses are atomic, then this safety requirement is
> +///   trivially fulfilled.
> +///
> +/// [`LKMM`]: srctree/tools/memory-model
> +pub unsafe fn atomic_per_byte_memcpy(src: *const u8, dst: *mut u8, len: usize) {
> +    // SAFETY: By the safety requirements of this function, the following operation will not:
> +    //  - Trap.
> +    //  - Invalidate any reference invariants.
> +    //  - Race with any operation by the Rust AM, as `bindings::memcpy` is a byte-wise atomic
> +    //    operation and all operations by the Rust AM to the involved memory areas use byte-wise
> +    //    atomic semantics.
> +    unsafe {
> +        bindings::memcpy(
> +            dst.cast::<kernel::ffi::c_void>(),
> +            src.cast::<kernel::ffi::c_void>(),
> +            len,
> +        )
> +    };
> +}
> 
> ---
> base-commit: 63804fed149a6750ffd28610c5c1c98cce6bd377
> change-id: 20260130-page-volatile-io-05ff595507d3
> 
> Best regards,
> -- 
> Andreas Hindborg <a.hindborg@kernel.org>
> 
> 

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

"Boqun Feng" <boqun@kernel.org> writes:

> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
>> When copying data from buffers that are mapped to user space, it is
>> impossible to guarantee absence of concurrent memory operations on those
>> buffers. Copying data to/from `Page` from/to these buffers would be
>> undefined behavior if no special considerations are made.
>>
>> Add methods on `Page` to read and write the contents using byte-wise atomic
>> operations.
>>
>
> Thank you, but in this patch we still have "the given IO memory" and use
> memcpy_{from,to}io() as the implementation, is that intended?

No that is a mistake. Please pretend it is regular `memcpy`. I'll send
v3 soon.

Best regards,
Andreas Hindborg

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

Andreas Hindborg <a.hindborg@kernel.org> writes:

> "Boqun Feng" <boqun@kernel.org> writes:
>
>> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
>>> When copying data from buffers that are mapped to user space, it is
>>> impossible to guarantee absence of concurrent memory operations on those
>>> buffers. Copying data to/from `Page` from/to these buffers would be
>>> undefined behavior if no special considerations are made.
>>>
>>> Add methods on `Page` to read and write the contents using byte-wise atomic
>>> operations.
>>>
>>
>> Thank you, but in this patch we still have "the given IO memory" and use
>> memcpy_{from,to}io() as the implementation, is that intended?
>
> No that is a mistake. Please pretend it is regular `memcpy`. I'll send
> v3 soon.

Sorry, pretend it is `kernel::sync::atomic_per_byte_memcpy` ofc.


Best regards,
Andreas Hindborg

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
> When copying data from buffers that are mapped to user space, it is
> impossible to guarantee absence of concurrent memory operations on those
> buffers. Copying data to/from `Page` from/to these buffers would be
> undefined behavior if no special considerations are made.
> 
> Add methods on `Page` to read and write the contents using byte-wise atomic
> operations.

I'm completely failing to understand. What!?

There is no such thing as an 'atomic' byte load, nor does it help one
whit against concurrent modification of the memory you're copying.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Greg KH]

On Fri, Feb 13, 2026 at 10:55:57AM +0100, Peter Zijlstra wrote:
> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
> > When copying data from buffers that are mapped to user space, it is
> > impossible to guarantee absence of concurrent memory operations on those
> > buffers. Copying data to/from `Page` from/to these buffers would be
> > undefined behavior if no special considerations are made.
> > 
> > Add methods on `Page` to read and write the contents using byte-wise atomic
> > operations.
> 
> I'm completely failing to understand. What!?
> 
> There is no such thing as an 'atomic' byte load, nor does it help one
> whit against concurrent modification of the memory you're copying.

I too am totally confused when reading this patch, and the previous
ones.  Shouldn't the "normal" copy_from_user() stuff be used here
correctly?  Why is anything new needed?

thanks,

greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

"Greg KH" <gregkh@linuxfoundation.org> writes:

> On Fri, Feb 13, 2026 at 10:55:57AM +0100, Peter Zijlstra wrote:
>> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
>> > When copying data from buffers that are mapped to user space, it is
>> > impossible to guarantee absence of concurrent memory operations on those
>> > buffers. Copying data to/from `Page` from/to these buffers would be
>> > undefined behavior if no special considerations are made.
>> >
>> > Add methods on `Page` to read and write the contents using byte-wise atomic
>> > operations.
>>
>> I'm completely failing to understand. What!?
>>
>> There is no such thing as an 'atomic' byte load, nor does it help one
>> whit against concurrent modification of the memory you're copying.
>
> I too am totally confused when reading this patch, and the previous
> ones.  Shouldn't the "normal" copy_from_user() stuff be used here
> correctly?  Why is anything new needed?

One use for this is copying data out of a `struct bio_vec`. As far as I
know, there is no way to know where the pages backing a bio_vec are
mapped. They could be mapped to user space.

Best regards,
Andreas Hindborg

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Greg KH]

On Fri, Feb 13, 2026 at 01:58:16PM +0100, Andreas Hindborg wrote:
> "Greg KH" <gregkh@linuxfoundation.org> writes:
> 
> > On Fri, Feb 13, 2026 at 10:55:57AM +0100, Peter Zijlstra wrote:
> >> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
> >> > When copying data from buffers that are mapped to user space, it is
> >> > impossible to guarantee absence of concurrent memory operations on those
> >> > buffers. Copying data to/from `Page` from/to these buffers would be
> >> > undefined behavior if no special considerations are made.
> >> >
> >> > Add methods on `Page` to read and write the contents using byte-wise atomic
> >> > operations.
> >>
> >> I'm completely failing to understand. What!?
> >>
> >> There is no such thing as an 'atomic' byte load, nor does it help one
> >> whit against concurrent modification of the memory you're copying.
> >
> > I too am totally confused when reading this patch, and the previous
> > ones.  Shouldn't the "normal" copy_from_user() stuff be used here
> > correctly?  Why is anything new needed?
> 
> One use for this is copying data out of a `struct bio_vec`. As far as I
> know, there is no way to know where the pages backing a bio_vec are
> mapped. They could be mapped to user space.

And how does C code do this today?  Surely there's a function that is
used there that we should also be using here, right?  Why do we need
something different?

thanks,

greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

"Greg KH" <gregkh@linuxfoundation.org> writes:

> On Fri, Feb 13, 2026 at 01:58:16PM +0100, Andreas Hindborg wrote:
>> "Greg KH" <gregkh@linuxfoundation.org> writes:
>>
>> > On Fri, Feb 13, 2026 at 10:55:57AM +0100, Peter Zijlstra wrote:
>> >> On Thu, Feb 12, 2026 at 03:51:24PM +0100, Andreas Hindborg wrote:
>> >> > When copying data from buffers that are mapped to user space, it is
>> >> > impossible to guarantee absence of concurrent memory operations on those
>> >> > buffers. Copying data to/from `Page` from/to these buffers would be
>> >> > undefined behavior if no special considerations are made.
>> >> >
>> >> > Add methods on `Page` to read and write the contents using byte-wise atomic
>> >> > operations.
>> >>
>> >> I'm completely failing to understand. What!?
>> >>
>> >> There is no such thing as an 'atomic' byte load, nor does it help one
>> >> whit against concurrent modification of the memory you're copying.
>> >
>> > I too am totally confused when reading this patch, and the previous
>> > ones.  Shouldn't the "normal" copy_from_user() stuff be used here
>> > correctly?  Why is anything new needed?
>>
>> One use for this is copying data out of a `struct bio_vec`. As far as I
>> know, there is no way to know where the pages backing a bio_vec are
>> mapped. They could be mapped to user space.
>
> And how does C code do this today?  Surely there's a function that is
> used there that we should also be using here, right?  Why do we need
> something different?

C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
driver [2].

Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
informed these are not safe to use if source or destination may incur
data races, and that we need an operation that is volatile or byte-wise
atomic [3].

Thus, we build an abstraction relying on byte-wise atomicity that does
not race under LKMM.

Best regards,
Andreas Hindborg

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/block/bio.c?h=v6.19#n1458
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/block/null_blk/main.c?h=v6.19#n1162
[3] https://lore.kernel.org/r/aYFKbWfQmTInYy91@tardis.local

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:

> C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
> driver [2].

Right. And that is *fine*.

> Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
> informed these are not safe to use if source or destination may incur
> data races, and that we need an operation that is volatile or byte-wise
> atomic [3].

Safe how? It should just copy N bytes. Whatever it thinks those bytes
are.

Nothing can guard against concurrent modification. If there is, you get
to keep the pieces. Pretending anything else is delusional.

Suppose the memory was 'AAAA' and while you're reading it, it is written
to be 'BBBB'. The resulting copy can be any combination of
'[AB][AB][AB][AB]'. Not one of them is better than the other.

No byte wise volatile barrier using nonsense is going to make this any
better.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Greg KH]

On Fri, Feb 13, 2026 at 03:26:08PM +0100, Peter Zijlstra wrote:
> On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:
> 
> > C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
> > driver [2].
> 
> Right. And that is *fine*.
> 
> > Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
> > informed these are not safe to use if source or destination may incur
> > data races, and that we need an operation that is volatile or byte-wise
> > atomic [3].
> 
> Safe how? It should just copy N bytes. Whatever it thinks those bytes
> are.
> 
> Nothing can guard against concurrent modification. If there is, you get
> to keep the pieces. Pretending anything else is delusional.
> 
> Suppose the memory was 'AAAA' and while you're reading it, it is written
> to be 'BBBB'. The resulting copy can be any combination of
> '[AB][AB][AB][AB]'. Not one of them is better than the other.
> 
> No byte wise volatile barrier using nonsense is going to make this any
> better.
> 

I'm with Peter, just call memcpy() like the C code does, and you will be
"fine" (with a note that "fine" better include checking the data really
is what you think it is if you are going to do anything based on it and
not just pass it off to the hardware.)

thanks,

greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Boqun Feng]

On Fri, Feb 13, 2026 at 04:34:04PM +0100, Greg KH wrote:
> On Fri, Feb 13, 2026 at 03:26:08PM +0100, Peter Zijlstra wrote:
> > On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:
> > 
> > > C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
> > > driver [2].
> > 
> > Right. And that is *fine*.
> > 

Yes, that's fine because memcpy() in C is volatile and per-byte atomic.

> > > Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
> > > informed these are not safe to use if source or destination may incur
> > > data races, and that we need an operation that is volatile or byte-wise
> > > atomic [3].
> > 
> > Safe how? It should just copy N bytes. Whatever it thinks those bytes
> > are.
> > 
> > Nothing can guard against concurrent modification. If there is, you get
> > to keep the pieces. Pretending anything else is delusional.
> > 
> > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > to be 'BBBB'. The resulting copy can be any combination of
> > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > 

The idea is if using Rust's own `core::ptr::copy()` or
`core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
not semantically guaranteed atomic per byte (i.e. tearing can happen at
bit level, because they are not designed for using in case of data
races, and there is no defined asm implementation of them, compilers can
do anything).

> > No byte wise volatile barrier using nonsense is going to make this any
> > better.

It's byte-wise atomic [1], which should be guaranteed using asm to
implement, hence at least at byte level, they are atomic (and volatile
in our case).

[1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1478r5.html

> > 
> 
> I'm with Peter, just call memcpy() like the C code does, and you will be
> "fine" (with a note that "fine" better include checking the data really

We are. See v3, we actually use `memcpy()` for the copy (as I already
pointed out, Andreas made a mistake in this version), it's just
because it's per-byte atomic. What this "byte-wise atomic" does is
clearing things out.

Regards,
Boqun

> is what you think it is if you are going to do anything based on it and
> not just pass it off to the hardware.)
> 
> thanks,
> 
> greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Greg KH]

On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> On Fri, Feb 13, 2026 at 04:34:04PM +0100, Greg KH wrote:
> > On Fri, Feb 13, 2026 at 03:26:08PM +0100, Peter Zijlstra wrote:
> > > On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:
> > > 
> > > > C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
> > > > driver [2].
> > > 
> > > Right. And that is *fine*.
> > > 
> 
> Yes, that's fine because memcpy() in C is volatile and per-byte atomic.
> 
> > > > Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
> > > > informed these are not safe to use if source or destination may incur
> > > > data races, and that we need an operation that is volatile or byte-wise
> > > > atomic [3].
> > > 
> > > Safe how? It should just copy N bytes. Whatever it thinks those bytes
> > > are.
> > > 
> > > Nothing can guard against concurrent modification. If there is, you get
> > > to keep the pieces. Pretending anything else is delusional.
> > > 
> > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > to be 'BBBB'. The resulting copy can be any combination of
> > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > 
> 
> The idea is if using Rust's own `core::ptr::copy()` or
> `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> not semantically guaranteed atomic per byte (i.e. tearing can happen at
> bit level, because they are not designed for using in case of data
> races, and there is no defined asm implementation of them, compilers can
> do anything).

Then why not just call the proper, in-kernel, arch specific, patched and
tested to the end-of-the-earth, memcpy()?

> > > No byte wise volatile barrier using nonsense is going to make this any
> > > better.
> 
> It's byte-wise atomic [1], which should be guaranteed using asm to
> implement, hence at least at byte level, they are atomic (and volatile
> in our case).
> 
> [1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1478r5.html

Again, just use memcpy() please.

> > > 
> > 
> > I'm with Peter, just call memcpy() like the C code does, and you will be
> > "fine" (with a note that "fine" better include checking the data really
> 
> We are. See v3, we actually use `memcpy()` for the copy (as I already
> pointed out, Andreas made a mistake in this version), it's just
> because it's per-byte atomic. What this "byte-wise atomic" does is
> clearing things out.

clear what out?  It shouldn't need anything special for a memcpy.

thanks,

greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Boqun Feng]

On Fri, Feb 13, 2026 at 04:58:54PM +0100, Greg KH wrote:
> On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> > On Fri, Feb 13, 2026 at 04:34:04PM +0100, Greg KH wrote:
> > > On Fri, Feb 13, 2026 at 03:26:08PM +0100, Peter Zijlstra wrote:
> > > > On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:
> > > > 
> > > > > C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
> > > > > driver [2].
> > > > 
> > > > Right. And that is *fine*.
> > > > 
> > 
> > Yes, that's fine because memcpy() in C is volatile and per-byte atomic.
> > 
> > > > > Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
> > > > > informed these are not safe to use if source or destination may incur
> > > > > data races, and that we need an operation that is volatile or byte-wise
> > > > > atomic [3].
> > > > 
> > > > Safe how? It should just copy N bytes. Whatever it thinks those bytes
> > > > are.
> > > > 
> > > > Nothing can guard against concurrent modification. If there is, you get
> > > > to keep the pieces. Pretending anything else is delusional.
> > > > 
> > > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > > to be 'BBBB'. The resulting copy can be any combination of
> > > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > > 
> > 
> > The idea is if using Rust's own `core::ptr::copy()` or
> > `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> > not semantically guaranteed atomic per byte (i.e. tearing can happen at
> > bit level, because they are not designed for using in case of data
> > races, and there is no defined asm implementation of them, compilers can
> > do anything).
> 
> Then why not just call the proper, in-kernel, arch specific, patched and
> tested to the end-of-the-earth, memcpy()?
> 

I believe you hadn't read my reply that we indeed call memcpy() here. So
I'm not going to reply this in case you mean something else.

> > > > No byte wise volatile barrier using nonsense is going to make this any
> > > > better.
> > 
> > It's byte-wise atomic [1], which should be guaranteed using asm to
> > implement, hence at least at byte level, they are atomic (and volatile
> > in our case).
> > 
> > [1]: https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p1478r5.html
> 
> Again, just use memcpy() please.
> 
> > > > 
> > > 
> > > I'm with Peter, just call memcpy() like the C code does, and you will be
> > > "fine" (with a note that "fine" better include checking the data really
> > 
> > We are. See v3, we actually use `memcpy()` for the copy (as I already
> > pointed out, Andreas made a mistake in this version), it's just
> > because it's per-byte atomic. What this "byte-wise atomic" does is
> > clearing things out.
> 
> clear what out?  It shouldn't need anything special for a memcpy.
> 

Well, in standard C, technically memcpy() has the same problem as Rust's
`core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
are vulnerable to data races. Our in-kernel memcpy() on the other hand
doesn't have this problem. Why? Because it's volatile byte-wise atomic
per the implementation.

So here, the clearing out is needed to say: this is not Rust's `copy()`
and this is not C's `memcpy()`, this is the kernel version, and it's
fine not because magic or kernel people believe it, but because its
implementation. The concept of byte-wise atomic at least describes this
correctly.

Regards,
Boqun

> thanks,
> 
> greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Gary Guo]

On 2026-02-13 15:34, Greg KH wrote:
> On Fri, Feb 13, 2026 at 03:26:08PM +0100, Peter Zijlstra wrote:
>> On Fri, Feb 13, 2026 at 03:13:01PM +0100, Andreas Hindborg wrote:
>> 
>> > C uses memcpy as seen in `bio_copy_data_iter` [1] and in the null_blk
>> > driver [2].
>> 
>> Right. And that is *fine*.
>> 
>> > Rust has `core::ptr::copy` and `core::ptr::copy_nonoverlapping`. I was
>> > informed these are not safe to use if source or destination may incur
>> > data races, and that we need an operation that is volatile or byte-wise
>> > atomic [3].
>> 
>> Safe how? It should just copy N bytes. Whatever it thinks those bytes
>> are.
>> 
>> Nothing can guard against concurrent modification. If there is, you get
>> to keep the pieces. Pretending anything else is delusional.
>> 
>> Suppose the memory was 'AAAA' and while you're reading it, it is written
>> to be 'BBBB'. The resulting copy can be any combination of
>> '[AB][AB][AB][AB]'. Not one of them is better than the other.
>> 
>> No byte wise volatile barrier using nonsense is going to make this any
>> better.
>> 
> 
> I'm with Peter, just call memcpy() like the C code does, and you will be
> "fine" (with a note that "fine" better include checking the data really
> is what you think it is if you are going to do anything based on it and
> not just pass it off to the hardware.)

As Boqun already pointed out in other thread, this *is* the direction that we're
heading for. We're not going to add a new implementation, just to add an API
with documented semantics which is distinct from existing Rsut
"copy_nonoverlapping", which in many cases is lowered to memcpy, but does not
have the correct semantics, as it is a compiler builtin, so compiler can
optimize on it and understand it as non-atomic non-volatile operation.

BTW, note that calling "memcpy" from C side doesn't actually have the right
semantics, as C, unlike Rust where compiler builtins are explicit, recognize
them by function names. So if you write "memcpy" without adding `-fno-builtin`
(or `-fno-builtin-memcpy`) flag, it *is* recognized by compiler as a normal,
non-atomic and non-volatile memcpy. The compiler is free to turn it to something
that is not an actual function call to "memcpy". It happens that when doing BIO
and copying a page, this is large enough that compilers always defer to actual
"memcpy" implementation, but this is another case that "it works in practice".

Best,
Gary

> 
> thanks,
> 
> greg k-h

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> Well, in standard C, technically memcpy() has the same problem as Rust's
> `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> are vulnerable to data races. Our in-kernel memcpy() on the other hand
> doesn't have this problem. Why? Because it's volatile byte-wise atomic
> per the implementation.

Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
byte-wise. Also, not a single atomic operation in sight.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:

> > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > to be 'BBBB'. The resulting copy can be any combination of
> > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > 
> 
> The idea is if using Rust's own `core::ptr::copy()` or
> `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> not semantically guaranteed atomic per byte (i.e. tearing can happen at
> bit level, because they are not designed for using in case of data
> races, and there is no defined asm implementation of them, compilers can
> do anything).

How the heck would they do out-of-thin-air? Any memcpy() implementation
that can cause that is insane and broken.

Compilers are broken crap if they do this.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 10:17:23AM +0100, Peter Zijlstra wrote:
> On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> 
> > > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > > to be 'BBBB'. The resulting copy can be any combination of
> > > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > > 
> > 
> > The idea is if using Rust's own `core::ptr::copy()` or
> > `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> > not semantically guaranteed atomic per byte (i.e. tearing can happen at
> > bit level, because they are not designed for using in case of data
> > races, and there is no defined asm implementation of them, compilers can
> > do anything).
> 
> How the heck would they do out-of-thin-air? Any memcpy() implementation
> that can cause that is insane and broken.
> 
> Compilers are broken crap if they do this.

If rust core code can cause this, I stand by my earlier position that we
should not want that anywhere near the kernel. There is a reason our C
code is free standing.

So fix Rust to not be broken or take it out.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > Well, in standard C, technically memcpy() has the same problem as Rust's
> > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > per the implementation.
> 
> Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> byte-wise.

movq is a valid implementation of 8 byte-wise copies.

> Also, not a single atomic operation in sight.

Relaxed atomics are just mov ops.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 10:17:23AM +0100, Peter Zijlstra wrote:
> On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> 
> > > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > > to be 'BBBB'. The resulting copy can be any combination of
> > > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > > 
> > 
> > The idea is if using Rust's own `core::ptr::copy()` or
> > `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> > not semantically guaranteed atomic per byte (i.e. tearing can happen at
> > bit level, because they are not designed for using in case of data
> > races, and there is no defined asm implementation of them, compilers can
> > do anything).
> 
> How the heck would they do out-of-thin-air? Any memcpy() implementation
> that can cause that is insane and broken.
> 
> Compilers are broken crap if they do this.

The only way I know for a compiler to cause out-of-thin-air is
speculative stores -- those MUST NOT happen. Just like hardware must not
do them.

We must all collectively say NO to a compiler that does these. Life is
too short to try and write concurrent programs with a compiler that does
this.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 10:23:43AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 10:17:23AM +0100, Peter Zijlstra wrote:
> > On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> > 
> > > > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > > > to be 'BBBB'. The resulting copy can be any combination of
> > > > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > > > 
> > > 
> > > The idea is if using Rust's own `core::ptr::copy()` or
> > > `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> > > not semantically guaranteed atomic per byte (i.e. tearing can happen at
> > > bit level, because they are not designed for using in case of data
> > > races, and there is no defined asm implementation of them, compilers can
> > > do anything).
> > 
> > How the heck would they do out-of-thin-air? Any memcpy() implementation
> > that can cause that is insane and broken.
> > 
> > Compilers are broken crap if they do this.
> 
> If rust core code can cause this, I stand by my earlier position that we
> should not want that anywhere near the kernel. There is a reason our C
> code is free standing.
> 
> So fix Rust to not be broken or take it out.

It can cause this to exactly the same extent as C can, and that's why
you all came up with READ_ONCE() and similar.

Rust's copy_nonoverlapping() might not call memcpy() if it's a small
fixed amount of bytes. E.g. a copy_nonoverlapping() of 8 bytes might
just emit a movq instruction or any other way the compiler can come up
with to copy 8 bytes. Like C compilers, that includes emitting more than
one read, for the same reasons as why C might emit multiple reads for a
single read when READ_ONCE() is not used.

After all, this logic all comes from LLVM which clang and rustc share.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 09:33:40AM +0000, Alice Ryhl wrote:
> On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> > On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > > Well, in standard C, technically memcpy() has the same problem as Rust's
> > > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > > per the implementation.
> > 
> > Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> > byte-wise.
> 
> movq is a valid implementation of 8 byte-wise copies.
> 
> > Also, not a single atomic operation in sight.
> 
> Relaxed atomics are just mov ops.

They are not atomics at all.

Somewhere along the line 'atomic' seems to have lost any and all meaning
:-(

It must be this C committee and their weasel speak for fear of reality
that has infected everyone or somesuch.

Anyway, all you really want is a normal memcpy and somehow Rust cannot
provide? WTF?!

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 09:37:46AM +0000, Alice Ryhl wrote:
> On Tue, Feb 17, 2026 at 10:23:43AM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 17, 2026 at 10:17:23AM +0100, Peter Zijlstra wrote:
> > > On Fri, Feb 13, 2026 at 07:45:19AM -0800, Boqun Feng wrote:
> > > 
> > > > > > Suppose the memory was 'AAAA' and while you're reading it, it is written
> > > > > > to be 'BBBB'. The resulting copy can be any combination of
> > > > > > '[AB][AB][AB][AB]'. Not one of them is better than the other.
> > > > > > 
> > > > 
> > > > The idea is if using Rust's own `core::ptr::copy()` or
> > > > `core::ptr::copy_nonoverlapping()`, you may get `CCCC`, because they are
> > > > not semantically guaranteed atomic per byte (i.e. tearing can happen at
> > > > bit level, because they are not designed for using in case of data
> > > > races, and there is no defined asm implementation of them, compilers can
> > > > do anything).
> > > 
> > > How the heck would they do out-of-thin-air? Any memcpy() implementation
> > > that can cause that is insane and broken.
> > > 
> > > Compilers are broken crap if they do this.
> > 
> > If rust core code can cause this, I stand by my earlier position that we
> > should not want that anywhere near the kernel. There is a reason our C
> > code is free standing.
> > 
> > So fix Rust to not be broken or take it out.
> 
> It can cause this to exactly the same extent as C can, and that's why
> you all came up with READ_ONCE() and similar.

So READ_ONCE() was mostly about inhibiting the compiler from re-loading
the value. We later added the no-tearing thing.

> Rust's copy_nonoverlapping() might not call memcpy() if it's a small
> fixed amount of bytes. E.g. a copy_nonoverlapping() of 8 bytes might
> just emit a movq instruction or any other way the compiler can come up
> with to copy 8 bytes. Like C compilers, that includes emitting more than
> one read, for the same reasons as why C might emit multiple reads for a
> single read when READ_ONCE() is not used.
> 
> After all, this logic all comes from LLVM which clang and rustc share.

Right, compiler is allowed to write code instead of emit memcpy() call.
But that code had better not be silly.

And since the C thing isn't silly, why would the Rust thing be silly?

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 10:45:15AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 09:33:40AM +0000, Alice Ryhl wrote:
> > On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> > > On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > > > Well, in standard C, technically memcpy() has the same problem as Rust's
> > > > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > > > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > > > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > > > per the implementation.
> > > 
> > > Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> > > byte-wise.
> > 
> > movq is a valid implementation of 8 byte-wise copies.
> > 
> > > Also, not a single atomic operation in sight.
> > 
> > Relaxed atomics are just mov ops.
> 
> They are not atomics at all.

Atomic loads and stores are just mov ops, right? Sure, RMW operations do
more complex stuff, but I'm pretty sure that relaxed atomic loads/stores
generally are compiled as mov ops.

> Somewhere along the line 'atomic' seems to have lost any and all meaning
> :-(
> 
> It must be this C committee and their weasel speak for fear of reality
> that has infected everyone or somesuch.
> 
> Anyway, all you really want is a normal memcpy and somehow Rust cannot
> provide? WTF?!

Forget about Rust for a moment.

Consider this code:

	// Is this ok?
	unsigned long *a, b;
	b = *a;
	if is_valid(b) {
	    // do stuff
	}

I can easily imagine that LLVM might optimize this into:

	// Uh oh!
	unsigned long *a, b;
	b = *a;
	if is_valid(*a) {  // <- this was "optimized"
	    // do stuff
	}

the argument being that you used an ordinary load of `a`, so it can be
assumed that there are no concurrent writes, so both reads are
guaranteed to return the same value.

So if `a` might be concurrently modified, then we are unhappy.

Of course, if *a is replaced with an atomic load such as READ_ONCE(a) an
optimization would no longer occur.

	// OK!
	unsigned long *a, b;
	b = READ_ONCE(a);
	if is_valid(b) {
	    // do stuff
	}

Now consider the following code:

	// Is this ok?
	unsigned long *a, b;
	memcpy(a, &b, sizeof(unsigned long));
	if is_valid(b) {
	    // do stuff
	}

If LLVM understands the memcpy in the same way as how it understands

	b = *a; // same as memcpy, right?

then by above discussion, the memcpy is not enough either. And Rust
documents that it may treat copy_nonoverlapping() in exactly that way,
which is why we want a memcpy where reading the values more than once is
not a permitted optimization. In most discussions of that topic, that's
called a per-byte atomic memcpy.

Does this optimization happen in the real world? I have no clue. I'd
rather not find out.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 10:01:56AM +0000, Alice Ryhl wrote:
> On Tue, Feb 17, 2026 at 10:45:15AM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 17, 2026 at 09:33:40AM +0000, Alice Ryhl wrote:
> > > On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> > > > On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > > > > Well, in standard C, technically memcpy() has the same problem as Rust's
> > > > > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > > > > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > > > > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > > > > per the implementation.
> > > > 
> > > > Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> > > > byte-wise.
> > > 
> > > movq is a valid implementation of 8 byte-wise copies.
> > > 
> > > > Also, not a single atomic operation in sight.
> > > 
> > > Relaxed atomics are just mov ops.
> > 
> > They are not atomics at all.
> 
> Atomic loads and stores are just mov ops, right? Sure, RMW operations do
> more complex stuff, but I'm pretty sure that relaxed atomic loads/stores
> generally are compiled as mov ops.

Yeah, because they're not in fact atomic. I have, on various occasions,
told people to not use atomic_t if all they end up doing is atomic_set()
and atomic_read(). They're just loads and stores, nothing atomic about
them.

They are just there to complete the interactions with the actual RmW
operations.

> > Somewhere along the line 'atomic' seems to have lost any and all meaning
> > :-(
> > 
> > It must be this C committee and their weasel speak for fear of reality
> > that has infected everyone or somesuch.
> > 
> > Anyway, all you really want is a normal memcpy and somehow Rust cannot
> > provide? WTF?!
> 
> Forget about Rust for a moment.
> 
> Consider this code:
> 
> 	// Is this ok?
> 	unsigned long *a, b;
> 	b = *a;
> 	if is_valid(b) {
> 	    // do stuff
> 	}

Syntax error on is_valid(), need opening ( after if.

> I can easily imagine that LLVM might optimize this into:
> 
> 	// Uh oh!
> 	unsigned long *a, b;
> 	b = *a;
> 	if is_valid(*a) {  // <- this was "optimized"
> 	    // do stuff
> 	}

Well, compiler would not do anything, since it wouldn't compile :-) But
sure, that is valid transform.

> the argument being that you used an ordinary load of `a`, so it can be
> assumed that there are no concurrent writes, so both reads are
> guaranteed to return the same value.
> 
> So if `a` might be concurrently modified, then we are unhappy.
> 
> Of course, if *a is replaced with an atomic load such as READ_ONCE(a) an
> optimization would no longer occur.

Stop using atomic for this. Is not atomic.

Key here is volatile, that indicates value can change outside of scope
and thus re-load is not valid. And I know C language people hates
volatile, but there it is.

> 	// OK!
> 	unsigned long *a, b;
> 	b = READ_ONCE(a);
> 	if is_valid(b) {
> 	    // do stuff
> 	}
> 
> Now consider the following code:
> 
> 	// Is this ok?
> 	unsigned long *a, b;
> 	memcpy(a, &b, sizeof(unsigned long));
> 	if is_valid(b) {
> 	    // do stuff
> 	}

Why the hell would you want to write that? But sure. I think similar but
less weird example would be with structures, where value copies end up
being similar to memcpy.

And in that case, you can still use volatile and compiler must not do
silly.

> If LLVM understands the memcpy in the same way as how it understands
> 
> 	b = *a; // same as memcpy, right?
> 
> then by above discussion, the memcpy is not enough either. And Rust
> documents that it may treat copy_nonoverlapping() in exactly that way,
> which is why we want a memcpy where reading the values more than once is
> not a permitted optimization. In most discussions of that topic, that's
> called a per-byte atomic memcpy.
> 
> Does this optimization happen in the real world? I have no clue. I'd
> rather not find out.

OK, but none of this has anything to do with atomic or byte-wise.

The whole byte-wise thing turns out to be about not allowing
out-of-thin-air. Nothing should ever allow that.

Anyway, normal userspace copies don't suffer this because accessing
userspace has enough magical crap around it to inhibit this optimization
in any case.

If its a shared mapping/DMA, you'd typically end up with barriers
anyway, and those have a memory clobber on them which tell the compiler
reloads aren't good.

So I'm still not exactly sure why this is a problem all of a sudden?

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 11:25:57AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 10:01:56AM +0000, Alice Ryhl wrote:
> > On Tue, Feb 17, 2026 at 10:45:15AM +0100, Peter Zijlstra wrote:
> > > On Tue, Feb 17, 2026 at 09:33:40AM +0000, Alice Ryhl wrote:
> > > > On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> > > > > On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > > > > > Well, in standard C, technically memcpy() has the same problem as Rust's
> > > > > > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > > > > > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > > > > > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > > > > > per the implementation.
> > > > > 
> > > > > Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> > > > > byte-wise.
> > > > 
> > > > movq is a valid implementation of 8 byte-wise copies.
> > > > 
> > > > > Also, not a single atomic operation in sight.
> > > > 
> > > > Relaxed atomics are just mov ops.
> > > 
> > > They are not atomics at all.
> > 
> > Atomic loads and stores are just mov ops, right? Sure, RMW operations do
> > more complex stuff, but I'm pretty sure that relaxed atomic loads/stores
> > generally are compiled as mov ops.
> 
> Yeah, because they're not in fact atomic. I have, on various occasions,
> told people to not use atomic_t if all they end up doing is atomic_set()
> and atomic_read(). They're just loads and stores, nothing atomic about
> them.
> 
> They are just there to complete the interactions with the actual RmW
> operations.
> 
> > > Somewhere along the line 'atomic' seems to have lost any and all meaning
> > > :-(
> > > 
> > > It must be this C committee and their weasel speak for fear of reality
> > > that has infected everyone or somesuch.
> > > 
> > > Anyway, all you really want is a normal memcpy and somehow Rust cannot
> > > provide? WTF?!
> > 
> > Forget about Rust for a moment.
> > 
> > Consider this code:
> > 
> > 	// Is this ok?
> > 	unsigned long *a, b;
> > 	b = *a;
> > 	if is_valid(b) {
> > 	    // do stuff
> > 	}
> 
> Syntax error on is_valid(), need opening ( after if.

Oops, too much Rust for me :)

> > I can easily imagine that LLVM might optimize this into:
> > 
> > 	// Uh oh!
> > 	unsigned long *a, b;
> > 	b = *a;
> > 	if is_valid(*a) {  // <- this was "optimized"
> > 	    // do stuff
> > 	}
> 
> Well, compiler would not do anything, since it wouldn't compile :-) But
> sure, that is valid transform.
> 
> > the argument being that you used an ordinary load of `a`, so it can be
> > assumed that there are no concurrent writes, so both reads are
> > guaranteed to return the same value.
> > 
> > So if `a` might be concurrently modified, then we are unhappy.
> > 
> > Of course, if *a is replaced with an atomic load such as READ_ONCE(a) an
> > optimization would no longer occur.
> 
> Stop using atomic for this. Is not atomic.
> 
> Key here is volatile, that indicates value can change outside of scope
> and thus re-load is not valid. And I know C language people hates
> volatile, but there it is.

Well, don't complain to me about this. I sent a patch to add READ_ONCE()/
WRITE_ONCE() impls for Rust and was told to just use atomics instead,
see: https://lwn.net/Articles/1053142/

> > 	// OK!
> > 	unsigned long *a, b;
> > 	b = READ_ONCE(a);
> > 	if is_valid(b) {
> > 	    // do stuff
> > 	}
> > 
> > Now consider the following code:
> > 
> > 	// Is this ok?
> > 	unsigned long *a, b;
> > 	memcpy(a, &b, sizeof(unsigned long));
> > 	if is_valid(b) {
> > 	    // do stuff
> > 	}
> 
> Why the hell would you want to write that? But sure. I think similar but
> less weird example would be with structures, where value copies end up
> being similar to memcpy.

I mean sure, let's say that it was a structure or whatever instead of a
long. The point is that the general pattern of memcpy, then checking the
bytes you copied, then use the bytes you copied, is potentially
susceptible to this exacty optimization.

> And in that case, you can still use volatile and compiler must not do
> silly.

What you mean by "volatile" here is the same as what this patch means
when it says "per-byte atomic". If you agree that a "volatile memcpy"
would be a good idea to use in this scenario, then it sounds like you
agree with the patch except for its naming / terminology.

> > If LLVM understands the memcpy in the same way as how it understands
> > 
> > 	b = *a; // same as memcpy, right?
> > 
> > then by above discussion, the memcpy is not enough either. And Rust
> > documents that it may treat copy_nonoverlapping() in exactly that way,
> > which is why we want a memcpy where reading the values more than once is
> > not a permitted optimization. In most discussions of that topic, that's
> > called a per-byte atomic memcpy.
> > 
> > Does this optimization happen in the real world? I have no clue. I'd
> > rather not find out.
> 
> OK, but none of this has anything to do with atomic or byte-wise.
> 
> The whole byte-wise thing turns out to be about not allowing
> out-of-thin-air. Nothing should ever allow that.

It's not just about out-of-thin-air, it's also the kind of optimization
I mentioned.

> Anyway, normal userspace copies don't suffer this because accessing
> userspace has enough magical crap around it to inhibit this optimization
> in any case.
> 
> If its a shared mapping/DMA, you'd typically end up with barriers
> anyway, and those have a memory clobber on them which tell the compiler
> reloads aren't good.
> 
> So I'm still not exactly sure why this is a problem all of a sudden?

I mean, this is for `struct page` specifically. If you have the struct
page for a page that might also be mapped into a userspace vma, then the
way to perform a "copy_from_user" operation is to:

1. kmap_local_page()
2. memcpy()
3. kunmap_local()

Correct me if I'm wrong, but my understanding is that on 64-bit systems,
kmap/kunmap are usually complete no-ops since you have enough address
space to simply map all pages into the kernel's address space. Not even
a barrier - just a `static inline` with an empty body.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 10:47:03AM +0000, Alice Ryhl wrote:

> > Stop using atomic for this. Is not atomic.
> > 
> > Key here is volatile, that indicates value can change outside of scope
> > and thus re-load is not valid. And I know C language people hates
> > volatile, but there it is.
> 
> Well, don't complain to me about this. I sent a patch to add READ_ONCE()/
> WRITE_ONCE() impls for Rust and was told to just use atomics instead,
> see: https://lwn.net/Articles/1053142/

*groan*

> > > 	// OK!
> > > 	unsigned long *a, b;
> > > 	b = READ_ONCE(a);
> > > 	if is_valid(b) {
> > > 	    // do stuff
> > > 	}
> > > 
> > > Now consider the following code:
> > > 
> > > 	// Is this ok?
> > > 	unsigned long *a, b;
> > > 	memcpy(a, &b, sizeof(unsigned long));
> > > 	if is_valid(b) {
> > > 	    // do stuff
> > > 	}
> > 
> > Why the hell would you want to write that? But sure. I think similar but
> > less weird example would be with structures, where value copies end up
> > being similar to memcpy.
> 
> I mean sure, let's say that it was a structure or whatever instead of a
> long. The point is that the general pattern of memcpy, then checking the
> bytes you copied, then use the bytes you copied, is potentially
> susceptible to this exacty optimization.

> > And in that case, you can still use volatile and compiler must not do
> > silly.
> 
> What you mean by "volatile" here is the same as what this patch means
> when it says "per-byte atomic". If you agree that a "volatile memcpy"
> would be a good idea to use in this scenario, then it sounds like you
> agree with the patch except for its naming / terminology.

  struct foo {
    int a, b;
  };

  struct foo *ptr, val;

  val = *(volatile struct foo *)ptr;

why would we need a an explicit new memcpy for this?

> > So I'm still not exactly sure why this is a problem all of a sudden?
> 
> I mean, this is for `struct page` specifically. If you have the struct
> page for a page that might also be mapped into a userspace vma, then the
> way to perform a "copy_from_user" operation is to:
> 
> 1. kmap_local_page()
> 2. memcpy()
> 3. kunmap_local()
> 
> Correct me if I'm wrong, but my understanding is that on 64-bit systems,
> kmap/kunmap are usually complete no-ops since you have enough address
> space to simply map all pages into the kernel's address space. Not even
> a barrier - just a `static inline` with an empty body.

That is all correct -- however that cannot be all you do.

Any shared memory will involved memory barriers of a sort. You cannot
just memcpy() and think you're done.

So yeah, on x86_64 those 1,2,3 are insufficient to inhibit the re-load,
but nobody should ever just do 1,2,3 and think job-done. There must
always be more.

If it is a ring-buffer like thing, you get:

         *   if (LOAD ->data_tail) {            LOAD ->data_head
         *                      (A)             smp_rmb()       (C)
         *      STORE $data                     LOAD $data
         *      smp_wmb()       (B)             smp_mb()        (D)
         *      STORE ->data_head               STORE ->data_tail
         *   }

if it is a seqlock like thing you get that.

If it is DMA, you need dma fences.

And the moment you use any of that, the re-load goes out the window.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 12:09:11PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 10:47:03AM +0000, Alice Ryhl wrote:
> > > > 	// OK!
> > > > 	unsigned long *a, b;
> > > > 	b = READ_ONCE(a);
> > > > 	if is_valid(b) {
> > > > 	    // do stuff
> > > > 	}
> > > > 
> > > > Now consider the following code:
> > > > 
> > > > 	// Is this ok?
> > > > 	unsigned long *a, b;
> > > > 	memcpy(a, &b, sizeof(unsigned long));
> > > > 	if is_valid(b) {
> > > > 	    // do stuff
> > > > 	}
> > > 
> > > Why the hell would you want to write that? But sure. I think similar but
> > > less weird example would be with structures, where value copies end up
> > > being similar to memcpy.
> > 
> > I mean sure, let's say that it was a structure or whatever instead of a
> > long. The point is that the general pattern of memcpy, then checking the
> > bytes you copied, then use the bytes you copied, is potentially
> > susceptible to this exacty optimization.
> 
> > > And in that case, you can still use volatile and compiler must not do
> > > silly.
> > 
> > What you mean by "volatile" here is the same as what this patch means
> > when it says "per-byte atomic". If you agree that a "volatile memcpy"
> > would be a good idea to use in this scenario, then it sounds like you
> > agree with the patch except for its naming / terminology.
> 
>   struct foo {
>     int a, b;
>   };
> 
>   struct foo *ptr, val;
> 
>   val = *(volatile struct foo *)ptr;
> 
> why would we need a an explicit new memcpy for this?

In my experience with dealing with `struct page` that is mapped into a
vma, you need memcpy because the struct might be split across two
different pages in the vma. The pages are adjacent in userspace's
address space, but not necessarily adjacent from the kernel's POV.

So you might end up with something that looks like this:

struct foo val;
void *ptr1 = kmap_local_page(p1);
void *ptr2 = kmap_local_page(p2);
memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
kunmap_local(ptr2);
kunmap_local(ptr1);

if (is_valid(&val)) {
    // use val
}

This exact thing happens in Binder. It has to be a memcpy.

> > > So I'm still not exactly sure why this is a problem all of a sudden?
> > 
> > I mean, this is for `struct page` specifically. If you have the struct
> > page for a page that might also be mapped into a userspace vma, then the
> > way to perform a "copy_from_user" operation is to:
> > 
> > 1. kmap_local_page()
> > 2. memcpy()
> > 3. kunmap_local()
> > 
> > Correct me if I'm wrong, but my understanding is that on 64-bit systems,
> > kmap/kunmap are usually complete no-ops since you have enough address
> > space to simply map all pages into the kernel's address space. Not even
> > a barrier - just a `static inline` with an empty body.
> 
> That is all correct -- however that cannot be all you do.
> 
> Any shared memory will involved memory barriers of a sort. You cannot
> just memcpy() and think you're done.
> 
> So yeah, on x86_64 those 1,2,3 are insufficient to inhibit the re-load,
> but nobody should ever just do 1,2,3 and think job-done. There must
> always be more.
> 
> If it is a ring-buffer like thing, you get:
> 
>          *   if (LOAD ->data_tail) {            LOAD ->data_head
>          *                      (A)             smp_rmb()       (C)
>          *      STORE $data                     LOAD $data
>          *      smp_wmb()       (B)             smp_mb()        (D)
>          *      STORE ->data_head               STORE ->data_tail
>          *   }
> 
> if it is a seqlock like thing you get that.
> 
> If it is DMA, you need dma fences.
> 
> And the moment you use any of that, the re-load goes out the window.

I don't know how Andreas is using this, but the usage pattern I'm
familiar with for `struct page` from my work on Binder is this one:

1. memcpy into the page
2. return from ioctl
3. userspace reads from vma

or

1. userspace writes to vma
2. call ioctl
3. kernel reads from page

which needs no barriers whatsoever. There is nothing to prevent this
kind of optimization in this kind of code, so an evil userspace could
trigger TOCTOU bugs in the kernel that are not present in the source
code if the code was optimized like I described.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 11:51:20AM +0000, Alice Ryhl wrote:

> In my experience with dealing with `struct page` that is mapped into a
> vma, you need memcpy because the struct might be split across two
> different pages in the vma. The pages are adjacent in userspace's
> address space, but not necessarily adjacent from the kernel's POV.
> 
> So you might end up with something that looks like this:
> 
> struct foo val;
> void *ptr1 = kmap_local_page(p1);
> void *ptr2 = kmap_local_page(p2);
> memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
> memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
> kunmap_local(ptr2);
> kunmap_local(ptr1);

  barrier();

> if (is_valid(&val)) {
>     // use val
> }
> 
> This exact thing happens in Binder. It has to be a memcpy.

Sure, but then stick that one barrier() in and you're good.


> I don't know how Andreas is using this, but the usage pattern I'm
> familiar with for `struct page` from my work on Binder is this one:
> 
> 1. memcpy into the page
> 2. return from ioctl
> 3. userspace reads from vma
> 
> or
> 
> 1. userspace writes to vma
> 2. call ioctl
> 3. kernel reads from page
> 
> which needs no barriers whatsoever. There is nothing to prevent this
> kind of optimization in this kind of code, so an evil userspace could
> trigger TOCTOU bugs in the kernel that are not present in the source
> code if the code was optimized like I described.

Then stick in barrier().

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 01:09:20PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 11:51:20AM +0000, Alice Ryhl wrote:
> 
> > In my experience with dealing with `struct page` that is mapped into a
> > vma, you need memcpy because the struct might be split across two
> > different pages in the vma. The pages are adjacent in userspace's
> > address space, but not necessarily adjacent from the kernel's POV.
> > 
> > So you might end up with something that looks like this:
> > 
> > struct foo val;
> > void *ptr1 = kmap_local_page(p1);
> > void *ptr2 = kmap_local_page(p2);
> > memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
> > memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
> > kunmap_local(ptr2);
> > kunmap_local(ptr1);
> 
>   barrier();
> 
> > if (is_valid(&val)) {
> >     // use val
> > }
> > 
> > This exact thing happens in Binder. It has to be a memcpy.
> 
> Sure, but then stick that one barrier() in and you're good.

Anyway, I don't think something like the below is an unreasonable patch.

It ensures all accesses to the ptr obtained from kmap_local_*() and
released by kunmap_local() stays inside those two.

---
diff --git a/include/linux/highmem-internal.h b/include/linux/highmem-internal.h
index 0574c21ca45d..2fe71b715a46 100644
--- a/include/linux/highmem-internal.h
+++ b/include/linux/highmem-internal.h
@@ -185,31 +185,42 @@ static inline void kunmap(const struct page *page)
 
 static inline void *kmap_local_page(const struct page *page)
 {
-	return page_address(page);
+	void *addr = page_address(page);
+	barrier();
+	return addr;
 }
 
 static inline void *kmap_local_page_try_from_panic(const struct page *page)
 {
-	return page_address(page);
+	void *addr = page_address(page);
+	barrier();
+	return addr;
 }
 
 static inline void *kmap_local_folio(const struct folio *folio, size_t offset)
 {
-	return folio_address(folio) + offset;
+	void *addr = folio_address(folio) + offset;
+	barrier();
+	return addr;
 }
 
 static inline void *kmap_local_page_prot(const struct page *page, pgprot_t prot)
 {
-	return kmap_local_page(page);
+	void *addr = kmap_local_page(page);
+	barrier();
+	return addr;
 }
 
 static inline void *kmap_local_pfn(unsigned long pfn)
 {
-	return kmap_local_page(pfn_to_page(pfn));
+	void *addr = kmap_local_page(pfn_to_page(pfn));
+	barrier();
+	return addr;
 }
 
 static inline void __kunmap_local(const void *addr)
 {
+	barrier();
 #ifdef ARCH_HAS_FLUSH_ON_KUNMAP
 	kunmap_flush_on_unmap(PTR_ALIGN_DOWN(addr, PAGE_SIZE));
 #endif

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Tue, Feb 17, 2026 at 01:09:20PM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 11:51:20AM +0000, Alice Ryhl wrote:
> 
> > In my experience with dealing with `struct page` that is mapped into a
> > vma, you need memcpy because the struct might be split across two
> > different pages in the vma. The pages are adjacent in userspace's
> > address space, but not necessarily adjacent from the kernel's POV.
> > 
> > So you might end up with something that looks like this:
> > 
> > struct foo val;
> > void *ptr1 = kmap_local_page(p1);
> > void *ptr2 = kmap_local_page(p2);
> > memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
> > memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
> > kunmap_local(ptr2);
> > kunmap_local(ptr1);
> 
>   barrier();
> 
> > if (is_valid(&val)) {
> >     // use val
> > }
> > 
> > This exact thing happens in Binder. It has to be a memcpy.
> 
> Sure, but then stick that one barrier() in and you're good.

Are we really good? Consider this code:

	bool is_valid(struct foo *val)
	{
		// for the sake of example
		return val->my_field != 0;
	}

	struct foo val;

	void *ptr = kmap_local_page(p1);
	memcpy(ptr, val, sizeof(struct foo));
	kunmap_local(p);
	barrier();
	if (is_valid(&val)) {
	    // use val
	}

optimize it into this first:

	struct foo val;
	int my_field_copy;

	void *ptr = kmap_local_page(p1);
	memcpy(ptr, val, sizeof(struct foo));
	my_field_copy = val->my_field;
	kunmap_local(p);
	barrier();
	if (my_field_copy != 0) {
	    // use val
	}

then optimize it into:

	struct foo val;
	int my_field_copy;

	void *ptr = kmap_local_page(p1);
	memcpy(ptr, val, sizeof(struct foo));
	my_field_copy = ((struct foo *) ptr)->my_field;
	kunmap_local(p);
	barrier();
	if (my_field_copy != 0) {
	    // use val
	}

oops!

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Danilo Krummrich]

On Tue Feb 17, 2026 at 2:00 PM CET, Peter Zijlstra wrote:
> Anyway, I don't think something like the below is an unreasonable patch.
>
> It ensures all accesses to the ptr obtained from kmap_local_*() and
> released by kunmap_local() stays inside those two.

I'd argue that not ensuring this is a feature, as I don't see why we would want
to ensure this if !CONFIG_HIGHMEM.

I think this is not about not escaping a critical scope, but about ensuring to
read exactly once.

> ---
> diff --git a/include/linux/highmem-internal.h b/include/linux/highmem-internal.h
> index 0574c21ca45d..2fe71b715a46 100644
> --- a/include/linux/highmem-internal.h
> +++ b/include/linux/highmem-internal.h
> @@ -185,31 +185,42 @@ static inline void kunmap(const struct page *page)
>  
>  static inline void *kmap_local_page(const struct page *page)
>  {
> -	return page_address(page);
> +	void *addr = page_address(page);
> +	barrier();
> +	return addr;
>  }
>  
>  static inline void *kmap_local_page_try_from_panic(const struct page *page)
>  {
> -	return page_address(page);
> +	void *addr = page_address(page);
> +	barrier();
> +	return addr;
>  }
>  
>  static inline void *kmap_local_folio(const struct folio *folio, size_t offset)
>  {
> -	return folio_address(folio) + offset;
> +	void *addr = folio_address(folio) + offset;
> +	barrier();
> +	return addr;
>  }
>  
>  static inline void *kmap_local_page_prot(const struct page *page, pgprot_t prot)
>  {
> -	return kmap_local_page(page);
> +	void *addr = kmap_local_page(page);
> +	barrier();
> +	return addr;
>  }
>  
>  static inline void *kmap_local_pfn(unsigned long pfn)
>  {
> -	return kmap_local_page(pfn_to_page(pfn));
> +	void *addr = kmap_local_page(pfn_to_page(pfn));
> +	barrier();
> +	return addr;
>  }
>  
>  static inline void __kunmap_local(const void *addr)
>  {
> +	barrier();
>  #ifdef ARCH_HAS_FLUSH_ON_KUNMAP
>  	kunmap_flush_on_unmap(PTR_ALIGN_DOWN(addr, PAGE_SIZE));
>  #endif

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

"Alice Ryhl" <aliceryhl@google.com> writes:

> On Tue, Feb 17, 2026 at 12:09:11PM +0100, Peter Zijlstra wrote:
>> On Tue, Feb 17, 2026 at 10:47:03AM +0000, Alice Ryhl wrote:
>> > > > 	// OK!
>> > > > 	unsigned long *a, b;
>> > > > 	b = READ_ONCE(a);
>> > > > 	if is_valid(b) {
>> > > > 	    // do stuff
>> > > > 	}
>> > > >
>> > > > Now consider the following code:
>> > > >
>> > > > 	// Is this ok?
>> > > > 	unsigned long *a, b;
>> > > > 	memcpy(a, &b, sizeof(unsigned long));
>> > > > 	if is_valid(b) {
>> > > > 	    // do stuff
>> > > > 	}
>> > >
>> > > Why the hell would you want to write that? But sure. I think similar but
>> > > less weird example would be with structures, where value copies end up
>> > > being similar to memcpy.
>> >
>> > I mean sure, let's say that it was a structure or whatever instead of a
>> > long. The point is that the general pattern of memcpy, then checking the
>> > bytes you copied, then use the bytes you copied, is potentially
>> > susceptible to this exacty optimization.
>>
>> > > And in that case, you can still use volatile and compiler must not do
>> > > silly.
>> >
>> > What you mean by "volatile" here is the same as what this patch means
>> > when it says "per-byte atomic". If you agree that a "volatile memcpy"
>> > would be a good idea to use in this scenario, then it sounds like you
>> > agree with the patch except for its naming / terminology.
>>
>>   struct foo {
>>     int a, b;
>>   };
>>
>>   struct foo *ptr, val;
>>
>>   val = *(volatile struct foo *)ptr;
>>
>> why would we need a an explicit new memcpy for this?
>
> In my experience with dealing with `struct page` that is mapped into a
> vma, you need memcpy because the struct might be split across two
> different pages in the vma. The pages are adjacent in userspace's
> address space, but not necessarily adjacent from the kernel's POV.
>
> So you might end up with something that looks like this:
>
> struct foo val;
> void *ptr1 = kmap_local_page(p1);
> void *ptr2 = kmap_local_page(p2);
> memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
> memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
> kunmap_local(ptr2);
> kunmap_local(ptr1);
>
> if (is_valid(&val)) {
>     // use val
> }
>
> This exact thing happens in Binder. It has to be a memcpy.
>
>> > > So I'm still not exactly sure why this is a problem all of a sudden?
>> >
>> > I mean, this is for `struct page` specifically. If you have the struct
>> > page for a page that might also be mapped into a userspace vma, then the
>> > way to perform a "copy_from_user" operation is to:
>> >
>> > 1. kmap_local_page()
>> > 2. memcpy()
>> > 3. kunmap_local()
>> >
>> > Correct me if I'm wrong, but my understanding is that on 64-bit systems,
>> > kmap/kunmap are usually complete no-ops since you have enough address
>> > space to simply map all pages into the kernel's address space. Not even
>> > a barrier - just a `static inline` with an empty body.
>>
>> That is all correct -- however that cannot be all you do.
>>
>> Any shared memory will involved memory barriers of a sort. You cannot
>> just memcpy() and think you're done.
>>
>> So yeah, on x86_64 those 1,2,3 are insufficient to inhibit the re-load,
>> but nobody should ever just do 1,2,3 and think job-done. There must
>> always be more.
>>
>> If it is a ring-buffer like thing, you get:
>>
>>          *   if (LOAD ->data_tail) {            LOAD ->data_head
>>          *                      (A)             smp_rmb()       (C)
>>          *      STORE $data                     LOAD $data
>>          *      smp_wmb()       (B)             smp_mb()        (D)
>>          *      STORE ->data_head               STORE ->data_tail
>>          *   }
>>
>> if it is a seqlock like thing you get that.
>>
>> If it is DMA, you need dma fences.
>>
>> And the moment you use any of that, the re-load goes out the window.
>
> I don't know how Andreas is using this, but the usage pattern I'm
> familiar with for `struct page` from my work on Binder is this one:
>
> 1. memcpy into the page
> 2. return from ioctl
> 3. userspace reads from vma
>
> or
>
> 1. userspace writes to vma
> 2. call ioctl
> 3. kernel reads from page
>
> which needs no barriers whatsoever. There is nothing to prevent this
> kind of optimization in this kind of code, so an evil userspace could
> trigger TOCTOU bugs in the kernel that are not present in the source
> code if the code was optimized like I described.

I'm processing disk IO in the Rust null block driver. The pages backing
the IO requests may be simultaneously mapped to user space, so there is
no way to guarantee that there is no concurrent memory operation to/from
the memory area. User space programs can do whatever.

I don't have any control flow depending on the data I copy. I just store
it somewhere and return it if a read IO for the same sector arrive.

Let me add a bit of context as to why I sent this patch. I am not an
expert on the finer details of this subject, so I rely on available
expertise in our community. It is my understanding that copying the
memory in the situation outlined above, without special consideration
(in Rust) would be undefined behavior. Such are the rules of the
language. Of course I don't want undefined behavior in the Rust null
block driver. When asked how to solve this, the experts suggested
defining this byte-wise atomic memory copy function, A function that
would have well defined behavior in this particular situation.

That seems like a reasonable course of action to me. I don't understand
why this is such a big deal, and I don't understand the need to use
aggressive language and swearing.

I would suggest that if we are failing to understand each other, or if
we are miscommunicating, let's be curious instead of angry. It get's us
to where we want to be, way faster. And it is a much more pleasant
journey.


Best regards,
Andreas Hindborg

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 01:09:39PM +0000, Alice Ryhl wrote:
> On Tue, Feb 17, 2026 at 01:09:20PM +0100, Peter Zijlstra wrote:
> > On Tue, Feb 17, 2026 at 11:51:20AM +0000, Alice Ryhl wrote:
> > 
> > > In my experience with dealing with `struct page` that is mapped into a
> > > vma, you need memcpy because the struct might be split across two
> > > different pages in the vma. The pages are adjacent in userspace's
> > > address space, but not necessarily adjacent from the kernel's POV.
> > > 
> > > So you might end up with something that looks like this:
> > > 
> > > struct foo val;
> > > void *ptr1 = kmap_local_page(p1);
> > > void *ptr2 = kmap_local_page(p2);
> > > memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
> > > memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
> > > kunmap_local(ptr2);
> > > kunmap_local(ptr1);
> > 
> >   barrier();
> > 
> > > if (is_valid(&val)) {
> > >     // use val
> > > }
> > > 
> > > This exact thing happens in Binder. It has to be a memcpy.
> > 
> > Sure, but then stick that one barrier() in and you're good.
> 
> Are we really good? Consider this code:
> 
> 	bool is_valid(struct foo *val)
> 	{
> 		// for the sake of example
> 		return val->my_field != 0;
> 	}
> 
> 	struct foo val;
> 
> 	void *ptr = kmap_local_page(p1);
> 	memcpy(ptr, val, sizeof(struct foo));
> 	kunmap_local(p);
> 	barrier();
> 	if (is_valid(&val)) {
> 	    // use val
> 	}
> 
> optimize it into this first:
> 
> 	struct foo val;
> 	int my_field_copy;
> 
> 	void *ptr = kmap_local_page(p1);
> 	memcpy(ptr, val, sizeof(struct foo));
> 	my_field_copy = val->my_field;
> 	kunmap_local(p);
> 	barrier();
> 	if (my_field_copy != 0) {
> 	    // use val
> 	}
> 
> then optimize it into:
> 
> 	struct foo val;
> 	int my_field_copy;
> 
> 	void *ptr = kmap_local_page(p1);
> 	memcpy(ptr, val, sizeof(struct foo));
> 	my_field_copy = ((struct foo *) ptr)->my_field;
> 	kunmap_local(p);
> 	barrier();
> 	if (my_field_copy != 0) {
> 	    // use val
> 	}

I don;t think this is allowed. You're lifting the load over the
barrier(), that is invalid.

So the initial version is:

	ptr = kmap_local_page()
	memcpy(ptr, val, sizeof(val))
	kunmap_local(ptr);

	barrier()

	if (val.field)
	  // do stuff

So the 'val.field' load is after the barrier(); and it must stay there,
because the barrier() just told the compiler that all of memory changed
-- this is what barrier() does.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 02:54:30PM +0100, Danilo Krummrich wrote:
> On Tue Feb 17, 2026 at 2:00 PM CET, Peter Zijlstra wrote:
> > Anyway, I don't think something like the below is an unreasonable patch.
> >
> > It ensures all accesses to the ptr obtained from kmap_local_*() and
> > released by kunmap_local() stays inside those two.
> 
> I'd argue that not ensuring this is a feature, as I don't see why we would want
> to ensure this if !CONFIG_HIGHMEM.

Because of the principle of least surprise. For the HIGHMEM case 'ptr'
only lives between kmap and kunmap and any access must necessarily be
confined in between those. Having the code behave differently for
!HIGHMEM is surprising.

> I think this is not about not escaping a critical scope, but about ensuring to
> read exactly once.

It ends up being the same thing.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Boqun Feng]

On Tue, Feb 17, 2026 at 10:13:48AM +0100, Peter Zijlstra wrote:
> On Fri, Feb 13, 2026 at 08:19:17AM -0800, Boqun Feng wrote:
> > Well, in standard C, technically memcpy() has the same problem as Rust's
> > `core::ptr::copy()` and `core::ptr::copy_nonoverlapping()`, i.e. they
> > are vulnerable to data races. Our in-kernel memcpy() on the other hand
> > doesn't have this problem. Why? Because it's volatile byte-wise atomic
> > per the implementation.
> 
> Look at arch/x86/lib/memcpy_64.S, plenty of movq variants there. Not
> byte-wise. Also, not a single atomic operation in sight.
> 

Sigh, byte-wise means you at least need to byte-wise atomic, if movq is
atomic at 32bit or 64 bit level in x86, then it's fine implementation
for byte wise. Or you are saying on x86 it can tear at byte level?

Regards,
Boqun

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 02:56:40PM +0100, Andreas Hindborg wrote:

> I'm processing disk IO in the Rust null block driver. The pages backing
> the IO requests may be simultaneously mapped to user space, so there is
> no way to guarantee that there is no concurrent memory operation to/from
> the memory area. User space programs can do whatever.
> 
> I don't have any control flow depending on the data I copy. I just store
> it somewhere and return it if a read IO for the same sector arrive.

Right, so IIRC the old DIO code used to have this problem. We'd end up
writing whatever random state to disk if you did DIO of an mmap().

And if IIRC the current state of things is better in that we ensure the
mapping becomes RO and we have writes fault and wait until the writeback
is complete, ensuring things are somewhat more consistent.

But you'd better ask Jens or someone that has looked at the various IO
paths in the past 10 years or so :-)

> Let me add a bit of context as to why I sent this patch. I am not an
> expert on the finer details of this subject, so I rely on available
> expertise in our community. It is my understanding that copying the
> memory in the situation outlined above, without special consideration
> (in Rust) would be undefined behavior. Such are the rules of the
> language. Of course I don't want undefined behavior in the Rust null
> block driver. When asked how to solve this, the experts suggested
> defining this byte-wise atomic memory copy function, A function that
> would have well defined behavior in this particular situation.

Yeah, so being a C programmer, stepping in UB and tearing up the spec is
what you do on the daily. Its called reality :-)

In reality it is *really* hard to have memcpy() not be sane. And if the
Rust spec doesn't outlaw out-of-thin-air, then the Rust spec is wrong.
It really is that simple.

> That seems like a reasonable course of action to me. I don't understand
> why this is such a big deal, and I don't understand the need to use
> aggressive language and swearing.

Feh, there hardly was any o that :-) Call it cultural differences and
show how inclusive you are by being open to how other people have
different norms, whahaha :-)

I've heard it said that in Scottish 'fucking' is the mere announcement
of a noun.

> I would suggest that if we are failing to understand each other, or if
> we are miscommunicating, let's be curious instead of angry. It get's us
> to where we want to be, way faster. And it is a much more pleasant
> journey.

/me mumbles something and quickly drops the mic before the PC police get
here...

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Danilo Krummrich]

On Tue Feb 17, 2026 at 4:50 PM CET, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 02:54:30PM +0100, Danilo Krummrich wrote:
>> On Tue Feb 17, 2026 at 2:00 PM CET, Peter Zijlstra wrote:
>> > Anyway, I don't think something like the below is an unreasonable patch.
>> >
>> > It ensures all accesses to the ptr obtained from kmap_local_*() and
>> > released by kunmap_local() stays inside those two.
>> 
>> I'd argue that not ensuring this is a feature, as I don't see why we would want
>> to ensure this if !CONFIG_HIGHMEM.
>
> Because of the principle of least surprise. For the HIGHMEM case 'ptr'
> only lives between kmap and kunmap and any access must necessarily be
> confined in between those. Having the code behave differently for
> !HIGHMEM is surprising.

Well, I'd argue that those are two different things.

I agree that correct code does only ever access the pointer between kmap and
kunmap (and in Rust we could even enforce this at compile time through the type
system).

But the memory barrier is not required, i.e. if the compiler reorders things if
!CONFIG_HIGHMEM, that's fine.

I think it would be wrong if users would - for their code - rely on
"kunmap_local() already issues a memory barrier for me". It's an implementation
detail that depends on CONFIG_HIGHMEM, not an API guarantee.

>> I think this is not about not escaping a critical scope, but about ensuring to
>> read exactly once.
>
> It ends up being the same thing.

Reading a value multiple times within a certain scope and reading it exactly
once is a difference, no?

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Andreas Hindborg]

"Peter Zijlstra" <peterz@infradead.org> writes:

> On Tue, Feb 17, 2026 at 02:56:40PM +0100, Andreas Hindborg wrote:
>
>> I'm processing disk IO in the Rust null block driver. The pages backing
>> the IO requests may be simultaneously mapped to user space, so there is
>> no way to guarantee that there is no concurrent memory operation to/from
>> the memory area. User space programs can do whatever.
>>
>> I don't have any control flow depending on the data I copy. I just store
>> it somewhere and return it if a read IO for the same sector arrive.
>
> Right, so IIRC the old DIO code used to have this problem. We'd end up
> writing whatever random state to disk if you did DIO of an mmap().
>
> And if IIRC the current state of things is better in that we ensure the
> mapping becomes RO and we have writes fault and wait until the writeback
> is complete, ensuring things are somewhat more consistent.
>
> But you'd better ask Jens or someone that has looked at the various IO
> paths in the past 10 years or so :-)

Oh, this is a really important detail that I did not find while trying
to follow the code path from user space.

Cc: Jens Axboe <axboe@kernel.dk>

@Jens is this so, are pages from user space that are part of a write
request mapped RO during the IO operation?

>
>> Let me add a bit of context as to why I sent this patch. I am not an
>> expert on the finer details of this subject, so I rely on available
>> expertise in our community. It is my understanding that copying the
>> memory in the situation outlined above, without special consideration
>> (in Rust) would be undefined behavior. Such are the rules of the
>> language. Of course I don't want undefined behavior in the Rust null
>> block driver. When asked how to solve this, the experts suggested
>> defining this byte-wise atomic memory copy function, A function that
>> would have well defined behavior in this particular situation.
>
> Yeah, so being a C programmer, stepping in UB and tearing up the spec is
> what you do on the daily. Its called reality :-)

I see. From my observations, "Rust people" in general have a somewhat
different approach to UB. An approach where we avoid it. We would rather
fix the language so that we can do what we need to do, in a well defined
manner.

>
> In reality it is *really* hard to have memcpy() not be sane. And if the
> Rust spec doesn't outlaw out-of-thin-air, then the Rust spec is wrong.
> It really is that simple.

I'm at the far end of my knowledge here, but I believe that I read that
the theoretical model allows OOTA but that this is considered a bug of
the model. I'm not sure what to do with this information though.

>
>> That seems like a reasonable course of action to me. I don't understand
>> why this is such a big deal, and I don't understand the need to use
>> aggressive language and swearing.
>
> Feh, there hardly was any o that :-) Call it cultural differences and
> show how inclusive you are by being open to how other people have
> different norms, whahaha :-)

Touche.


Best regards,
Andreas Hindborg

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Jens Axboe]

On 2/17/26 11:43 AM, Andreas Hindborg wrote:
> "Peter Zijlstra" <peterz@infradead.org> writes:
> 
>> On Tue, Feb 17, 2026 at 02:56:40PM +0100, Andreas Hindborg wrote:
>>
>>> I'm processing disk IO in the Rust null block driver. The pages backing
>>> the IO requests may be simultaneously mapped to user space, so there is
>>> no way to guarantee that there is no concurrent memory operation to/from
>>> the memory area. User space programs can do whatever.
>>>
>>> I don't have any control flow depending on the data I copy. I just store
>>> it somewhere and return it if a read IO for the same sector arrive.
>>
>> Right, so IIRC the old DIO code used to have this problem. We'd end up
>> writing whatever random state to disk if you did DIO of an mmap().
>>
>> And if IIRC the current state of things is better in that we ensure the
>> mapping becomes RO and we have writes fault and wait until the writeback
>> is complete, ensuring things are somewhat more consistent.
>>
>> But you'd better ask Jens or someone that has looked at the various IO
>> paths in the past 10 years or so :-)
> 
> Oh, this is a really important detail that I did not find while trying
> to follow the code path from user space.
> 
> Cc: Jens Axboe <axboe@kernel.dk>
> 
> @Jens is this so, are pages from user space that are part of a write
> request mapped RO during the IO operation?

No, there's no such thing. It'd make O_DIRECT writes much slower.

Also see the recent stable pages merge that went into upstream this
merge window:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4adc13ed7c281c16152a700e47b65d17de07321a

which only really deals with the checksumming problem, but regardless
it's in the same area of "userspace modifies page(s) while IO is
in-flight". If you don't do checksums, and you rely on bufferA ending up
on stable storage while having it in-flight and also modifying bufferA
simultaneously from your application, you get to keep both broken pieces

-- 
Jens Axboe

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Gary Guo]

On 2026-02-17 15:48, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 01:09:39PM +0000, Alice Ryhl wrote:
>> On Tue, Feb 17, 2026 at 01:09:20PM +0100, Peter Zijlstra wrote:
>> > On Tue, Feb 17, 2026 at 11:51:20AM +0000, Alice Ryhl wrote:
>> > 
>> > > In my experience with dealing with `struct page` that is mapped into a
>> > > vma, you need memcpy because the struct might be split across two
>> > > different pages in the vma. The pages are adjacent in userspace's
>> > > address space, but not necessarily adjacent from the kernel's POV.
>> > > 
>> > > So you might end up with something that looks like this:
>> > > 
>> > > struct foo val;
>> > > void *ptr1 = kmap_local_page(p1);
>> > > void *ptr2 = kmap_local_page(p2);
>> > > memcpy(ptr1 + offset, val, PAGE_SIZE - offset);
>> > > memcpy(ptr2, val + offset, sizeof(struct foo) - (PAGE_SIZE - offset));
>> > > kunmap_local(ptr2);
>> > > kunmap_local(ptr1);
>> > 
>> >   barrier();
>> > 
>> > > if (is_valid(&val)) {
>> > >     // use val
>> > > }
>> > > 
>> > > This exact thing happens in Binder. It has to be a memcpy.
>> > 
>> > Sure, but then stick that one barrier() in and you're good.
>> 
>> Are we really good? Consider this code:
>> 
>> 	bool is_valid(struct foo *val)
>> 	{
>> 		// for the sake of example
>> 		return val->my_field != 0;
>> 	}
>> 
>> 	struct foo val;
>> 
>> 	void *ptr = kmap_local_page(p1);
>> 	memcpy(ptr, val, sizeof(struct foo));
>> 	kunmap_local(p);
>> 	barrier();
>> 	if (is_valid(&val)) {
>> 	    // use val
>> 	}
>> 
>> optimize it into this first:
>> 
>> 	struct foo val;
>> 	int my_field_copy;
>> 
>> 	void *ptr = kmap_local_page(p1);
>> 	memcpy(ptr, val, sizeof(struct foo));
>> 	my_field_copy = val->my_field;
>> 	kunmap_local(p);
>> 	barrier();
>> 	if (my_field_copy != 0) {
>> 	    // use val
>> 	}
>> 
>> then optimize it into:
>> 
>> 	struct foo val;
>> 	int my_field_copy;
>> 
>> 	void *ptr = kmap_local_page(p1);
>> 	memcpy(ptr, val, sizeof(struct foo));
>> 	my_field_copy = ((struct foo *) ptr)->my_field;
>> 	kunmap_local(p);
>> 	barrier();
>> 	if (my_field_copy != 0) {
>> 	    // use val
>> 	}
> 
> I don;t think this is allowed. You're lifting the load over the
> barrier(), that is invalid.

This is allowed. Compilers perform escape analysis and find out that
"val" does not escape the function and therefore nothing can change "val".

A simple example to demonstrate this effect is that

    int x = 0;
    x = 1;
    barrier();
    do_something(x);

is happily optimized into

    barrier();
    do_something(1);

by both GCC and Clang. The fact that the local variable here is a struct and
memcpy is used to assign the value here does not make a fundamental difference.

barrier() does nothing to local variables if pointers to them do not escape the
local function.

>> 	int my_field_copy;
>> 
>> 	void *ptr = kmap_local_page(p1);
>> 	memcpy(ptr, val, sizeof(struct foo));
>> 	my_field_copy = ((struct foo *) ptr)->my_field;
>> 	kunmap_local(p);
>> 	barrier();
>> 	if (my_field_copy != 0) {
>> 	    // use val
>> 	}

Best,
Gary

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Tue, Feb 17, 2026 at 11:39:18PM +0000, Gary Guo wrote:

> >> Are we really good? Consider this code:
> >> 
> >> 	bool is_valid(struct foo *val)
> >> 	{
> >> 		// for the sake of example
> >> 		return val->my_field != 0;
> >> 	}
> >> 
> >> 	struct foo val;
> >> 
> >> 	void *ptr = kmap_local_page(p1);
> >> 	memcpy(ptr, val, sizeof(struct foo));
> >> 	kunmap_local(p);
> >> 	barrier();
> >> 	if (is_valid(&val)) {
> >> 	    // use val
> >> 	}
> >> 
> >> optimize it into this first:
> >> 
> >> 	struct foo val;
> >> 	int my_field_copy;
> >> 
> >> 	void *ptr = kmap_local_page(p1);
> >> 	memcpy(ptr, val, sizeof(struct foo));
> >> 	my_field_copy = val->my_field;
> >> 	kunmap_local(p);
> >> 	barrier();
> >> 	if (my_field_copy != 0) {
> >> 	    // use val
> >> 	}
> >> 
> >> then optimize it into:
> >> 
> >> 	struct foo val;
> >> 	int my_field_copy;
> >> 
> >> 	void *ptr = kmap_local_page(p1);
> >> 	memcpy(ptr, val, sizeof(struct foo));
> >> 	my_field_copy = ((struct foo *) ptr)->my_field;
> >> 	kunmap_local(p);
> >> 	barrier();
> >> 	if (my_field_copy != 0) {
> >> 	    // use val
> >> 	}
> > 
> > I don;t think this is allowed. You're lifting the load over the
> > barrier(), that is invalid.
> 
> This is allowed. Compilers perform escape analysis and find out that
> "val" does not escape the function and therefore nothing can change "val".
> 
> A simple example to demonstrate this effect is that
> 
>     int x = 0;
>     x = 1;
>     barrier();
>     do_something(x);
> 
> is happily optimized into
> 
>     barrier();
>     do_something(1);
> 
> by both GCC and Clang. The fact that the local variable here is a struct and
> memcpy is used to assign the value here does not make a fundamental difference.
> 
> barrier() does nothing to local variables if pointers to them do not escape the
> local function.

So how do we stop the compiler from doing this? Because I'm thinking
there's quite a bit of code that would be broken if this were done.

Must we really go write things like:

	struct foo val, *ptr;

	ptr = kmap_local_page(page);
	memcpy(ptr, val, sizeof(val));
	kunmap_local(ptr);

	ptr = RELOC_HIDE(&val, 0);

	if (ptr->field) {
		...
	}

That seems 'unfortunate'. It basically means we must never use local
stack for copies or somesuch.

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Alice Ryhl]

On Wed, Feb 18, 2026 at 09:37:54AM +0100, Peter Zijlstra wrote:
> On Tue, Feb 17, 2026 at 11:39:18PM +0000, Gary Guo wrote:
> 
> > >> Are we really good? Consider this code:
> > >> 
> > >> 	bool is_valid(struct foo *val)
> > >> 	{
> > >> 		// for the sake of example
> > >> 		return val->my_field != 0;
> > >> 	}
> > >> 
> > >> 	struct foo val;
> > >> 
> > >> 	void *ptr = kmap_local_page(p1);
> > >> 	memcpy(ptr, val, sizeof(struct foo));
> > >> 	kunmap_local(p);
> > >> 	barrier();
> > >> 	if (is_valid(&val)) {
> > >> 	    // use val
> > >> 	}
> > >> 
> > >> optimize it into this first:
> > >> 
> > >> 	struct foo val;
> > >> 	int my_field_copy;
> > >> 
> > >> 	void *ptr = kmap_local_page(p1);
> > >> 	memcpy(ptr, val, sizeof(struct foo));
> > >> 	my_field_copy = val->my_field;
> > >> 	kunmap_local(p);
> > >> 	barrier();
> > >> 	if (my_field_copy != 0) {
> > >> 	    // use val
> > >> 	}
> > >> 
> > >> then optimize it into:
> > >> 
> > >> 	struct foo val;
> > >> 	int my_field_copy;
> > >> 
> > >> 	void *ptr = kmap_local_page(p1);
> > >> 	memcpy(ptr, val, sizeof(struct foo));
> > >> 	my_field_copy = ((struct foo *) ptr)->my_field;
> > >> 	kunmap_local(p);
> > >> 	barrier();
> > >> 	if (my_field_copy != 0) {
> > >> 	    // use val
> > >> 	}
> > > 
> > > I don;t think this is allowed. You're lifting the load over the
> > > barrier(), that is invalid.
> > 
> > This is allowed. Compilers perform escape analysis and find out that
> > "val" does not escape the function and therefore nothing can change "val".
> > 
> > A simple example to demonstrate this effect is that
> > 
> >     int x = 0;
> >     x = 1;
> >     barrier();
> >     do_something(x);
> > 
> > is happily optimized into
> > 
> >     barrier();
> >     do_something(1);
> > 
> > by both GCC and Clang. The fact that the local variable here is a struct and
> > memcpy is used to assign the value here does not make a fundamental difference.
> > 
> > barrier() does nothing to local variables if pointers to them do not escape the
> > local function.
> 
> So how do we stop the compiler from doing this? Because I'm thinking
> there's quite a bit of code that would be broken if this were done.
> 
> Must we really go write things like:
> 
> 	struct foo val, *ptr;
> 
> 	ptr = kmap_local_page(page);
> 	memcpy(ptr, val, sizeof(val));
> 	kunmap_local(ptr);
> 
> 	ptr = RELOC_HIDE(&val, 0);
> 
> 	if (ptr->field) {
> 		...
> 	}
> 
> That seems 'unfortunate'. It basically means we must never use local
> stack for copies or somesuch.

No I don't think RELOC_HIDE is what you want to be using here.

The way to stop the compiler from doing this is to ensure that, in
LLVM's eyes, the memcpy is either a volatile memcpy, an atomic memcpy,
or an opaque function call. According to Gary's reply to my email on V3,
it sounds like an explicit call to memcpy like this apparently falls into
opaque function call, so it should be okay.

Anyway, that's why using Rust's copy_nonoverlapping() isn't okay here.
It emits a non-volatile non-atomic LLVM memcpy intrinsic, which permits
these optimizations that we don't want in this case. By calling
bindings::memcpy(), it falls into the opaque function call category
instead, which fixes the problem.

Alice

Re: [PATCH v2] rust: page: add byte-wise atomic memory copy methods

[Peter Zijlstra]

On Wed, Feb 18, 2026 at 09:31:13AM +0000, Alice Ryhl wrote:

> > Must we really go write things like:
> > 
> > 	struct foo val, *ptr;
> > 
> > 	ptr = kmap_local_page(page);
> > 	memcpy(ptr, val, sizeof(val));
> > 	kunmap_local(ptr);
> > 
> > 	ptr = RELOC_HIDE(&val, 0);
> > 
> > 	if (ptr->field) {
> > 		...
> > 	}
> > 
> > That seems 'unfortunate'. It basically means we must never use local
> > stack for copies or somesuch.
> 
> No I don't think RELOC_HIDE is what you want to be using here.
> 
> The way to stop the compiler from doing this is to ensure that, in
> LLVM's eyes, the memcpy is either a volatile memcpy, an atomic memcpy,
> or an opaque function call. According to Gary's reply to my email on V3,
> it sounds like an explicit call to memcpy like this apparently falls into
> opaque function call, so it should be okay.

Ah, so we should write:

 	struct foo val, *ptr;
 
 	ptr = kmap_local_page(page);
 	__memcpy(ptr, val, sizeof(val));
 	kunmap_local(ptr);
 
 	if (val.field) {
 		...
 	}

That forces things to be a function call since it elides the C memcpy
intrinsic for not having the right name.

But that also elides KASAN and such that want to hijack the various mem
functions for instrumentation purposes.