Re: [LSF/MM/BPF TOPIC] Buffered atomic writes

[Ojaswin Mujoo <ojaswin@linux.ibm.com>]

On Fri, Feb 13, 2026 at 11:20:36AM +0100, Pankaj Raghav wrote:
> Hi all,
> 
> Atomic (untorn) writes for Direct I/O have successfully landed in kernel
> for ext4 and XFS[1][2]. However, extending this support to Buffered I/O
> remains a contentious topic, with previous discussions often stalling due to
> concerns about complexity versus utility.
> 
> I would like to propose a session to discuss the concrete use cases for
> buffered atomic writes and if possible, talk about the outstanding
> architectural blockers blocking the current RFCs[3][4].

Hi Pankaj,

Thanks for the proposal and glad to hear there is a wider interest in
this topic. We have also been actively working on this and I in middle
of testing and ironing out bugs in my RFC v2 for buffered atomic
writes, which is largely based on Dave's suggestions to maintain atomic
write mappings in FS layer (aka XFS COW fork). Infact I was going to
propose a discussion on this myself :) 

> 
> ## Use Case:
> 
> A recurring objection to buffered atomics is the lack of a convincing use
> case, with the argument that databases should simply migrate to direct I/O.
> We have been working with PostgreSQL developer Andres Freund, who has
> highlighted a specific architectural requirement where buffered I/O remains
> preferable in certain scenarios.

Looks like you have some nice insights to cover from postgres side which
filesystem community has been asking for. As I've also been working on
the kernel implementation side of it, do you think we could do a joint
session on this topic?

> 
> While Postgres recently started to support direct I/O, optimal performance
> requires a large, statically configured user-space buffer pool. This becomes
> problematic when running many Postgres instances on the same hardware, a
> common deployment scenario. Statically partitioning RAM for direct I/O
> caches across many instances is inefficient compared to allowing the kernel
> page cache to dynamically balance memory pressure between instances.
> 
> The other use case is using postgres as part of a larger workload on one
> instance. Using up enough memory for postgres' buffer pool to make DIO use
> viable is often not realistic, because some deployments require a lot of
> memory to cache database IO, while others need a lot of memory for
> non-database caching.
> 
> Enabling atomic writes for this buffered workload would allow Postgres to
> disable full-page writes [5]. For direct I/O, this has shown to reduce
> transaction variability; for buffered I/O, we expect similar gains,
> alongside decreased WAL bandwidth and storage costs for WAL archival. As a
> side note, for most workloads full page writes occupy  a significant portion
> of WAL volume.
> 
> Andres has agreed to attend LSFMM this year to discuss these requirements.

Glad to hear people from postgres would also be joining!

> 
> ## Discussion:
> 
> We currently have RFCs posted by John Garry and Ojaswin Mujoo, and there
> was a previous LSFMM proposal about untorn buffered writes from Ted Tso.
> Based on the conversation/blockers we had before, the discussion at LSFMM
> should focus on the following blocking issues:
> 
> - Handling Short Writes under Memory Pressure[6]: A buffered atomic
>   write might span page boundaries. If memory pressure causes a page
>   fault or reclaim mid-copy, the write could be torn inside the page
>   cache before it even reaches the filesystem.
>     - The current RFC uses a "pinning" approach: pinning user pages and
>       creating a BVEC to ensure the full copy can proceed atomically.
>       This adds complexity to the write path.
>     - Discussion: Is this acceptable? Should we consider alternatives,
>       such as requiring userspace to mlock the I/O buffers before
>       issuing the write to guarantee atomic copy in the page cache?

Right, I chose this approach because we only get to know about the short
copy after it has actually happened in copy_folio_from_iter_atomic()
and it seemed simpler to just not let the short copy happen. This is
inspired from how dio pins the pages for DMA, just that we do it
for a shorter time.

It does add slight complexity to the path but I'm not sure if it's complex
enough to justify adding a hard requirement of having pages mlock'd.

> 
> - Page Cache Model vs. Filesystem CoW: The current RFC introduces a
>   PG_atomic page flag to track dirty pages requiring atomic writeback.
>   This faced pushback due to page flags being a scarce resource[7].
>   Furthermore, it was argued that atomic model does not fit the buffered
>   I/O model because data sitting in the page cache is vulnerable to
>   modification before writeback occurs, and writeback does not preserve
>   application ordering[8].
>     -  Dave Chinner has proposed leveraging the filesystem's CoW path
>        where we always allocate new blocks for the atomic write (forced
>        CoW). If the hardware supports it (e.g., NVMe atomic limits), the
>        filesystem can optimize the writeback to use REQ_ATOMIC in place,
>        avoiding the CoW overhead while maintaining the architectural
>        separation.

Right, this is what I'm doing in the new RFC where we maintain the
mappings for atomic write in COW fork. This way we are able to utilize a
lot of existing infrastructure, however it does add some complexity to
->iomap_begin() and ->writeback_range() callbacks of the FS. I believe
it is a tradeoff since the general consesus was mostly to avoid adding
too much complexity to iomap layer.

Another thing that came up is to consider using write through semantics 
for buffered atomic writes, where we are able to transition page to
writeback state immediately after the write and avoid any other users to
modify the data till writeback completes. This might affect performance
since we won't be able to batch similar atomic IOs but maybe
applications like postgres would not mind this too much. If we go with
this approach, we will be able to avoid worrying too much about other
users changing atomic data underneath us. 

An argument against this however is that it is user's responsibility to
not do non atomic IO over an atomic range and this shall be considered a
userspace usage error. This is similar to how there are ways users can
tear a dio if they perform overlapping writes. [1]. 

That being said, I think these points are worth discussing and it would
be helpful to have people from postgres around while discussing these
semantics with the FS community members.

As for ordering of writes, I'm not sure if that is something that
we should guarantee via the RWF_ATOMIC api. Ensuring ordering has mostly
been the task of userspace via fsync() and friends.


[1] https://lore.kernel.org/fstests/0af205d9-6093-4931-abe9-f236acae8d44@oracle.com/

>     - Discussion: While the CoW approach fits XFS and other CoW
>       filesystems well, it presents challenges for filesystems like ext4
>       which lack CoW capabilities for data. Should this be a filesystem
>       specific feature?

I believe your question is if we should have a hard dependency on COW
mappings for atomic writes. Currently, COW in atomic write context in
XFS, is used for these 2 things:

1. COW fork holds atomic write ranges.

This is not strictly a COW feature, just that we are repurposing the COW
fork to hold our atomic ranges. Basically a way for writeback path to
know that atomic write was done here.

COW fork is one way to do this but I believe every FS has a version of
in memory extent trees where such ephemeral atomic write mappings can be
held. The extent status cache is ext4's version of this, and can be used
to manage the atomic write ranges. 

There is an alternate suggestion that came up from discussions with Ted
and Darrick that we can instead use a generic side-car structure which
holds atomic write ranges. FSes can populate these during atomic writes
and query these in their writeback paths. 

This means for any FS operation (think truncate, falloc, mwrite, write
...) we would need to keep this structure in sync, which can become pretty
complex pretty fast. I'm yet to implement this so not sure how it would
look in practice though.

2. COW feature as a whole enables software based atomic writes.

This is something that ext4 won't be able to support (right now), just
like how we don't support software writes for dio.

I believe Baokun and Yi and working on a feature that can eventually
enable COW writes in ext4 [2]. Till we have something like that, we
would have to rely on hardware support.

Regardless, I don't think the ability to support or not support
software atomic writes largely depends on the filesystem so I'm not
sure how we can lift this up to a generic layer anyways.

[2] https://lore.kernel.org/linux-ext4/9666679c-c9f7-435c-8b67-c67c2f0c19ab@huawei.com/


Thanks,
Ojaswin
> 
> Comments or Curses, all are welcome.
> 
> --
> Pankaj
> 
> [1] https://lwn.net/Articles/1009298/
> [2] https://docs.kernel.org/6.17/filesystems/ext4/atomic_writes.html
> [3] https://lore.kernel.org/linux-fsdevel/20240422143923.3927601-1-john.g.garry@oracle.com/
> [4] https://lore.kernel.org/all/cover.1762945505.git.ojaswin@linux.ibm.com
> [5] https://www.postgresql.org/docs/16/runtime-config-wal.html#GUC-FULL-PAGE-WRITES
> [6]
> https://lore.kernel.org/linux-fsdevel/ZiZ8XGZz46D3PRKr@casper.infradead.org/
> [7]
> https://lore.kernel.org/linux-fsdevel/aRSuH82gM-8BzPCU@casper.infradead.org/
> [8]
> https://lore.kernel.org/linux-fsdevel/aRmHRk7FGD4nCT0s@dread.disaster.area/
>