Re: [LSF/MM/BPF TOPIC] Buffered atomic writes

[Ojaswin Mujoo <ojaswin@linux.ibm.com>]

On Tue, Feb 17, 2026 at 11:13:07AM -0500, Andres Freund wrote:
> Hi,
> 
> On 2026-02-17 13:06:04 +0100, Jan Kara wrote:
> > On Mon 16-02-26 10:45:40, Andres Freund wrote:
> > > (*) As it turns out, it often seems to improves write throughput as well, if
> > > writeback is triggered by memory pressure instead of SYNC_FILE_RANGE_WRITE,
> > > linux seems to often trigger a lot more small random IO.
> > >
> > > > So immediately writing them might be ok as long as we don't remove those
> > > > pages from the page cache like we do in RWF_UNCACHED.
> > >
> > > Yes, it might.  I actually often have wished for something like a
> > > RWF_WRITEBACK flag...
> >
> > I'd call it RWF_WRITETHROUGH but otherwise it makes sense.
> 
> Heh, that makes sense. I think that's what I actually was thinking of.
> 
> 
> > > > > An argument against this however is that it is user's responsibility to
> > > > > not do non atomic IO over an atomic range and this shall be considered a
> > > > > userspace usage error. This is similar to how there are ways users can
> > > > > tear a dio if they perform overlapping writes. [1].
> > >
> > > Hm, the scope of the prohibition here is not clear to me. Would it just
> > > be forbidden to do:
> > >
> > > P1: start pwritev(fd, [blocks 1-10], RWF_ATOMIC)
> > > P2: pwrite(fd, [any block in 1-10]), non-atomically
> > > P1: complete pwritev(fd, ...)
> > >
> > > or is it also forbidden to do:
> > >
> > > P1: pwritev(fd, [blocks 1-10], RWF_ATOMIC) start & completes
> > > Kernel: starts writeback but doesn't complete it
> > > P1: pwrite(fd, [any block in 1-10]), non-atomically
> > > Kernel: completes writeback
> > >
> > > The former is not at all an issue for postgres' use case, the pages in
> > > our buffer pool that are undergoing IO are locked, preventing additional
> > > IO (be it reads or writes) to those blocks.
> > >
> > > The latter would be a problem, since userspace wouldn't even know that
> > > here is still "atomic writeback" going on, afaict the only way we could
> > > avoid it would be to issue an f[data]sync(), which likely would be
> > > prohibitively expensive.
> >
> > It somewhat depends on what outcome you expect in terms of crash safety :)
> > Unless we are careful, the RWF_ATOMIC write in your latter example can end
> > up writing some bits of the data from the second write because the second
> > write may be copying data to the pages as we issue DMA from them to the
> > device.
> 
> Hm. It's somewhat painful to not know when we can write in what mode again -
> with DIO that's not an issue. I guess we could use
> sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE) if we really needed to know?
> Although the semantics of the SFR flags aren't particularly clear, so maybe
> not?
> 
> 
> > I expect this isn't really acceptable because if you crash before
> > the second write fully makes it to the disk, you will have inconsistent
> > data.
> 

> The scenarios that I can think that would lead us to doing something like
> this, are when we are overwriting data without regard for the prior contents,
> e.g:
> 
> An already partially filled page is filled with more rows, we write that page
> out, then all the rows are deleted, and we re-fill the page with new content
> from scratch. Write it out again.  With our existing logic we treat the second
> write differently, because the entire contents of the page will be in the
> journal, as there is no prior content that we care about.

Hi Andres,

From my mental model and very high level understanding of Postgres' WAL
model [1] I am under the impression that for moving from full page
writes to RWF_ATOMIC, we would need to ensure that the **disk** write IO
of any data buffer should go in an untorn fashion.

Now, coming to your example, IIUC here we can actually tolerate to do
the 2nd write above non atomically because it is already a sort of full
page write in the journal.

So lets say if we do something like:

0. Buffer has some initial value on disk
1. Write new rows into buffer
2. Write the buffer as RWF_ATOMIC
3. Overwrite the complete buffer which will journal all the contents
4. Write the buffer as non RWF_ATOMIC
5. Crash

I think it is still possible to satisfy my assumption of **disk** IO
being untorn. Example, here we can have an RWF_ATOMIC implementation
where the data on disk after crash could either be in initial state 0.
or be the new value after 4. This is not strictly the old or new
semantic but still ensures the data is consistent. 

My naive understanding says that as long as disk has consistent/untorn
data, like above, we can recover via the journal. In this case the
kernel implementation should be able to tolerate mixing of atomic and
non atomic writes, but again I might be wrong here.

However, if the above guarantees are not enough and actually care about
true old or new semantic, we would need something like RWF_WRITETHROUGH
to ensure we get truely old or new.


[1] https://www.interdb.jp/pg/pgsql09/01.html


> 
> A second scenario in which we might not use RWF_ATOMIC, if we carry today's
> logic forward, is if a newly created relation is bulk loaded in the same
> transaction that created the relation. If a crash were to happen while that
> bulk load is ongoing, we don't care about the contents of the file(s), as it
> will never be visible to anyone after crash recovery.  In this case we won't
> have prio RWF_ATOMIC writes - but we could have the opposite, i.e. an
> RWF_ATOMIC write while there already is non-RWF_ATOMIC dirty data in the page
> cache. Would that be an issue?

I think this is same discussion as above.

> 
> 
> It's possible we should just always use RWF_ATOMIC, even in the cases where
> it's not needed from our side, to avoid potential performance penalties and
> "undefined behaviour".  I guess that will really depend on the performance
> penalty that RWF_ATOMIC will carry and whether multiple-atomicity-mode will
> eventually be supported (as doing small writes during bulk loading is quite
> expensive).
> 
> 
> Greetings,
> 
> Andres Freund