Re: [LSF/MM/BPF TOPIC] Buffered atomic writes

[Andres Freund <andres@anarazel.de>]

Hi,

On 2026-02-17 13:06:04 +0100, Jan Kara wrote:
> On Mon 16-02-26 10:45:40, Andres Freund wrote:
> > (*) As it turns out, it often seems to improves write throughput as well, if
> > writeback is triggered by memory pressure instead of SYNC_FILE_RANGE_WRITE,
> > linux seems to often trigger a lot more small random IO.
> >
> > > So immediately writing them might be ok as long as we don't remove those
> > > pages from the page cache like we do in RWF_UNCACHED.
> >
> > Yes, it might.  I actually often have wished for something like a
> > RWF_WRITEBACK flag...
>
> I'd call it RWF_WRITETHROUGH but otherwise it makes sense.

Heh, that makes sense. I think that's what I actually was thinking of.


> > > > An argument against this however is that it is user's responsibility to
> > > > not do non atomic IO over an atomic range and this shall be considered a
> > > > userspace usage error. This is similar to how there are ways users can
> > > > tear a dio if they perform overlapping writes. [1].
> >
> > Hm, the scope of the prohibition here is not clear to me. Would it just
> > be forbidden to do:
> >
> > P1: start pwritev(fd, [blocks 1-10], RWF_ATOMIC)
> > P2: pwrite(fd, [any block in 1-10]), non-atomically
> > P1: complete pwritev(fd, ...)
> >
> > or is it also forbidden to do:
> >
> > P1: pwritev(fd, [blocks 1-10], RWF_ATOMIC) start & completes
> > Kernel: starts writeback but doesn't complete it
> > P1: pwrite(fd, [any block in 1-10]), non-atomically
> > Kernel: completes writeback
> >
> > The former is not at all an issue for postgres' use case, the pages in
> > our buffer pool that are undergoing IO are locked, preventing additional
> > IO (be it reads or writes) to those blocks.
> >
> > The latter would be a problem, since userspace wouldn't even know that
> > here is still "atomic writeback" going on, afaict the only way we could
> > avoid it would be to issue an f[data]sync(), which likely would be
> > prohibitively expensive.
>
> It somewhat depends on what outcome you expect in terms of crash safety :)
> Unless we are careful, the RWF_ATOMIC write in your latter example can end
> up writing some bits of the data from the second write because the second
> write may be copying data to the pages as we issue DMA from them to the
> device.

Hm. It's somewhat painful to not know when we can write in what mode again -
with DIO that's not an issue. I guess we could use
sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE) if we really needed to know?
Although the semantics of the SFR flags aren't particularly clear, so maybe
not?


> I expect this isn't really acceptable because if you crash before
> the second write fully makes it to the disk, you will have inconsistent
> data.

The scenarios that I can think that would lead us to doing something like
this, are when we are overwriting data without regard for the prior contents,
e.g:

An already partially filled page is filled with more rows, we write that page
out, then all the rows are deleted, and we re-fill the page with new content
from scratch. Write it out again.  With our existing logic we treat the second
write differently, because the entire contents of the page will be in the
journal, as there is no prior content that we care about.

A second scenario in which we might not use RWF_ATOMIC, if we carry today's
logic forward, is if a newly created relation is bulk loaded in the same
transaction that created the relation. If a crash were to happen while that
bulk load is ongoing, we don't care about the contents of the file(s), as it
will never be visible to anyone after crash recovery.  In this case we won't
have prio RWF_ATOMIC writes - but we could have the opposite, i.e. an
RWF_ATOMIC write while there already is non-RWF_ATOMIC dirty data in the page
cache. Would that be an issue?


It's possible we should just always use RWF_ATOMIC, even in the cases where
it's not needed from our side, to avoid potential performance penalties and
"undefined behaviour".  I guess that will really depend on the performance
penalty that RWF_ATOMIC will carry and whether multiple-atomicity-mode will
eventually be supported (as doing small writes during bulk loading is quite
expensive).


Greetings,

Andres Freund