Re: [LSF/MM/BPF TOPIC] Buffered atomic writes

[Jan Kara <jack@suse.cz>]

On Tue 17-02-26 11:13:07, Andres Freund wrote:
> > > P1: pwritev(fd, [blocks 1-10], RWF_ATOMIC) start & completes
> > > Kernel: starts writeback but doesn't complete it
> > > P1: pwrite(fd, [any block in 1-10]), non-atomically
> > > Kernel: completes writeback
> > >
> > > The former is not at all an issue for postgres' use case, the pages in
> > > our buffer pool that are undergoing IO are locked, preventing additional
> > > IO (be it reads or writes) to those blocks.
> > >
> > > The latter would be a problem, since userspace wouldn't even know that
> > > here is still "atomic writeback" going on, afaict the only way we could
> > > avoid it would be to issue an f[data]sync(), which likely would be
> > > prohibitively expensive.
> >
> > It somewhat depends on what outcome you expect in terms of crash safety :)
> > Unless we are careful, the RWF_ATOMIC write in your latter example can end
> > up writing some bits of the data from the second write because the second
> > write may be copying data to the pages as we issue DMA from them to the
> > device.
> 
> Hm. It's somewhat painful to not know when we can write in what mode again -
> with DIO that's not an issue. I guess we could use
> sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE) if we really needed to know?
> Although the semantics of the SFR flags aren't particularly clear, so maybe
> not?

If you used RWF_WRITETHROUGH for your writes (so you are sure IO has
already started) then sync_file_range(SYNC_FILE_RANGE_WAIT_BEFORE) would
indeed be a safe way of waiting for that IO to complete (or just wait for
the write(2) syscall itself to complete if we make RWF_WRITETHROUGH wait
for IO completion as Dave suggests - but I guess writes may happen from
multiple threads so that may be not very convenient and sync_file_range(2)
might be actually easier).

> > I expect this isn't really acceptable because if you crash before
> > the second write fully makes it to the disk, you will have inconsistent
> > data.
> 
> The scenarios that I can think that would lead us to doing something like
> this, are when we are overwriting data without regard for the prior contents,
> e.g:
> 
> An already partially filled page is filled with more rows, we write that page
> out, then all the rows are deleted, and we re-fill the page with new content
> from scratch. Write it out again.  With our existing logic we treat the second
> write differently, because the entire contents of the page will be in the
> journal, as there is no prior content that we care about.
> 
> A second scenario in which we might not use RWF_ATOMIC, if we carry today's
> logic forward, is if a newly created relation is bulk loaded in the same
> transaction that created the relation. If a crash were to happen while that
> bulk load is ongoing, we don't care about the contents of the file(s), as it
> will never be visible to anyone after crash recovery.  In this case we won't
> have prio RWF_ATOMIC writes - but we could have the opposite, i.e. an
> RWF_ATOMIC write while there already is non-RWF_ATOMIC dirty data in the page
> cache. Would that be an issue?

No, this should be fine. But as I'm thinking about it what seems the most
natural is that RWF_WRITETHROUGH writes will wait on any pages under
writeback in the target range before proceeding with the write. That will
give user proper serialization with other RWF_WRITETHROUGH writes to the
overlapping range as well as writeback from previous normal writes. So the
only case that needs handling - either by userspace or kernel forcing
stable writes - would be RWF_WRITETHROUGH write followed by a normal write.

> It's possible we should just always use RWF_ATOMIC, even in the cases where
> it's not needed from our side, to avoid potential performance penalties and
> "undefined behaviour".  I guess that will really depend on the performance
> penalty that RWF_ATOMIC will carry and whether multiple-atomicity-mode will
> eventually be supported (as doing small writes during bulk loading is quite
> expensive).

Sure, that's a possibility as well. I guess it requires some
experimentation and benchmarking to pick a proper tradeoff.

									Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR