Re: [LSF/MM/BPF TOPIC] Buffered atomic writes

[Jan Kara <jack@suse.cz>]

On Mon 16-02-26 10:45:40, Andres Freund wrote:
> > Hmm, IIUC, postgres will write their dirty buffer cache by combining
> > multiple DB pages based on `io_combine_limit` (typically 128kb).
> 
> We will try to do that, but it's obviously far from always possible, in some
> workloads [parts of ]the data in the buffer pool rarely will be dirtied in
> consecutive blocks.
> 
> FWIW, postgres already tries to force some just-written pages into
> writeback. For sources of writes that can be plentiful and are done in the
> background, we default to issuing sync_file_range(SYNC_FILE_RANGE_WRITE),
> after 256kB-512kB of writes, as otherwise foreground latency can be
> significantly impacted by the kernel deciding to suddenly write back (due to
> dirty_writeback_centisecs, dirty_background_bytes, ...) and because otherwise
> the fsyncs at the end of a checkpoint can be unpredictably slow.  For
> foreground writes we do not default to that, as there are users that won't
> (because they don't know, because they overcommit hardware, ...) size
> postgres' buffer pool to be big enough and thus will often re-dirty pages that
> have already recently been written out to the operating systems.  But for many
> workloads it's recommened that users turn on
> sync_file_range(SYNC_FILE_RANGE_WRITE) for foreground writes as well (*).
> 
> So for many workloads it'd be fine to just always start writeback for atomic
> writes immediately. It's possible, but I am not at all sure, that for most of
> the other workloads, the gains from atomic writes will outstrip the cost of
> more frequently writing data back.

OK, good. Then I think it's worth a try.

> (*) As it turns out, it often seems to improves write throughput as well, if
> writeback is triggered by memory pressure instead of SYNC_FILE_RANGE_WRITE,
> linux seems to often trigger a lot more small random IO.
> 
> > So immediately writing them might be ok as long as we don't remove those
> > pages from the page cache like we do in RWF_UNCACHED.
> 
> Yes, it might.  I actually often have wished for something like a
> RWF_WRITEBACK flag...

I'd call it RWF_WRITETHROUGH but otherwise it makes sense.

> > > An argument against this however is that it is user's responsibility to
> > > not do non atomic IO over an atomic range and this shall be considered a
> > > userspace usage error. This is similar to how there are ways users can
> > > tear a dio if they perform overlapping writes. [1].
> 
> Hm, the scope of the prohibition here is not clear to me. Would it just
> be forbidden to do:
> 
> P1: start pwritev(fd, [blocks 1-10], RWF_ATOMIC)
> P2: pwrite(fd, [any block in 1-10]), non-atomically
> P1: complete pwritev(fd, ...)
> 
> or is it also forbidden to do:
> 
> P1: pwritev(fd, [blocks 1-10], RWF_ATOMIC) start & completes
> Kernel: starts writeback but doesn't complete it
> P1: pwrite(fd, [any block in 1-10]), non-atomically
> Kernel: completes writeback
> 
> The former is not at all an issue for postgres' use case, the pages in
> our buffer pool that are undergoing IO are locked, preventing additional
> IO (be it reads or writes) to those blocks.
> 
> The latter would be a problem, since userspace wouldn't even know that
> here is still "atomic writeback" going on, afaict the only way we could
> avoid it would be to issue an f[data]sync(), which likely would be
> prohibitively expensive.

It somewhat depends on what outcome you expect in terms of crash safety :)
Unless we are careful, the RWF_ATOMIC write in your latter example can end
up writing some bits of the data from the second write because the second
write may be copying data to the pages as we issue DMA from them to the
device. I expect this isn't really acceptable because if you crash before
the second write fully makes it to the disk, you will have inconsistent
data. So what we can offer is to enable "stable pages" feature for the
filesystem (support for buffered atomic writes would be conditioned by
that) - that will block the second write until the IO is done so torn
writes cannot happen. If quick overwrites are rare, this should be a fine
option. If they are frequent, we'd need to come up with some bounce
buffering but things get ugly quickly there.

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR