Re: [Lsf-pc] [LSF/MM/BPF TOPIC] Measuring limits and enhancing buffered IO

[Chris Mason <clm@meta.com>]

On 2/28/24 2:48 AM, Amir Goldstein wrote:
> On Wed, Feb 28, 2024 at 12:42 AM Dave Chinner via Lsf-pc

>> Essentially, we need to explicitly give POSIX the big finger and
>> state that there are no atomicity guarantees given for write() calls
>> of any size, nor are there any guarantees for data coherency for
>> any overlapping concurrent buffered IO operations.
>>
> 
> I have disabled read vs. write atomicity (out-of-tree) to make xfs behave
> as the other fs ever since Jan has added the invalidate_lock and I believe
> that Meta kernel has done that way before.

Hmmm, you might be thinking of my patch to prevent kswapd from getting
stuck on XFS inode reclaim, but I don't think we've ever messed with
write concurrency.  I'm comfortable with the concurrency change in
general, but it's not somewhere I'd be excited about differing from
upstream.

Total tangent, but we only carry two XFS patches right now that aren't
upstream.  I dropped the inode reclaim patch; the problem stopped
showing up in our profiles, and the impacted workloads changed to
rocksdb for other reasons.

We flip XFS discards back to synchronous.  Async disards without any
kind of metering saturate drives when we do bulk deletes, leading to
latency spikes on reads and writes.  There's probably a class of flash
that can handle this, but we don't have it.

Unfortunately I also disable large folios on XFS.  They are corrupting
xarrays on our v5.19 kernel, with large folios from multiple files
interleaved together in the same file.  We'll try again with them on
v6.4 or maybe v6.8, but the repro needs thousands of machines making NFS
noises just to trigger one failure, and I won't be able to debug it
until I can make a more reasonable reproduction.

-chris