Re: O_DIRECT vs BLK_FEAT_STABLE_WRITES, was Re: [PATCH] btrfs: never trust the bio from direct IO

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2025/10/21 22:00, Johannes Thumshirn 写道:
> On 10/21/25 10:15 AM, Qu Wenruo wrote:
>>
>> 在 2025/10/21 18:18, Christoph Hellwig 写道:
>>> On Tue, Oct 21, 2025 at 01:47:03PM +1030, Qu Wenruo wrote:
>>>> Off-topic a little, mind to share the performance drop with PI enabled on
>>>> XFS?
>>> If the bandwith of the SSDs get close or exceeds the DRAM bandwith
>>> buffered I/O can be 50% or less of the direct I/O performance.
>> In my case, the DRAM is way faster than the SSD (tens of GiB/s vs less
>> than 5GiB/s).
>>
>>>> With this patch I'm able to enable direct IO for inodes with checksums.
>>>> I thought it would easily improve the performance, but the truth is, it's
>>>> not that different from buffered IO fall back.
>>> That's because you still copy data.
>> Enabling the extra copy for direct IO only drops around 15~20%
>> performance, but that's on no csum case.
>>
>> So far the calculation matches your estimation, but...
>>
>>>> So I start wondering if it's the checksum itself causing the miserable
>>>> performance numbers.
>>> Only indirectly by touching all the cachelines.  But once you copy you
>>> touch them again.  Especially if not done in small chunks.
>> As long as I enable checksum verification, even with the bouncing page
>> direct IO, the result is not any better than buffered IO fallback, all
>> around 10% (not by 10%, at 10%) of the direct IO speed (no matter
>> bouncing or not).
>>
>> Maybe I need to check if the proper hardware accelerated CRC32 is
>> utilized...
> 
> 
> You could also hack in a NULL-csum for testing. Something that writes a
> fixed value every time. This would then rule out all the cost of the
> csum generation and only test the affected IO paths.
> 

It turns out to be checksum, and particularly my VM setup.

My VM is using kvm64 CPU type, which blocks quite a lot of CPU features, 
thus the CRC32 performance is pretty poor.

I just tried a short hack to always make direct IO to fallback to 
buffered IO, the nodatasum performance is the same as the bouncing page 
solution, so the slow down is not page cache itself but really the checksum.

With CPU features all passed to the VM, the falling-back-to-buffered 
direct IO performance is only slightly worse (10~20%) than nodatasum cases.

Really sorry for the noise.

Thanks,
Qu