Re: [ISSUE] Read performance regression when using RWF_DONTCACHE form 8026e49 "mm/filemap: add read support for RWF_DONTCACHE"

[Mingyu He <mingyu.he@shopee.com>]

Hi Kiryl,

I will list my phy sec and fs block size at the tail of this email.

I have 2 types of hard disk on my Linux. SSD NVME and HDD.
And I tested buffer_size with range from 1k, 4k, 16k, 64k, 128k.  And
also with/without cgroup.

On both type of hard disk I got same output: RWF_DONTCACHE has very
low performance.

Strongly guessing this is due to readahead.   Pages are dropped after
reading. So system need another I/O to get the next part of the data.
However, I dont test the cases with Kswapd strongly working (But this
is not the core of the question.)


I guess this case needs optimization. But I am not sure it needs an
optimization or just I got wrong using cases, as I am not a proficient
kernel developer.
So I need the advice from experts like you to make sure.
If this is a case worth optimizing, I'd like to do that optimization
 ( But I think many people might have noticed this problem, so I'm not
sure I could finish the optimization before those proficient
developers )


RWF_DONTCACHE Performance Comparison (MiB/s)

+--------------+-------------+------------------------+------------------+
| Device Type  | Buffer Size | RWF_DONTCACHE (MiB/s)  | Normal (MiB/s)   |
+--------------+-------------+------------------------+------------------+
| HDD          | 4K          | 119.6                  | 2268.1           |
| HDD          | 16K         | 1568.6                 | 3814.7           |
| HDD          | 64K         | 2351.0                 | 4161.8           |
| HDD          | 128K        | 2951.4                 | 4061.0           |
+--------------+-------------+------------------------+------------------+
| NVMe         | 4K          | 148.7                  | 1556.1           |
| NVMe         | 16K         | 619.0                  | 1601.5           |
| NVMe         | 64K         | 1139.6                 | 1618.6           |
| NVMe         | 128K        | 1725.4                 | 1579.2
  |- NVMe @ 128K is the only case where RWF_DONTCACHE > Normal
+--------------+-------------+------------------------+------------------+







# lsblk -o NAME,FSTYPE,SIZE,FSUSED,FSUSE%,ROTA,MODEL,MOUNTPOINT

NAME    FSTYPE  SIZE FSUSED FSUSE% ROTA MODEL
    MOUNTPOIN
sda             1.1T                  1 PERC H750 Adp
├─sda1            4M                  1
├─sda2  vfat    110M   6.1M     6%    1
    /boot/efi
├─sda3  ext4      2G 517.1M    27%    1                                    /boot
└─sda4  xfs     1.1T  70.4G     6%    1                                    /
nvme0n1 ext4    1.7T     5G     0%    0 Dell Ent NVMe v2 AGN RI U.2 1.92TB /data


# lsblk -o NAME,PHY-SEC,LOG-SEC
NAME    PHY-SEC LOG-SEC
sda         512     512
├─sda1      512     512
├─sda2      512     512
├─sda3      512     512
└─sda4      512     512
nvme0n1     512     512

# dumpe2fs /dev/nvme0n1 | grep "Block size"
dumpe2fs 1.47.0 (5-Feb-2023)
Block size:               4096

# xfs_info /
meta-data=/dev/sda4              isize=512    agcount=566, agsize=516864 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1 inobtcount=1
data     =                       bsize=4096   blocks=292326651, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


On Wed, Apr 15, 2026 at 6:05 PM Kiryl Shutsemau <kirill@shutemov.name> wrote:
>
> On Wed, Apr 15, 2026 at 03:28:27PM +0800, Mingyu He wrote:
> > The smaller the buffer_size in the test program, the more the
> > performance dropped. Initially, I used a 4k buffer_size, and the
> > performance decreased significantly. When the buffer_size was
> > increased to 128K, the read performance with RWF_DONTCACHE actually
> > surpassed the non-flagged version by about 10%.
>
> Maybe you have block size larger than 4k? Core-mm will allocate larger
> folios for page cache if filesystem asks it to. And if you try to access
> it with 4k buffer it gets multiple read-discard cycles for the same
> block with RWF_DONTCACHE. Without RWF_DONTCACHE only the first access to
> the block will lead to I/O, following accesses are served from page
> cache.
>
> --
>   Kiryl Shutsemau / Kirill A. Shutemov