Re: Direct I/O performance problems with 1GB pages

[Andres Freund <andres@anarazel.de>]

Hi,

On 2025-01-27 17:09:57 +0100, David Hildenbrand wrote:
> > Andres shared some gists, but I don't want to send those to a
> > mailing list without permission.  Here's the kernel part of the
> > perf report:
> >
> >      14.04%  postgres         [kernel.kallsyms]          [k] try_grab_folio_fast
> >              |
> >               --14.04%--try_grab_folio_fast
> >                         gup_fast_fallback
> >                         |
> >                          --13.85%--iov_iter_extract_pages
> >                                    bio_iov_iter_get_pages
> >                                    iomap_dio_bio_iter
> >                                    __iomap_dio_rw
> >                                    iomap_dio_rw
> >                                    xfs_file_dio_read
> >                                    xfs_file_read_iter
> >                                    __io_read
> >                                    io_read
> >                                    io_issue_sqe
> >                                    io_submit_sqes
> >                                    __do_sys_io_uring_enter
> >                                    do_syscall_64
> >
> > Now, since postgres is using io_uring, perhaps there could be a path
> > which registers the memory with the iouring (doing the refcount/pincount
> > dance once), and then use that pinned memory for each I/O.  Maybe that
> > already exists; I'm not keeping up with io_uring development and I can't
> > seem to find any documentation on what things like io_provide_buffers()
> > actually do.

Worth noting that we'll not always use io_uring. Partially for portability to
other platforms, partially because it turns out that io_uring is disabled in
enough environments that we can't rely on it. The generic fallback
implementation is a pool of worker processes connected via shared memory. The
worker process approach did run into this issue, fwiw.

That's not to say that a legit answer to this scalability issue can't be "use
fixed bufs with io_uring", just wanted to give context.


> That's precisely what io-uring fixed buffers do :)

I looked at using them at some point - unfortunately it seems that there is
just {READ,WRITE}_FIXED not {READV,WRITEV}_FIXED. It's *exceedingly* common
for us to do reads/writes where source/target buffers aren't wholly
contiguous. Thus - unless I am misunderstanding something, entirely plausible
- using fixed buffers would unfortunately increase the number of IOs
noticeably.

Should have sent an email about that...

I guess we could add some heuristic to use _FIXED if it doesn't require
splitting an IO into too many sub-ios. But that seems pretty gnarly.


I dimly recall that I also ran into some around using fixed buffers as a
non-root user. It might just be the accounting of registered buffers as
mlocked memory and the difficulty of configuring that across
distributions. But I unfortunately don't remember any details anymore.


Greetings,

Andres Freund