fix swap on NFS

fix swap on NFS

[Christoph Hellwig]

Hi all,

as of 6.10-rc swap can swap out larger than PAGE_SIZE chunks, which
trips a VM_BUG_ON in NFS.  This trivial patch seems to fix it to me,
but I haven't really figured out what Fixes tag this should get.

Also cifs has the same VM_BUG_ON and will also need a similar fix, but
I'll leave it to people with a working test setup.

[PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Christoph Hellwig]

As of Linux 6.10-rc the MM can swap out larger than page size chunks.
NFS has all code ready to handle this, but has a VM_BUG_ON that
triggers when this happens.  Simply remove the VM_BUG_ON to fix this
use case.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 fs/nfs/direct.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index bb2f583eb28bf1..90079ca134dd3c 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -141,8 +141,6 @@ int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter)
 {
 	ssize_t ret;
 
-	VM_BUG_ON(iov_iter_count(iter) != PAGE_SIZE);
-
 	if (iov_iter_rw(iter) == READ)
 		ret = nfs_file_direct_read(iocb, iter, true);
 	else
-- 
2.43.0

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Jeff Layton]

On Fri, 2024-06-14 at 12:03 +0200, Christoph Hellwig wrote:
> As of Linux 6.10-rc the MM can swap out larger than page size chunks.
> NFS has all code ready to handle this, but has a VM_BUG_ON that
> triggers when this happens.  Simply remove the VM_BUG_ON to fix this
> use case.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  fs/nfs/direct.c | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
> index bb2f583eb28bf1..90079ca134dd3c 100644
> --- a/fs/nfs/direct.c
> +++ b/fs/nfs/direct.c
> @@ -141,8 +141,6 @@ int nfs_swap_rw(struct kiocb *iocb, struct
> iov_iter *iter)
>  {
>  	ssize_t ret;
>  
> -	VM_BUG_ON(iov_iter_count(iter) != PAGE_SIZE);
> -
>  	if (iov_iter_rw(iter) == READ)
>  		ret = nfs_file_direct_read(iocb, iter, true);
>  	else


This definitely seems wrong in a large folio world.

Reviewed-by: Jeff Layton <jlayton@kernel.org>

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Andrew Morton]

On Fri, 14 Jun 2024 12:03:25 +0200 Christoph Hellwig <hch@lst.de> wrote:

> As of Linux 6.10-rc the MM can swap out larger than page size chunks.
> NFS has all code ready to handle this, but has a VM_BUG_ON that
> triggers when this happens.  Simply remove the VM_BUG_ON to fix this
> use case.
> 
> ...
>
> --- a/fs/nfs/direct.c
> +++ b/fs/nfs/direct.c
> @@ -141,8 +141,6 @@ int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter)
>  {
>  	ssize_t ret;
>  
> -	VM_BUG_ON(iov_iter_count(iter) != PAGE_SIZE);
> -
>  	if (iov_iter_rw(iter) == READ)
>  		ret = nfs_file_direct_read(iocb, iter, true);
>  	else

I'm thinking this should precede "mm: swap: entirely map large folios
found in swapcache", or be a part of it.

Barry/Chuanhua, any opinions?

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Sun, Jun 16, 2024 at 5:55 AM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Fri, 14 Jun 2024 12:03:25 +0200 Christoph Hellwig <hch@lst.de> wrote:
>
> > As of Linux 6.10-rc the MM can swap out larger than page size chunks.
> > NFS has all code ready to handle this, but has a VM_BUG_ON that
> > triggers when this happens.  Simply remove the VM_BUG_ON to fix this
> > use case.

As I understand it, this isn't happening because we don't support
mTHP swapping out to a swapfile, whether it's on NFS or any
other filesystem.
static int scan_swap_map_slots(struct swap_info_struct *si,
                               unsigned char usage, int nr,
                               swp_entry_t slots[], int order)
{
        ...
        if (order > 0) {
                ...

                /*
                 * Swapfile is not block device or not using clusters so unable
                 * to allocate large entries.
                 */
                if (!(si->flags & SWP_BLKDEV) || !si->cluster_info)
                        return 0;
        }
}

However, I'm pleased to see this patch, as we might need it someday.

> >
> > ...
> >
> > --- a/fs/nfs/direct.c
> > +++ b/fs/nfs/direct.c
> > @@ -141,8 +141,6 @@ int nfs_swap_rw(struct kiocb *iocb, struct iov_iter *iter)
> >  {
> >       ssize_t ret;
> >
> > -     VM_BUG_ON(iov_iter_count(iter) != PAGE_SIZE);
> > -
> >       if (iov_iter_rw(iter) == READ)
> >               ret = nfs_file_direct_read(iocb, iter, true);
> >       else
>
> I'm thinking this should precede "mm: swap: entirely map large folios
> found in swapcache", or be a part of it.
>
> Barry/Chuanhua, any opinions?

The patch “mm: swap: entirely map large folios found in swapcache” does not
perform any I/O operations; it only maps the accessed large folio within the
swapcache. Therefore, this patch can be considered independent.

BTW, we haven't supported swapping out mTHP to nfs swapfile.

So, this patch is future-proof and can serve as a precedent if we
want to extend mthp swapout/swpin to non-blkdev systems.
Please correct me if I'm wrong, Ryan.

Thanks
Barry

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Christoph Hellwig]

On Sun, Jun 16, 2024 at 12:16:10PM +1200, Barry Song wrote:
> As I understand it, this isn't happening because we don't support
> mTHP swapping out to a swapfile, whether it's on NFS or any
> other filesystem.

It does happen.  The reason why I sent this patch is becaue I observed
the BUG_ON trigger on a trivial swap generation workload (usemem.c from
xfstests).

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Sun, Jun 16, 2024 at 4:54 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Sun, Jun 16, 2024 at 12:16:10PM +1200, Barry Song wrote:
> > As I understand it, this isn't happening because we don't support
> > mTHP swapping out to a swapfile, whether it's on NFS or any
> > other filesystem.
>
> It does happen.  The reason why I sent this patch is becaue I observed
> the BUG_ON trigger on a trivial swap generation workload (usemem.c from
> xfstests).

This is quite unusual. Could you share your setup and backtrace? I'd
like to reproduce the issue, as the mm code only supports mTHP
swapout on block devices. What is your swap device or swap file?
Additionally, on what kind of filesystem is the executable file built
from usemem.c located?

>

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Christoph Hellwig]

On Sun, Jun 16, 2024 at 06:23:16PM +0800, Barry Song wrote:
> This is quite unusual. Could you share your setup and backtrace?

Do this in an x86 kvm vm with 1G of memory:

mkfs.xfs -f /dev/vdb

mount /dev/vdb /mnt/nfs1

# this requires the following line in /etc/exports to actually export the
# file system: "/mnt/nfs2	localhost(rw,no_root_squash)"
systemctl restart nfs-kernel-server

mount -t nfs 127.0.0.1:/mnt/nfs1/ /mnt/test/

cd /mnt/test
dd if=/dev/zero of=swapfile bs=1024 count=524288
mkswap swapfile
chmod 0600 swapfile
swapon swapfile

# xfstests-dev is a checked out and built xfstests repo
~/xfstests-dev/src/usemem 1024

The backtrace is at the end of the mail.

> I'd
> like to reproduce the issue, as the mm code only supports mTHP
> swapout on block devices. What is your swap device or swap file?

The above swap fіle on NFS is the only swap in the system

> Additionally, on what kind of filesystem is the executable file built
> from usemem.c located?

Doesn't matter.  I've tried xfs, ext4 and nfs itself.

root@testvm:~# ./xfstests-dev/src/usemem 1024
[   65.065292] ------------[ cut here ]------------
[   65.065528] kernel BUG at fs/nfs/direct.c:144!
[   65.065724] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[   65.065988] CPU: 0 PID: 71 Comm: kswapd0 Not tainted 6.10.0-rc3+ #2587
[   65.066270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[   65.066683] RIP: 0010:nfs_swap_rw+0x3f/0x50
[   65.066893] Code: 00 00 74 13 e8 e2 fb ff ff 31 d2 48 85 c0 48 0f 4f c2 c3 cc cc cc cc e8 9f f8 ff ff 31 0
[   65.067698] RSP: 0018:ffffc900001b79c8 EFLAGS: 00010206
[   65.067949] RAX: ffffffff8162afc0 RBX: ffff88800ca24c00 RCX: 0000000000000004
[   65.068274] RDX: ffff88800ca24c30 RSI: ffffc900001b79d0 RDI: ffff88800ca24c00
[   65.068588] RBP: ffff88800dd710e8 R08: 0000000000004000 R09: 0000000000000010
[   65.068911] R10: 0000000000035a40 R11: 000000000000001d R12: ffffc900001b7a78
[   65.069225] R13: ffffc900001b7db0 R14: ffffc900001b7a68 R15: ffffea00003f1c40
[   65.069542] FS:  0000000000000000(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000
[   65.069904] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   65.070157] CR2: 00005606ddec4dd0 CR3: 000000000365a003 CR4: 0000000000770ef0
[   65.070472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   65.070784] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400
[   65.071282] PKRU: 55555554
[   65.071407] Call Trace:
[   65.071519]  <TASK>
[   65.071619]  ? die+0x31/0x80
[   65.071754]  ? do_trap+0xd7/0x100
[   65.071915]  ? do_error_trap+0x65/0x90
[   65.072084]  ? nfs_swap_rw+0x3f/0x50
[   65.072247]  ? exc_invalid_op+0x50/0x70
[   65.072420]  ? nfs_swap_rw+0x3f/0x50
[   65.072581]  ? asm_exc_invalid_op+0x1a/0x20
[   65.072770]  ? __pfx_nfs_swap_rw+0x10/0x10
[   65.072966]  ? nfs_swap_rw+0x3f/0x50
[   65.073125]  swap_write_unplug+0x57/0x90
[   65.073307]  shrink_folio_list+0x3a1/0xed0
[   65.073500]  ? __mod_zone_page_state+0x61/0xa0
[   65.073700]  ? isolate_lru_folios+0x2f3/0x400
[   65.073902]  shrink_lruvec+0x587/0xb60
[   65.074071]  ? shrink_slab+0x3ad/0x3c0
[   65.074239]  ? shrink_slab+0x3ad/0x3c0
[   65.074408]  shrink_node+0x2b7/0x7e0
[   65.074569]  balance_pgdat+0x291/0x6f0

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Mon, Jun 17, 2024 at 5:32 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Sun, Jun 16, 2024 at 06:23:16PM +0800, Barry Song wrote:
> > This is quite unusual. Could you share your setup and backtrace?
>
> Do this in an x86 kvm vm with 1G of memory:
>
> mkfs.xfs -f /dev/vdb
>
> mount /dev/vdb /mnt/nfs1
>
> # this requires the following line in /etc/exports to actually export the
> # file system: "/mnt/nfs2       localhost(rw,no_root_squash)"
> systemctl restart nfs-kernel-server
>
> mount -t nfs 127.0.0.1:/mnt/nfs1/ /mnt/test/
>
> cd /mnt/test
> dd if=/dev/zero of=swapfile bs=1024 count=524288
> mkswap swapfile
> chmod 0600 swapfile
> swapon swapfile

I did everything the same as above,
but always got failure at the last step to swapon:
/mnt/test # swapon swapfile
swapon: /mnt/test/swapfile: swapon failed: Invalid argument

the mount status:
/dev/vdb on /mnt/nfs1 type xfs
(rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
127.0.0.1:/mnt/nfs1 on /mnt/test type nfs4
(rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

As long as the swapfile is not on nfs, I am ok to swapon:
/mnt # swapon swapfile
[ 1150.302641] Adding 65532k swap on /mnt/swapfile.  Priority:-2
extents:1 across:65532k

Anything I am missing?

>
> # xfstests-dev is a checked out and built xfstests repo
> ~/xfstests-dev/src/usemem 1024
>
> The backtrace is at the end of the mail.
>
> > I'd
> > like to reproduce the issue, as the mm code only supports mTHP
> > swapout on block devices. What is your swap device or swap file?
>
> The above swap fіle on NFS is the only swap in the system
>
> > Additionally, on what kind of filesystem is the executable file built
> > from usemem.c located?
>
> Doesn't matter.  I've tried xfs, ext4 and nfs itself.
>
> root@testvm:~# ./xfstests-dev/src/usemem 1024
> [   65.065292] ------------[ cut here ]------------
> [   65.065528] kernel BUG at fs/nfs/direct.c:144!
> [   65.065724] Oops: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
> [   65.065988] CPU: 0 PID: 71 Comm: kswapd0 Not tainted 6.10.0-rc3+ #2587
> [   65.066270] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [   65.066683] RIP: 0010:nfs_swap_rw+0x3f/0x50
> [   65.066893] Code: 00 00 74 13 e8 e2 fb ff ff 31 d2 48 85 c0 48 0f 4f c2 c3 cc cc cc cc e8 9f f8 ff ff 31 0
> [   65.067698] RSP: 0018:ffffc900001b79c8 EFLAGS: 00010206
> [   65.067949] RAX: ffffffff8162afc0 RBX: ffff88800ca24c00 RCX: 0000000000000004
> [   65.068274] RDX: ffff88800ca24c30 RSI: ffffc900001b79d0 RDI: ffff88800ca24c00
> [   65.068588] RBP: ffff88800dd710e8 R08: 0000000000004000 R09: 0000000000000010
> [   65.068911] R10: 0000000000035a40 R11: 000000000000001d R12: ffffc900001b7a78
> [   65.069225] R13: ffffc900001b7db0 R14: ffffc900001b7a68 R15: ffffea00003f1c40
> [   65.069542] FS:  0000000000000000(0000) GS:ffff88803ec00000(0000) knlGS:0000000000000000
> [   65.069904] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   65.070157] CR2: 00005606ddec4dd0 CR3: 000000000365a003 CR4: 0000000000770ef0
> [   65.070472] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   65.070784] DR3: 0000000000000000 DR6: 00000000ffff07f0 DR7: 0000000000000400
> [   65.071282] PKRU: 55555554
> [   65.071407] Call Trace:
> [   65.071519]  <TASK>
> [   65.071619]  ? die+0x31/0x80
> [   65.071754]  ? do_trap+0xd7/0x100
> [   65.071915]  ? do_error_trap+0x65/0x90
> [   65.072084]  ? nfs_swap_rw+0x3f/0x50
> [   65.072247]  ? exc_invalid_op+0x50/0x70
> [   65.072420]  ? nfs_swap_rw+0x3f/0x50
> [   65.072581]  ? asm_exc_invalid_op+0x1a/0x20
> [   65.072770]  ? __pfx_nfs_swap_rw+0x10/0x10
> [   65.072966]  ? nfs_swap_rw+0x3f/0x50
> [   65.073125]  swap_write_unplug+0x57/0x90
> [   65.073307]  shrink_folio_list+0x3a1/0xed0
> [   65.073500]  ? __mod_zone_page_state+0x61/0xa0
> [   65.073700]  ? isolate_lru_folios+0x2f3/0x400
> [   65.073902]  shrink_lruvec+0x587/0xb60
> [   65.074071]  ? shrink_slab+0x3ad/0x3c0
> [   65.074239]  ? shrink_slab+0x3ad/0x3c0
> [   65.074408]  shrink_node+0x2b7/0x7e0
> [   65.074569]  balance_pgdat+0x291/0x6f0

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Christoph Hellwig]

On Mon, Jun 17, 2024 at 08:02:47PM +1200, Barry Song wrote:
> I did everything the same as above,
> but always got failure at the last step to swapon:
> /mnt/test # swapon swapfile
> swapon: /mnt/test/swapfile: swapon failed: Invalid argument

You are probably missing

CONFIG_NFS_SWAP=y

in your .config.

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Tue, Jun 18, 2024 at 5:52 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Mon, Jun 17, 2024 at 08:02:47PM +1200, Barry Song wrote:
> > I did everything the same as above,
> > but always got failure at the last step to swapon:
> > /mnt/test # swapon swapfile
> > swapon: /mnt/test/swapfile: swapon failed: Invalid argument
>
> You are probably missing
>
> CONFIG_NFS_SWAP=y
>
> in your .config.

Yes, that was exactly what I missed. I then figured it out, reproduced
the issue,
and discovered that the root cause was unrelated to large folios. It
was actually
due to a batched bio plugging optimization from 2022. You can find the new patch
here:

https://lore.kernel.org/linux-mm/20240617220135.43563-1-21cnbao@gmail.com/


>

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Christoph Hellwig]

On Tue, Jun 18, 2024 at 06:05:33PM +1200, Barry Song wrote:
> Yes, that was exactly what I missed. I then figured it out, reproduced
> the issue,
> and discovered that the root cause was unrelated to large folios. It
> was actually
> due to a batched bio plugging optimization from 2022. You can find the new patch
> here:
> 
> https://lore.kernel.org/linux-mm/20240617220135.43563-1-21cnbao@gmail.com/

I don't really see any point in keeping the VM_BUG_ON.  The underlying
direct I/O code doesn't really care about the size at all.

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Tue, Jun 18, 2024 at 6:13 PM Christoph Hellwig <hch@lst.de> wrote:
>
> On Tue, Jun 18, 2024 at 06:05:33PM +1200, Barry Song wrote:
> > Yes, that was exactly what I missed. I then figured it out, reproduced
> > the issue,
> > and discovered that the root cause was unrelated to large folios. It
> > was actually
> > due to a batched bio plugging optimization from 2022. You can find the new patch
> > here:
> >
> > https://lore.kernel.org/linux-mm/20240617220135.43563-1-21cnbao@gmail.com/
>
> I don't really see any point in keeping the VM_BUG_ON.  The underlying
> direct I/O code doesn't really care about the size at all.

I am perfectly fine with dropping the VM_BUG/WARN. I was keeping it
for debugging
purposes while we add mTHP swapout support for the swapfile. Large folios might
be in bio vec, this can help verify things are still normal after
having large folios there.

But I agree that this is not important for the NFS code itself. I can
add this warning
locally for debugging when needed.

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Ryan Roberts]

On 16/06/2024 11:23, Barry Song wrote:
> On Sun, Jun 16, 2024 at 4:54 PM Christoph Hellwig <hch@lst.de> wrote:
>>
>> On Sun, Jun 16, 2024 at 12:16:10PM +1200, Barry Song wrote:
>>> As I understand it, this isn't happening because we don't support
>>> mTHP swapping out to a swapfile, whether it's on NFS or any
>>> other filesystem.
>>
>> It does happen.  The reason why I sent this patch is becaue I observed
>> the BUG_ON trigger on a trivial swap generation workload (usemem.c from
>> xfstests).
> 
> This is quite unusual. Could you share your setup and backtrace? I'd
> like to reproduce the issue, as the mm code only supports mTHP
> swapout on block devices. What is your swap device or swap file?
> Additionally, on what kind of filesystem is the executable file built
> from usemem.c located?

Yes, I'm also confused by this, since as Barry says, the swap-out changes to
support mTHP are only intended to be activated when the swap device is a
non-rotating block device - swap files on file systems are explicitly not
supported and all swapping should be done page-by-page in that case. This
constraint is exactly the same as for the pre-existing PMD-size THP swap-out
support. So if you are seeing large folios being written after the mTHP swap-out
change, you should also be seeing large folios before this change.

Hopefully the stack trace will tell us what's going on here.

(Sorry for my slow responses/lack of engagement over the last month; its been a
combination of paternity leave/lack of sleep/working on other things. I'm hoping
to get properly back into this stuff within the next couple of weeks).

Thanks,
Ryan

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Barry Song]

On Mon, Jun 17, 2024 at 8:03 PM Ryan Roberts <ryan.roberts@arm.com> wrote:
>
> On 16/06/2024 11:23, Barry Song wrote:
> > On Sun, Jun 16, 2024 at 4:54 PM Christoph Hellwig <hch@lst.de> wrote:
> >>
> >> On Sun, Jun 16, 2024 at 12:16:10PM +1200, Barry Song wrote:
> >>> As I understand it, this isn't happening because we don't support
> >>> mTHP swapping out to a swapfile, whether it's on NFS or any
> >>> other filesystem.
> >>
> >> It does happen.  The reason why I sent this patch is becaue I observed
> >> the BUG_ON trigger on a trivial swap generation workload (usemem.c from
> >> xfstests).
> >
> > This is quite unusual. Could you share your setup and backtrace? I'd
> > like to reproduce the issue, as the mm code only supports mTHP
> > swapout on block devices. What is your swap device or swap file?
> > Additionally, on what kind of filesystem is the executable file built
> > from usemem.c located?
>
> Yes, I'm also confused by this, since as Barry says, the swap-out changes to
> support mTHP are only intended to be activated when the swap device is a
> non-rotating block device - swap files on file systems are explicitly not
> supported and all swapping should be done page-by-page in that case. This
> constraint is exactly the same as for the pre-existing PMD-size THP swap-out
> support. So if you are seeing large folios being written after the mTHP swap-out
> change, you should also be seeing large folios before this change.
>
> Hopefully the stack trace will tell us what's going on here.

Hi Ryan, Christoph,

I am able to reproduce the issue now. I am debugging and will update
the root cause
with you this week.

Initial investigation shows the issue might *not* be related to THP_SWPOUT.

I am even able to reproduce it after disabling thp and mthp, entirely by
small folios:

[  215.925069] folio_alloc_swap folio nr:1 anon:1 swapbacked:1
[  215.926383] vmscan: shrink_folio_list folio nr:1 anon:1 swapbacked:1
[  215.927008] folio_alloc_swap folio nr:1 anon:1 swapbacked:1
[  215.929368] ------------[ cut here ]------------
[  215.929824] kernel BUG at fs/nfs/direct.c:144!
[  215.930403] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
[  215.931264] Modules linked in:
[  215.932328] CPU: 3 PID: 214 Comm: mthp_swpout_tes Not tainted
6.10.0-rc3-ga12328d9fb85-dirty #292
[  215.932953] Hardware name: linux,dummy-virt (DT)
[  215.933461] pstate: 21400005 (nzCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
[  215.934030] pc : nfs_swap_rw+0x60/0x70
[  215.935079] lr : swap_write_unplug+0x64/0xb0
[  215.935559] sp : ffff800087363280
[  215.935958] x29: ffff800087363280 x28: ffff0000c3241800 x27: fffffdffc323a4c0
[  215.937012] x26: fffffdffc323a4c8 x25: ffff0001b4a51500 x24: ffff80008250f670
[  215.937893] x23: 0000000000000001 x22: ffff0000c0b2da00 x21: 0000000000020000
[  215.938734] x20: ffff0000c46a8bd8 x19: ffff0000c154f800 x18: ffffffffffffffff
[  215.939594] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800107363097
[  215.940591] x14: 0000000000000000 x13: 313a64656b636162 x12: 7061777320313a6e
[  215.941621] x11: 6f6e6120313a726e x10: ffff800083e86318 x9 : ffff8000803e9ad4
[  215.942673] x8 : ffff800087363168 x7 : 0000000000000000 x6 : ffff0001adbfa4c6
[  215.943674] x5 : 0000000000000002 x4 : 0000000000020000 x3 : 0000000000020000
[  215.944673] x2 : ffff8000806015e8 x1 : ffff8000873632a0 x0 : ffff0000c154f800
[  215.945568] Call trace:
[  215.945906]  nfs_swap_rw+0x60/0x70
[  215.946351]  __swap_writepage+0x2e8/0x328
[  215.946775]  swap_writepage+0x68/0xd0
[  215.947184]  pageout+0xe4/0x430
[  215.947587]  shrink_folio_list+0x9bc/0xf60
[  215.947992]  reclaim_folio_list+0x8c/0x168
[  215.948454]  reclaim_pages+0xfc/0x178
[  215.948843]  madvise_cold_or_pageout_pte_range+0x8d8/0xf28
[  215.949285]  walk_pgd_range+0x390/0x808
[  215.949660]  __walk_page_range+0x1e0/0x1f0
[  215.950040]  walk_page_range+0x1f0/0x2c8
[  215.950458]  madvise_pageout+0xf8/0x280
[  215.950905]  madvise_vma_behavior+0x314/0xa20
[  215.951361]  madvise_walk_vmas+0xc0/0x128
[  215.951807]  do_madvise.part.0+0x110/0x558
[  215.952298]  __arm64_sys_madvise+0x68/0x88
[  215.952723]  invoke_syscall+0x50/0x128
[  215.953148]  el0_svc_common.constprop.0+0x48/0xf8
[  215.953592]  do_el0_svc+0x28/0x40
[  215.954036]  el0_svc+0x50/0x150
[  215.954610]  el0t_64_sync_handler+0x13c/0x158
[  215.955070]  el0t_64_sync+0x1a4/0x1a8
[  215.955685] Code: a8c17bfd d50323bf 9a9fd000 d65f03c0 (d4210000)
[  215.956510] ---[ end trace 0000000000000000 ]---


>
> (Sorry for my slow responses/lack of engagement over the last month; its been a
> combination of paternity leave/lack of sleep/working on other things. I'm hoping
> to get properly back into this stuff within the next couple of weeks).
>
> Thanks,
> Ryan
>

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Ryan Roberts]

On 17/06/2024 10:40, Barry Song wrote:
> On Mon, Jun 17, 2024 at 8:03 PM Ryan Roberts <ryan.roberts@arm.com> wrote:
>>
>> On 16/06/2024 11:23, Barry Song wrote:
>>> On Sun, Jun 16, 2024 at 4:54 PM Christoph Hellwig <hch@lst.de> wrote:
>>>>
>>>> On Sun, Jun 16, 2024 at 12:16:10PM +1200, Barry Song wrote:
>>>>> As I understand it, this isn't happening because we don't support
>>>>> mTHP swapping out to a swapfile, whether it's on NFS or any
>>>>> other filesystem.
>>>>
>>>> It does happen.  The reason why I sent this patch is becaue I observed
>>>> the BUG_ON trigger on a trivial swap generation workload (usemem.c from
>>>> xfstests).
>>>
>>> This is quite unusual. Could you share your setup and backtrace? I'd
>>> like to reproduce the issue, as the mm code only supports mTHP
>>> swapout on block devices. What is your swap device or swap file?
>>> Additionally, on what kind of filesystem is the executable file built
>>> from usemem.c located?
>>
>> Yes, I'm also confused by this, since as Barry says, the swap-out changes to
>> support mTHP are only intended to be activated when the swap device is a
>> non-rotating block device - swap files on file systems are explicitly not
>> supported and all swapping should be done page-by-page in that case. This
>> constraint is exactly the same as for the pre-existing PMD-size THP swap-out
>> support. So if you are seeing large folios being written after the mTHP swap-out
>> change, you should also be seeing large folios before this change.
>>
>> Hopefully the stack trace will tell us what's going on here.
> 
> Hi Ryan, Christoph,
> 
> I am able to reproduce the issue now. I am debugging and will update
> the root cause
> with you this week.

Ahh great; for some reason I'm not receiving Chrostoph's mails so didn't see the
stack trace and instructions until you replied to it. I had a go are repro'ing
too but am failing to even get the systemd nfs service to start. I'll leave it
to you.

> 
> Initial investigation shows the issue might *not* be related to THP_SWPOUT.
> 
> I am even able to reproduce it after disabling thp and mthp, entirely by
> small folios:
> 
> [  215.925069] folio_alloc_swap folio nr:1 anon:1 swapbacked:1
> [  215.926383] vmscan: shrink_folio_list folio nr:1 anon:1 swapbacked:1
> [  215.927008] folio_alloc_swap folio nr:1 anon:1 swapbacked:1
> [  215.929368] ------------[ cut here ]------------
> [  215.929824] kernel BUG at fs/nfs/direct.c:144!
> [  215.930403] Internal error: Oops - BUG: 00000000f2000800 [#1] PREEMPT SMP
> [  215.931264] Modules linked in:
> [  215.932328] CPU: 3 PID: 214 Comm: mthp_swpout_tes Not tainted
> 6.10.0-rc3-ga12328d9fb85-dirty #292
> [  215.932953] Hardware name: linux,dummy-virt (DT)
> [  215.933461] pstate: 21400005 (nzCv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
> [  215.934030] pc : nfs_swap_rw+0x60/0x70
> [  215.935079] lr : swap_write_unplug+0x64/0xb0
> [  215.935559] sp : ffff800087363280
> [  215.935958] x29: ffff800087363280 x28: ffff0000c3241800 x27: fffffdffc323a4c0
> [  215.937012] x26: fffffdffc323a4c8 x25: ffff0001b4a51500 x24: ffff80008250f670
> [  215.937893] x23: 0000000000000001 x22: ffff0000c0b2da00 x21: 0000000000020000
> [  215.938734] x20: ffff0000c46a8bd8 x19: ffff0000c154f800 x18: ffffffffffffffff
> [  215.939594] x17: 0000000000000000 x16: 0000000000000000 x15: ffff800107363097
> [  215.940591] x14: 0000000000000000 x13: 313a64656b636162 x12: 7061777320313a6e
> [  215.941621] x11: 6f6e6120313a726e x10: ffff800083e86318 x9 : ffff8000803e9ad4
> [  215.942673] x8 : ffff800087363168 x7 : 0000000000000000 x6 : ffff0001adbfa4c6
> [  215.943674] x5 : 0000000000000002 x4 : 0000000000020000 x3 : 0000000000020000
> [  215.944673] x2 : ffff8000806015e8 x1 : ffff8000873632a0 x0 : ffff0000c154f800
> [  215.945568] Call trace:
> [  215.945906]  nfs_swap_rw+0x60/0x70
> [  215.946351]  __swap_writepage+0x2e8/0x328
> [  215.946775]  swap_writepage+0x68/0xd0
> [  215.947184]  pageout+0xe4/0x430
> [  215.947587]  shrink_folio_list+0x9bc/0xf60
> [  215.947992]  reclaim_folio_list+0x8c/0x168
> [  215.948454]  reclaim_pages+0xfc/0x178
> [  215.948843]  madvise_cold_or_pageout_pte_range+0x8d8/0xf28
> [  215.949285]  walk_pgd_range+0x390/0x808
> [  215.949660]  __walk_page_range+0x1e0/0x1f0
> [  215.950040]  walk_page_range+0x1f0/0x2c8
> [  215.950458]  madvise_pageout+0xf8/0x280
> [  215.950905]  madvise_vma_behavior+0x314/0xa20
> [  215.951361]  madvise_walk_vmas+0xc0/0x128
> [  215.951807]  do_madvise.part.0+0x110/0x558
> [  215.952298]  __arm64_sys_madvise+0x68/0x88
> [  215.952723]  invoke_syscall+0x50/0x128
> [  215.953148]  el0_svc_common.constprop.0+0x48/0xf8
> [  215.953592]  do_el0_svc+0x28/0x40
> [  215.954036]  el0_svc+0x50/0x150
> [  215.954610]  el0t_64_sync_handler+0x13c/0x158
> [  215.955070]  el0t_64_sync+0x1a4/0x1a8
> [  215.955685] Code: a8c17bfd d50323bf 9a9fd000 d65f03c0 (d4210000)
> [  215.956510] ---[ end trace 0000000000000000 ]---
> 
> 
>>
>> (Sorry for my slow responses/lack of engagement over the last month; its been a
>> combination of paternity leave/lack of sleep/working on other things. I'm hoping
>> to get properly back into this stuff within the next couple of weeks).
>>
>> Thanks,
>> Ryan
>>

Re: [PATCH] nfs: fix nfs_swap_rw for large-folio swap

[Martin Wege]

On Fri, Jun 14, 2024 at 12:38 PM Christoph Hellwig <hch@lst.de> wrote:
>
> As of Linux 6.10-rc the MM can swap out larger than page size chunks.
> NFS has all code ready to handle this, but has a VM_BUG_ON that
> triggers when this happens.  Simply remove the VM_BUG_ON to fix this
> use case.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  fs/nfs/direct.c | 2 --
>  1 file changed, 2 deletions(-)
>
> diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
> index bb2f583eb28bf1..90079ca134dd3c 100644

+1 for the patch, and thank you for fixing NFS swap

Thanks,
Martin