[regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Salvatore Bonaccorso]

Hi

[Apologies if this has been reported already but I have not found an
already filled corresponding report]

After updating from the 6.1.129 based version to 6.1.133, various
users have reported that their VMs do not boot anymore up (both KVM
and under Xen) if pci-passthrough is involved. The reports are at:

https://bugs.debian.org/1102889
https://bugs.debian.org/1102914
https://bugs.debian.org/1103153

Milan Broz bisected the issues and found that the commit introducing
the problems can be tracked down to backport of c8070b787519 ("mm:
Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got
backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in
pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60

#regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774

476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit
commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
Author: David Howells <dhowells@redhat.com>
Date:   Fri May 26 22:41:40 2023 +0100

    mm: Don't pin ZERO_PAGE in pin_user_pages()

    [ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]

    Make pin_user_pages*() leave a ZERO_PAGE unpinned if it extracts a pointer
    to it from the page tables and make unpin_user_page*() correspondingly
    ignore a ZERO_PAGE when unpinning.  We don't want to risk overrunning a
    zero page's refcount as we're only allowed ~2 million pins on it -
    something that userspace can conceivably trigger.

    Add a pair of functions to test whether a page or a folio is a ZERO_PAGE.

    Signed-off-by: David Howells <dhowells@redhat.com>
    cc: Christoph Hellwig <hch@infradead.org>
    cc: David Hildenbrand <david@redhat.com>
    cc: Lorenzo Stoakes <lstoakes@gmail.com>
    cc: Andrew Morton <akpm@linux-foundation.org>
    cc: Jens Axboe <axboe@kernel.dk>
    cc: Al Viro <viro@zeniv.linux.org.uk>
    cc: Matthew Wilcox <willy@infradead.org>
    cc: Jan Kara <jack@suse.cz>
    cc: Jeff Layton <jlayton@kernel.org>
    cc: Jason Gunthorpe <jgg@nvidia.com>
    cc: Logan Gunthorpe <logang@deltatee.com>
    cc: Hillf Danton <hdanton@sina.com>
    cc: Christian Brauner <brauner@kernel.org>
    cc: Linus Torvalds <torvalds@linux-foundation.org>
    cc: linux-fsdevel@vger.kernel.org
    cc: linux-block@vger.kernel.org
    cc: linux-kernel@vger.kernel.org
    cc: linux-mm@kvack.org
    Reviewed-by: Lorenzo Stoakes <lstoakes@gmail.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>
    Acked-by: David Hildenbrand <david@redhat.com>
    Link: https://lore.kernel.org/r/20230526214142.958751-2-dhowells@redhat.com
    Signed-off-by: Jens Axboe <axboe@kernel.dk>
    Stable-dep-of: bddf10d26e6e ("uprobes: Reject the shared zeropage in uprobe_write_opcode()")
    Signed-off-by: Sasha Levin <sashal@kernel.org>

 Documentation/core-api/pin_user_pages.rst |  6 ++++++
 include/linux/mm.h                        | 26 ++++++++++++++++++++++++--
 mm/gup.c                                  | 31 ++++++++++++++++++++++++++++++-
 3 files changed, 60 insertions(+), 3 deletions(-)

Milan verified that the issue persists in 6.1.134 so far and the patch
itself cannot be just reverted.

The failures all have a similar pattern, when pci-passthrough is used
for a pci devide, for instance under qemu the bootup will fail with:

qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container

(in the case as reported by Milan).

Any ideas here?

Regards,
Salvatore

Re: [regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Salvatore Bonaccorso]

[resent with correct stable@vger.kernel.org list]

On Tue, Apr 15, 2025 at 08:59:19PM +0200, Salvatore Bonaccorso wrote:
> Hi
> 
> [Apologies if this has been reported already but I have not found an
> already filled corresponding report]
> 
> After updating from the 6.1.129 based version to 6.1.133, various
> users have reported that their VMs do not boot anymore up (both KVM
> and under Xen) if pci-passthrough is involved. The reports are at:
> 
> https://bugs.debian.org/1102889
> https://bugs.debian.org/1102914
> https://bugs.debian.org/1103153
> 
> Milan Broz bisected the issues and found that the commit introducing
> the problems can be tracked down to backport of c8070b787519 ("mm:
> Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got
> backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in
> pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
> 
> #regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> 
> 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit
> commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> Author: David Howells <dhowells@redhat.com>
> Date:   Fri May 26 22:41:40 2023 +0100
> 
>     mm: Don't pin ZERO_PAGE in pin_user_pages()
> 
>     [ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]
> 
>     Make pin_user_pages*() leave a ZERO_PAGE unpinned if it extracts a pointer
>     to it from the page tables and make unpin_user_page*() correspondingly
>     ignore a ZERO_PAGE when unpinning.  We don't want to risk overrunning a
>     zero page's refcount as we're only allowed ~2 million pins on it -
>     something that userspace can conceivably trigger.
> 
>     Add a pair of functions to test whether a page or a folio is a ZERO_PAGE.
> 
>     Signed-off-by: David Howells <dhowells@redhat.com>
>     cc: Christoph Hellwig <hch@infradead.org>
>     cc: David Hildenbrand <david@redhat.com>
>     cc: Lorenzo Stoakes <lstoakes@gmail.com>
>     cc: Andrew Morton <akpm@linux-foundation.org>
>     cc: Jens Axboe <axboe@kernel.dk>
>     cc: Al Viro <viro@zeniv.linux.org.uk>
>     cc: Matthew Wilcox <willy@infradead.org>
>     cc: Jan Kara <jack@suse.cz>
>     cc: Jeff Layton <jlayton@kernel.org>
>     cc: Jason Gunthorpe <jgg@nvidia.com>
>     cc: Logan Gunthorpe <logang@deltatee.com>
>     cc: Hillf Danton <hdanton@sina.com>
>     cc: Christian Brauner <brauner@kernel.org>
>     cc: Linus Torvalds <torvalds@linux-foundation.org>
>     cc: linux-fsdevel@vger.kernel.org
>     cc: linux-block@vger.kernel.org
>     cc: linux-kernel@vger.kernel.org
>     cc: linux-mm@kvack.org
>     Reviewed-by: Lorenzo Stoakes <lstoakes@gmail.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>     Acked-by: David Hildenbrand <david@redhat.com>
>     Link: https://lore.kernel.org/r/20230526214142.958751-2-dhowells@redhat.com
>     Signed-off-by: Jens Axboe <axboe@kernel.dk>
>     Stable-dep-of: bddf10d26e6e ("uprobes: Reject the shared zeropage in uprobe_write_opcode()")
>     Signed-off-by: Sasha Levin <sashal@kernel.org>
> 
>  Documentation/core-api/pin_user_pages.rst |  6 ++++++
>  include/linux/mm.h                        | 26 ++++++++++++++++++++++++--
>  mm/gup.c                                  | 31 ++++++++++++++++++++++++++++++-
>  3 files changed, 60 insertions(+), 3 deletions(-)
> 
> Milan verified that the issue persists in 6.1.134 so far and the patch
> itself cannot be just reverted.
> 
> The failures all have a similar pattern, when pci-passthrough is used
> for a pci devide, for instance under qemu the bootup will fail with:
> 
> qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
> qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container
> 
> (in the case as reported by Milan).
> 
> Any ideas here?
> 
> Regards,
> Salvatore

Re: [regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Milan Broz]

On 4/15/25 8:59 PM, Salvatore Bonaccorso wrote:
> Milan verified that the issue persists in 6.1.134 so far and the patch
> itself cannot be just reverted.

Just an update:

With 6.1.134 stable as a base, I reverted that commit (upstream c8070b78751955e59b42457b974bea4a4fe00187)
+ applied upstream commit 873aefb376bbc0ed1dd2381ea1d6ec88106fdbd4 (vfio/type1: Unpin zero pages)
+ reverted upstream bddf10d26e6e5114e7415a0e442ec6f51a559468 (uprobes: Reject the shared zeropage in uprobe_write_opcode())
to be able compile code without errors.

With these changes, kernel works with my NVMe passthrough test again. This confirms the issue.

Milan

> 
> The failures all have a similar pattern, when pci-passthrough is used
> for a pci devide, for instance under qemu the bootup will fail with:
> 
> qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: VFIO_MAP_DMA failed: Cannot allocate memory
> qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:03:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0"}: vfio 0000:03:00.0: failed to setup container
> 
> (in the case as reported by Milan).
> 
> Any ideas here?
> 
> Regards,
> Salvatore

Re: [regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Alex Williamson]

On Tue, 15 Apr 2025 20:59:19 +0200
Salvatore Bonaccorso <carnil@debian.org> wrote:

> Hi
> 
> [Apologies if this has been reported already but I have not found an
> already filled corresponding report]
> 
> After updating from the 6.1.129 based version to 6.1.133, various
> users have reported that their VMs do not boot anymore up (both KVM
> and under Xen) if pci-passthrough is involved. The reports are at:
> 
> https://bugs.debian.org/1102889
> https://bugs.debian.org/1102914
> https://bugs.debian.org/1103153
> 
> Milan Broz bisected the issues and found that the commit introducing
> the problems can be tracked down to backport of c8070b787519 ("mm:
> Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got
> backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in
> pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
> 
> #regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> 
> 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit
> commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> Author: David Howells <dhowells@redhat.com>
> Date:   Fri May 26 22:41:40 2023 +0100
> 
>     mm: Don't pin ZERO_PAGE in pin_user_pages()
> 
>     [ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]

It's a bad backport, I've debugged and posted the fix for stable here:

https://lore.kernel.org/all/20250416202441.3911142-1-alex.williamson@redhat.com/

Thanks,
Alex

Re: [regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Salvatore Bonaccorso]

Hi Alex,

On Wed, Apr 16, 2025 at 02:26:45PM -0600, Alex Williamson wrote:
> On Tue, 15 Apr 2025 20:59:19 +0200
> Salvatore Bonaccorso <carnil@debian.org> wrote:
> 
> > Hi
> > 
> > [Apologies if this has been reported already but I have not found an
> > already filled corresponding report]
> > 
> > After updating from the 6.1.129 based version to 6.1.133, various
> > users have reported that their VMs do not boot anymore up (both KVM
> > and under Xen) if pci-passthrough is involved. The reports are at:
> > 
> > https://bugs.debian.org/1102889
> > https://bugs.debian.org/1102914
> > https://bugs.debian.org/1103153
> > 
> > Milan Broz bisected the issues and found that the commit introducing
> > the problems can be tracked down to backport of c8070b787519 ("mm:
> > Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got
> > backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in
> > pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
> > 
> > #regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> > 
> > 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit
> > commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
> > Author: David Howells <dhowells@redhat.com>
> > Date:   Fri May 26 22:41:40 2023 +0100
> > 
> >     mm: Don't pin ZERO_PAGE in pin_user_pages()
> > 
> >     [ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]
> 
> It's a bad backport, I've debugged and posted the fix for stable here:
> 
> https://lore.kernel.org/all/20250416202441.3911142-1-alex.williamson@redhat.com/

Thank you, that worked (replying here as well mainly to fix my mistake
in the CC to stable@vger.kernel.org, which got truncated to
table@vger.kernel.org in my initial submission).


> 
> Thanks,
> Alex
> 

-- 
  .-.  Salvatore Bonaccorso --------------- Debian GNU/Linux Developer
  oo|  ----------------------------------------- http://www.debian.org
 /`'\  GPG key ID: 0x789D6F057FD863FE --------------------------------
(\_;/) Fingerprint: 04A4 407C B914 2C23 030C  17AE 789D 6F05 7FD8 63FE

Re: [regression 6.1.y] Regression from 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in pin_user_pages()") with pci-passthrough for both KVM VMs and booting in xen DomU

[Milan Broz]

On 4/16/25 11:50 PM, Salvatore Bonaccorso wrote:
> Hi Alex,
> 
> On Wed, Apr 16, 2025 at 02:26:45PM -0600, Alex Williamson wrote:
>> On Tue, 15 Apr 2025 20:59:19 +0200
>> Salvatore Bonaccorso <carnil@debian.org> wrote:
>>
>>> Hi
>>>
>>> [Apologies if this has been reported already but I have not found an
>>> already filled corresponding report]
>>>
>>> After updating from the 6.1.129 based version to 6.1.133, various
>>> users have reported that their VMs do not boot anymore up (both KVM
>>> and under Xen) if pci-passthrough is involved. The reports are at:
>>>
>>> https://bugs.debian.org/1102889
>>> https://bugs.debian.org/1102914
>>> https://bugs.debian.org/1103153
>>>
>>> Milan Broz bisected the issues and found that the commit introducing
>>> the problems can be tracked down to backport of c8070b787519 ("mm:
>>> Don't pin ZERO_PAGE in pin_user_pages()") from 6.5-rc1 which got
>>> backported as 476c1dfefab8 ("mm: Don't pin ZERO_PAGE in
>>> pin_user_pages()") in 6.1.130. See https://bugs.debian.org/1102914#60
>>>
>>> #regzbot introduced: 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
>>>
>>> 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774 is the first bad commit
>>> commit 476c1dfefab8b98ae9c3e3ad283c2ac10d30c774
>>> Author: David Howells <dhowells@redhat.com>
>>> Date:   Fri May 26 22:41:40 2023 +0100
>>>
>>>      mm: Don't pin ZERO_PAGE in pin_user_pages()
>>>
>>>      [ Upstream commit c8070b78751955e59b42457b974bea4a4fe00187 ]
>>
>> It's a bad backport, I've debugged and posted the fix for stable here:
>>
>> https://lore.kernel.org/all/20250416202441.3911142-1-alex.williamson@redhat.com/
> 
> Thank you, that worked (replying here as well mainly to fix my mistake
> in the CC to stable@vger.kernel.org, which got truncated to
> table@vger.kernel.org in my initial submission).

I can also confirm the fix works, thanks for the quick response!

Tested-by: Milan Broz <gmazyland@gmail.com>

Milan