[PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Mikulas Patocka]

The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
flags") breaks the device mapper VDO target. The VDO target calls vmalloc
with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
flags.

There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
let's add this flag to GFP_VMALLOC_SUPPORTED.

Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
Fixes: 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")
Cc: stable@vger.kernel.org	# v6.19

---
 mm/vmalloc.c |    1 +
 1 file changed, 1 insertion(+)

Index: linux-2.6/mm/vmalloc.c
===================================================================
--- linux-2.6.orig/mm/vmalloc.c
+++ linux-2.6/mm/vmalloc.c
@@ -3928,6 +3928,7 @@ fail:
  */
 #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
 				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
+				__GFP_RETRY_MAYFAIL |\
 				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
 				GFP_USER | __GFP_NOLOCKDEP)
 

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[SeongJae Park]

On Thu, 12 Feb 2026 17:33:30 +0100 (CET) Mikulas Patocka <mpatocka@redhat.com> wrote:

> The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> flags.
> 
> There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> let's add this flag to GFP_VMALLOC_SUPPORTED.
> 
> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
> Fixes: 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")
> Cc: stable@vger.kernel.org	# v6.19

Acked-by: SeongJae Park <sj@kernel.org>

> 
> ---
>  mm/vmalloc.c |    1 +
>  1 file changed, 1 insertion(+)
> 
> Index: linux-2.6/mm/vmalloc.c
> ===================================================================
> --- linux-2.6.orig/mm/vmalloc.c
> +++ linux-2.6/mm/vmalloc.c
> @@ -3928,6 +3928,7 @@ fail:
>   */
>  #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
>  				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
> +				__GFP_RETRY_MAYFAIL |\
>  				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
>  				GFP_USER | __GFP_NOLOCKDEP)

By the way, I think this list might better to be alphabetically sorted in
future.  Obviously just a trivial and random thought that never required for
this patch.


Thanks,
SJ

[...]

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Anshuman Khandual]

On 21/02/26 6:49 AM, SeongJae Park wrote:
> On Thu, 12 Feb 2026 17:33:30 +0100 (CET) Mikulas Patocka <mpatocka@redhat.com> wrote:
> 
>> The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
>> flags") breaks the device mapper VDO target. The VDO target calls vmalloc
>> with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
>> flags.
>>
>> There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
>> let's add this flag to GFP_VMALLOC_SUPPORTED.
>>
>> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
>> Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
>> Fixes: 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")
>> Cc: stable@vger.kernel.org	# v6.19
> 
> Acked-by: SeongJae Park <sj@kernel.org>
> 
>>
>> ---
>>  mm/vmalloc.c |    1 +
>>  1 file changed, 1 insertion(+)
>>
>> Index: linux-2.6/mm/vmalloc.c
>> ===================================================================
>> --- linux-2.6.orig/mm/vmalloc.c
>> +++ linux-2.6/mm/vmalloc.c
>> @@ -3928,6 +3928,7 @@ fail:
>>   */
>>  #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
>>  				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
>> +				__GFP_RETRY_MAYFAIL |\
>>  				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
>>  				GFP_USER | __GFP_NOLOCKDEP)
> 
> By the way, I think this list might better to be alphabetically sorted in

Agreed.

> future.  Obviously just a trivial and random thought that never required for
> this patch.

Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Vishal Moola (Oracle)]

On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> flags.
> 
> There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> let's add this flag to GFP_VMALLOC_SUPPORTED.

My only skepticism about this comes from the line in the
vmalloc_node_range() doc: 
"and %__GFP_RETRY_MAYFAIL are not supported."

I myself don't know why that may be. Could you elaborate on if/why the
doc is wrong please?

> Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
> Fixes: 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")
> Cc: stable@vger.kernel.org	# v6.19
> 
> ---
>  mm/vmalloc.c |    1 +
>  1 file changed, 1 insertion(+)
> 
> Index: linux-2.6/mm/vmalloc.c
> ===================================================================
> --- linux-2.6.orig/mm/vmalloc.c
> +++ linux-2.6/mm/vmalloc.c
> @@ -3928,6 +3928,7 @@ fail:
>   */
>  #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
>  				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
> +				__GFP_RETRY_MAYFAIL |\
>  				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
>  				GFP_USER | __GFP_NOLOCKDEP)
>  
> 

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Mikulas Patocka]

On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:

> On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > flags.
> > 
> > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > let's add this flag to GFP_VMALLOC_SUPPORTED.
> 
> My only skepticism about this comes from the line in the
> vmalloc_node_range() doc: 
> "and %__GFP_RETRY_MAYFAIL are not supported."
> 
> I myself don't know why that may be. Could you elaborate on if/why the
> doc is wrong please?

This statement was added by Michal Hocko in the commit 
b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
you think that __GFP_RETRY_MAYFAIL is not supported?

The VDO module needs to allocate large amounts of memory and it doesn't 
want to trigger the OOM killer (which would kill some innocent task and 
wouldn't solve the out of memory condition at all), so I think that 
__GFP_RETRY_MAYFAIL is appropriate.

Mikulas

> > Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
> > Reported-by: Zdenek Kabelac <zkabelac@redhat.com>
> > Fixes: 07003531e03c ("mm/vmalloc: warn on invalid vmalloc gfp flags")
> > Cc: stable@vger.kernel.org	# v6.19
> > 
> > ---
> >  mm/vmalloc.c |    1 +
> >  1 file changed, 1 insertion(+)
> > 
> > Index: linux-2.6/mm/vmalloc.c
> > ===================================================================
> > --- linux-2.6.orig/mm/vmalloc.c
> > +++ linux-2.6/mm/vmalloc.c
> > @@ -3928,6 +3928,7 @@ fail:
> >   */
> >  #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
> >  				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
> > +				__GFP_RETRY_MAYFAIL |\
> >  				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
> >  				GFP_USER | __GFP_NOLOCKDEP)
> >  
> > 
> 

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Uladzislau Rezki]

On Mon, Feb 23, 2026 at 08:25:38PM +0100, Mikulas Patocka wrote:
> 
> 
> On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> 
> > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > flags.
> > > 
> > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > 
> > My only skepticism about this comes from the line in the
> > vmalloc_node_range() doc: 
> > "and %__GFP_RETRY_MAYFAIL are not supported."
> > 
> > I myself don't know why that may be. Could you elaborate on if/why the
> > doc is wrong please?
> 
> This statement was added by Michal Hocko in the commit 
> b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> you think that __GFP_RETRY_MAYFAIL is not supported?
> 
> The VDO module needs to allocate large amounts of memory and it doesn't 
> want to trigger the OOM killer (which would kill some innocent task and 
> wouldn't solve the out of memory condition at all), so I think that 
> __GFP_RETRY_MAYFAIL is appropriate.
> 
I do not see any problems using this with vmalloc from the first glance.
KASAN path was hard-coded when it comes to GFP but it is not anymore.
 
--
Uladzislau Rezki

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Michal Hocko]

On Mon 23-02-26 20:25:38, Mikulas Patocka wrote:
> 
> 
> On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> 
> > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > flags.
> > > 
> > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > 
> > My only skepticism about this comes from the line in the
> > vmalloc_node_range() doc: 
> > "and %__GFP_RETRY_MAYFAIL are not supported."
> > 
> > I myself don't know why that may be. Could you elaborate on if/why the
> > doc is wrong please?
> 
> This statement was added by Michal Hocko in the commit 
> b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> you think that __GFP_RETRY_MAYFAIL is not supported?

The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully
supported. While pages that back the allocation can be easily made aware
of this failure mode there are page table allocations which are
hardocded GFP_KERNEL and there is no sensible way to extend the API to
change that (as we have learned several time over years).

> The VDO module needs to allocate large amounts of memory and it doesn't 
> want to trigger the OOM killer (which would kill some innocent task and 
> wouldn't solve the out of memory condition at all), so I think that 
> __GFP_RETRY_MAYFAIL is appropriate.

Understood. But as said the very page table allocation could be the
trigger for the unwanted OOM. The same applies to __GFP_NORETRY
unfortunately as well. vmalloc has just recently gained support of
GFP_NOWAIT allocation mode, though. This will make the allocation
failure much more likely though so I am not entirely sure this is a
proper solution for your problem.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Uladzislau Rezki]

On Mon, Feb 23, 2026 at 11:08:02PM +0100, Michal Hocko wrote:
> On Mon 23-02-26 20:25:38, Mikulas Patocka wrote:
> > 
> > 
> > On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> > 
> > > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > > flags.
> > > > 
> > > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > > 
> > > My only skepticism about this comes from the line in the
> > > vmalloc_node_range() doc: 
> > > "and %__GFP_RETRY_MAYFAIL are not supported."
> > > 
> > > I myself don't know why that may be. Could you elaborate on if/why the
> > > doc is wrong please?
> > 
> > This statement was added by Michal Hocko in the commit 
> > b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> > you think that __GFP_RETRY_MAYFAIL is not supported?
> 
> The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully
> supported. While pages that back the allocation can be easily made aware
> of this failure mode there are page table allocations which are
> hardocded GFP_KERNEL and there is no sensible way to extend the API to
> change that (as we have learned several time over years).
> 
> > The VDO module needs to allocate large amounts of memory and it doesn't 
> > want to trigger the OOM killer (which would kill some innocent task and 
> > wouldn't solve the out of memory condition at all), so I think that 
> > __GFP_RETRY_MAYFAIL is appropriate.
> 
> Understood. But as said the very page table allocation could be the
> trigger for the unwanted OOM. The same applies to __GFP_NORETRY
> unfortunately as well. vmalloc has just recently gained support of
> GFP_NOWAIT allocation mode, though. This will make the allocation
> failure much more likely though so I am not entirely sure this is a
> proper solution for your problem.
>
Yes, the page-table manipulation entries are hard-coded and it looks
like it is the last path which is not wired properly with gfp-flags.

Since we grow PTEs and never release it might not be a big issue for
the __GFP_RETRY_MAYFAIL usage. But it is still not valid in noted path.

--
Uladzislau Rezki

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Michal Hocko]

On Tue 24-02-26 12:39:33, Uladzislau Rezki wrote:
> On Mon, Feb 23, 2026 at 11:08:02PM +0100, Michal Hocko wrote:
> > On Mon 23-02-26 20:25:38, Mikulas Patocka wrote:
> > > 
> > > 
> > > On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> > > 
> > > > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > > > flags.
> > > > > 
> > > > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > > > 
> > > > My only skepticism about this comes from the line in the
> > > > vmalloc_node_range() doc: 
> > > > "and %__GFP_RETRY_MAYFAIL are not supported."
> > > > 
> > > > I myself don't know why that may be. Could you elaborate on if/why the
> > > > doc is wrong please?
> > > 
> > > This statement was added by Michal Hocko in the commit 
> > > b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> > > you think that __GFP_RETRY_MAYFAIL is not supported?
> > 
> > The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully
> > supported. While pages that back the allocation can be easily made aware
> > of this failure mode there are page table allocations which are
> > hardocded GFP_KERNEL and there is no sensible way to extend the API to
> > change that (as we have learned several time over years).
> > 
> > > The VDO module needs to allocate large amounts of memory and it doesn't 
> > > want to trigger the OOM killer (which would kill some innocent task and 
> > > wouldn't solve the out of memory condition at all), so I think that 
> > > __GFP_RETRY_MAYFAIL is appropriate.
> > 
> > Understood. But as said the very page table allocation could be the
> > trigger for the unwanted OOM. The same applies to __GFP_NORETRY
> > unfortunately as well. vmalloc has just recently gained support of
> > GFP_NOWAIT allocation mode, though. This will make the allocation
> > failure much more likely though so I am not entirely sure this is a
> > proper solution for your problem.
> >
> Yes, the page-table manipulation entries are hard-coded and it looks
> like it is the last path which is not wired properly with gfp-flags.
> 
> Since we grow PTEs and never release it might not be a big issue for
> the __GFP_RETRY_MAYFAIL usage. But it is still not valid in noted path.

One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
__GFP_NORETRY is to use NOWAIT allocation semantic for page table
allocations as those could be achieved by scoped allocation context.
This could cause pre-mature failure after the whole bunch of memory has
already been allocated for the backing pages but considering that page
table allocations should be more and more rare over system runtime it
might be just a reasonable workaround. WDYT?
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Christoph Hellwig]

On Tue, Feb 24, 2026 at 01:22:36PM +0100, Michal Hocko wrote:
> One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
> __GFP_NORETRY is to use NOWAIT allocation semantic for page table
> allocations as those could be achieved by scoped allocation context.
> This could cause pre-mature failure after the whole bunch of memory has
> already been allocated for the backing pages but considering that page
> table allocations should be more and more rare over system runtime it
> might be just a reasonable workaround. WDYT?

Why bother?  __GFP_RETRY_MAYFAIL has pretty lose semantics.  Trying
too hard to allocate PTEs is not breaking the overall concept.

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Shakeel Butt]

On Tue, Feb 24, 2026 at 06:03:13AM -0800, Christoph Hellwig wrote:
> On Tue, Feb 24, 2026 at 01:22:36PM +0100, Michal Hocko wrote:
> > One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
> > __GFP_NORETRY is to use NOWAIT allocation semantic for page table
> > allocations as those could be achieved by scoped allocation context.
> > This could cause pre-mature failure after the whole bunch of memory has
> > already been allocated for the backing pages but considering that page
> > table allocations should be more and more rare over system runtime it
> > might be just a reasonable workaround. WDYT?
> 
> Why bother?  __GFP_RETRY_MAYFAIL has pretty lose semantics.  Trying
> too hard to allocate PTEs is not breaking the overall concept.
> 

One thing __GFP_RETRY_MAYFAIL is very clear about is to not trigger the
oom-killer which is not the case for GFP_KERNEL. There are users who explicitly
use __GFP_RETRY_MAYFAIL to avoid oom-killer.

Mikulas, is that the reason you are using __GFP_RETRY_MAYFAIL in your use-case?

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Christoph Hellwig]

On Tue, Feb 24, 2026 at 06:22:15AM -0800, Shakeel Butt wrote:
> > Why bother?  __GFP_RETRY_MAYFAIL has pretty lose semantics.  Trying
> > too hard to allocate PTEs is not breaking the overall concept.
> > 
> 
> One thing __GFP_RETRY_MAYFAIL is very clear about is to not trigger the
> oom-killer which is not the case for GFP_KERNEL. There are users who explicitly
> use __GFP_RETRY_MAYFAIL to avoid oom-killer.

That is a good point.  Compared to other options documenting that it
might would still seem like the best option after just not using
__GFP_RETRY_MAYFAIL with vmalloc.

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Michal Hocko]

On Tue 24-02-26 06:22:15, Shakeel Butt wrote:
> On Tue, Feb 24, 2026 at 06:03:13AM -0800, Christoph Hellwig wrote:
> > On Tue, Feb 24, 2026 at 01:22:36PM +0100, Michal Hocko wrote:
> > > One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
> > > __GFP_NORETRY is to use NOWAIT allocation semantic for page table
> > > allocations as those could be achieved by scoped allocation context.
> > > This could cause pre-mature failure after the whole bunch of memory has
> > > already been allocated for the backing pages but considering that page
> > > table allocations should be more and more rare over system runtime it
> > > might be just a reasonable workaround. WDYT?
> > 
> > Why bother?  __GFP_RETRY_MAYFAIL has pretty lose semantics.  Trying
> > too hard to allocate PTEs is not breaking the overall concept.
> > 
> 
> One thing __GFP_RETRY_MAYFAIL is very clear about is to not trigger the
> oom-killer which is not the case for GFP_KERNEL. There are users who explicitly
> use __GFP_RETRY_MAYFAIL to avoid oom-killer.
> 
> Mikulas, is that the reason you are using __GFP_RETRY_MAYFAIL in your use-case?

yes https://lore.kernel.org/all/32bd9bed-a939-69c4-696d-f7f9a5fe31d8@redhat.com/T/#u
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Uladzislau Rezki]

On Tue, Feb 24, 2026 at 01:22:36PM +0100, Michal Hocko wrote:
> On Tue 24-02-26 12:39:33, Uladzislau Rezki wrote:
> > On Mon, Feb 23, 2026 at 11:08:02PM +0100, Michal Hocko wrote:
> > > On Mon 23-02-26 20:25:38, Mikulas Patocka wrote:
> > > > 
> > > > 
> > > > On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> > > > 
> > > > > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > > > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > > > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > > > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > > > > flags.
> > > > > > 
> > > > > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > > > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > > > > 
> > > > > My only skepticism about this comes from the line in the
> > > > > vmalloc_node_range() doc: 
> > > > > "and %__GFP_RETRY_MAYFAIL are not supported."
> > > > > 
> > > > > I myself don't know why that may be. Could you elaborate on if/why the
> > > > > doc is wrong please?
> > > > 
> > > > This statement was added by Michal Hocko in the commit 
> > > > b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> > > > you think that __GFP_RETRY_MAYFAIL is not supported?
> > > 
> > > The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully
> > > supported. While pages that back the allocation can be easily made aware
> > > of this failure mode there are page table allocations which are
> > > hardocded GFP_KERNEL and there is no sensible way to extend the API to
> > > change that (as we have learned several time over years).
> > > 
> > > > The VDO module needs to allocate large amounts of memory and it doesn't 
> > > > want to trigger the OOM killer (which would kill some innocent task and 
> > > > wouldn't solve the out of memory condition at all), so I think that 
> > > > __GFP_RETRY_MAYFAIL is appropriate.
> > > 
> > > Understood. But as said the very page table allocation could be the
> > > trigger for the unwanted OOM. The same applies to __GFP_NORETRY
> > > unfortunately as well. vmalloc has just recently gained support of
> > > GFP_NOWAIT allocation mode, though. This will make the allocation
> > > failure much more likely though so I am not entirely sure this is a
> > > proper solution for your problem.
> > >
> > Yes, the page-table manipulation entries are hard-coded and it looks
> > like it is the last path which is not wired properly with gfp-flags.
> > 
> > Since we grow PTEs and never release it might not be a big issue for
> > the __GFP_RETRY_MAYFAIL usage. But it is still not valid in noted path.
> 
> One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
> __GFP_NORETRY is to use NOWAIT allocation semantic for page table
> allocations as those could be achieved by scoped allocation context.
> This could cause pre-mature failure after the whole bunch of memory has
> already been allocated for the backing pages but considering that page
> table allocations should be more and more rare over system runtime it
> might be just a reasonable workaround. WDYT?
>
As far as i understand, Mikulas uses __GFP_RETRY_MAYFAIL with vmalloc
for some time already. I have not seen any reports about that a PTE/xxx
alloc path triggers OOM killer thus i tend to say that it is not easy
to trigger.

I do not have a strong opinion about workaround you noted. Maybe Mikulas
can switch to NOWAIT flag instead.

--
Uladzislau Rezki

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Michal Hocko]

On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
> On Tue, Feb 24, 2026 at 01:22:36PM +0100, Michal Hocko wrote:
> > On Tue 24-02-26 12:39:33, Uladzislau Rezki wrote:
> > > On Mon, Feb 23, 2026 at 11:08:02PM +0100, Michal Hocko wrote:
> > > > On Mon 23-02-26 20:25:38, Mikulas Patocka wrote:
> > > > > 
> > > > > 
> > > > > On Mon, 23 Feb 2026, Vishal Moola (Oracle) wrote:
> > > > > 
> > > > > > On Thu, Feb 12, 2026 at 05:33:30PM +0100, Mikulas Patocka wrote:
> > > > > > > The commit 07003531e03c8 ("mm/vmalloc: warn on invalid vmalloc gfp
> > > > > > > flags") breaks the device mapper VDO target. The VDO target calls vmalloc
> > > > > > > with __GFP_RETRY_MAYFAIL and this flag is not in the mask of allowed
> > > > > > > flags.
> > > > > > > 
> > > > > > > There is no reason why vmalloc couldn't support __GFP_RETRY_MAYFAIL, so 
> > > > > > > let's add this flag to GFP_VMALLOC_SUPPORTED.
> > > > > > 
> > > > > > My only skepticism about this comes from the line in the
> > > > > > vmalloc_node_range() doc: 
> > > > > > "and %__GFP_RETRY_MAYFAIL are not supported."
> > > > > > 
> > > > > > I myself don't know why that may be. Could you elaborate on if/why the
> > > > > > doc is wrong please?
> > > > > 
> > > > > This statement was added by Michal Hocko in the commit 
> > > > > b7d90e7a5ea8d64e668d5685925900d33d3884d5. Michal, could you explain why do 
> > > > > you think that __GFP_RETRY_MAYFAIL is not supported?
> > > > 
> > > > The problem with __GFP_RETRY_MAYFAIL is that it cannot be fully
> > > > supported. While pages that back the allocation can be easily made aware
> > > > of this failure mode there are page table allocations which are
> > > > hardocded GFP_KERNEL and there is no sensible way to extend the API to
> > > > change that (as we have learned several time over years).
> > > > 
> > > > > The VDO module needs to allocate large amounts of memory and it doesn't 
> > > > > want to trigger the OOM killer (which would kill some innocent task and 
> > > > > wouldn't solve the out of memory condition at all), so I think that 
> > > > > __GFP_RETRY_MAYFAIL is appropriate.
> > > > 
> > > > Understood. But as said the very page table allocation could be the
> > > > trigger for the unwanted OOM. The same applies to __GFP_NORETRY
> > > > unfortunately as well. vmalloc has just recently gained support of
> > > > GFP_NOWAIT allocation mode, though. This will make the allocation
> > > > failure much more likely though so I am not entirely sure this is a
> > > > proper solution for your problem.
> > > >
> > > Yes, the page-table manipulation entries are hard-coded and it looks
> > > like it is the last path which is not wired properly with gfp-flags.
> > > 
> > > Since we grow PTEs and never release it might not be a big issue for
> > > the __GFP_RETRY_MAYFAIL usage. But it is still not valid in noted path.
> > 
> > One thing that we could do to improve __GFP_RETRY_MAYFAIL resp.
> > __GFP_NORETRY is to use NOWAIT allocation semantic for page table
> > allocations as those could be achieved by scoped allocation context.
> > This could cause pre-mature failure after the whole bunch of memory has
> > already been allocated for the backing pages but considering that page
> > table allocations should be more and more rare over system runtime it
> > might be just a reasonable workaround. WDYT?
> >
> As far as i understand, Mikulas uses __GFP_RETRY_MAYFAIL with vmalloc
> for some time already. I have not seen any reports about that a PTE/xxx
> alloc path triggers OOM killer thus i tend to say that it is not easy
> to trigger.

Yes, triggering OOM killer is usually not all that easy. But rare OOMs
are likely even worse because they are harder to test for. So I can see
reasosning to avoid it as much as possible and rather fail early.

> I do not have a strong opinion about workaround you noted. Maybe Mikulas
> can switch to NOWAIT flag instead.

using NOWAIT for the full vmalloc allocation would be just too easy to
fail under moderate memory pressure.

The real question is whether we want to provide some sort of backoff
early but not way too easily allocation semantic for vmalloc. If yes we
need to get creative in the vmalloc internals rather than expect callers
to be working around that on their side. History has proven that this
just leads to tech. dept and more work later on.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Michal Hocko]

On Tue 24-02-26 16:44:43, Michal Hocko wrote:
> On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
[...]
> > I do not have a strong opinion about workaround you noted. Maybe Mikulas
> > can switch to NOWAIT flag instead.
> 
> using NOWAIT for the full vmalloc allocation would be just too easy to
> fail under moderate memory pressure.
> 
> The real question is whether we want to provide some sort of backoff
> early but not way too easily allocation semantic for vmalloc. If yes we
> need to get creative in the vmalloc internals rather than expect callers
> to be working around that on their side. History has proven that this
> just leads to tech. dept and more work later on.

Just to make sure we are on the same page I mean something like this 
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 61caa55a4402..791366fe44e2 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)
  * non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save()
  * GFP_NOFS - memalloc_nofs_save()
  * GFP_NOIO - memalloc_noio_save()
+ * __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent
+ * OOMs
  *
  * Returns a flag cookie to pair with restore.
  */
@@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask)
 {
 	unsigned int flags = 0;
 
-	if (!gfpflags_allow_blocking(gfp_mask))
+	if (!gfpflags_allow_blocking(gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY)))
 		flags = memalloc_noreclaim_save();
 	else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
 		flags = memalloc_nofs_save();
@@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
  * GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP.
  */
 #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
-				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
+				__GFP_NOFAIL |  __GFP_ZERO | |\
+				__GFP_NORETRY | __GFP_RETRY_MAYFAIL |\
 				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
 				GFP_USER | __GFP_NOLOCKDEP)
 
@@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)
  * virtual range with protection @prot.
  *
  * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT,
- * %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported.
+ * %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO.
+ * Zone modifiers are not supported.
  * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only
  * by __vmalloc().
  *
- * Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY
- * and %__GFP_RETRY_MAYFAIL are not supported.
+ * Retry modifiers: only %__GFP_NOFAIL is fully supported; 
+ * %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation,
+ * i.e. page tables are allocated with NOWAIT semantic so they might fail
+ * under moderate memory pressure. 
  *
  * %__GFP_NOWARN can be used to suppress failure messages.
  *

-- 
Michal Hocko
SUSE Labs

Re: [PATCH] mm: allow __GFP_RETRY_MAYFAIL in vmalloc

[Uladzislau Rezki]

On Tue, Feb 24, 2026 at 04:51:35PM +0100, Michal Hocko wrote:
> On Tue 24-02-26 16:44:43, Michal Hocko wrote:
> > On Tue 24-02-26 16:38:01, Uladzislau Rezki wrote:
> [...]
> > > I do not have a strong opinion about workaround you noted. Maybe Mikulas
> > > can switch to NOWAIT flag instead.
> > 
> > using NOWAIT for the full vmalloc allocation would be just too easy to
> > fail under moderate memory pressure.
> > 
> > The real question is whether we want to provide some sort of backoff
> > early but not way too easily allocation semantic for vmalloc. If yes we
> > need to get creative in the vmalloc internals rather than expect callers
> > to be working around that on their side. History has proven that this
> > just leads to tech. dept and more work later on.
> 
> Just to make sure we are on the same page I mean something like this 
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index 61caa55a4402..791366fe44e2 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -3798,6 +3798,8 @@ static void defer_vm_area_cleanup(struct vm_struct *area)
>   * non-blocking (no __GFP_DIRECT_RECLAIM) - memalloc_noreclaim_save()
>   * GFP_NOFS - memalloc_nofs_save()
>   * GFP_NOIO - memalloc_noio_save()
> + * __GFP_RETRY_MAYFAIL, __GFP_NORETRY - memalloc_noreclaim_save() to prevent
> + * OOMs
>   *
>   * Returns a flag cookie to pair with restore.
>   */
> @@ -3806,7 +3808,7 @@ memalloc_apply_gfp_scope(gfp_t gfp_mask)
>  {
>  	unsigned int flags = 0;
>  
> -	if (!gfpflags_allow_blocking(gfp_mask))
> +	if (!gfpflags_allow_blocking(gfp_mask) || (gfp_mask & (__GFP_RETRY_MAYFAIL|__GFP_NORETRY)))
>  		flags = memalloc_noreclaim_save();
>  	else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
>  		flags = memalloc_nofs_save();
> @@ -3940,7 +3942,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask,
>   * GFP_KERNEL_ACCOUNT. Xfs uses __GFP_NOLOCKDEP.
>   */
>  #define GFP_VMALLOC_SUPPORTED (GFP_KERNEL | GFP_ATOMIC | GFP_NOWAIT |\
> -				__GFP_NOFAIL |  __GFP_ZERO | __GFP_NORETRY |\
> +				__GFP_NOFAIL |  __GFP_ZERO | |\
> +				__GFP_NORETRY | __GFP_RETRY_MAYFAIL |\
>  				GFP_NOFS | GFP_NOIO | GFP_KERNEL_ACCOUNT |\
>  				GFP_USER | __GFP_NOLOCKDEP)
>  
> @@ -3971,12 +3974,15 @@ static gfp_t vmalloc_fix_flags(gfp_t flags)
>   * virtual range with protection @prot.
>   *
>   * Supported GFP classes: %GFP_KERNEL, %GFP_ATOMIC, %GFP_NOWAIT,
> - * %GFP_NOFS and %GFP_NOIO. Zone modifiers are not supported.
> + * %__GFP_RETRY_MAYFAIL, %__GFP_NORETRY, %GFP_NOFS and %GFP_NOIO.
> + * Zone modifiers are not supported.
>   * Please note %GFP_ATOMIC and %GFP_NOWAIT are supported only
>   * by __vmalloc().
>   *
> - * Retry modifiers: only %__GFP_NOFAIL is supported; %__GFP_NORETRY
> - * and %__GFP_RETRY_MAYFAIL are not supported.
> + * Retry modifiers: only %__GFP_NOFAIL is fully supported; 
> + * %__GFP_NORETRY and %__GFP_RETRY_MAYFAIL are supported with limitation,
> + * i.e. page tables are allocated with NOWAIT semantic so they might fail
> + * under moderate memory pressure. 
>   *
>   * %__GFP_NOWARN can be used to suppress failure messages.
>   *
> 
Yep, i got your intention correctly. It would eliminate the problem
with page tables allocations :)

--
Uladzislau Rezki