[PATCH v2 0/2] kasan: vmalloc: Fix incorrect tag assignment with multiple vm_structs

[PATCH v2 0/2] kasan: vmalloc: Fix incorrect tag assignment with multiple vm_structs

[Maciej Wieczor-Retman]

A KASAN tag mismatch, possibly resulting in a kernel panic, can be
observed on systems with a tag-based KASAN enabled and with multiple
NUMA nodes. Initially it was only noticed on x86 [1] but later a similar
issue was also reported on arm64 [2].

Specifically the problem is related to how vm_structs interact with
pcpu_chunks - both when they are allocated, assigned and when pcpu_chunk
addresses are derived.

When vm_structs are allocated they are tagged if vmalloc support is
enabled along the KASAN mode. Later when first pcpu chunk is allocated
it gets its 'base_addr' field set to the first allocated vm_struct.
With that it inherits that vm_struct's tag.

When pcpu_chunk addresses are later derived (by pcpu_chunk_addr(), for
example in pcpu_alloc_noprof()) the base_addr field is used and offsets
are added to it. If the initial conditions are satisfied then some of
the offsets will point into memory allocated with a different vm_struct.
So while the lower bits will get accurately derived the tag bits in the
top of the pointer won't match the shadow memory contents.

The solution (proposed at v2 of the x86 KASAN series [3]) is to tag the
vm_structs the same when allocating them for the per cpu allocator (in
pcpu_get_vm_areas()).

Originally these patches were part of the x86 KASAN series [4].

The series is based on 6.18.

[1] https://lore.kernel.org/all/e7e04692866d02e6d3b32bb43b998e5d17092ba4.1738686764.git.maciej.wieczor-retman@intel.com/
[2] https://lore.kernel.org/all/aMUrW1Znp1GEj7St@MiWiFi-R3L-srv/
[3] https://lore.kernel.org/all/CAPAsAGxDRv_uFeMYu9TwhBVWHCCtkSxoWY4xmFB_vowMbi8raw@mail.gmail.com/
[4] https://lore.kernel.org/all/cover.1761763681.git.m.wieczorretman@pm.me/

Changes v2:
- Redid the patches since last version wasn't an actual refactor as the
  patch promised.
- Also fixed multiple mistakes and retested everything.

Maciej Wieczor-Retman (2):
  kasan: Refactor pcpu kasan vmalloc unpoison
  kasan: Unpoison vms[area] addresses with a common tag

 include/linux/kasan.h | 16 +++++++++++++---
 mm/kasan/common.c     | 18 ++++++++++++++++++
 mm/kasan/hw_tags.c    | 21 ++++++++++++++++++---
 mm/kasan/shadow.c     | 25 ++++++++++++++++++++++---
 mm/vmalloc.c          |  4 +---
 5 files changed, 72 insertions(+), 12 deletions(-)

-- 
2.52.0

[PATCH v2 1/2] kasan: Refactor pcpu kasan vmalloc unpoison

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>

A KASAN tag mismatch, possibly causing a kernel panic, can be observed
on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
It was reported on arm64 and reproduced on x86. It can be explained in
the following points:

	1. There can be more than one virtual memory chunk.
	2. Chunk's base address has a tag.
	3. The base address points at the first chunk and thus inherits
	   the tag of the first chunk.
	4. The subsequent chunks will be accessed with the tag from the
	   first chunk.
	5. Thus, the subsequent chunks need to have their tag set to
	   match that of the first chunk.

Refactor code by reusing __kasan_unpoison_vmalloc in a new helper in
preparation for the actual fix.

Changelog v1 (after splitting of from the KASAN series):
- Rewrite first paragraph of the patch message to point at the user
  impact of the issue.
- Move helper to common.c so it can be compiled in all KASAN modes.

Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS")
Cc: <stable@vger.kernel.org> # 6.1+
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
---
Changelog v2:
- Redo the whole patch so it's an actual refactor.

 include/linux/kasan.h | 16 +++++++++++++---
 mm/kasan/common.c     | 17 +++++++++++++++++
 mm/kasan/hw_tags.c    | 15 +++++++++++++--
 mm/kasan/shadow.c     | 16 ++++++++++++++--
 mm/vmalloc.c          |  4 +---
 5 files changed, 58 insertions(+), 10 deletions(-)

diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index d12e1a5f5a9a..4a3d3dba9764 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -595,14 +595,14 @@ static inline void kasan_release_vmalloc(unsigned long start,
 
 #endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
 
-void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
-			       kasan_vmalloc_flags_t flags);
+void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
+				      kasan_vmalloc_flags_t flags);
 static __always_inline void *kasan_unpoison_vmalloc(const void *start,
 						unsigned long size,
 						kasan_vmalloc_flags_t flags)
 {
 	if (kasan_enabled())
-		return __kasan_unpoison_vmalloc(start, size, flags);
+		return __kasan_random_unpoison_vmalloc(start, size, flags);
 	return (void *)start;
 }
 
@@ -614,6 +614,11 @@ static __always_inline void kasan_poison_vmalloc(const void *start,
 		__kasan_poison_vmalloc(start, size);
 }
 
+void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
+				  kasan_vmalloc_flags_t flags, u8 tag);
+void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
+			       kasan_vmalloc_flags_t flags);
+
 #else /* CONFIG_KASAN_VMALLOC */
 
 static inline void kasan_populate_early_vm_area_shadow(void *start,
@@ -638,6 +643,11 @@ static inline void *kasan_unpoison_vmalloc(const void *start,
 static inline void kasan_poison_vmalloc(const void *start, unsigned long size)
 { }
 
+static __always_inline void
+kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
+			  kasan_vmalloc_flags_t flags)
+{ }
+
 #endif /* CONFIG_KASAN_VMALLOC */
 
 #if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index d4c14359feaf..7884ea7d13f9 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -28,6 +28,7 @@
 #include <linux/string.h>
 #include <linux/types.h>
 #include <linux/bug.h>
+#include <linux/vmalloc.h>
 
 #include "kasan.h"
 #include "../slab.h"
@@ -582,3 +583,19 @@ bool __kasan_check_byte(const void *address, unsigned long ip)
 	}
 	return true;
 }
+
+#ifdef CONFIG_KASAN_VMALLOC
+void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
+			       kasan_vmalloc_flags_t flags)
+{
+	unsigned long size;
+	void *addr;
+	int area;
+
+	for (area = 0 ; area < nr_vms ; area++) {
+		size = vms[area]->size;
+		addr = vms[area]->addr;
+		vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
+	}
+}
+#endif
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 1c373cc4b3fa..4b7936a2bd6f 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -316,8 +316,8 @@ static void init_vmalloc_pages(const void *start, unsigned long size)
 	}
 }
 
-void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
-				kasan_vmalloc_flags_t flags)
+static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
+				      kasan_vmalloc_flags_t flags)
 {
 	u8 tag;
 	unsigned long redzone_start, redzone_size;
@@ -387,6 +387,12 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 	return (void *)start;
 }
 
+void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
+				      kasan_vmalloc_flags_t flags)
+{
+	return __kasan_unpoison_vmalloc(start, size, flags);
+}
+
 void __kasan_poison_vmalloc(const void *start, unsigned long size)
 {
 	/*
@@ -396,6 +402,11 @@ void __kasan_poison_vmalloc(const void *start, unsigned long size)
 	 */
 }
 
+void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
+				  kasan_vmalloc_flags_t flags, u8 tag)
+{
+	return __kasan_unpoison_vmalloc(addr, size, flags);
+}
 #endif
 
 void kasan_enable_hw_tags(void)
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 5d2a876035d6..0a8d8bf6e9cf 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -624,8 +624,8 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
 	}
 }
 
-void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
-			       kasan_vmalloc_flags_t flags)
+static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
+				      kasan_vmalloc_flags_t flags)
 {
 	/*
 	 * Software KASAN modes unpoison both VM_ALLOC and non-VM_ALLOC
@@ -653,6 +653,18 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 	return (void *)start;
 }
 
+void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
+				      kasan_vmalloc_flags_t flags)
+{
+	return __kasan_unpoison_vmalloc(start, size, flags);
+}
+
+void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
+				  kasan_vmalloc_flags_t flags, u8 tag)
+{
+	return __kasan_unpoison_vmalloc(addr, size, flags);
+}
+
 /*
  * Poison the shadow for a vmalloc region. Called as part of the
  * freeing process at the time the region is freed.
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 798b2ed21e46..32ecdb8cd4b8 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -4870,9 +4870,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
 	 * With hardware tag-based KASAN, marking is skipped for
 	 * non-VM_ALLOC mappings, see __kasan_unpoison_vmalloc().
 	 */
-	for (area = 0; area < nr_vms; area++)
-		vms[area]->addr = kasan_unpoison_vmalloc(vms[area]->addr,
-				vms[area]->size, KASAN_VMALLOC_PROT_NORMAL);
+	kasan_unpoison_vmap_areas(vms, nr_vms, KASAN_VMALLOC_PROT_NORMAL);
 
 	kfree(vas);
 	return vms;
-- 
2.52.0

[PATCH v2 2/2] kasan: Unpoison vms[area] addresses with a common tag

[Maciej Wieczor-Retman]

From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>

A KASAN tag mismatch, possibly causing a kernel panic, can be observed
on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
It was reported on arm64 and reproduced on x86. It can be explained in
the following points:

	1. There can be more than one virtual memory chunk.
	2. Chunk's base address has a tag.
	3. The base address points at the first chunk and thus inherits
	   the tag of the first chunk.
	4. The subsequent chunks will be accessed with the tag from the
	   first chunk.
	5. Thus, the subsequent chunks need to have their tag set to
	   match that of the first chunk.

Use the modified __kasan_unpoison_vmalloc() to pass the tag of the first
vm_struct's address when vm_structs are unpoisoned in
pcpu_get_vm_areas(). Assigning a common tag resolves the pcpu chunk
address mismatch.

Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS")
Cc: <stable@vger.kernel.org> # 6.1+
Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
---
Changelog v2:
- Revise the whole patch to match the fixed refactorization from the
  first patch.

Changelog v1:
- Rewrite the patch message to point at the user impact of the issue.
- Move helper to common.c so it can be compiled in all KASAN modes.

 mm/kasan/common.c  |  3 ++-
 mm/kasan/hw_tags.c | 12 ++++++++----
 mm/kasan/shadow.c  | 15 +++++++++++----
 3 files changed, 21 insertions(+), 9 deletions(-)

diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index 7884ea7d13f9..e5a867a5670b 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -591,11 +591,12 @@ void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
 	unsigned long size;
 	void *addr;
 	int area;
+	u8 tag = get_tag(vms[0]->addr);
 
 	for (area = 0 ; area < nr_vms ; area++) {
 		size = vms[area]->size;
 		addr = vms[area]->addr;
-		vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
+		vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags, tag);
 	}
 }
 #endif
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 4b7936a2bd6f..2a02b898b9d8 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -317,7 +317,7 @@ static void init_vmalloc_pages(const void *start, unsigned long size)
 }
 
 static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
-				      kasan_vmalloc_flags_t flags)
+				      kasan_vmalloc_flags_t flags, int unpoison_tag)
 {
 	u8 tag;
 	unsigned long redzone_start, redzone_size;
@@ -361,7 +361,11 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 		return (void *)start;
 	}
 
-	tag = kasan_random_tag();
+	if (unpoison_tag < 0)
+		tag = kasan_random_tag();
+	else
+		tag = unpoison_tag;
+
 	start = set_tag(start, tag);
 
 	/* Unpoison and initialize memory up to size. */
@@ -390,7 +394,7 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
 				      kasan_vmalloc_flags_t flags)
 {
-	return __kasan_unpoison_vmalloc(start, size, flags);
+	return __kasan_unpoison_vmalloc(start, size, flags, -1);
 }
 
 void __kasan_poison_vmalloc(const void *start, unsigned long size)
@@ -405,7 +409,7 @@ void __kasan_poison_vmalloc(const void *start, unsigned long size)
 void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
 				  kasan_vmalloc_flags_t flags, u8 tag)
 {
-	return __kasan_unpoison_vmalloc(addr, size, flags);
+	return __kasan_unpoison_vmalloc(addr, size, flags, tag);
 }
 #endif
 
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index 0a8d8bf6e9cf..7a66ffc1d5b3 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -625,8 +625,10 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
 }
 
 static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
-				      kasan_vmalloc_flags_t flags)
+				      kasan_vmalloc_flags_t flags, int unpoison_tag)
 {
+	u8 tag;
+
 	/*
 	 * Software KASAN modes unpoison both VM_ALLOC and non-VM_ALLOC
 	 * mappings, so the KASAN_VMALLOC_VM_ALLOC flag is ignored.
@@ -648,7 +650,12 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 	    !(flags & KASAN_VMALLOC_PROT_NORMAL))
 		return (void *)start;
 
-	start = set_tag(start, kasan_random_tag());
+	if (unpoison_tag < 0)
+		tag = kasan_random_tag();
+	else
+		tag = unpoison_tag;
+
+	start = set_tag(start, tag);
 	kasan_unpoison(start, size, false);
 	return (void *)start;
 }
@@ -656,13 +663,13 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
 void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
 				      kasan_vmalloc_flags_t flags)
 {
-	return __kasan_unpoison_vmalloc(start, size, flags);
+	return __kasan_unpoison_vmalloc(start, size, flags, -1);
 }
 
 void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
 				  kasan_vmalloc_flags_t flags, u8 tag)
 {
-	return __kasan_unpoison_vmalloc(addr, size, flags);
+	return __kasan_unpoison_vmalloc(addr, size, flags, tag);
 }
 
 /*
-- 
2.52.0

Re: [PATCH v2 0/2] kasan: vmalloc: Fix incorrect tag assignment with multiple vm_structs

[Andrew Morton]

On Tue, 02 Dec 2025 14:27:56 +0000 Maciej Wieczor-Retman <m.wieczorretman@pm.me> wrote:

> A KASAN tag mismatch, possibly resulting in a kernel panic, can be
> observed on systems with a tag-based KASAN enabled and with multiple
> NUMA nodes. Initially it was only noticed on x86 [1] but later a similar
> issue was also reported on arm64 [2].
> 
> Specifically the problem is related to how vm_structs interact with
> pcpu_chunks - both when they are allocated, assigned and when pcpu_chunk
> addresses are derived.
> 
> When vm_structs are allocated they are tagged if vmalloc support is
> enabled along the KASAN mode. Later when first pcpu chunk is allocated
> it gets its 'base_addr' field set to the first allocated vm_struct.
> With that it inherits that vm_struct's tag.
> 
> When pcpu_chunk addresses are later derived (by pcpu_chunk_addr(), for
> example in pcpu_alloc_noprof()) the base_addr field is used and offsets
> are added to it. If the initial conditions are satisfied then some of
> the offsets will point into memory allocated with a different vm_struct.
> So while the lower bits will get accurately derived the tag bits in the
> top of the pointer won't match the shadow memory contents.
> 
> The solution (proposed at v2 of the x86 KASAN series [3]) is to tag the
> vm_structs the same when allocating them for the per cpu allocator (in
> pcpu_get_vm_areas()).
> 
> Originally these patches were part of the x86 KASAN series [4].
> 
> The series is based on 6.18.

This series overlaps a lot with

https://lkml.kernel.org/r/20251128111516.244497-1-jiayuan.chen@linux.dev

Please discuss!

> [1] https://lore.kernel.org/all/e7e04692866d02e6d3b32bb43b998e5d17092ba4.1738686764.git.maciej.wieczor-retman@intel.com/
> [2] https://lore.kernel.org/all/aMUrW1Znp1GEj7St@MiWiFi-R3L-srv/
> [3] https://lore.kernel.org/all/CAPAsAGxDRv_uFeMYu9TwhBVWHCCtkSxoWY4xmFB_vowMbi8raw@mail.gmail.com/
> [4] https://lore.kernel.org/all/cover.1761763681.git.m.wieczorretman@pm.me/
> 

Re: [PATCH v2 0/2] kasan: vmalloc: Fix incorrect tag assignment with multiple vm_structs

[Maciej Wieczór-Retman]

On 2025-12-02 at 08:35:22 -0800, Andrew Morton wrote:
>On Tue, 02 Dec 2025 14:27:56 +0000 Maciej Wieczor-Retman <m.wieczorretman@pm.me> wrote:
...
>
>This series overlaps a lot with
>
>https://lkml.kernel.org/r/20251128111516.244497-1-jiayuan.chen@linux.dev
>
>Please discuss!
>

Thanks for pointing it out, I'll test if Jiayuan's version of
__kasan_unpoison_vmalloc fixes the issue I'm seeing.

kind regards
Maciej Wieczór-Retman

Re: [PATCH v2 2/2] kasan: Unpoison vms[area] addresses with a common tag

[Andrey Konovalov]

On Tue, Dec 2, 2025 at 3:29 PM Maciej Wieczor-Retman
<m.wieczorretman@pm.me> wrote:
>
> From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
>
> A KASAN tag mismatch, possibly causing a kernel panic, can be observed
> on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
> It was reported on arm64 and reproduced on x86. It can be explained in
> the following points:
>
>         1. There can be more than one virtual memory chunk.
>         2. Chunk's base address has a tag.
>         3. The base address points at the first chunk and thus inherits
>            the tag of the first chunk.
>         4. The subsequent chunks will be accessed with the tag from the
>            first chunk.
>         5. Thus, the subsequent chunks need to have their tag set to
>            match that of the first chunk.
>
> Use the modified __kasan_unpoison_vmalloc() to pass the tag of the first
> vm_struct's address when vm_structs are unpoisoned in
> pcpu_get_vm_areas(). Assigning a common tag resolves the pcpu chunk
> address mismatch.
>
> Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS")
> Cc: <stable@vger.kernel.org> # 6.1+
> Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
> ---
> Changelog v2:
> - Revise the whole patch to match the fixed refactorization from the
>   first patch.
>
> Changelog v1:
> - Rewrite the patch message to point at the user impact of the issue.
> - Move helper to common.c so it can be compiled in all KASAN modes.
>
>  mm/kasan/common.c  |  3 ++-
>  mm/kasan/hw_tags.c | 12 ++++++++----
>  mm/kasan/shadow.c  | 15 +++++++++++----
>  3 files changed, 21 insertions(+), 9 deletions(-)
>
> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> index 7884ea7d13f9..e5a867a5670b 100644
> --- a/mm/kasan/common.c
> +++ b/mm/kasan/common.c
> @@ -591,11 +591,12 @@ void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
>         unsigned long size;
>         void *addr;
>         int area;
> +       u8 tag = get_tag(vms[0]->addr);
>
>         for (area = 0 ; area < nr_vms ; area++) {
>                 size = vms[area]->size;
>                 addr = vms[area]->addr;
> -               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
> +               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags, tag);

I'm thinking what you can do here is:

vms[area]->addr = set_tag(addr, tag);
__kasan_unpoison_vmalloc(addr, size, flags | KASAN_VMALLOC_KEEP_TAG);

This is with the assumption that Jiayuan's patch is changed to add
KASAN_VMALLOC_KEEP_TAG to kasan_vmalloc_flags_t.

Then you should not need that extra __kasan_random_unpoison_vmalloc helper.


>         }
>  }
>  #endif
> diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
> index 4b7936a2bd6f..2a02b898b9d8 100644
> --- a/mm/kasan/hw_tags.c
> +++ b/mm/kasan/hw_tags.c
> @@ -317,7 +317,7 @@ static void init_vmalloc_pages(const void *start, unsigned long size)
>  }
>
>  static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> -                                     kasan_vmalloc_flags_t flags)
> +                                     kasan_vmalloc_flags_t flags, int unpoison_tag)
>  {
>         u8 tag;
>         unsigned long redzone_start, redzone_size;
> @@ -361,7 +361,11 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>                 return (void *)start;
>         }
>
> -       tag = kasan_random_tag();
> +       if (unpoison_tag < 0)
> +               tag = kasan_random_tag();
> +       else
> +               tag = unpoison_tag;
> +
>         start = set_tag(start, tag);
>
>         /* Unpoison and initialize memory up to size. */
> @@ -390,7 +394,7 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>  void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
>                                       kasan_vmalloc_flags_t flags)
>  {
> -       return __kasan_unpoison_vmalloc(start, size, flags);
> +       return __kasan_unpoison_vmalloc(start, size, flags, -1);
>  }
>
>  void __kasan_poison_vmalloc(const void *start, unsigned long size)
> @@ -405,7 +409,7 @@ void __kasan_poison_vmalloc(const void *start, unsigned long size)
>  void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
>                                   kasan_vmalloc_flags_t flags, u8 tag)
>  {
> -       return __kasan_unpoison_vmalloc(addr, size, flags);
> +       return __kasan_unpoison_vmalloc(addr, size, flags, tag);
>  }
>  #endif
>
> diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
> index 0a8d8bf6e9cf..7a66ffc1d5b3 100644
> --- a/mm/kasan/shadow.c
> +++ b/mm/kasan/shadow.c
> @@ -625,8 +625,10 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
>  }
>
>  static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> -                                     kasan_vmalloc_flags_t flags)
> +                                     kasan_vmalloc_flags_t flags, int unpoison_tag)
>  {
> +       u8 tag;
> +
>         /*
>          * Software KASAN modes unpoison both VM_ALLOC and non-VM_ALLOC
>          * mappings, so the KASAN_VMALLOC_VM_ALLOC flag is ignored.
> @@ -648,7 +650,12 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>             !(flags & KASAN_VMALLOC_PROT_NORMAL))
>                 return (void *)start;
>
> -       start = set_tag(start, kasan_random_tag());
> +       if (unpoison_tag < 0)
> +               tag = kasan_random_tag();
> +       else
> +               tag = unpoison_tag;
> +
> +       start = set_tag(start, tag);
>         kasan_unpoison(start, size, false);
>         return (void *)start;
>  }
> @@ -656,13 +663,13 @@ static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>  void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
>                                       kasan_vmalloc_flags_t flags)
>  {
> -       return __kasan_unpoison_vmalloc(start, size, flags);
> +       return __kasan_unpoison_vmalloc(start, size, flags, -1);
>  }
>
>  void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
>                                   kasan_vmalloc_flags_t flags, u8 tag)
>  {
> -       return __kasan_unpoison_vmalloc(addr, size, flags);
> +       return __kasan_unpoison_vmalloc(addr, size, flags, tag);
>  }
>
>  /*
> --
> 2.52.0
>
>

Re: [PATCH v2 1/2] kasan: Refactor pcpu kasan vmalloc unpoison

[Andrey Konovalov]

On Tue, Dec 2, 2025 at 3:29 PM Maciej Wieczor-Retman
<m.wieczorretman@pm.me> wrote:
>
> From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
>
> A KASAN tag mismatch, possibly causing a kernel panic, can be observed
> on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
> It was reported on arm64 and reproduced on x86. It can be explained in
> the following points:
>
>         1. There can be more than one virtual memory chunk.
>         2. Chunk's base address has a tag.
>         3. The base address points at the first chunk and thus inherits
>            the tag of the first chunk.
>         4. The subsequent chunks will be accessed with the tag from the
>            first chunk.
>         5. Thus, the subsequent chunks need to have their tag set to
>            match that of the first chunk.
>
> Refactor code by reusing __kasan_unpoison_vmalloc in a new helper in
> preparation for the actual fix.
>
> Changelog v1 (after splitting of from the KASAN series):
> - Rewrite first paragraph of the patch message to point at the user
>   impact of the issue.
> - Move helper to common.c so it can be compiled in all KASAN modes.
>
> Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS")
> Cc: <stable@vger.kernel.org> # 6.1+
> Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
> ---
> Changelog v2:
> - Redo the whole patch so it's an actual refactor.
>
>  include/linux/kasan.h | 16 +++++++++++++---
>  mm/kasan/common.c     | 17 +++++++++++++++++
>  mm/kasan/hw_tags.c    | 15 +++++++++++++--
>  mm/kasan/shadow.c     | 16 ++++++++++++++--
>  mm/vmalloc.c          |  4 +---
>  5 files changed, 58 insertions(+), 10 deletions(-)
>
> diff --git a/include/linux/kasan.h b/include/linux/kasan.h
> index d12e1a5f5a9a..4a3d3dba9764 100644
> --- a/include/linux/kasan.h
> +++ b/include/linux/kasan.h
> @@ -595,14 +595,14 @@ static inline void kasan_release_vmalloc(unsigned long start,
>
>  #endif /* CONFIG_KASAN_GENERIC || CONFIG_KASAN_SW_TAGS */
>
> -void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> -                              kasan_vmalloc_flags_t flags);
> +void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
> +                                     kasan_vmalloc_flags_t flags);
>  static __always_inline void *kasan_unpoison_vmalloc(const void *start,
>                                                 unsigned long size,
>                                                 kasan_vmalloc_flags_t flags)
>  {
>         if (kasan_enabled())
> -               return __kasan_unpoison_vmalloc(start, size, flags);
> +               return __kasan_random_unpoison_vmalloc(start, size, flags);
>         return (void *)start;
>  }
>
> @@ -614,6 +614,11 @@ static __always_inline void kasan_poison_vmalloc(const void *start,
>                 __kasan_poison_vmalloc(start, size);
>  }
>
> +void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
> +                                 kasan_vmalloc_flags_t flags, u8 tag);
> +void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
> +                              kasan_vmalloc_flags_t flags);
> +
>  #else /* CONFIG_KASAN_VMALLOC */
>
>  static inline void kasan_populate_early_vm_area_shadow(void *start,
> @@ -638,6 +643,11 @@ static inline void *kasan_unpoison_vmalloc(const void *start,
>  static inline void kasan_poison_vmalloc(const void *start, unsigned long size)
>  { }
>
> +static __always_inline void
> +kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
> +                         kasan_vmalloc_flags_t flags)
> +{ }
> +
>  #endif /* CONFIG_KASAN_VMALLOC */
>
>  #if (defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)) && \
> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
> index d4c14359feaf..7884ea7d13f9 100644
> --- a/mm/kasan/common.c
> +++ b/mm/kasan/common.c
> @@ -28,6 +28,7 @@
>  #include <linux/string.h>
>  #include <linux/types.h>
>  #include <linux/bug.h>
> +#include <linux/vmalloc.h>
>
>  #include "kasan.h"
>  #include "../slab.h"
> @@ -582,3 +583,19 @@ bool __kasan_check_byte(const void *address, unsigned long ip)
>         }
>         return true;
>  }
> +
> +#ifdef CONFIG_KASAN_VMALLOC
> +void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
> +                              kasan_vmalloc_flags_t flags)

kasan_unpoison_vmap_areas() needs to be defined in
inclunde/linux/kasan.h and call __kasan_unpoison_vmap_areas() when
kasan_enabled() == true, similar to the other wrappers.

And check my comment for patch #2: with that, you should not need to
add so many new __helpers: just __kasan_unpoison_vmalloc and
__kasan_unpoison_vmap_areas should suffice.


> +{
> +       unsigned long size;
> +       void *addr;
> +       int area;
> +
> +       for (area = 0 ; area < nr_vms ; area++) {
> +               size = vms[area]->size;
> +               addr = vms[area]->addr;
> +               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
> +       }
> +}
> +#endif
> diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
> index 1c373cc4b3fa..4b7936a2bd6f 100644
> --- a/mm/kasan/hw_tags.c
> +++ b/mm/kasan/hw_tags.c
> @@ -316,8 +316,8 @@ static void init_vmalloc_pages(const void *start, unsigned long size)
>         }
>  }
>
> -void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> -                               kasan_vmalloc_flags_t flags)
> +static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> +                                     kasan_vmalloc_flags_t flags)
>  {
>         u8 tag;
>         unsigned long redzone_start, redzone_size;
> @@ -387,6 +387,12 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>         return (void *)start;
>  }
>
> +void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
> +                                     kasan_vmalloc_flags_t flags)
> +{
> +       return __kasan_unpoison_vmalloc(start, size, flags);
> +}
> +
>  void __kasan_poison_vmalloc(const void *start, unsigned long size)
>  {
>         /*
> @@ -396,6 +402,11 @@ void __kasan_poison_vmalloc(const void *start, unsigned long size)
>          */
>  }
>
> +void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
> +                                 kasan_vmalloc_flags_t flags, u8 tag)
> +{
> +       return __kasan_unpoison_vmalloc(addr, size, flags);
> +}
>  #endif
>
>  void kasan_enable_hw_tags(void)
> diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
> index 5d2a876035d6..0a8d8bf6e9cf 100644
> --- a/mm/kasan/shadow.c
> +++ b/mm/kasan/shadow.c
> @@ -624,8 +624,8 @@ void kasan_release_vmalloc(unsigned long start, unsigned long end,
>         }
>  }
>
> -void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> -                              kasan_vmalloc_flags_t flags)
> +static void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
> +                                     kasan_vmalloc_flags_t flags)
>  {
>         /*
>          * Software KASAN modes unpoison both VM_ALLOC and non-VM_ALLOC
> @@ -653,6 +653,18 @@ void *__kasan_unpoison_vmalloc(const void *start, unsigned long size,
>         return (void *)start;
>  }
>
> +void *__kasan_random_unpoison_vmalloc(const void *start, unsigned long size,
> +                                     kasan_vmalloc_flags_t flags)
> +{
> +       return __kasan_unpoison_vmalloc(start, size, flags);
> +}
> +
> +void *__kasan_unpoison_vmap_areas(void *addr, unsigned long size,
> +                                 kasan_vmalloc_flags_t flags, u8 tag)
> +{
> +       return __kasan_unpoison_vmalloc(addr, size, flags);
> +}
> +
>  /*
>   * Poison the shadow for a vmalloc region. Called as part of the
>   * freeing process at the time the region is freed.
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index 798b2ed21e46..32ecdb8cd4b8 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -4870,9 +4870,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
>          * With hardware tag-based KASAN, marking is skipped for
>          * non-VM_ALLOC mappings, see __kasan_unpoison_vmalloc().
>          */
> -       for (area = 0; area < nr_vms; area++)
> -               vms[area]->addr = kasan_unpoison_vmalloc(vms[area]->addr,
> -                               vms[area]->size, KASAN_VMALLOC_PROT_NORMAL);
> +       kasan_unpoison_vmap_areas(vms, nr_vms, KASAN_VMALLOC_PROT_NORMAL);
>
>         kfree(vas);
>         return vms;
> --
> 2.52.0
>
>

Re: [PATCH v2 2/2] kasan: Unpoison vms[area] addresses with a common tag

[Maciej Wieczór-Retman]

On 2025-12-03 at 16:53:01 +0100, Andrey Konovalov wrote:
>On Tue, Dec 2, 2025 at 3:29 PM Maciej Wieczor-Retman
><m.wieczorretman@pm.me> wrote:
>>
>> From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
>>
>> A KASAN tag mismatch, possibly causing a kernel panic, can be observed
>> on systems with a tag-based KASAN enabled and with multiple NUMA nodes.
>> It was reported on arm64 and reproduced on x86. It can be explained in
>> the following points:
>>
>>         1. There can be more than one virtual memory chunk.
>>         2. Chunk's base address has a tag.
>>         3. The base address points at the first chunk and thus inherits
>>            the tag of the first chunk.
>>         4. The subsequent chunks will be accessed with the tag from the
>>            first chunk.
>>         5. Thus, the subsequent chunks need to have their tag set to
>>            match that of the first chunk.
>>
>> Use the modified __kasan_unpoison_vmalloc() to pass the tag of the first
>> vm_struct's address when vm_structs are unpoisoned in
>> pcpu_get_vm_areas(). Assigning a common tag resolves the pcpu chunk
>> address mismatch.
>>
>> Fixes: 1d96320f8d53 ("kasan, vmalloc: add vmalloc tagging for SW_TAGS")
>> Cc: <stable@vger.kernel.org> # 6.1+
>> Signed-off-by: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
>> ---
>> Changelog v2:
>> - Revise the whole patch to match the fixed refactorization from the
>>   first patch.
>>
>> Changelog v1:
>> - Rewrite the patch message to point at the user impact of the issue.
>> - Move helper to common.c so it can be compiled in all KASAN modes.
>>
>>  mm/kasan/common.c  |  3 ++-
>>  mm/kasan/hw_tags.c | 12 ++++++++----
>>  mm/kasan/shadow.c  | 15 +++++++++++----
>>  3 files changed, 21 insertions(+), 9 deletions(-)
>>
>> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
>> index 7884ea7d13f9..e5a867a5670b 100644
>> --- a/mm/kasan/common.c
>> +++ b/mm/kasan/common.c
>> @@ -591,11 +591,12 @@ void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
>>         unsigned long size;
>>         void *addr;
>>         int area;
>> +       u8 tag = get_tag(vms[0]->addr);
>>
>>         for (area = 0 ; area < nr_vms ; area++) {
>>                 size = vms[area]->size;
>>                 addr = vms[area]->addr;
>> -               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
>> +               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags, tag);
>
>I'm thinking what you can do here is:
>
>vms[area]->addr = set_tag(addr, tag);
>__kasan_unpoison_vmalloc(addr, size, flags | KASAN_VMALLOC_KEEP_TAG);


I noticed that something like this wouldn't work once I started trying
to rebase my work onto Jiayuan's. The line:
+       u8 tag = get_tag(vms[0]->addr);
is wrong and should be
+       u8 tag = kasan_random_tag();
I was sure the vms[0]->addr was already tagged (I recall checking this
so I'm not sure if something changed or my previous check was wrong) but
the problem here is that vms[0]->addr, vms[1]->addr ... were unpoisoned
with random addresses, specifically different random addresses. So then
later in the pcpu chunk code vms[1] related pointers would get the tag
from vms[0]->addr.

So I think we still need a separate way to do __kasan_unpoison_vmalloc
with a specific tag.

>
>This is with the assumption that Jiayuan's patch is changed to add
>KASAN_VMALLOC_KEEP_TAG to kasan_vmalloc_flags_t.
>
>Then you should not need that extra __kasan_random_unpoison_vmalloc helper.

I already rewrote the patch rebased onto Jiayuan's patch. I was able to
ditch the __kasan_random_unpoison_vmalloc but I needed to add
__kasan_unpoison_vrealloc - so I can pass the tag of the start pointer
to __kasan_unpoison_vmalloc. I was hoping to post it today/tomorrow so
Jiayuan can check my changes don't break his solution. I'm just waiting
to check it compiles against all the fun kernel configs.

-- 
kind regards
Maciej Wieczór-Retman

Re: [PATCH v2 1/2] kasan: Refactor pcpu kasan vmalloc unpoison

[Maciej Wieczór-Retman]

On 2025-12-03 at 16:53:04 +0100, Andrey Konovalov wrote:
>On Tue, Dec 2, 2025 at 3:29 PM Maciej Wieczor-Retman
><m.wieczorretman@pm.me> wrote:
>>
>> From: Maciej Wieczor-Retman <maciej.wieczor-retman@intel.com>
>>
...
>> diff --git a/mm/kasan/common.c b/mm/kasan/common.c
>> index d4c14359feaf..7884ea7d13f9 100644
>> --- a/mm/kasan/common.c
>> +++ b/mm/kasan/common.c
>> @@ -28,6 +28,7 @@
>>  #include <linux/string.h>
>>  #include <linux/types.h>
>>  #include <linux/bug.h>
>> +#include <linux/vmalloc.h>
>>
>>  #include "kasan.h"
>>  #include "../slab.h"
>> @@ -582,3 +583,19 @@ bool __kasan_check_byte(const void *address, unsigned long ip)
>>         }
>>         return true;
>>  }
>> +
>> +#ifdef CONFIG_KASAN_VMALLOC
>> +void kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
>> +                              kasan_vmalloc_flags_t flags)
>
>kasan_unpoison_vmap_areas() needs to be defined in
>inclunde/linux/kasan.h and call __kasan_unpoison_vmap_areas() when
>kasan_enabled() == true, similar to the other wrappers.
>
>And check my comment for patch #2: with that, you should not need to
>add so many new __helpers: just __kasan_unpoison_vmalloc and
>__kasan_unpoison_vmap_areas should suffice.

Okay, I think I see what you mean. I was trying to avoid using
__kasan_unpoison_vmalloc() here so that it compiled properly, but that
was before I added the ifdef guard. Now there is not reason not to use
it here.

I'll make the changes you mentioned.

Kind regards
Maciej Wieczór-Retman

>
>> +{
>> +       unsigned long size;
>> +       void *addr;
>> +       int area;
>> +
>> +       for (area = 0 ; area < nr_vms ; area++) {
>> +               size = vms[area]->size;
>> +               addr = vms[area]->addr;
>> +               vms[area]->addr = __kasan_unpoison_vmap_areas(addr, size, flags);
>> +       }
>> +}
>> +#endif

Re: [PATCH v2 2/2] kasan: Unpoison vms[area] addresses with a common tag

[Andrey Konovalov]

On Wed, Dec 3, 2025 at 5:24 PM Maciej Wieczór-Retman
<m.wieczorretman@pm.me> wrote:
>
> >I'm thinking what you can do here is:
> >
> >vms[area]->addr = set_tag(addr, tag);
> >__kasan_unpoison_vmalloc(addr, size, flags | KASAN_VMALLOC_KEEP_TAG);
>
>
> I noticed that something like this wouldn't work once I started trying
> to rebase my work onto Jiayuan's. The line:
> +       u8 tag = get_tag(vms[0]->addr);
> is wrong and should be
> +       u8 tag = kasan_random_tag();

Ah, right.

> I was sure the vms[0]->addr was already tagged (I recall checking this
> so I'm not sure if something changed or my previous check was wrong) but
> the problem here is that vms[0]->addr, vms[1]->addr ... were unpoisoned
> with random addresses, specifically different random addresses. So then
> later in the pcpu chunk code vms[1] related pointers would get the tag
> from vms[0]->addr.
>
> So I think we still need a separate way to do __kasan_unpoison_vmalloc
> with a specific tag.

Why?

Assuming KASAN_VMALLOC_KEEP_TAG takes the tag from the pointer, just do:

tag = kasan_random_tag();
for (area = 0; ...) {
    vms[area]->addr = set_tag(vms[area]->addr, tag);
    __kasan_unpoison_vmalloc(vms[area]->addr, vms[area]->size, flags |
KASAN_VMALLOC_KEEP_TAG);
}

Or maybe even better:

vms[0]->addr = __kasan_unpoison_vmalloc(vms[0]->addr, vms[0]->size, flags);
tag = get_tag(vms[0]->addr);
for (area = 1; ...) {
    vms[area]->addr = set_tag(vms[area]->addr, tag);
    __kasan_unpoison_vmalloc(vms[area]->addr, vms[area]->size, flags |
KASAN_VMALLOC_KEEP_TAG);
}

This way we won't assign a random tag unless it's actually needed
(i.e. when KASAN_VMALLOC_PROT_NORMAL is not provided; assuming we care
to support that case).

Re: [PATCH v2 2/2] kasan: Unpoison vms[area] addresses with a common tag

[Maciej Wieczór-Retman]

On 2025-12-04 at 01:43:36 +0100, Andrey Konovalov wrote:
>On Wed, Dec 3, 2025 at 5:24 PM Maciej Wieczór-Retman
><m.wieczorretman@pm.me> wrote:
>> I was sure the vms[0]->addr was already tagged (I recall checking this
>> so I'm not sure if something changed or my previous check was wrong) but
>> the problem here is that vms[0]->addr, vms[1]->addr ... were unpoisoned
>> with random addresses, specifically different random addresses. So then
>> later in the pcpu chunk code vms[1] related pointers would get the tag
>> from vms[0]->addr.
>>
>> So I think we still need a separate way to do __kasan_unpoison_vmalloc
>> with a specific tag.
>
>Why?
>
>Assuming KASAN_VMALLOC_KEEP_TAG takes the tag from the pointer, just do:
>
>tag = kasan_random_tag();
>for (area = 0; ...) {
>    vms[area]->addr = set_tag(vms[area]->addr, tag);
>    __kasan_unpoison_vmalloc(vms[area]->addr, vms[area]->size, flags |
>KASAN_VMALLOC_KEEP_TAG);
>}
>
>Or maybe even better:
>
>vms[0]->addr = __kasan_unpoison_vmalloc(vms[0]->addr, vms[0]->size, flags);
>tag = get_tag(vms[0]->addr);
>for (area = 1; ...) {
>    vms[area]->addr = set_tag(vms[area]->addr, tag);
>    __kasan_unpoison_vmalloc(vms[area]->addr, vms[area]->size, flags |
>KASAN_VMALLOC_KEEP_TAG);
>}
>
>This way we won't assign a random tag unless it's actually needed
>(i.e. when KASAN_VMALLOC_PROT_NORMAL is not provided; assuming we care
>to support that case).

Oh, right yes, that would work nicely. I thought putting these behind
helpers would end up clean but this is very neat too.

I suppose I'll wait for Jiayuan to update his patch and then I'll make
these changes on top of that.

Thanks! :)

-- 
Kind regards
Maciej Wieczór-Retman