* [PATCH 1/3] mm,slub,x86: decouple size of struct page from CONFIG_CMPXCHG_LOCAL
2011-11-10 14:04 [PATCH 0/3] CMPXCHG config options changes Heiko Carstens
@ 2011-11-10 14:04 ` Heiko Carstens
2011-11-10 15:00 ` Christoph Lameter
2011-11-10 14:04 ` [PATCH 2/3] mm,x86,um: move CMPXCHG_LOCAL config option Heiko Carstens
2011-11-10 14:04 ` [PATCH 3/3] mm,x86,um: move CMPXCHG_DOUBLE " Heiko Carstens
2 siblings, 1 reply; 7+ messages in thread
From: Heiko Carstens @ 2011-11-10 14:04 UTC (permalink / raw)
To: Pekka Enberg, Andrew Morton
Cc: Jeff Dike, Ingo Molnar, Christoph Lameter, linux-mm,
linux-kernel, Heiko Carstens
If an architecture supports CMPXCHG_LOCAL this shouldn't result automatically
in larger struct pages if the SLUB allocator is used. Instead introduce a new
config option "HAVE_ALIGNED_STRUCT_PAGE" which can be selected if a double
word aligned struct page is required.
Also update x86 Kconfig so that it should work as before.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
---
arch/Kconfig | 8 ++++++++
arch/x86/Kconfig | 1 +
include/linux/mm_types.h | 9 ++++-----
mm/slub.c | 6 +++---
4 files changed, 16 insertions(+), 8 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index 4b0669c..4b4a140 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -181,4 +181,12 @@ config HAVE_RCU_TABLE_FREE
config ARCH_HAVE_NMI_SAFE_CMPXCHG
bool
+config HAVE_ALIGNED_STRUCT_PAGE
+ bool
+ help
+ This makes sure that struct pages are double word aligned and that
+ e.g. the SLUB allocator can perform double word atomic operations
+ on a struct page for better performance. However selecting this
+ might increase the size of a struct page by a word.
+
source "kernel/gcov/Kconfig"
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index cb9a104..5115ce4 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -75,6 +75,7 @@ config X86
select HAVE_BPF_JIT if (X86_64 && NET)
select CLKEVT_I8253
select ARCH_HAVE_NMI_SAFE_CMPXCHG
+ select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5b42f1b..3cc3062 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -151,12 +151,11 @@ struct page {
#endif
}
/*
- * If another subsystem starts using the double word pairing for atomic
- * operations on struct page then it must change the #if to ensure
- * proper alignment of the page struct.
+ * The struct page can be forced to be double word aligned so that atomic ops
+ * on double words work. The SLUB allocator can make use of such a feature.
*/
-#if defined(CONFIG_SLUB) && defined(CONFIG_CMPXCHG_LOCAL)
- __attribute__((__aligned__(2*sizeof(unsigned long))))
+#ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE
+ __aligned(2 * sizeof(unsigned long))
#endif
;
diff --git a/mm/slub.c b/mm/slub.c
index 7d2a996..7669b4c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -366,7 +366,7 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page
const char *n)
{
VM_BUG_ON(!irqs_disabled());
-#ifdef CONFIG_CMPXCHG_DOUBLE
+#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (s->flags & __CMPXCHG_DOUBLE) {
if (cmpxchg_double(&page->freelist,
freelist_old, counters_old,
@@ -400,7 +400,7 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page,
void *freelist_new, unsigned long counters_new,
const char *n)
{
-#ifdef CONFIG_CMPXCHG_DOUBLE
+#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (s->flags & __CMPXCHG_DOUBLE) {
if (cmpxchg_double(&page->freelist,
freelist_old, counters_old,
@@ -2990,7 +2990,7 @@ static int kmem_cache_open(struct kmem_cache *s,
}
}
-#ifdef CONFIG_CMPXCHG_DOUBLE
+#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (system_has_cmpxchg_double() && (s->flags & SLAB_DEBUG_FLAGS) == 0)
/* Enable fast mode */
s->flags |= __CMPXCHG_DOUBLE;
--
1.7.7.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 1/3] mm,slub,x86: decouple size of struct page from CONFIG_CMPXCHG_LOCAL
2011-11-10 14:04 ` [PATCH 1/3] mm,slub,x86: decouple size of struct page from CONFIG_CMPXCHG_LOCAL Heiko Carstens
@ 2011-11-10 15:00 ` Christoph Lameter
0 siblings, 0 replies; 7+ messages in thread
From: Christoph Lameter @ 2011-11-10 15:00 UTC (permalink / raw)
To: Heiko Carstens
Cc: Pekka Enberg, Andrew Morton, Jeff Dike, Ingo Molnar, linux-mm,
linux-kernel
On Thu, 10 Nov 2011, Heiko Carstens wrote:
> If an architecture supports CMPXCHG_LOCAL this shouldn't result automatically
> in larger struct pages if the SLUB allocator is used. Instead introduce a new
> config option "HAVE_ALIGNED_STRUCT_PAGE" which can be selected if a double
> word aligned struct page is required.
> Also update x86 Kconfig so that it should work as before.
Acked-by: Christoph Lameter <cl@linux.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/3] mm,x86,um: move CMPXCHG_LOCAL config option
2011-11-10 14:04 [PATCH 0/3] CMPXCHG config options changes Heiko Carstens
2011-11-10 14:04 ` [PATCH 1/3] mm,slub,x86: decouple size of struct page from CONFIG_CMPXCHG_LOCAL Heiko Carstens
@ 2011-11-10 14:04 ` Heiko Carstens
2011-11-10 15:01 ` Christoph Lameter
2011-11-10 14:04 ` [PATCH 3/3] mm,x86,um: move CMPXCHG_DOUBLE " Heiko Carstens
2 siblings, 1 reply; 7+ messages in thread
From: Heiko Carstens @ 2011-11-10 14:04 UTC (permalink / raw)
To: Pekka Enberg, Andrew Morton
Cc: Jeff Dike, Ingo Molnar, Christoph Lameter, linux-mm,
linux-kernel, Heiko Carstens
Move CMPXCHG_LOCAL and rename it to HAVE_CMPXCHG_LOCAL so architectures can
simply select the option if it is supported.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
---
arch/Kconfig | 3 +++
arch/x86/Kconfig | 1 +
arch/x86/Kconfig.cpu | 3 ---
arch/x86/um/Kconfig | 4 ----
mm/vmstat.c | 2 +-
5 files changed, 5 insertions(+), 8 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index 4b4a140..f5e749b 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -189,4 +189,7 @@ config HAVE_ALIGNED_STRUCT_PAGE
on a struct page for better performance. However selecting this
might increase the size of a struct page by a word.
+config HAVE_CMPXCHG_LOCAL
+ bool
+
source "kernel/gcov/Kconfig"
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 5115ce4..71aebf5 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -76,6 +76,7 @@ config X86
select CLKEVT_I8253
select ARCH_HAVE_NMI_SAFE_CMPXCHG
select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386
+ select HAVE_CMPXCHG_LOCAL if !M386
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu
index e3ca7e0..99d2ab8 100644
--- a/arch/x86/Kconfig.cpu
+++ b/arch/x86/Kconfig.cpu
@@ -309,9 +309,6 @@ config X86_INTERNODE_CACHE_SHIFT
config X86_CMPXCHG
def_bool X86_64 || (X86_32 && !M386)
-config CMPXCHG_LOCAL
- def_bool X86_64 || (X86_32 && !M386)
-
config CMPXCHG_DOUBLE
def_bool y
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index 1d97bd8..a62bfc6 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -6,10 +6,6 @@ menu "UML-specific options"
menu "Host processor type and features"
-config CMPXCHG_LOCAL
- bool
- default n
-
config CMPXCHG_DOUBLE
bool
default n
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 8fd603b..f600557 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -295,7 +295,7 @@ void __dec_zone_page_state(struct page *page, enum zone_stat_item item)
}
EXPORT_SYMBOL(__dec_zone_page_state);
-#ifdef CONFIG_CMPXCHG_LOCAL
+#ifdef CONFIG_HAVE_CMPXCHG_LOCAL
/*
* If we have cmpxchg_local support then we do not need to incur the overhead
* that comes with local_irq_save/restore if we use this_cpu_cmpxchg.
--
1.7.7.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 2/3] mm,x86,um: move CMPXCHG_LOCAL config option
2011-11-10 14:04 ` [PATCH 2/3] mm,x86,um: move CMPXCHG_LOCAL config option Heiko Carstens
@ 2011-11-10 15:01 ` Christoph Lameter
0 siblings, 0 replies; 7+ messages in thread
From: Christoph Lameter @ 2011-11-10 15:01 UTC (permalink / raw)
To: Heiko Carstens
Cc: Pekka Enberg, Andrew Morton, Jeff Dike, Ingo Molnar, linux-mm,
linux-kernel
On Thu, 10 Nov 2011, Heiko Carstens wrote:
> Move CMPXCHG_LOCAL and rename it to HAVE_CMPXCHG_LOCAL so architectures can
> simply select the option if it is supported.
Acked-by: Christoph Lameter <cl@linux.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/3] mm,x86,um: move CMPXCHG_DOUBLE config option
2011-11-10 14:04 [PATCH 0/3] CMPXCHG config options changes Heiko Carstens
2011-11-10 14:04 ` [PATCH 1/3] mm,slub,x86: decouple size of struct page from CONFIG_CMPXCHG_LOCAL Heiko Carstens
2011-11-10 14:04 ` [PATCH 2/3] mm,x86,um: move CMPXCHG_LOCAL config option Heiko Carstens
@ 2011-11-10 14:04 ` Heiko Carstens
2011-11-10 15:02 ` Christoph Lameter
2 siblings, 1 reply; 7+ messages in thread
From: Heiko Carstens @ 2011-11-10 14:04 UTC (permalink / raw)
To: Pekka Enberg, Andrew Morton
Cc: Jeff Dike, Ingo Molnar, Christoph Lameter, linux-mm,
linux-kernel, Heiko Carstens
Move CMPXCHG_DOUBLE and rename it to HAVE_CMPXCHG_DOUBLE so architectures can
simply select the option if it is supported.
Signed-off-by: Heiko Carstens <heiko.carstens@de.ibm.com>
---
arch/Kconfig | 3 +++
arch/x86/Kconfig | 1 +
arch/x86/Kconfig.cpu | 3 ---
arch/x86/um/Kconfig | 4 ----
mm/slub.c | 9 ++++++---
5 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index f5e749b..a5d7e7a 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -192,4 +192,7 @@ config HAVE_ALIGNED_STRUCT_PAGE
config HAVE_CMPXCHG_LOCAL
bool
+config HAVE_CMPXCHG_DOUBLE
+ bool
+
source "kernel/gcov/Kconfig"
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 71aebf5..6f38b61 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -77,6 +77,7 @@ config X86
select ARCH_HAVE_NMI_SAFE_CMPXCHG
select HAVE_ALIGNED_STRUCT_PAGE if SLUB && !M386
select HAVE_CMPXCHG_LOCAL if !M386
+ select HAVE_CMPXCHG_DOUBLE
config INSTRUCTION_DECODER
def_bool (KPROBES || PERF_EVENTS)
diff --git a/arch/x86/Kconfig.cpu b/arch/x86/Kconfig.cpu
index 99d2ab8..3c57033 100644
--- a/arch/x86/Kconfig.cpu
+++ b/arch/x86/Kconfig.cpu
@@ -309,9 +309,6 @@ config X86_INTERNODE_CACHE_SHIFT
config X86_CMPXCHG
def_bool X86_64 || (X86_32 && !M386)
-config CMPXCHG_DOUBLE
- def_bool y
-
config X86_L1_CACHE_SHIFT
int
default "7" if MPENTIUM4 || MPSC
diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig
index a62bfc6..b2b54d2 100644
--- a/arch/x86/um/Kconfig
+++ b/arch/x86/um/Kconfig
@@ -6,10 +6,6 @@ menu "UML-specific options"
menu "Host processor type and features"
-config CMPXCHG_DOUBLE
- bool
- default n
-
source "arch/x86/Kconfig.cpu"
endmenu
diff --git a/mm/slub.c b/mm/slub.c
index 7669b4c..7fd4e00 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -366,7 +366,8 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page
const char *n)
{
VM_BUG_ON(!irqs_disabled());
-#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
+ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (s->flags & __CMPXCHG_DOUBLE) {
if (cmpxchg_double(&page->freelist,
freelist_old, counters_old,
@@ -400,7 +401,8 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page,
void *freelist_new, unsigned long counters_new,
const char *n)
{
-#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
+ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (s->flags & __CMPXCHG_DOUBLE) {
if (cmpxchg_double(&page->freelist,
freelist_old, counters_old,
@@ -2990,7 +2992,8 @@ static int kmem_cache_open(struct kmem_cache *s,
}
}
-#if defined(CONFIG_CMPXCHG_DOUBLE) && defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
+#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \
+ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE)
if (system_has_cmpxchg_double() && (s->flags & SLAB_DEBUG_FLAGS) == 0)
/* Enable fast mode */
s->flags |= __CMPXCHG_DOUBLE;
--
1.7.7.1
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH 3/3] mm,x86,um: move CMPXCHG_DOUBLE config option
2011-11-10 14:04 ` [PATCH 3/3] mm,x86,um: move CMPXCHG_DOUBLE " Heiko Carstens
@ 2011-11-10 15:02 ` Christoph Lameter
0 siblings, 0 replies; 7+ messages in thread
From: Christoph Lameter @ 2011-11-10 15:02 UTC (permalink / raw)
To: Heiko Carstens
Cc: Pekka Enberg, Andrew Morton, Jeff Dike, Ingo Molnar, linux-mm,
linux-kernel
On Thu, 10 Nov 2011, Heiko Carstens wrote:
> Move CMPXCHG_DOUBLE and rename it to HAVE_CMPXCHG_DOUBLE so architectures can
> simply select the option if it is supported.
Acked-by: Christoph Lameter <cl@linux.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 7+ messages in thread