From: cl@linux-foundation.org
To: akpm@linux-foundation.org
Cc: linux-mm@kvack.org, Tejun Heo <tj@kernel.org>,
mingo@elte.hu, rusty@rustcorp.com.au, davem@davemloft.net,
Pekka Enberg <penberg@cs.helsinki.fi>
Subject: [this_cpu_xx V3 16/19] SLUB: Get rid of dynamic DMA kmalloc cache allocation
Date: Thu, 01 Oct 2009 13:40:49 -0400 [thread overview]
Message-ID: <20091001174122.602309040@gentwo.org> (raw)
In-Reply-To: <20091001174033.576397715@gentwo.org>
[-- Attachment #1: this_cpu_slub_static_dma_kmalloc --]
[-- Type: text/plain, Size: 3913 bytes --]
Dynamic DMA kmalloc cache allocation is troublesome since the
new percpu allocator does not support allocations in atomic contexts.
Reserve some statically allocated kmalloc_cpu structures instead.
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
---
include/linux/slub_def.h | 19 +++++++++++--------
mm/slub.c | 24 ++++++++++--------------
2 files changed, 21 insertions(+), 22 deletions(-)
Index: linux-2.6/include/linux/slub_def.h
===================================================================
--- linux-2.6.orig/include/linux/slub_def.h 2009-09-29 11:42:06.000000000 -0500
+++ linux-2.6/include/linux/slub_def.h 2009-09-29 11:43:18.000000000 -0500
@@ -131,11 +131,21 @@ struct kmem_cache {
#define SLUB_PAGE_SHIFT (PAGE_SHIFT + 2)
+#ifdef CONFIG_ZONE_DMA
+#define SLUB_DMA __GFP_DMA
+/* Reserve extra caches for potential DMA use */
+#define KMALLOC_CACHES (2 * SLUB_PAGE_SHIFT - 6)
+#else
+/* Disable DMA functionality */
+#define SLUB_DMA (__force gfp_t)0
+#define KMALLOC_CACHES SLUB_PAGE_SHIFT
+#endif
+
/*
* We keep the general caches in an array of slab caches that are used for
* 2^x bytes of allocations.
*/
-extern struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT];
+extern struct kmem_cache kmalloc_caches[KMALLOC_CACHES];
/*
* Sorry that the following has to be that ugly but some versions of GCC
@@ -203,13 +213,6 @@ static __always_inline struct kmem_cache
return &kmalloc_caches[index];
}
-#ifdef CONFIG_ZONE_DMA
-#define SLUB_DMA __GFP_DMA
-#else
-/* Disable DMA functionality */
-#define SLUB_DMA (__force gfp_t)0
-#endif
-
void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
void *__kmalloc(size_t size, gfp_t flags);
Index: linux-2.6/mm/slub.c
===================================================================
--- linux-2.6.orig/mm/slub.c 2009-09-29 11:42:06.000000000 -0500
+++ linux-2.6/mm/slub.c 2009-09-29 11:43:18.000000000 -0500
@@ -2090,7 +2090,7 @@ static inline int alloc_kmem_cache_cpus(
{
int cpu;
- if (s < kmalloc_caches + SLUB_PAGE_SHIFT && s >= kmalloc_caches)
+ if (s < kmalloc_caches + KMALLOC_CACHES && s >= kmalloc_caches)
/*
* Boot time creation of the kmalloc array. Use static per cpu data
* since the per cpu allocator is not available yet.
@@ -2537,7 +2537,7 @@ EXPORT_SYMBOL(kmem_cache_destroy);
* Kmalloc subsystem
*******************************************************************/
-struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT] __cacheline_aligned;
+struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;
EXPORT_SYMBOL(kmalloc_caches);
static int __init setup_slub_min_order(char *str)
@@ -2627,6 +2627,7 @@ static noinline struct kmem_cache *dma_k
char *text;
size_t realsize;
unsigned long slabflags;
+ int i;
s = kmalloc_caches_dma[index];
if (s)
@@ -2647,18 +2648,13 @@ static noinline struct kmem_cache *dma_k
text = kasprintf(flags & ~SLUB_DMA, "kmalloc_dma-%d",
(unsigned int)realsize);
- if (flags & __GFP_WAIT)
- s = kmalloc(kmem_size, flags & ~SLUB_DMA);
- else {
- int i;
+ s = NULL;
+ for (i = 0; i < KMALLOC_CACHES; i++)
+ if (kmalloc_caches[i].size)
+ break;
- s = NULL;
- for (i = 0; i < SLUB_PAGE_SHIFT; i++)
- if (kmalloc_caches[i].size) {
- s = kmalloc_caches + i;
- break;
- }
- }
+ BUG_ON(i >= KMALLOC_CACHES);
+ s = kmalloc_caches + i;
/*
* Must defer sysfs creation to a workqueue because we don't know
@@ -2672,7 +2668,7 @@ static noinline struct kmem_cache *dma_k
if (!s || !text || !kmem_cache_open(s, flags, text,
realsize, ARCH_KMALLOC_MINALIGN, slabflags, NULL)) {
- kfree(s);
+ s->size = 0;
kfree(text);
goto unlock_out;
}
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2009-10-01 17:03 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-10-01 17:40 [this_cpu_xx V3 00/19] Introduce per cpu atomic operations and avoid per cpu address arithmetic cl
2009-10-01 17:40 ` [this_cpu_xx V3 01/19] Introduce this_cpu_ptr() and generic this_cpu_* operations cl
2009-10-01 17:40 ` [this_cpu_xx V3 02/19] this_cpu: X86 optimized this_cpu operations cl
2009-10-01 17:40 ` [this_cpu_xx V3 03/19] Use this_cpu operations for SNMP statistics cl
2009-10-01 17:40 ` [this_cpu_xx V3 04/19] Use this_cpu operations for NFS statistics cl
2009-10-01 17:40 ` [this_cpu_xx V3 05/19] use this_cpu ops for network statistics cl
2009-10-01 19:35 ` David Miller
2009-10-01 17:40 ` [this_cpu_xx V3 06/19] this_cpu_ptr: Straight transformations cl
2009-10-01 17:40 ` [this_cpu_xx V3 07/19] this_cpu_ptr: Eliminate get/put_cpu cl
2009-10-01 17:40 ` [this_cpu_xx V3 08/19] this_cpu_ptr: xfs_icsb_modify_counters does not need "cpu" variable cl
2009-10-01 17:40 ` [this_cpu_xx V3 09/19] Use this_cpu_ptr in crypto subsystem cl
2009-10-01 17:40 ` [this_cpu_xx V3 10/19] Use this_cpu ops for VM statistics cl
2009-10-01 17:40 ` [this_cpu_xx V3 11/19] RCU: Use this_cpu operations cl
2009-10-01 19:17 ` Ingo Molnar
2009-10-01 17:40 ` [this_cpu_xx V3 12/19] Move early initialization of pagesets out of zone_wait_table_init() cl
2009-10-01 17:40 ` [this_cpu_xx V3 13/19] this_cpu_ops: page allocator conversion cl
2009-10-01 17:40 ` [this_cpu_xx V3 14/19] this_cpu ops: Remove pageset_notifier cl
2009-10-01 17:40 ` [this_cpu_xx V3 15/19] Use this_cpu operations in slub cl
2009-10-01 17:40 ` cl [this message]
2009-10-01 17:40 ` [this_cpu_xx V3 17/19] this_cpu: Remove slub kmem_cache fields cl
2009-10-01 17:40 ` [this_cpu_xx V3 18/19] Make slub statistics use this_cpu_inc cl
2009-10-01 17:40 ` [this_cpu_xx V3 19/19] this_cpu: slub aggressive use of this_cpu operations in the hotpaths cl
2009-10-02 9:29 ` [this_cpu_xx V3 00/19] Introduce per cpu atomic operations and avoid per cpu address arithmetic Ingo Molnar
2009-10-02 9:31 ` Ingo Molnar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20091001174122.602309040@gentwo.org \
--to=cl@linux-foundation.org \
--cc=akpm@linux-foundation.org \
--cc=davem@davemloft.net \
--cc=linux-mm@kvack.org \
--cc=mingo@elte.hu \
--cc=penberg@cs.helsinki.fi \
--cc=rusty@rustcorp.com.au \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox