* [SLUB 0/3] SLUB: The unqueued slab allocator V4
@ 2007-03-07 2:35 Christoph Lameter
2007-03-07 2:35 ` [SLUB 1/3] SLUB core Christoph Lameter
` (3 more replies)
0 siblings, 4 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 2:35 UTC (permalink / raw)
To: akpm
Cc: Marcelo Tosatti, linux-kernel, linux-mm, Christoph Lameter, mpm,
Manfred Spraul
[PATCH] SLUB The unqueued slab allocator v4
V3->V4
- Rename /proc/slabinfo to /proc/slubinfo. We have a different format after
all.
- More bug fixes and stabilization of diagnostic functions. This seems
to be finally something that works wherever we test it.
- Serialize kmem_cache_create and kmem_cache_destroy via slub_lock (Adrian's
idea)
- Add two new modifications (separate patches) to guarantee
a mininum number of objects per slab and to pass through large
allocations.
Note that SLUB will warn on zero sized allocations. SLAB just allocates
some memory. So some traces from the usb subsystem etc should be expected.
There are very likely also issues remaining in SLUB.
V2->V3
- Debugging and diagnostic support. This is runtime enabled and not compile
time enabled. Runtime debugging can be controlled via kernel boot options
on an individual slab cache basis or globally.
- Slab Trace support (For individual slab caches).
- Resiliency support: If basic sanity checks are enabled (via F f.e.)
(boot option) then SLUB will do the best to perform diagnostics and
then continue (i.e. mark corrupted objects as used).
- Fix up numerous issues including clash of SLUBs use of page
flags with i386 arch use for pmd and pgds (which are managed
as slab caches, sigh).
- Dynamic per CPU array sizing.
- Explain SLUB slabcache flags
V1->V2
- Fix up various issues. Tested on i386 UP, X86_64 SMP, ia64 NUMA.
- Provide NUMA support by splitting partial lists per node.
- Better Slab cache merge support (now at around 50% of slabs)
- List slab cache aliases if slab caches are merged.
- Updated descriptions /proc/slabinfo output
This is a new slab allocator which was motivated by the complexity of the
existing code in mm/slab.c. It attempts to address a variety of concerns
with the existing implementation.
A. Management of object queues
A particular concern was the complex management of the numerous object
queues in SLAB. SLUB has no such queues. Instead we dedicate a slab for
each allocating CPU and use objects from a slab directly instead of
queueing them up.
B. Storage overhead of object queues
SLAB Object queues exist per node, per CPU. The alien cache queue even
has a queue array that contain a queue for each processor on each
node. For very large systems the number of queues and the number of
objects that may be caught in those queues grows exponentially. On our
systems with 1k nodes / processors we have several gigabytes just tied up
for storing references to objects for those queues This does not include
the objects that could be on those queues. One fears that the whole
memory of the machine could one day be consumed by those queues.
C. SLAB meta data overhead
SLAB has overhead at the beginning of each slab. This means that data
cannot be naturally aligned at the beginning of a slab block. SLUB keeps
all meta data in the corresponding page_struct. Objects can be naturally
aligned in the slab. F.e. a 128 byte object will be aligned at 128 byte
boundaries and can fit tightly into a 4k page with no bytes left over.
SLAB cannot do this.
D. SLAB has a complex cache reaper
SLUB does not need a cache reaper for UP systems. On SMP systems
the per CPU slab may be pushed back into partial list but that
operation is simple and does not require an iteration over a list
of objects. SLAB expires per CPU, shared and alien object queues
during cache reaping which may cause strange hold offs.
E. SLAB has complex NUMA policy layer support
SLUB pushes NUMA policy handling into the page allocator. This means that
allocation is coarser (SLUB does interleave on a page level) but that
situation was also present before 2.6.13. SLABs application of
policies to individual slab objects allocated in SLAB is
certainly a performance concern due to the frequent references to
memory policies which may lead a sequence of objects to come from
one node after another. SLUB will get a slab full of objects
from one node and then will switch to the next.
F. Reduction of the size of partial slab lists
SLAB has per node partial lists. This means that over time a large
number of partial slabs may accumulate on those lists. These can
only be reused if allocator occur on specific nodes. SLUB has a global
pool of partial slabs and will consume slabs from that pool to
decrease fragmentation.
G. Tunables
SLAB has sophisticated tuning abilities for each slab cache. One can
manipulate the queue sizes in detail. However, filling the queues still
requires the uses of the spin lock to check out slabs. SLUB has a global
parameter (min_slab_order) for tuning. Increasing the minimum slab
order can decrease the locking overhead. The bigger the slab order the
less motions of pages between per CPU and partial lists occur and the
better SLUB will be scaling.
G. Slab merging
We often have slab caches with similar parameters. SLUB detects those
on boot up and merges them into the corresponding general caches. This
leads to more effective memory use. About 50% of all caches can
be eliminated through slab merging. This will also decrease
slab fragmentation because partial allocated slabs can be filled
up again. Slab merging can be switched off by specifying
slub_nomerge on boot up.
Note that merging can expose heretofore unknown bugs in the kernel
because corrupted objects may now be placed differently and corrupt
differing neighboring objects. Enable sanity checks to find those.
H. Diagnostics
The current slab diagnostics are difficult to use and require a
recompilation of the kernel. SLUB contains debugging code that
is always available (but is kept out of the hot code paths).
SLUB diagnostics can be enabled via the "slab_debug" option.
Parameters can be specified to select a single or a group of
slab caches for diagnostics. This means that the system is running
with the usual performance and it is much more likely that
race conditions can be reproduced.
I. Resiliency
If basic sanity checks are on then SLUB is capable of detecting
common error conditions and recover as best as possible to allow the
system to continue.
J. Tracing
Tracing can be enabled via the slab_debug=T,<slabcache> option
during boot. SLUB will then protocol each action on that slabcache
and dump the object contents on free.
K. On demand DMA cache creation.
Generally DMA caches are not needed. If a kmalloc is used with
__GFP_DMA then just create this single slabcache that is needed.
For systems that have no ZONE_DMA requirement the support is
completely eliminated.
Tested on:
i386 SMP, x86_64 UP + SMP + NUMA emulation, IA64 NUMA + Simulator
SLUB Boot options
slub_nomerge Disable merging of slabs
slub_min_order=x Require a minimum order for slab caches. This
increases the managed chunk size and therefore
reduces meta data and locking overhead.
slub_debug Enable all diagnostics for all caches
slub_debug=<options> Enable selective options for all caches
slub_debug=<o>,<cache> Enable selective options for a certain set of
caches
slub_min_objects Mininum objects per slab. Default is 8.
Available Debug options
F Double Free checking, sanity and resiliency
R Red zoning
P Object / padding poisoning
U Track last free / alloc
T Trace all allocs / frees (only use on individual slabs).
To use SLUB: Apply this patch and then select SLUB as the default slab
allocator. The output of /proc/slabinfo will then change. Here is a
sample (this is an UP/SMP format. The NUMA display will show on which nodes
the slabs were allocated). Flags are
a Cpucache Align requested
A Hardware Align required
C Constructor
d DMA cache
D Destructor
F Double free checking/Sanity
p Panic on failure
P Poisoning
r Objects are reclaimable
R RCU destroy
S Memory Spreading
U User Tracking
T Tracing
Z Red Zone
Thanks to Adrian Drzewiecki <z@drze.net> for many ideas and spotting many
bugs.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* [SLUB 1/3] SLUB core
2007-03-07 2:35 [SLUB 0/3] SLUB: The unqueued slab allocator V4 Christoph Lameter
@ 2007-03-07 2:35 ` Christoph Lameter
2007-03-07 2:35 ` [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs Christoph Lameter
` (2 subsequent siblings)
3 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 2:35 UTC (permalink / raw)
To: akpm
Cc: Marcelo Tosatti, linux-kernel, linux-mm, Christoph Lameter, mpm,
Manfred Spraul
SLUB core
Basic new slab allocator. See overview for details
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Index: linux-2.6.21-rc2-mm1/fs/proc/proc_misc.c
===================================================================
--- linux-2.6.21-rc2-mm1.orig/fs/proc/proc_misc.c 2007-03-06 17:59:44.000000000 -0800
+++ linux-2.6.21-rc2-mm1/fs/proc/proc_misc.c 2007-03-06 18:03:49.000000000 -0800
@@ -399,6 +399,21 @@ static const struct file_operations proc
};
#endif
+#ifdef CONFIG_SLUB
+extern struct seq_operations slubinfo_op;
+static int slubinfo_open(struct inode *inode, struct file *file)
+{
+ return seq_open(file, &slubinfo_op);
+}
+static const struct file_operations proc_slubinfo_operations = {
+ .open = slubinfo_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+#endif
+
+
#ifdef CONFIG_SLAB
static int slabinfo_open(struct inode *inode, struct file *file)
{
@@ -789,6 +804,9 @@ void __init proc_misc_init(void)
#endif
create_seq_entry("stat", 0, &proc_stat_operations);
create_seq_entry("interrupts", 0, &proc_interrupts_operations);
+#ifdef CONFIG_SLUB
+ create_seq_entry("slubinfo",S_IWUSR|S_IRUGO,&proc_slubinfo_operations);
+#endif
#ifdef CONFIG_SLAB
create_seq_entry("slabinfo",S_IWUSR|S_IRUGO,&proc_slabinfo_operations);
#ifdef CONFIG_DEBUG_SLAB_LEAK
Index: linux-2.6.21-rc2-mm1/include/linux/mm_types.h
===================================================================
--- linux-2.6.21-rc2-mm1.orig/include/linux/mm_types.h 2007-03-06 17:59:44.000000000 -0800
+++ linux-2.6.21-rc2-mm1/include/linux/mm_types.h 2007-03-06 18:03:49.000000000 -0800
@@ -19,10 +19,16 @@ struct page {
unsigned long flags; /* Atomic flags, some possibly
* updated asynchronously */
atomic_t _count; /* Usage count, see below. */
- atomic_t _mapcount; /* Count of ptes mapped in mms,
+ union {
+ atomic_t _mapcount; /* Count of ptes mapped in mms,
* to show when page is mapped
* & limit reverse map searches.
*/
+ struct { /* SLUB uses */
+ short unsigned int inuse;
+ short unsigned int offset;
+ };
+ };
union {
struct {
unsigned long private; /* Mapping-private opaque data:
@@ -43,8 +49,15 @@ struct page {
#if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
spinlock_t ptl;
#endif
+ struct { /* SLUB uses */
+ struct page *first_page; /* Compound pages */
+ struct kmem_cache *slab; /* Pointer to slab */
+ };
+ };
+ union {
+ pgoff_t index; /* Our offset within mapping. */
+ void *freelist; /* SLUB: pointer to free object */
};
- pgoff_t index; /* Our offset within mapping. */
struct list_head lru; /* Pageout list, eg. active_list
* protected by zone->lru_lock !
*/
Index: linux-2.6.21-rc2-mm1/include/linux/slab.h
===================================================================
--- linux-2.6.21-rc2-mm1.orig/include/linux/slab.h 2007-03-06 17:59:44.000000000 -0800
+++ linux-2.6.21-rc2-mm1/include/linux/slab.h 2007-03-06 18:03:49.000000000 -0800
@@ -32,6 +32,7 @@ typedef struct kmem_cache kmem_cache_t _
#define SLAB_PANIC 0x00040000UL /* Panic if kmem_cache_create() fails */
#define SLAB_DESTROY_BY_RCU 0x00080000UL /* Defer freeing slabs to RCU */
#define SLAB_MEM_SPREAD 0x00100000UL /* Spread some memory over cpuset */
+#define SLAB_TRACE 0x00200000UL /* Trace allocations and frees */
/* Flags passed to a constructor functions */
#define SLAB_CTOR_CONSTRUCTOR 0x001UL /* If not set, then deconstructor */
@@ -95,9 +96,14 @@ static inline void *kcalloc(size_t n, si
* the appropriate general cache at compile time.
*/
-#ifdef CONFIG_SLAB
+#if defined(CONFIG_SLAB) || defined(CONFIG_SLUB)
+#ifdef CONFIG_SLUB
+#include <linux/slub_def.h>
+#else
#include <linux/slab_def.h>
+#endif /* !CONFIG_SLUB */
#else
+
/*
* Fallback definitions for an allocator not wanting to provide
* its own optimized kmalloc definitions (like SLOB).
Index: linux-2.6.21-rc2-mm1/include/linux/slub_def.h
===================================================================
--- /dev/null 1970-01-01 00:00:00.000000000 +0000
+++ linux-2.6.21-rc2-mm1/include/linux/slub_def.h 2007-03-06 18:03:49.000000000 -0800
@@ -0,0 +1,168 @@
+#ifndef _LINUX_SLUB_DEF_H
+#define _LINUX_SLUB_DEF_H
+
+/*
+ * SLUB : A Slab allocator without object queues.
+ *
+ * (C) 2007 SGI, Christoph Lameter <clameter@sgi.com>
+ */
+#include <linux/types.h>
+#include <linux/gfp.h>
+#include <linux/workqueue.h>
+
+struct kmem_cache_node {
+ spinlock_t list_lock; /* Protect partial list and nr_partial */
+ unsigned long nr_partial;
+ atomic_long_t nr_slabs;
+ struct list_head partial;
+};
+
+/*
+ * Slab cache management.
+ */
+struct kmem_cache {
+ int offset; /* Free pointer offset. */
+ unsigned int order;
+ unsigned long flags;
+ int size; /* Total size of an object */
+ int objects; /* Number of objects in slab */
+ struct kmem_cache_node local_node;
+ int refcount; /* Refcount for destroy */
+ void (*ctor)(void *, struct kmem_cache *, unsigned long);
+ void (*dtor)(void *, struct kmem_cache *, unsigned long);
+
+ int objsize; /* The size of an object that is in a chunk */
+ int inuse; /* Used portion of the chunk */
+ const char *name; /* Name (only for display!) */
+ char *aliases; /* Slabs merged into this one */
+ struct list_head list; /* List of slabs */
+#ifdef CONFIG_SMP
+ struct mutex flushing;
+ atomic_t cpu_slabs; /*
+ * if >0 then flusher is scheduled. Also used
+ * to count remaining cpus if flushing
+ */
+ struct delayed_work flush;
+#endif
+#ifdef CONFIG_NUMA
+ struct kmem_cache_node *node[MAX_NUMNODES];
+#endif
+ struct page *cpu_slab[NR_CPUS];
+};
+
+/*
+ * Kmalloc subsystem.
+ */
+#define KMALLOC_SHIFT_LOW 3
+
+#define KMALLOC_SHIFT_HIGH 18
+
+#if L1_CACHE_BYTES <= 64
+#define KMALLOC_EXTRAS 2
+#define KMALLOC_EXTRA
+#else
+#define KMALLOC_EXTRAS 0
+#endif
+
+#define KMALLOC_NR_CACHES (KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW \
+ + 1 + KMALLOC_EXTRAS)
+/*
+ * We keep the general caches in an array of slab caches that are used for
+ * 2^x bytes of allocations.
+ */
+extern struct kmem_cache kmalloc_caches[KMALLOC_NR_CACHES];
+
+/*
+ * Sorry that the following has to be that ugly but some versions of GCC
+ * have trouble with constant propagation and loops.
+ */
+static inline int kmalloc_index(int size)
+{
+ if (size <= 8) return 3;
+ if (size <= 16) return 4;
+ if (size <= 32) return 5;
+ if (size <= 64) return 6;
+#ifdef KMALLOC_EXTRA
+ if (size <= 96) return KMALLOC_SHIFT_HIGH + 1;
+#endif
+ if (size <= 128) return 7;
+#ifdef KMALLOC_EXTRA
+ if (size <= 192) return KMALLOC_SHIFT_HIGH + 2;
+#endif
+ if (size <= 256) return 8;
+ if (size <= 512) return 9;
+ if (size <= 1024) return 10;
+ if (size <= 2048) return 11;
+ if (size <= 4096) return 12;
+ if (size <= 8 * 1024) return 13;
+ if (size <= 16 * 1024) return 14;
+ if (size <= 32 * 1024) return 15;
+ if (size <= 64 * 1024) return 16;
+ if (size <= 128 * 1024) return 17;
+ if (size <= 256 * 1024) return 18;
+ return -1;
+}
+
+/*
+ * Find the slab cache for a given combination of allocation flags and size.
+ *
+ * This ought to end up with a global pointer to the right cache
+ * in kmalloc_caches.
+ */
+static inline struct kmem_cache *kmalloc_slab(size_t size)
+{
+ int index = kmalloc_index(size) - KMALLOC_SHIFT_LOW;
+
+ if (index < 0) {
+ /*
+ * Generate a link failure. Would be great if we could
+ * do something to stop the compile here.
+ */
+ extern void __kmalloc_size_too_large(void);
+ __kmalloc_size_too_large();
+ }
+ return &kmalloc_caches[index];
+}
+
+#ifdef CONFIG_ZONE_DMA
+#define SLUB_DMA __GFP_DMA
+#else
+/* Disable SLAB functionality */
+#define SLUB_DMA 0
+#endif
+
+static inline void *kmalloc(size_t size, gfp_t flags)
+{
+ if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+ struct kmem_cache *s = kmalloc_slab(size);
+
+ return kmem_cache_alloc(s, flags);
+ } else
+ return __kmalloc(size, flags);
+}
+
+static inline void *kzalloc(size_t size, gfp_t flags)
+{
+ if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+ struct kmem_cache *s = kmalloc_slab(size);
+
+ return kmem_cache_zalloc(s, flags);
+ } else
+ return __kzalloc(size, flags);
+}
+
+#ifdef CONFIG_NUMA
+extern void *__kmalloc_node(size_t size, gfp_t flags, int node);
+
+static inline void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+ if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+ struct kmem_cache *s = kmalloc_slab(size);
+
+ return kmem_cache_alloc_node(s, flags, node);
+ } else
+ return __kmalloc_node(size, flags, node);
+}
+#endif
+
+#endif /* _LINUX_SLUB_DEF_H */
Index: linux-2.6.21-rc2-mm1/init/Kconfig
===================================================================
--- linux-2.6.21-rc2-mm1.orig/init/Kconfig 2007-03-06 17:59:44.000000000 -0800
+++ linux-2.6.21-rc2-mm1/init/Kconfig 2007-03-06 18:03:49.000000000 -0800
@@ -481,15 +481,6 @@ config SHMEM
option replaces shmem and tmpfs with the much simpler ramfs code,
which may be appropriate on small systems without swap.
-config SLAB
- default y
- bool "Use full SLAB allocator" if (EMBEDDED && !SMP && !SPARSEMEM)
- help
- Disabling this replaces the advanced SLAB allocator and
- kmalloc support with the drastically simpler SLOB allocator.
- SLOB is more space efficient but does not scale well and is
- more susceptible to fragmentation.
-
config VM_EVENT_COUNTERS
default y
bool "Enable VM event counters for /proc/vmstat" if EMBEDDED
@@ -537,6 +528,46 @@ config RCU_TRACE
Say Y here if you want to enable RCU tracing
Say N if you are unsure.
+choice
+ prompt "Choose SLAB allocator"
+ default SLAB
+ help
+ This option allows to select a slab allocator.
+
+config SLAB
+ bool "SLAB"
+ help
+ The regular slab allocator that is established and known to work
+ well in all environments. It organizes chache hot objects in
+ per cpu and per node queues. SLAB is the default choice for
+ slab allocator.
+
+config SLUB
+ depends on EXPERIMENTAL
+ bool "SLUB (Unqueued Allocator)"
+ help
+ SLUB is a slab allocator that minimizes cache line usage
+ instead of managing queues of cached objects (SLAB approach).
+ Per cpu caching is realized using slabs of objects instead
+ of queues of objects. SLUB can use memory in the most efficient
+ way and has enhanced diagnostics.
+
+config SLOB
+#
+# SLOB cannot support SMP because SLAB_DESTROY_BY_RCU does not work
+# properly.
+#
+ depends on EMBEDDED && !SMP && !SPARSEMEM
+ bool "SLOB (Simple Allocator)"
+ help
+ SLOB replaces the SLAB allocator with a drastically simpler
+ allocator. SLOB is more space efficient that SLAB but does not
+ scale well (single lock for all operations) and is more susceptible
+ to fragmentation. SLOB it is a great choice to reduce
+ memory usage and code size.
+
+endchoice
+
endmenu # General setup
config RT_MUTEXES
@@ -552,10 +583,6 @@ config BASE_SMALL
default 0 if BASE_FULL
default 1 if !BASE_FULL
-config SLOB
- default !SLAB
- bool
-
menu "Loadable module support"
config MODULES
Index: linux-2.6.21-rc2-mm1/mm/Makefile
===================================================================
--- linux-2.6.21-rc2-mm1.orig/mm/Makefile 2007-03-06 17:59:44.000000000 -0800
+++ linux-2.6.21-rc2-mm1/mm/Makefile 2007-03-06 18:03:49.000000000 -0800
@@ -26,6 +26,7 @@ obj-$(CONFIG_TMPFS_POSIX_ACL) += shmem_a
obj-$(CONFIG_TINY_SHMEM) += tiny-shmem.o
obj-$(CONFIG_SLOB) += slob.o
obj-$(CONFIG_SLAB) += slab.o
+obj-$(CONFIG_SLUB) += slub.o
obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
obj-$(CONFIG_FS_XIP) += filemap_xip.o
obj-$(CONFIG_MIGRATION) += migrate.o
Index: linux-2.6.21-rc2-mm1/mm/slub.c
===================================================================
--- /dev/null 1970-01-01 00:00:00.000000000 +0000
+++ linux-2.6.21-rc2-mm1/mm/slub.c 2007-03-06 18:05:36.000000000 -0800
@@ -0,0 +1,2253 @@
+/*
+ * SLUB: A slab allocator that limits cache line use instead of queuing
+ * objects in per cpu and per node lists.
+ *
+ * The allocator synchronizes using per slab locks and only
+ * uses a centralized lock to manage a pool of partial slabs.
+ *
+ * (C) 2007 SGI, Christoph Lameter <clameter@sgi.com>
+ */
+
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/bit_spinlock.h>
+#include <linux/interrupt.h>
+#include <linux/bitops.h>
+#include <linux/slab.h>
+#include <linux/seq_file.h>
+#include <linux/cpu.h>
+#include <linux/cpuset.h>
+#include <linux/mempolicy.h>
+#include <linux/ctype.h>
+
+/*
+ * Lock order:
+ * 1. slab_lock(page)
+ * 2. slab->list_lock
+ *
+ * SLUB assigns one slab for allocation to each processor.
+ * Allocations only occur from these slabs called cpu slabs.
+ *
+ * If a cpu slab exists then a workqueue thread checks every 10
+ * seconds if the cpu slab is still in use. The cpu slab is pushed back
+ * to the list if inactive [only needed for SMP].
+ *
+ * Slabs with free elements are kept on a partial list.
+ * There is no list for full slabs. If an object in a full slab is
+ * freed then the slab will show up again on the partial lists.
+ * Otherwise there is no need to track full slabs (but we keep a counter).
+ *
+ * Slabs are freed when they become empty. Teardown and setup is
+ * minimal so we rely on the page allocators per cpu caches for
+ * fast frees and allocs.
+ *
+ * Overloading of page flags that are otherwise used for LRU management.
+ *
+ * PageActive The slab is used as a cpu cache. Allocations
+ * may be performed from the slab. The slab is not
+ * on a partial list.
+ *
+ * PageReferenced The per cpu slab was used recently. This is used
+ * to push back per cpu slabs if they are unused
+ * for a longer time period.
+ *
+ * PageError Slab requires special handling due to debug
+ * options set or a single page slab. This moves
+ * slab handling out of the fast path.
+ */
+
+/*
+ * Flags from the regular SLAB that SLUB does not support:
+ */
+#define SLUB_UNIMPLEMENTED (SLAB_DEBUG_INITIAL)
+
+#define DEBUG_DEFAULT_FLAGS (SLAB_DEBUG_FREE | SLAB_RED_ZONE | \
+ SLAB_POISON)
+/*
+ * Set of flags that will prevent slab merging
+ */
+#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
+ SLAB_TRACE)
+
+#define SLUB_MERGE_SAME (SLAB_DEBUG_FREE | SLAB_DESTROY_BY_RCU | \
+ SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA)
+
+#ifndef ARCH_KMALLOC_MINALIGN
+#define ARCH_KMALLOC_MINALIGN sizeof(void *)
+#endif
+
+#ifndef ARCH_SLAB_MINALIGN
+#define ARCH_SLAB_MINALIGN sizeof(void *)
+#endif
+
+static int kmem_size = sizeof(struct kmem_cache);
+
+#ifdef CONFIG_SMP
+static struct notifier_block slab_notifier;
+#endif
+
+static enum {
+ DOWN, /* No slab functionality available */
+ PARTIAL, /* kmem_cache_open() works but kmalloc does not */
+ UP /* Everything works */
+} slab_state = DOWN;
+
+int slab_is_available(void) {
+ return slab_state == UP;
+}
+
+/* A list of all slab caches on the system */
+static DECLARE_RWSEM(slub_lock);
+LIST_HEAD(slab_caches);
+
+/********************************************************************
+ * Core slab cache functions
+ *******************************************************************/
+
+struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
+{
+#ifdef CONFIG_NUMA
+ return s->node[node];
+#else
+ return &s->local_node;
+#endif
+}
+
+/*
+ * Object debugging
+ */
+static void print_section(char *text, u8 *addr, unsigned int length)
+{
+ int i, offset;
+ int newline = 1;
+ char ascii[17];
+
+ if (length > 128)
+ length = 128;
+ ascii[16] = 0;
+
+ for (i = 0; i < length; i++) {
+ if (newline) {
+ printk(KERN_ERR "%10s %p: ", text, addr + i);
+ newline = 0;
+ }
+ printk(" %02x", addr[i]);
+ offset = i % 16;
+ ascii[offset] = isgraph(addr[i]) ? addr[i] : '.';
+ if (offset == 15) {
+ printk(" %s\n",ascii);
+ newline = 1;
+ }
+ }
+ if (!newline) {
+ i %= 16;
+ while (i < 16) {
+ printk(" ");
+ ascii[i] = ' ';
+ i++;
+ }
+ printk(" %s\n", ascii);
+ }
+}
+
+/*
+ * Slow version of get and set free pointer.
+ *
+ * This requires touching the cache lines of kmem_cache.
+ * The offset can also be obtained from the page. In that
+ * case it is in the cacheline that we already need to touch.
+ */
+static void *get_freepointer(struct kmem_cache *s, void *object)
+{
+ return *(void **)(object + s->offset);
+}
+
+static void set_freepointer(struct kmem_cache *s, void *object, void *fp)
+{
+ *(void **)(object + s->offset) = fp;
+}
+
+/*
+ * Tracking user of a slab.
+ */
+static void *get_track(struct kmem_cache *s, void *object, int alloc)
+{
+ void **p = object + s->inuse + sizeof(void *);
+
+ return p[alloc];
+}
+
+static void set_track(struct kmem_cache *s, void *object,
+ int alloc, void *addr)
+{
+ void **p = object + s->inuse + sizeof(void *);
+
+ p[alloc] = addr;
+}
+
+#define set_tracking(__s, __o, __a) set_track(__s, __o, __a, \
+ __builtin_return_address(0))
+
+static void init_tracking(struct kmem_cache *s, void *object)
+{
+ if (s->flags & SLAB_STORE_USER) {
+ set_track(s, object, 0, NULL);
+ set_track(s, object, 1, NULL);
+ }
+}
+
+static void print_trailer(struct kmem_cache *s, u8 *p)
+{
+ unsigned int off; /* Offset of last byte */
+
+ if (s->offset)
+ off = s->offset + sizeof(void *);
+ else
+ off = s->inuse;
+
+ if (s->flags & SLAB_RED_ZONE)
+ print_section("Redzone", p + s->objsize,
+ s->inuse - s->objsize);
+
+ printk(KERN_ERR "FreePointer %p: %p\n",
+ p + s->offset,
+ get_freepointer(s, p));
+
+ if (s->flags & SLAB_STORE_USER) {
+ printk(KERN_ERR "Last Allocate from %p. Last Free from %p\n",
+ get_track(s, p, 0), get_track(s, p, 1));
+ off += 2 * sizeof(void *);
+ }
+
+ if (off != s->size)
+ /* Beginning of the filler is the free pointer */
+ print_section("Filler", p + off, s->size - off);
+}
+
+static void object_err(struct kmem_cache *s, struct page *page,
+ u8 *object, char *reason)
+{
+ u8 *addr = page_address(page);
+
+ printk(KERN_ERR "*** SLUB: %s in %s@%p Slab %p\n",
+ reason, s->name, object, page);
+ printk(KERN_ERR " offset=%u flags=%04lx inuse=%u freelist=%p\n",
+ (int)(object - addr), page->flags, page->inuse, page->freelist);
+ if (object > addr + 16)
+ print_section("Bytes b4", object - 16, 16);
+ print_section("Object", object, s->objsize);
+ print_trailer(s, object);
+ dump_stack();
+}
+
+static void init_object(struct kmem_cache *s, void *object, int active)
+{
+ u8 *p = object;
+
+ if (s->objects == 1)
+ return;
+
+ if (s->flags & SLAB_POISON) {
+ memset(p, POISON_FREE, s->objsize -1);
+ p[s->objsize -1] = POISON_END;
+ }
+
+ if (s->flags & SLAB_RED_ZONE)
+ memset(p + s->objsize,
+ active ? RED_ACTIVE : RED_INACTIVE,
+ s->inuse - s->objsize);
+}
+
+static int check_bytes(u8 *start, unsigned int value, unsigned int bytes)
+{
+ while (bytes) {
+ if (*start != (u8)value)
+ return 0;
+ start++;
+ bytes--;
+ }
+ return 1;
+}
+
+
+static int check_valid_pointer(struct kmem_cache *s, struct page *page,
+ void *object)
+{
+ void *base;
+
+ if (!object)
+ return 1;
+
+ base = page_address(page);
+ if (object < base || object >= base + s->objects * s->size ||
+ (object - base) % s->size) {
+ return 0;
+ }
+
+ return 1;
+}
+
+/*
+ * Object layout:
+ *
+ * object address
+ * Bytes of the object to be managed.
+ * If the freepointer may overlay the object then the free
+ * pointer is the first word of the object.
+ * Poisoning uses 0x6b (POISON_FREE) and the last byte is
+ * 0xa5 (POISON_END)
+ *
+ * object + s->objsize
+ * Padding to reach word boundary. This is also used for Redzoning.
+ * Padding is extended to word size if Redzoning is enabled
+ * and objsize == inuse.
+ * We fill with 0x71 (RED_INACTIVE) for inactive objects and with
+ * 0xa5 (RED_ACTIVE) for objects in use.
+ *
+ * object + s->inuse
+ * A. Free pointer (if we cannot overwrite object on free)
+ * B. Tracking data for SLAB_STORE_USER
+ * C. Padding to reach required alignment boundary
+ * Padding is done using 0x5a (POISON_INUSE)
+ *
+ * object + s->size
+ *
+ * If slabcaches are merged then the objsize and inuse boundaries are to be ignored.
+ * And therefore no slab options that rely on these boundaries may be used with
+ * merged slabcaches.
+ */
+
+static int check_pad_bytes(struct kmem_cache *s, struct page *page, u8 *p)
+{
+ unsigned long off = s->inuse; /* The end of info */
+
+ if (s->offset)
+ /* Freepointer is placed after the object. */
+ off += sizeof(void *);
+
+ if (s->flags & SLAB_STORE_USER)
+ /* We also have user information there */
+ off += 2 * sizeof(void *);
+
+ if (s->size == off)
+ return 1;
+
+ if (check_bytes(p + off, POISON_INUSE, s->size - off))
+ return 1;
+
+ object_err(s, page, p, "Object padding check fails");
+ return 0;
+}
+
+static int check_object(struct kmem_cache *s, struct page *page,
+ void *object, int active)
+{
+ u8 *p = object;
+ u8 *endobject = object + s->objsize;
+
+ /* Single object slabs do not get policed */
+ if (s->objects == 1)
+ return 1;
+
+ if (s->flags & SLAB_RED_ZONE) {
+ if (!check_bytes(endobject,
+ active ? RED_ACTIVE : RED_INACTIVE,
+ s->inuse - s->objsize)) {
+ object_err(s, page, object,
+ active ? "Redzone Active check fails" :
+ "Redzone Inactive check fails");
+ return 0;
+ }
+ } else
+ if ((s->flags & SLAB_POISON) &&
+ s->objsize < s->inuse &&
+ !check_bytes(endobject, POISON_INUSE, s->inuse - s->objsize))
+ object_err(s, page, p, "Alignment padding check fails");
+
+ if (s->flags & SLAB_POISON) {
+ if (!active && (!check_bytes(p, POISON_FREE, s->objsize - 1) ||
+ p[s->objsize -1] != POISON_END)) {
+ object_err(s, page, p, "Poison");
+ return 0;
+ }
+ if (!check_pad_bytes(s, page, p))
+ return 0;
+ }
+
+ if (!s->offset && active)
+ /*
+ * Object and freepointer overlap. Cannot check
+ * freepointer while object is allocated.
+ */
+ return 1;
+
+ /* Check free pointer validity */
+ if (!check_valid_pointer(s, page, get_freepointer(s, p))) {
+ object_err(s, page, p, "Freepointer corrupt");
+ /*
+ * No choice but to zap it. This may cause
+ * another error because the object count
+ * is now wrong.
+ */
+ set_freepointer(s, p, NULL);
+ return 0;
+ }
+ return 1;
+}
+
+static int check_slab(struct kmem_cache *s, struct page *page)
+{
+ if (!PageSlab(page)) {
+ printk(KERN_CRIT "SLUB: %s Not a valid slab page @%p flags=%lx"
+ " mapping=%p count=%d \n",
+ s->name, page, page->flags, page->mapping,
+ page_count(page));
+ return 0;
+ }
+ if (page->offset * sizeof(void *) != s->offset) {
+ printk(KERN_CRIT "SLUB: %s Corrupted offset %lu in slab @%p"
+ " flags=%lx mapping=%p count=%d\n",
+ s->name,
+ (unsigned long)(page->offset * sizeof(void *)),
+ page,
+ page->flags,
+ page->mapping,
+ page_count(page));
+ return 0;
+ }
+ if (page->inuse > s->objects) {
+ printk(KERN_CRIT "SLUB: %s Inuse %u > max %u in slab page @%p"
+ " flags=%lx mapping=%p count=%d\n",
+ s->name, page->inuse, s->objects, page, page->flags,
+ page->mapping, page_count(page));
+ return 0;
+ }
+ return 1;
+}
+
+/*
+ * Determine if a certain object on a page is on the freelist and
+ * therefore free. Must hold the slab lock for cpu slabs to
+ * guarantee that the chains are consistent.
+ */
+static int on_freelist(struct kmem_cache *s, struct page *page, void *search)
+{
+ int nr = 0;
+ void *fp = page->freelist;
+ void *object = NULL;
+
+ if (s->objects == 1)
+ return 0;
+
+ while (fp && nr <= s->objects) {
+ if (fp == search)
+ return 1;
+ if (!check_valid_pointer(s, page, fp)) {
+ if (object) {
+ object_err(s, page, object, "Freechain corrupt");
+ set_freepointer(s, object, NULL);
+ break;
+ } else {
+ printk(KERN_ERR "SLUB: %s slab %p freepointer %p corrupted.\n",
+ s->name, page, fp);
+ dump_stack();
+ page->freelist = NULL;
+ page->inuse = s->objects;
+ return 0;
+ }
+ break;
+ }
+ object = fp;
+ fp = get_freepointer(s, object);
+ nr++;
+ }
+
+ if (page->inuse != s->objects - nr) {
+ printk(KERN_CRIT "slab %s: page %p wrong object count."
+ " counter is %d but counted were %d\n",
+ s->name, page, page->inuse,
+ s->objects - nr);
+ page->inuse = s->objects - nr;
+ }
+ return 0;
+}
+
+static int alloc_object_checks(struct kmem_cache *s, struct page *page,
+ void *object)
+{
+ if (!check_slab(s, page))
+ goto bad;
+
+ if (object && !on_freelist(s, page, object)) {
+ printk(KERN_ERR "SLAB: %s Object %p@%p already allocated.\n",
+ s->name, object, page);
+ goto dump;
+ }
+
+ if (!check_valid_pointer(s, page, object)) {
+ object_err(s, page, object, "Freelist Pointer check fails");
+ goto dump;
+ }
+
+ if (!object)
+ return 1;
+
+ if (!check_object(s, page, object, 0))
+ goto bad;
+ init_object(s, object, 1);
+
+ if (s->flags & SLAB_TRACE) {
+ printk("SLUB-Trace %s alloc object=%p slab=%p inuse=%d"
+ " freelist=%p\n",
+ s->name, object, page, page->inuse,
+ page->freelist);
+ dump_stack();
+ }
+ return 1;
+dump:
+ dump_stack();
+bad:
+ /* Mark slab full */
+ page->inuse = s->objects;
+ page->freelist = NULL;
+ return 0;
+}
+
+static int free_object_checks(struct kmem_cache *s, struct page *page, void *object)
+{
+ if (!check_slab(s, page)) {
+ goto fail;
+ }
+
+ if (!check_valid_pointer(s, page, object)) {
+ printk(KERN_ERR "SLUB: %s slab %p invalid object pointer %p\n",
+ s->name, page, object);
+ goto fail;
+ }
+
+ if (on_freelist(s, page, object)) {
+ printk(KERN_CRIT "SLUB: %s slab %p object %p already free.\n",
+ s->name, page, object);
+ goto fail;
+ }
+
+ if (!check_object(s, page, object, 1))
+ return 0;
+
+ if (unlikely(s != page->slab)) {
+ if (!PageSlab(page))
+ printk(KERN_CRIT "slab_free %s size %d: attempt to"
+ "free object(%p) outside of slab.\n",
+ s->name, s->size, object);
+ else
+ if (!page->slab)
+ printk(KERN_CRIT
+ "slab_free : no slab(NULL) for object %p.\n",
+ object);
+ else
+ printk(KERN_CRIT "slab_free %s(%d): object at %p"
+ " belongs to slab %s(%d)\n",
+ s->name, s->size, object,
+ page->slab->name, page->slab->size);
+ goto fail;
+ }
+ if (s->flags & SLAB_TRACE) {
+ printk("SLUB-Trace %s free object=%p slab=%p"
+ "inuse=%d freelist=%p\n",
+ s->name, object, page, page->inuse,
+ page->freelist);
+ print_section("SLUB-Trace", object, s->objsize);
+ dump_stack();
+ }
+ init_object(s, object, 0);
+ return 1;
+fail:
+ dump_stack();
+ return 0;
+}
+
+/*
+ * Slab allocation and freeing
+ */
+static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
+{
+ struct page * page;
+ int pages = 1 << s->order;
+
+ if (s->order)
+ flags |= __GFP_COMP;
+
+ if (s->flags & SLUB_DMA)
+ flags |= GFP_DMA;
+
+ if (node == -1)
+ page = alloc_pages(flags, s->order);
+ else
+ page = alloc_pages_node(node, flags, s->order);
+
+ if (!page)
+ return NULL;
+
+ mod_zone_page_state(page_zone(page),
+ (s->flags & SLAB_RECLAIM_ACCOUNT) ?
+ NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
+ pages);
+
+ if (unlikely(s->ctor)) {
+ void *start = page_address(page);
+ void *end = start + (pages << PAGE_SHIFT);
+ void *p;
+ int mode = 1;
+
+ if (!(flags & __GFP_WAIT))
+ mode |= SLAB_CTOR_ATOMIC;
+
+ for (p = start; p <= end - s->size; p += s->size)
+ s->ctor(p, s, mode);
+ }
+ return page;
+}
+
+static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node)
+{
+ struct page *page;
+ struct kmem_cache_node *n;
+
+ BUG_ON(flags & ~(GFP_DMA | GFP_LEVEL_MASK | __GFP_NO_GROW));
+ if (flags & __GFP_NO_GROW)
+ return NULL;
+
+ if (flags & __GFP_WAIT)
+ local_irq_enable();
+
+ page = allocate_slab(s, flags & GFP_LEVEL_MASK, node);
+ if (!page)
+ goto out;
+
+ n = get_node(s, page_to_nid(page));
+ if (n)
+ atomic_long_inc(&n->nr_slabs);
+ page->offset = s->offset / sizeof(void *);
+ page->slab = s;
+ page->flags |= 1 << PG_slab;
+ if (s->flags & (SLAB_DEBUG_FREE | SLAB_RED_ZONE | SLAB_POISON |
+ SLAB_STORE_USER | SLAB_TRACE) ||
+ s->objects == 1)
+ page->flags |= 1 << PG_error;
+
+ if (s->objects > 1) {
+ void *start = page_address(page);
+ void *end = start + s->objects * s->size;
+ void *last = start;
+ void *p = start + s->size;
+
+ if (unlikely(s->flags & SLAB_POISON))
+ memset(start, POISON_INUSE, PAGE_SIZE << s->order);
+ while (p < end) {
+ if (PageError(page)) {
+ init_object(s, last, 0);
+ init_tracking(s, last);
+ }
+ set_freepointer(s, last, p);
+ last = p;
+ p += s->size;
+ }
+ set_freepointer(s, last, NULL);
+ page->freelist = start;
+ page->inuse = 0;
+ if (PageError(page)) {
+ init_object(s, last, 0);
+ init_tracking(s, last);
+ }
+ }
+
+out:
+ if (flags & __GFP_WAIT)
+ local_irq_disable();
+ return page;
+}
+
+
+static void __free_slab(struct kmem_cache *s, struct page *page)
+{
+ int pages = 1 << s->order;
+
+ if (unlikely(PageError(page) || s->dtor)) {
+ void *start = page_address(page);
+ void *end = start + (pages << PAGE_SHIFT);
+ void *p;
+ int n;
+
+ for (p = start; p <= end - s->size; p += s->size) {
+ if (s->dtor)
+ s->dtor(p, s, 0);
+ else
+ check_object(s, page, p, 0);
+ }
+ n = end - p;
+ if (n && (s->flags & SLAB_POISON) &&
+ check_bytes(p, POISON_INUSE, n)) {
+ printk(KERN_ERR "SLUB: %s slab %p: Slab"
+ "Padding fails check\n", s->name, p);
+ print_section("Slab Pad", p, n);
+ }
+ }
+
+ mod_zone_page_state(page_zone(page),
+ (s->flags & SLAB_RECLAIM_ACCOUNT) ?
+ NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
+ - pages);
+
+ __free_pages(page, s->order);
+}
+
+static void rcu_free_slab(struct rcu_head *h)
+{
+ struct page *page;
+ struct kmem_cache *s;
+
+ page = container_of((struct list_head *)h, struct page, lru);
+ s = (struct kmem_cache *)page->mapping;
+ page->mapping = NULL;
+ __free_slab(s, page);
+}
+
+static void free_slab(struct kmem_cache *s, struct page *page)
+{
+ if (unlikely(s->flags & SLAB_DESTROY_BY_RCU)) {
+ /*
+ * RCU free overloads the RCU head over the LRU
+ */
+ struct rcu_head *head = (void *)&page->lru;
+
+ page->mapping = (void *)s;
+ call_rcu(head, rcu_free_slab);
+ } else
+ __free_slab(s, page);
+}
+
+static void discard_slab(struct kmem_cache *s, struct page *page)
+{
+ struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+ atomic_long_dec(&n->nr_slabs);
+
+ page->mapping = NULL;
+ reset_page_mapcount(page);
+ page->flags &= ~(1 << PG_slab | 1 << PG_error);
+ free_slab(s, page);
+}
+
+/*
+ * Per slab locking using the pagelock
+ */
+static __always_inline void slab_lock(struct page *page)
+{
+#ifdef CONFIG_SMP
+ bit_spin_lock(PG_locked, &page->flags);
+#endif
+}
+
+static __always_inline void slab_unlock(struct page *page)
+{
+#ifdef CONFIG_SMP
+ bit_spin_unlock(PG_locked, &page->flags);
+#endif
+}
+
+static __always_inline int slab_trylock(struct page *page)
+{
+ int rc = 1;
+#ifdef CONFIG_SMP
+ rc = bit_spin_trylock(PG_locked, &page->flags);
+#endif
+ return rc;
+}
+
+/*
+ * Management of partially allocated slabs
+ */
+static void __always_inline add_partial(struct kmem_cache *s, struct page *page)
+{
+ struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+ spin_lock(&n->list_lock);
+ n->nr_partial++;
+ list_add_tail(&page->lru, &n->partial);
+ spin_unlock(&n->list_lock);
+}
+
+static void __always_inline remove_partial(struct kmem_cache *s,
+ struct page *page)
+{
+ struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+ spin_lock(&n->list_lock);
+ list_del(&page->lru);
+ n->nr_partial--;
+ spin_unlock(&n->list_lock);
+}
+
+/*
+ * Lock page and remove it from the partial list
+ *
+ * Must hold list_lock
+ */
+static __always_inline int lock_and_del_slab(struct kmem_cache_node *n,
+ struct page *page)
+{
+ if (slab_trylock(page)) {
+ list_del(&page->lru);
+ n->nr_partial--;
+ return 1;
+ }
+ return 0;
+}
+
+/*
+ * Try to get a partial slab from a specific node
+ */
+static struct page *get_partial_node(struct kmem_cache_node *n)
+{
+ struct page *page;
+
+ /*
+ * Racy check. If we mistakenly see no partial slabs then we
+ * just allocate an empty slab. If we mistakenly try to get a
+ * partial slab then get_partials() will return NULL.
+ */
+ if (!n || !n->nr_partial)
+ return NULL;
+
+ spin_lock(&n->list_lock);
+ list_for_each_entry(page, &n->partial, lru)
+ if (lock_and_del_slab(n, page))
+ goto out;
+ page = NULL;
+out:
+ spin_unlock(&n->list_lock);
+ return page;
+}
+
+/*
+ * Get a page from somewhere. Search in increasing NUMA
+ * distances.
+ */
+static struct page *get_any_partial(struct kmem_cache *s, gfp_t flags)
+{
+#ifdef CONFIG_NUMA
+ struct zonelist *zonelist = &NODE_DATA(slab_node(current->mempolicy))
+ ->node_zonelists[gfp_zone(flags)];
+ struct zone **z;
+ struct page *page;
+
+ for (z = zonelist->zones; *z; z++) {
+ struct kmem_cache_node *n;
+
+ n = get_node(s, zone_to_nid(*z));
+
+ if (n && cpuset_zone_allowed_hardwall(*z, flags) &&
+ n->nr_partial > 2) {
+ page = get_partial_node(n);
+ if (page)
+ return page;
+ }
+ }
+#endif
+ return NULL;
+}
+
+/*
+ * Get a partial page, lock it and return it.
+ */
+static struct page *get_partial(struct kmem_cache *s, gfp_t flags, int node)
+{
+ struct page *page;
+ int searchnode = (node == -1) ? numa_node_id() : node;
+
+ page = get_partial_node(get_node(s, searchnode));
+ if (page || (flags & __GFP_THISNODE))
+ return page;
+
+ return get_any_partial(s, flags);
+}
+
+/*
+ * Move a page back to the lists.
+ *
+ * Must be called with the slab lock held.
+ *
+ * On exit the slab lock will have been dropped.
+ */
+static void __always_inline putback_slab(struct kmem_cache *s, struct page *page)
+{
+ if (page->inuse) {
+ if (page->inuse < s->objects)
+ add_partial(s, page);
+ slab_unlock(page);
+ } else {
+ slab_unlock(page);
+ discard_slab(s, page);
+ }
+}
+
+/*
+ * Remove the cpu slab
+ */
+static void __always_inline deactivate_slab(struct kmem_cache *s,
+ struct page *page, int cpu)
+{
+ s->cpu_slab[cpu] = NULL;
+ ClearPageActive(page);
+ ClearPageReferenced(page);
+
+ putback_slab(s, page);
+}
+
+static void flush_slab(struct kmem_cache *s, struct page *page, int cpu)
+{
+ slab_lock(page);
+ deactivate_slab(s, page, cpu);
+}
+
+/*
+ * Flush cpu slab.
+ * Called from IPI handler with interrupts disabled.
+ */
+static void __flush_cpu_slab(struct kmem_cache *s, int cpu)
+{
+ struct page *page = s->cpu_slab[cpu];
+
+ if (likely(page))
+ flush_slab(s, page, cpu);
+}
+
+static void flush_cpu_slab(void *d)
+{
+ struct kmem_cache *s = d;
+ int cpu = smp_processor_id();
+
+ __flush_cpu_slab(s, cpu);
+}
+
+#ifdef CONFIG_SMP
+/*
+ * Called from IPI to check and flush cpu slabs.
+ */
+static void check_flush_cpu_slab(void *private)
+{
+ struct kmem_cache *s = private;
+ int cpu = smp_processor_id();
+ struct page *page = s->cpu_slab[cpu];
+
+ if (page) {
+ if (!TestClearPageReferenced(page))
+ return;
+ flush_slab(s, page, cpu);
+ }
+ atomic_dec(&s->cpu_slabs);
+}
+
+/*
+ * Called from eventd
+ */
+static void flusher(struct work_struct *w)
+{
+ struct kmem_cache *s = container_of(w, struct kmem_cache, flush.work);
+
+ if (!mutex_trylock(&s->flushing))
+ return;
+
+ atomic_set(&s->cpu_slabs, num_online_cpus());
+ on_each_cpu(check_flush_cpu_slab, s, 1, 1);
+ if (atomic_read(&s->cpu_slabs))
+ schedule_delayed_work(&s->flush, 30 * HZ);
+ mutex_unlock(&s->flushing);
+}
+
+static void flush_all(struct kmem_cache *s)
+{
+ if (atomic_read(&s->cpu_slabs)) {
+ mutex_lock(&s->flushing);
+ cancel_delayed_work(&s->flush);
+ atomic_set(&s->cpu_slabs, 0);
+ on_each_cpu(flush_cpu_slab, s, 1, 1);
+ mutex_unlock(&s->flushing);
+ }
+}
+#else
+static void flush_all(struct kmem_cache *s)
+{
+ unsigned long flags;
+
+ local_irq_save(flags);
+ flush_cpu_slab(s);
+ local_irq_restore(flags);
+}
+#endif
+
+static __always_inline void *slab_alloc(struct kmem_cache *s,
+ gfp_t gfpflags, int node)
+{
+ struct page *page;
+ void **object;
+ unsigned long flags;
+ int cpu;
+
+ local_irq_save(flags);
+ cpu = smp_processor_id();
+ page = s->cpu_slab[cpu];
+ if (!page)
+ goto new_slab;
+
+ slab_lock(page);
+ if (unlikely(node != -1 && page_to_nid(page) != node))
+ goto another_slab;
+redo:
+ if (unlikely(!page->freelist))
+ goto another_slab;
+ object = page->freelist;
+ if (unlikely(PageError(page))) {
+ if (!alloc_object_checks(s, page, object))
+ goto another_slab;
+ if (s->flags & SLAB_STORE_USER)
+ set_tracking(s, object, 0);
+ }
+ page->inuse++;
+ page->freelist = object[page->offset];
+ SetPageReferenced(page);
+ slab_unlock(page);
+ local_irq_restore(flags);
+ return object;
+
+another_slab:
+ deactivate_slab(s, page, cpu);
+
+new_slab:
+ page = get_partial(s, gfpflags, node);
+ if (unlikely(!page)) {
+
+ page = new_slab(s, gfpflags, node);
+ if (!page) {
+ local_irq_restore(flags);
+ return NULL;
+ }
+
+ if (s->objects == 1) {
+ local_irq_restore(flags);
+ return page_address(page);
+ }
+
+ if (s->cpu_slab[cpu]) {
+ /*
+ * Someone else populated the cpu_slab while
+ * we enabled interrupts. The page may not
+ * be on the required node.
+ */
+ if (node == -1 ||
+ page_to_nid(s->cpu_slab[cpu]) == node) {
+ /*
+ * Current cpuslab is acceptable and we
+ * want the current one since its cache hot
+ */
+ discard_slab(s, page);
+ page = s->cpu_slab[cpu];
+ slab_lock(page);
+ goto redo;
+ }
+ flush_slab(s, s->cpu_slab[cpu], cpu);
+ }
+ slab_lock(page);
+ }
+
+ s->cpu_slab[cpu] = page;
+ SetPageActive(page);
+
+#ifdef CONFIG_SMP
+ if (!atomic_read(&s->cpu_slabs) && keventd_up()) {
+ atomic_inc(&s->cpu_slabs);
+ schedule_delayed_work(&s->flush, 30 * HZ);
+ }
+#endif
+ goto redo;
+}
+
+void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
+{
+ return slab_alloc(s, gfpflags, -1);
+}
+EXPORT_SYMBOL(kmem_cache_alloc);
+
+#ifdef CONFIG_NUMA
+void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node)
+{
+ return slab_alloc(s, gfpflags, node);
+}
+EXPORT_SYMBOL(kmem_cache_alloc_node);
+#endif
+
+void kmem_cache_free(struct kmem_cache *s, void *x)
+{
+ struct page * page;
+ void *prior;
+ void **object = (void *)x;
+ unsigned long flags;
+
+ if (!object)
+ return;
+
+ page = virt_to_page(x);
+
+ if (unlikely(PageCompound(page)))
+ page = page->first_page;
+
+ if (!s)
+ s = page->slab;
+
+ local_irq_save(flags);
+
+ if (unlikely(PageError(page)) && s->objects == 1)
+ goto single_object_slab;
+
+ slab_lock(page);
+
+ if (unlikely(PageError(page))) {
+ if (!free_object_checks(s, page, x))
+ goto out_unlock;
+ if (s->flags & SLAB_STORE_USER)
+ set_tracking(s, object, 1);
+ }
+
+ prior = object[page->offset] = page->freelist;
+ page->freelist = object;
+ page->inuse--;
+
+ if (likely(PageActive(page) || (page->inuse && prior)))
+ goto out_unlock;
+
+ if (!prior) {
+ /*
+ * The slab was full before. It will have one free
+ * object now. So move to the partial list.
+ */
+ add_partial(s, page);
+ goto out_unlock;
+ }
+
+ /*
+ * All object have been freed.
+ */
+ remove_partial(s, page);
+ slab_unlock(page);
+single_object_slab:
+ discard_slab(s, page);
+ local_irq_restore(flags);
+ return;
+
+out_unlock:
+ slab_unlock(page);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(kmem_cache_free);
+
+/* Figure out on which slab object the object resides */
+static __always_inline struct page *get_object_page(const void *x)
+{
+ struct page *page = virt_to_page(x);
+
+ if (unlikely(PageCompound(page)))
+ page = page->first_page;
+
+ if (!PageSlab(page))
+ return NULL;
+
+ return page;
+}
+
+/*
+ * kmem_cache_open produces objects aligned at "size" and the first object
+ * is placed at offset 0 in the slab (We have no metainformation on the
+ * slab, all slabs are in essence "off slab").
+ *
+ * In order to get the desired alignment one just needs to align the
+ * size.
+ *
+ * Notice that the allocation order determines the sizes of the per cpu
+ * caches. Each processor has always one slab available for allocations.
+ * Increasing the allocation order reduces the number of times that slabs
+ * must be moved on and off the partial lists and therefore may influence
+ * locking overhead.
+ *
+ * The offset is used to relocate the free list link in each object. It is
+ * therefore possible to move the free list link behind the object. This
+ * is necessary for RCU to work properly and also useful for debugging.
+ *
+ * No freelists are necessary if there is only one element per slab.
+ */
+
+/*
+ * Mininum order of slab pages. This influences locking overhead and slab
+ * fragmentation. A higher order reduces the number of partial slabs
+ * and increases the number of allocations possible without having to
+ * take the list_lock.
+ */
+static int slub_min_order = 0;
+
+/*
+ * Merge control. If this is set then no merging of slab caches will occur.
+ */
+static int slub_nomerge = 0;
+
+/*
+ * Debug settings:
+ */
+static int slub_debug = 0;
+
+static char *slub_debug_slabs = NULL;
+
+static int calculate_order(int size)
+{
+ int order;
+ int rem;
+
+ if ((size & (size -1)) == 0) {
+ /*
+ * We can use the page allocator if the requested size
+ * is compatible with the page sizes supported.
+ */
+ int order = fls(size) - 1 - PAGE_SHIFT;
+
+ if (order >= 0)
+ return order;
+ }
+
+ for (order = max(slub_min_order, fls(size - 1) - PAGE_SHIFT);
+ order < MAX_ORDER; order++) {
+ unsigned long slab_size = PAGE_SIZE << order;
+
+ if (slab_size < size)
+ continue;
+
+ rem = slab_size % size;
+
+ if (rem * 8 <= PAGE_SIZE << order)
+ break;
+
+ }
+ if (order >= MAX_ORDER)
+ return -E2BIG;
+ return order;
+}
+
+static unsigned long calculate_alignment(unsigned long flags,
+ unsigned long align)
+{
+ if (flags & SLAB_HWCACHE_ALIGN)
+ return L1_CACHE_BYTES;
+ if (flags & SLAB_MUST_HWCACHE_ALIGN)
+ return max(align, (unsigned long)L1_CACHE_BYTES);
+
+ if (align < ARCH_SLAB_MINALIGN)
+ return ARCH_SLAB_MINALIGN;
+
+ return ALIGN(align, sizeof(void *));
+}
+
+static void free_kmem_cache_nodes(struct kmem_cache *s)
+{
+#ifdef CONFIG_NUMA
+ int node;
+
+ for_each_online_node(node) {
+ struct kmem_cache_node *n = s->node[node];
+ if (n && n != &s->local_node)
+ kfree(n);
+ s->node[node] = NULL;
+ }
+#endif
+}
+
+static void init_kmem_cache_node(struct kmem_cache_node *n)
+{
+ memset(n, 0, sizeof(struct kmem_cache_node));
+ atomic_long_set(&n->nr_slabs, 0);
+ spin_lock_init(&n->list_lock);
+ INIT_LIST_HEAD(&n->partial);
+}
+
+static int init_kmem_cache_nodes(struct kmem_cache *s, gfp_t gfpflags)
+{
+#ifdef CONFIG_NUMA
+ int node;
+ int local_node;
+
+ if (slab_state == UP)
+ local_node = page_to_nid(virt_to_page(s));
+ else
+ local_node = 0;
+
+ for_each_online_node(node) {
+ struct kmem_cache_node *n;
+
+ if (local_node == node)
+ n = &s->local_node;
+ else
+ if (slab_state == DOWN) {
+ /*
+ * No kmalloc_node yet so do it by hand.
+ * We know that this is the first slab on the
+ * node for this slabcache. There are no concurrent
+ * accesses possible. Which simplifies things.
+ */
+ unsigned long flags;
+ struct page *page;
+
+ BUG_ON(s->size < sizeof(struct kmem_cache_node));
+ local_irq_save(flags);
+ page = new_slab(s, gfpflags, node);
+
+ BUG_ON(!page);
+ n = page->freelist;
+ page->freelist = *(void **)page->freelist;
+ page->inuse++;
+ local_irq_restore(flags);
+ } else
+ n = kmalloc_node(sizeof(struct kmem_cache_node),
+ gfpflags, node);
+
+ if (!n) {
+ free_kmem_cache_nodes(s);
+ return 0;
+ }
+
+ s->node[node] = n;
+ init_kmem_cache_node(n);
+
+ if (slab_state == DOWN)
+ atomic_long_inc(&n->nr_slabs);
+ }
+#else
+ init_kmem_cache_node(&s->local_node);
+#endif
+ return 1;
+}
+
+static int kmem_cache_open(struct kmem_cache *s, gfp_t gfpflags,
+ const char *name, size_t size,
+ size_t align, unsigned long flags,
+ void (*ctor)(void *, struct kmem_cache *, unsigned long),
+ void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+ int tentative_size;
+ BUG_ON(flags & SLUB_UNIMPLEMENTED);
+
+ memset(s, 0, kmem_size);
+
+ /*
+ * Enable debugging if selected on the kernel commandline.
+ */
+ if (slub_debug &&
+ (!slub_debug_slabs ||
+ strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs)) == 0))
+ flags |= slub_debug;
+
+ if ((flags & SLAB_POISON) &&((flags & SLAB_DESTROY_BY_RCU) ||
+ ctor || dtor)) {
+ if (!(slub_debug & SLAB_POISON))
+ printk(KERN_WARNING "SLUB %s: Clearing SLAB_POISON "
+ "because de/constructor exists.\n",
+ s->name);
+ flags &= ~SLAB_POISON;
+ }
+
+ tentative_size = ALIGN(size, calculate_alignment(align, flags));
+
+ /*
+ * Single object slabs are passed through to the page allocator
+ * and therefore the checks we can do are limited.
+ */
+ if (size * 2 > (PAGE_SIZE << calculate_order(tentative_size)))
+ flags &= ~(SLAB_RED_ZONE | SLAB_DEBUG_FREE | \
+ SLAB_STORE_USER | SLAB_POISON);
+
+ s->name = name;
+ s->ctor = ctor;
+ s->dtor = dtor;
+ s->objsize = size;
+ s->flags = flags;
+
+ size = ALIGN(size, sizeof(void *));
+
+ /*
+ * If we redzone then check if we have space through above
+ * alignment. If not then add an additional word, so
+ * that we have a guard value to check for overwrites.
+ */
+ if ((s->flags & SLAB_RED_ZONE) && size == s->objsize)
+ size += sizeof(void *);
+
+ s->inuse = size;
+
+ if (size * 2 < (PAGE_SIZE << calculate_order(size)) &&
+ ((flags & (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
+ ctor || dtor)) {
+ /*
+ * Relocate free pointer after the object if it is not
+ * permitted to overwrite the first word of the object on
+ * kmem_cache_free.
+ *
+ * This is the case if we do RCU, have a constructor or
+ * destructor or are poisoning the objects.
+ */
+ s->offset = size;
+ size += sizeof(void *);
+ }
+
+ if (flags & SLAB_STORE_USER)
+ size += 2 * sizeof(void *);
+
+ align = calculate_alignment(flags, align);
+
+ size = ALIGN(size, align);
+ s->size = size;
+
+ s->order = calculate_order(size);
+ if (s->order < 0)
+ goto error;
+
+ s->objects = (PAGE_SIZE << s->order) / size;
+ if (!s->objects || s->objects > 65535)
+ goto error;
+
+ s->refcount = 1;
+
+#ifdef CONFIG_SMP
+ mutex_init(&s->flushing);
+ atomic_set(&s->cpu_slabs, 0);
+ INIT_DELAYED_WORK(&s->flush, flusher);
+#endif
+ if (init_kmem_cache_nodes(s, gfpflags & ~SLUB_DMA)) {
+ return 1;
+ }
+error:
+ if (flags & SLAB_PANIC)
+ panic("Cannot create slab %s size=%lu realsize=%u "
+ "order=%u offset=%u flags=%lx\n",
+ s->name, (unsigned long)size, s->size, s->order,
+ s->offset, flags);
+ return 0;
+}
+EXPORT_SYMBOL(kmem_cache_open);
+
+/*
+ * Check if a given pointer is valid
+ */
+int kmem_ptr_validate(struct kmem_cache *s, const void *object)
+{
+ struct page * page;
+ void *addr;
+
+ page = get_object_page(object);
+
+ if (!page || s != page->slab)
+ /* No slab or wrong slab */
+ return 0;
+
+ addr = page_address(page);
+ if (object < addr || object >= addr + s->objects * s->size)
+ /* Out of bounds */
+ return 0;
+
+ if ((object - addr) & s->size)
+ /* Improperly aligned */
+ return 0;
+
+ /*
+ * We could also check here if the object is on the slabs freelist.
+ * But this would be too expensive and it seems that the main
+ * purpose of kmem_ptr_valid is to check if the object belongs
+ * to a certain slab.
+ */
+ return 1;
+}
+EXPORT_SYMBOL(kmem_ptr_validate);
+
+/*
+ * Determine the size of a slab object
+ */
+unsigned int kmem_cache_size(struct kmem_cache *s)
+{
+ return s->objsize;
+}
+EXPORT_SYMBOL(kmem_cache_size);
+
+const char *kmem_cache_name(struct kmem_cache *s)
+{
+ return s->name;
+}
+EXPORT_SYMBOL(kmem_cache_name);
+
+static int free_list(struct kmem_cache *s, struct kmem_cache_node *n,
+ struct list_head *list)
+{
+ int slabs_inuse = 0;
+ unsigned long flags;
+ struct page *page, *h;
+
+ spin_lock_irqsave(&n->list_lock, flags);
+ list_for_each_entry_safe(page, h, list, lru)
+ if (!page->inuse) {
+ list_del(&page->lru);
+ discard_slab(s, page);
+ } else
+ slabs_inuse++;
+ spin_unlock_irqrestore(&n->list_lock, flags);
+ return slabs_inuse;
+}
+
+/*
+ * Release all resources used by slab cache
+ * (if possible...)
+ */
+static int kmem_cache_close(struct kmem_cache *s)
+{
+ int node;
+
+ flush_all(s);
+
+ /* Attempt to free all objects */
+ for_each_online_node(node) {
+ struct kmem_cache_node *n = get_node(s, node);
+
+ free_list(s, n, &n->partial);
+ if (atomic_long_read(&n->nr_slabs))
+ return 1;
+ }
+ free_kmem_cache_nodes(s);
+ return 0;
+}
+EXPORT_SYMBOL(kmem_cache_close);
+
+/*
+ * Close a cache and release the kmem_cache structure
+ * (must be used for caches created using kmem_cache_create)
+ */
+void kmem_cache_destroy(struct kmem_cache *s)
+{
+ down_write(&slub_lock);
+ if (s->refcount)
+ s->refcount--;
+ else {
+ list_del(&s->list);
+ BUG_ON(kmem_cache_close(s));
+ kfree(s);
+ }
+ up_write(&slub_lock);
+}
+EXPORT_SYMBOL(kmem_cache_destroy);
+
+static unsigned long slab_objects(struct kmem_cache *s,
+ unsigned long *p_total, unsigned long *p_cpu_slabs,
+ unsigned long *p_partial, unsigned long *nodes)
+{
+ int nr_slabs = 0;
+ int nr_partial_slabs = 0;
+ int nr_cpu_slabs = 0;
+ int in_cpu_slabs = 0;
+ int in_partial_slabs = 0;
+ int cpu;
+ int node;
+ unsigned long flags;
+ struct page *page;
+
+ for_each_online_node(node) {
+ struct kmem_cache_node *n = get_node(s, node);
+
+ nr_slabs += atomic_read(&n->nr_slabs);
+ nr_partial_slabs += n->nr_partial;
+
+ nodes[node] = atomic_read(&n->nr_slabs) +
+ n->nr_partial;
+
+ spin_lock_irqsave(&n->list_lock, flags);
+ list_for_each_entry(page, &n->partial, lru)
+ in_partial_slabs += page->inuse;
+ spin_unlock_irqrestore(&n->list_lock, flags);
+ }
+
+ for_each_possible_cpu(cpu) {
+ page = s->cpu_slab[cpu];
+ if (page) {
+ nr_cpu_slabs++;
+ in_cpu_slabs += page->inuse;
+ nodes[page_to_nid(page)]++;
+ }
+ }
+
+ if (p_partial)
+ *p_partial = nr_partial_slabs;
+
+ if (p_cpu_slabs)
+ *p_cpu_slabs = nr_cpu_slabs;
+
+ if (p_total)
+ *p_total = nr_slabs;
+
+ return in_partial_slabs + in_cpu_slabs +
+ (nr_slabs - nr_partial_slabs - nr_cpu_slabs) * s->objects;
+}
+
+/********************************************************************
+ * Kmalloc subsystem
+ *******************************************************************/
+
+struct kmem_cache kmalloc_caches[KMALLOC_NR_CACHES] __cacheline_aligned;
+EXPORT_SYMBOL(kmalloc_caches);
+
+#ifdef CONFIG_ZONE_DMA
+static struct kmem_cache *kmalloc_caches_dma[KMALLOC_NR_CACHES];
+#endif
+
+static int __init setup_slub_min_order(char *str)
+{
+ get_option (&str, &slub_min_order);
+
+ return 1;
+}
+
+__setup("slub_min_order=", setup_slub_min_order);
+
+static int __init setup_slub_nomerge(char *str)
+{
+ slub_nomerge = 1;
+ return 1;
+}
+
+__setup("slub_nomerge", setup_slub_nomerge);
+
+static int __init setup_slub_debug(char *str)
+{
+ if (!str || *str != '=')
+ slub_debug = DEBUG_DEFAULT_FLAGS;
+ else {
+ str++;
+ if (*str == 0 || *str == ',')
+ slub_debug = DEBUG_DEFAULT_FLAGS;
+ else
+ for( ;*str && *str != ','; str++)
+ switch (*str) {
+ case 'f' : case 'F' : slub_debug |= SLAB_DEBUG_FREE;break;
+ case 'z' : case 'Z' : slub_debug |= SLAB_RED_ZONE;break;
+ case 'p' : case 'P' : slub_debug |= SLAB_POISON;break;
+ case 'u' : case 'U' : slub_debug |= SLAB_STORE_USER;break;
+ case 't' : case 'T' : slub_debug |= SLAB_TRACE;break;
+ default:
+ printk(KERN_CRIT "slub_debug option '%c' unknown. skipped\n",*str);
+ }
+ }
+
+ if (*str == ',')
+ slub_debug_slabs = str + 1;
+ return 1;
+}
+
+__setup("slub_debug", setup_slub_debug);
+
+static struct kmem_cache *create_kmalloc_cache(struct kmem_cache *s,
+ const char *name, int size, gfp_t gfp_flags)
+{
+ unsigned int flags = 0;
+
+ if (gfp_flags & SLUB_DMA)
+ flags = SLAB_CACHE_DMA;
+
+ down_write(&slub_lock);
+ if (!kmem_cache_open(s, gfp_flags, name, size, ARCH_KMALLOC_MINALIGN,
+ flags, NULL, NULL))
+ panic("Creation of kmalloc slab %s size=%d failed.\n",
+ name, size);
+ list_add(&s->list, &slab_caches);
+ up_write(&slub_lock);
+ return s;
+}
+
+static struct kmem_cache *get_slab(size_t size, gfp_t flags)
+{
+ int index = kmalloc_index(size) - KMALLOC_SHIFT_LOW;
+
+ /* SLAB allows allocations with zero size. So warn on those */
+ WARN_ON(size == 0);
+ /* Allocation too large? */
+ BUG_ON(index < 0);
+
+#ifdef CONFIG_ZONE_DMA
+ if ((flags & SLUB_DMA)) {
+ struct kmem_cache *s;
+ struct kmem_cache *x;
+ char *text;
+ size_t realsize;
+
+ s = kmalloc_caches_dma[index];
+ if (s)
+ return s;
+
+ /* Dynamically create dma cache */
+ x = kmalloc(kmem_size, flags & ~SLUB_DMA);
+ if (!x)
+ panic("Unable to allocate memory for dma cache\n");
+
+#ifdef KMALLOC_EXTRA
+ if (index <= KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW)
+#endif
+ realsize = 1 << (index + KMALLOC_SHIFT_LOW);
+#ifdef KMALLOC_EXTRA
+ else {
+ index -= KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW +1;
+ if (!index)
+ realsize = 96;
+ else
+ realsize = 192;
+ }
+#endif
+
+ text = kasprintf(flags & ~SLUB_DMA, "kmalloc_dma-%d",
+ (unsigned int)realsize);
+ s = create_kmalloc_cache(x, text, realsize, flags);
+ kmalloc_caches_dma[index] = s;
+ return s;
+ }
+#endif
+ return &kmalloc_caches[index];
+}
+
+void *__kmalloc(size_t size, gfp_t flags)
+{
+ return kmem_cache_alloc(get_slab(size, flags), flags);
+}
+EXPORT_SYMBOL(__kmalloc);
+
+#ifdef CONFIG_NUMA
+void *__kmalloc_node(size_t size, gfp_t flags, int node)
+{
+ return kmem_cache_alloc_node(get_slab(size, flags),
+ flags, node);
+}
+EXPORT_SYMBOL(__kmalloc_node);
+#endif
+
+size_t ksize(const void *object)
+{
+ struct page *page = get_object_page(object);
+ struct kmem_cache *s;
+
+ BUG_ON(!page);
+ s = page->slab;
+ BUG_ON(!s);
+ if (s->flags & SLAB_RED_ZONE)
+ return s->objsize;
+ if (s->flags & (SLAB_DESTROY_BY_RCU | SLAB_STORE_USER))
+ return s->inuse;
+ return s->size;
+}
+EXPORT_SYMBOL(ksize);
+
+void kfree(const void *object)
+{
+ kmem_cache_free(NULL, (void *)object);
+}
+EXPORT_SYMBOL(kfree);
+
+/**
+ * krealloc - reallocate memory. The contents will remain unchanged.
+ *
+ * @p: object to reallocate memory for.
+ * @new_size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate.
+ *
+ * The contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes. If @p is %NULL, krealloc()
+ * behaves exactly like kmalloc(). If @size is 0 and @p is not a
+ * %NULL pointer, the object pointed to is freed.
+ */
+void *krealloc(const void *p, size_t new_size, gfp_t flags)
+{
+ struct kmem_cache *new_cache;
+ void *ret;
+ struct page *page;
+
+ if (unlikely(!p))
+ return kmalloc(new_size, flags);
+
+ if (unlikely(!new_size)) {
+ kfree(p);
+ return NULL;
+ }
+
+ page = virt_to_page(p);
+
+ if (unlikely(PageCompound(page)))
+ page = page->first_page;
+
+ new_cache = get_slab(new_size, flags);
+
+ /*
+ * If new size fits in the current cache, bail out.
+ */
+ if (likely(page->slab == new_cache))
+ return (void *)p;
+
+ /*
+ * We are on the slow-path here so do not use __cache_alloc
+ * because it bloats kernel text.
+ */
+ ret = kmalloc(new_size, flags);
+ if (ret) {
+ memcpy(ret, p, min(new_size, ksize(p)));
+ kfree(p);
+ }
+ return ret;
+}
+EXPORT_SYMBOL(krealloc);
+
+/********************************************************************
+ * Basic setup of slabs
+ *******************************************************************/
+
+void __init kmem_cache_init(void)
+{
+ int i;
+ int kmem_cache_node_cache =
+ kmalloc_index(sizeof(struct kmem_cache_node));
+
+ BUG_ON(kmem_cache_node_cache < 0 ||
+ kmem_cache_node_cache > KMALLOC_SHIFT_HIGH);
+
+ /*
+ * Must first have the slab cache available for the allocations of the
+ * struct kmalloc_cache_node's. There is special bootstrap code in
+ * kmem_cache_open for the situation when slab_state == DOWN.
+ */
+ create_kmalloc_cache(&kmalloc_caches[kmem_cache_node_cache
+ - KMALLOC_SHIFT_LOW],
+ "kmalloc",
+ 1 << kmem_cache_node_cache,
+ GFP_KERNEL);
+
+ /* Now we are able to allocate the per node structures */
+ slab_state = PARTIAL;
+
+ for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
+ if (i == kmem_cache_node_cache)
+ continue;
+
+ create_kmalloc_cache(
+ &kmalloc_caches[i - KMALLOC_SHIFT_LOW],
+ "kmalloc", 1 << i, GFP_KERNEL);
+ }
+
+#ifdef KMALLOC_EXTRA
+ /* Caches that are not of the two-to-the-power-of size */
+ create_kmalloc_cache(&kmalloc_caches
+ [KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW + 1],
+ "kmalloc-96", 96, GFP_KERNEL);
+ create_kmalloc_cache(&kmalloc_caches
+ [KMALLOC_SHIFT_HIGH - KMALLOC_SHIFT_LOW + 2],
+ "kmalloc-192", 192, GFP_KERNEL);
+#endif
+ slab_state = UP;
+
+ /* Provide the correct kmalloc names now that the caches are up */
+ for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
+ char *name = kasprintf(GFP_KERNEL, "kmalloc-%d", 1 << i);
+
+ BUG_ON(!name);
+ kmalloc_caches[i - KMALLOC_SHIFT_LOW].name = name;
+ };
+
+#ifdef CONFIG_SMP
+ register_cpu_notifier(&slab_notifier);
+#endif
+ if (nr_cpu_ids) /* Remove when nr_cpu_ids was fixed ! */
+ kmem_size = offsetof(struct kmem_cache, cpu_slab)
+ + nr_cpu_ids * sizeof(struct page *);
+
+ printk(KERN_INFO "SLUB V4: General Slabs=%d, HW alignment=%d, Processors=%d, Nodes=%d\n",
+ KMALLOC_SHIFT_HIGH + KMALLOC_EXTRAS + 1 - KMALLOC_SHIFT_LOW,
+ L1_CACHE_BYTES, nr_cpu_ids, nr_node_ids);
+}
+
+static struct kmem_cache *kmem_cache_dup(struct kmem_cache *s,
+ gfp_t flags, const char *name)
+{
+ if (s->refcount == 1) {
+ s->refcount++;
+ if (!s->aliases)
+ s->aliases = kstrdup(name, flags);
+ else {
+ char *x = s->aliases;
+ s->aliases = kasprintf(flags, "%s/%s", s->aliases, name);
+ kfree(x);
+ }
+ } else
+ s = NULL;
+ return s;
+}
+
+/*
+ * Find a mergeable slab cache
+ */
+static struct kmem_cache *find_mergeable(size_t size,
+ size_t align, unsigned long flags,
+ void (*ctor)(void *, struct kmem_cache *, unsigned long),
+ void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+ struct list_head *h;
+
+ if (slub_nomerge || (flags & SLUB_NEVER_MERGE))
+ return NULL;
+
+ if (ctor || dtor)
+ return NULL;
+
+ size = ALIGN(size, sizeof(void *));
+ align = calculate_alignment(flags, align);
+ size = ALIGN(size, align);
+
+ list_for_each(h, &slab_caches) {
+ struct kmem_cache *s =
+ container_of(h, struct kmem_cache, list);
+
+ if (size > s->size)
+ continue;
+
+ if (s->flags & SLUB_NEVER_MERGE)
+ continue;
+
+ if (s->dtor || s->ctor)
+ continue;
+
+ if (((flags | slub_debug) & SLUB_MERGE_SAME) !=
+ (s->flags & SLUB_MERGE_SAME))
+ continue;
+ /*
+ * Check if alignment is compatible.
+ * Courtesy of Adrian Drzewiecki
+ */
+ if ((s->size & ~(align -1)) != s->size)
+ continue;
+
+ if (s->size - size >= sizeof(void *))
+ continue;
+
+ return s;
+ }
+ return NULL;
+}
+
+struct kmem_cache *kmem_cache_create(const char *name, size_t size,
+ size_t align, unsigned long flags,
+ void (*ctor)(void *, struct kmem_cache *, unsigned long),
+ void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+ struct kmem_cache *s;
+
+ down_write(&slub_lock);
+ s = find_mergeable(size, align, flags, dtor, ctor);
+ if (s) {
+ s = kmem_cache_dup(s, GFP_KERNEL, name);
+ if (s)
+ goto out;
+
+ }
+ s = kmalloc(kmem_size, GFP_KERNEL);
+ if (s && kmem_cache_open(s, GFP_KERNEL, name,
+ size, align, flags, ctor, dtor)) {
+ list_add(&s->list, &slab_caches);
+ } else
+ kfree(s);
+out:
+ up_write(&slub_lock);
+ return s;
+}
+EXPORT_SYMBOL(kmem_cache_create);
+
+void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t flags)
+{
+ void *x;
+
+ x = kmem_cache_alloc(s, flags);
+ if (x)
+ memset(x, 0, s->objsize);
+ return x;
+}
+EXPORT_SYMBOL(kmem_cache_zalloc);
+
+/********************************************************************
+ * Slab proc interface
+ *******************************************************************/
+
+static void print_slubinfo_header(struct seq_file *m)
+{
+ /*
+ * Output format version, so at least we can change it
+ * without _too_ many complaints.
+ */
+ seq_puts(m, "slubinfo - version: 1.0\n");
+ seq_puts(m, "# name <objects> <order> <objsize> <objperslab>"
+ " <slabs>/<partial>/<cpu> <flags>");
+#ifdef CONFIG_NUMA
+ seq_puts(m, " <nodes>");
+#endif
+ seq_putc(m, '\n');
+}
+
+static void *s_start(struct seq_file *m, loff_t *pos)
+{
+ loff_t n = *pos;
+ struct list_head *p;
+
+ down_read(&slub_lock);
+ if (!n)
+ print_slubinfo_header(m);
+ p = slab_caches.next;
+ while (n--) {
+ p = p->next;
+ if (p == &slab_caches)
+ return NULL;
+ }
+ return list_entry(p, struct kmem_cache, list);
+}
+
+static void *s_next(struct seq_file *m, void *p, loff_t *pos)
+{
+ struct kmem_cache *s = p;
+ ++*pos;
+ return s->list.next == &slab_caches ?
+ NULL : list_entry(s->list.next, struct kmem_cache, list);
+}
+
+static void s_stop(struct seq_file *m, void *p)
+{
+ up_read(&slub_lock);
+}
+
+static void display_nodes(struct seq_file *m, unsigned long *nodes)
+{
+#ifdef CONFIG_NUMA
+ int node;
+
+ for_each_online_node(node)
+ if (nodes[node])
+ seq_printf(m, " N%d=%lu", node, nodes[node]);
+#endif
+}
+
+static int s_show(struct seq_file *m, void *p)
+{
+ struct kmem_cache *s = p;
+ unsigned long total_slabs;
+ unsigned long cpu_slabs;
+ unsigned long partial_slabs;
+ unsigned long objects;
+ unsigned char options[17];
+ char *d = options;
+ char *x;
+ unsigned long nodes[nr_node_ids];
+
+ objects = slab_objects(s, &total_slabs, &cpu_slabs,
+ &partial_slabs, nodes);
+ if (s->ctor)
+ *d++ = 'C';
+ if (s->dtor)
+ *d++ = 'D';
+ if (s->flags & SLAB_DESTROY_BY_RCU)
+ *d++ = 'R';
+ if (s->flags & SLAB_MEM_SPREAD)
+ *d++ = 'S';
+ if (s->flags & SLAB_CACHE_DMA)
+ *d++ = 'd';
+ if (s->flags & SLAB_RECLAIM_ACCOUNT)
+ *d++ = 'r';
+ if (s->flags & SLAB_PANIC)
+ *d++ = 'p';
+ if (s->flags & SLAB_HWCACHE_ALIGN)
+ *d++ = 'a';
+ if (s->flags & SLAB_MUST_HWCACHE_ALIGN)
+ *d++ = 'A';
+ if (s->flags & SLAB_DEBUG_FREE)
+ *d++ = 'F';
+ if (s->flags & SLAB_DEBUG_INITIAL)
+ *d++ = 'I';
+ if (s->flags & SLAB_STORE_USER)
+ *d++ = 'U';
+ if (s->flags & SLAB_RED_ZONE)
+ *d++ = 'Z';
+ if (s->flags & SLAB_POISON)
+ *d++ = 'P';
+ if (s->flags & SLAB_TRACE)
+ *d++ = 'T';
+
+ *d = 0;
+
+ x = kasprintf(GFP_KERNEL, "%lu/%lu/%lu", total_slabs, partial_slabs,
+ cpu_slabs);
+
+ seq_printf(m, "%-21s %6lu %1d %6u %4d %12s %7s",
+ s->name, objects, s->order, s->objsize, s->objects, x, options);
+
+ kfree(x);
+ display_nodes(m, nodes);
+ if (s->aliases) {
+ seq_putc(m, ' ');
+ seq_puts(m, s->aliases);
+ }
+ seq_putc(m, '\n');
+ return 0;
+}
+
+/*
+ * slabinfo_op - iterator that generates /proc/slabinfo
+ */
+struct seq_operations slubinfo_op = {
+ .start = s_start,
+ .next = s_next,
+ .stop = s_stop,
+ .show = s_show,
+};
+
+#ifdef CONFIG_SMP
+static void for_all_slabs(void (*func)(struct kmem_cache *, int), int cpu)
+{
+ struct list_head *h;
+
+ down_read(&slub_lock);
+ list_for_each(h, &slab_caches) {
+ struct kmem_cache *s =
+ container_of(h, struct kmem_cache, list);
+
+ func(s, cpu);
+ }
+ up_read(&slub_lock);
+}
+
+/*
+ * Use the cpu notifier to insure that the thresholds are recalculated
+ * when necessary.
+ */
+static int __cpuinit slab_cpuup_callback(struct notifier_block *nfb,
+ unsigned long action, void *hcpu)
+{
+ long cpu = (long)hcpu;
+
+ switch (action) {
+ case CPU_UP_CANCELED:
+ case CPU_DEAD:
+ for_all_slabs(__flush_cpu_slab, cpu);
+ break;
+ default:
+ break;
+ }
+ return NOTIFY_OK;
+}
+
+static struct notifier_block __cpuinitdata slab_notifier =
+ { &slab_cpuup_callback, NULL, 0 };
+
+#endif
+
+/***************************************************************
+ * Compatiblility definitions
+ **************************************************************/
+
+int kmem_cache_shrink(struct kmem_cache *s)
+{
+ flush_all(s);
+ return 0;
+}
+EXPORT_SYMBOL(kmem_cache_shrink);
+
+#ifdef CONFIG_NUMA
+
+/*****************************************************************
+ * Generic reaper used to support the page allocator
+ * (the cpu slabs are reaped by a per slab workqueue).
+ *
+ * Maybe move this to the page allocator?
+ ****************************************************************/
+
+static DEFINE_PER_CPU(unsigned long, reap_node);
+
+static void init_reap_node(int cpu)
+{
+ int node;
+
+ node = next_node(cpu_to_node(cpu), node_online_map);
+ if (node == MAX_NUMNODES)
+ node = first_node(node_online_map);
+
+ __get_cpu_var(reap_node) = node;
+}
+
+static void next_reap_node(void)
+{
+ int node = __get_cpu_var(reap_node);
+
+ /*
+ * Also drain per cpu pages on remote zones
+ */
+ if (node != numa_node_id())
+ drain_node_pages(node);
+
+ node = next_node(node, node_online_map);
+ if (unlikely(node >= MAX_NUMNODES))
+ node = first_node(node_online_map);
+ __get_cpu_var(reap_node) = node;
+}
+#else
+#define init_reap_node(cpu) do { } while (0)
+#define next_reap_node(void) do { } while (0)
+#endif
+
+#define REAPTIMEOUT_CPUC (2*HZ)
+
+#ifdef CONFIG_SMP
+static DEFINE_PER_CPU(struct delayed_work, reap_work);
+
+static void cache_reap(struct work_struct *unused)
+{
+ next_reap_node();
+ refresh_cpu_vm_stats(smp_processor_id());
+ schedule_delayed_work(&__get_cpu_var(reap_work),
+ REAPTIMEOUT_CPUC);
+}
+
+static void __devinit start_cpu_timer(int cpu)
+{
+ struct delayed_work *reap_work = &per_cpu(reap_work, cpu);
+
+ /*
+ * When this gets called from do_initcalls via cpucache_init(),
+ * init_workqueues() has already run, so keventd will be setup
+ * at that time.
+ */
+ if (keventd_up() && reap_work->work.func == NULL) {
+ init_reap_node(cpu);
+ INIT_DELAYED_WORK(reap_work, cache_reap);
+ schedule_delayed_work_on(cpu, reap_work, HZ + 3 * cpu);
+ }
+}
+
+static int __init cpucache_init(void)
+{
+ int cpu;
+
+ /*
+ * Register the timers that drain pcp pages and update vm statistics
+ */
+ for_each_online_cpu(cpu)
+ start_cpu_timer(cpu);
+ return 0;
+}
+__initcall(cpucache_init);
+#endif
+
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 2:35 [SLUB 0/3] SLUB: The unqueued slab allocator V4 Christoph Lameter
2007-03-07 2:35 ` [SLUB 1/3] SLUB core Christoph Lameter
@ 2007-03-07 2:35 ` Christoph Lameter
2007-03-07 2:40 ` Matt Mackall
2007-03-07 9:01 ` Peter Zijlstra
2007-03-07 2:35 ` [SLUB 3/3] Guarantee minimum number of objects in a slab Christoph Lameter
2007-03-08 10:54 ` [SLUB 0/3] SLUB: The unqueued slab allocator V4 Mel Gorman
3 siblings, 2 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 2:35 UTC (permalink / raw)
To: akpm
Cc: Marcelo Tosatti, linux-kernel, linux-mm, Christoph Lameter, mpm,
Manfred Spraul
Unlimited kmalloc size and removal of general caches >=4.
We can directly use the page allocator for all allocations 4K and larger. This
means that no general slabs are necessary and the size of the allocation passed
to kmalloc() can be arbitrarily large. Remove the useless general caches over 4k.
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Index: linux-2.6.21-rc2-mm1/mm/slub.c
===================================================================
--- linux-2.6.21-rc2-mm1.orig/mm/slub.c 2007-03-06 17:56:50.000000000 -0800
+++ linux-2.6.21-rc2-mm1/mm/slub.c 2007-03-06 17:57:11.000000000 -0800
@@ -1101,6 +1101,13 @@ void kmem_cache_free(struct kmem_cache *
if (unlikely(PageCompound(page)))
page = page->first_page;
+ if (unlikely(!PageSlab(page))) {
+ if (x == page_address(page)) {
+ put_page(page);
+ return;
+ }
+ }
+
if (!s)
s = page->slab;
@@ -1678,7 +1685,8 @@ static struct kmem_cache *get_slab(size_
/* SLAB allows allocations with zero size. So warn on those */
WARN_ON(size == 0);
/* Allocation too large? */
- BUG_ON(index < 0);
+ if (index < 0)
+ return NULL;
#ifdef CONFIG_ZONE_DMA
if ((flags & SLUB_DMA)) {
@@ -1722,15 +1730,32 @@ static struct kmem_cache *get_slab(size_
void *__kmalloc(size_t size, gfp_t flags)
{
- return kmem_cache_alloc(get_slab(size, flags), flags);
+ struct kmem_cache *s = get_slab(size, flags);
+ struct page *page;
+
+ if (s)
+ return kmem_cache_alloc(s, flags);
+
+ page = alloc_pages(flags, get_order(size));
+ if (!page)
+ return NULL;
+ return page_address(page);
}
EXPORT_SYMBOL(__kmalloc);
#ifdef CONFIG_NUMA
void *__kmalloc_node(size_t size, gfp_t flags, int node)
{
- return kmem_cache_alloc_node(get_slab(size, flags),
- flags, node);
+ struct kmem_cache *s = get_slab(size, flags);
+ struct page *page;
+
+ if (s)
+ return kmem_cache_alloc_node(s, flags, node);
+
+ page = alloc_pages_node(node, flags, get_order(size));
+ if (!page)
+ return NULL;
+ return page_address(page);
}
EXPORT_SYMBOL(__kmalloc_node);
#endif
Index: linux-2.6.21-rc2-mm1/include/linux/slub_def.h
===================================================================
--- linux-2.6.21-rc2-mm1.orig/include/linux/slub_def.h 2007-03-06 17:56:14.000000000 -0800
+++ linux-2.6.21-rc2-mm1/include/linux/slub_def.h 2007-03-06 17:57:11.000000000 -0800
@@ -55,7 +55,7 @@ struct kmem_cache {
*/
#define KMALLOC_SHIFT_LOW 3
-#define KMALLOC_SHIFT_HIGH 18
+#define KMALLOC_SHIFT_HIGH 11
#if L1_CACHE_BYTES <= 64
#define KMALLOC_EXTRAS 2
@@ -93,13 +93,6 @@ static inline int kmalloc_index(int size
if (size <= 512) return 9;
if (size <= 1024) return 10;
if (size <= 2048) return 11;
- if (size <= 4096) return 12;
- if (size <= 8 * 1024) return 13;
- if (size <= 16 * 1024) return 14;
- if (size <= 32 * 1024) return 15;
- if (size <= 64 * 1024) return 16;
- if (size <= 128 * 1024) return 17;
- if (size <= 256 * 1024) return 18;
return -1;
}
@@ -113,14 +106,8 @@ static inline struct kmem_cache *kmalloc
{
int index = kmalloc_index(size) - KMALLOC_SHIFT_LOW;
- if (index < 0) {
- /*
- * Generate a link failure. Would be great if we could
- * do something to stop the compile here.
- */
- extern void __kmalloc_size_too_large(void);
- __kmalloc_size_too_large();
- }
+ if (index < 0)
+ return NULL;
return &kmalloc_caches[index];
}
@@ -136,9 +123,10 @@ static inline void *kmalloc(size_t size,
if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
struct kmem_cache *s = kmalloc_slab(size);
- return kmem_cache_alloc(s, flags);
- } else
- return __kmalloc(size, flags);
+ if (s)
+ return kmem_cache_alloc(s, flags);
+ }
+ return __kmalloc(size, flags);
}
static inline void *kzalloc(size_t size, gfp_t flags)
@@ -146,9 +134,10 @@ static inline void *kzalloc(size_t size,
if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
struct kmem_cache *s = kmalloc_slab(size);
- return kmem_cache_zalloc(s, flags);
- } else
- return __kzalloc(size, flags);
+ if (s)
+ return kmem_cache_zalloc(s, flags);
+ }
+ return __kzalloc(size, flags);
}
#ifdef CONFIG_NUMA
@@ -159,9 +148,10 @@ static inline void *kmalloc_node(size_t
if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
struct kmem_cache *s = kmalloc_slab(size);
- return kmem_cache_alloc_node(s, flags, node);
- } else
- return __kmalloc_node(size, flags, node);
+ if (s)
+ return kmem_cache_alloc_node(s, flags, node);
+ }
+ return __kmalloc_node(size, flags, node);
}
#endif
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* [SLUB 3/3] Guarantee minimum number of objects in a slab
2007-03-07 2:35 [SLUB 0/3] SLUB: The unqueued slab allocator V4 Christoph Lameter
2007-03-07 2:35 ` [SLUB 1/3] SLUB core Christoph Lameter
2007-03-07 2:35 ` [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs Christoph Lameter
@ 2007-03-07 2:35 ` Christoph Lameter
2007-03-08 10:54 ` [SLUB 0/3] SLUB: The unqueued slab allocator V4 Mel Gorman
3 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 2:35 UTC (permalink / raw)
To: akpm
Cc: Marcelo Tosatti, linux-kernel, linux-mm, Christoph Lameter, mpm,
Manfred Spraul
Guarantee a mininum number of objects per slab
The number of objects per slab is important for SLUB because it determines
the number of allocations that can be performed without having to consult
per node slab lists. Add another boot option "min_objects=xx" that
allows the configuration of the objects per slab. This is similar
to SLABS queue configurations.
Set the default of objects to 4. This will increase the page order for
certain slab objects.
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Index: linux-2.6.21-rc2-mm1/mm/slub.c
===================================================================
--- linux-2.6.21-rc2-mm1.orig/mm/slub.c 2007-03-06 17:57:11.000000000 -0800
+++ linux-2.6.21-rc2-mm1/mm/slub.c 2007-03-06 17:57:15.000000000 -0800
@@ -1201,6 +1201,12 @@ static __always_inline struct page *get_
static int slub_min_order = 0;
/*
+ * Minumum number of objects per slab. This is necessary in order to
+ * reduce locking overhead. Similar to the queue size in SLAB.
+ */
+static int slub_min_objects = 4;
+
+/*
* Merge control. If this is set then no merging of slab caches will occur.
*/
static int slub_nomerge = 0;
@@ -1232,7 +1238,7 @@ static int calculate_order(int size)
order < MAX_ORDER; order++) {
unsigned long slab_size = PAGE_SIZE << order;
- if (slab_size < size)
+ if (slab_size < slub_min_objects * size)
continue;
rem = slab_size % size;
@@ -1624,6 +1630,15 @@ static int __init setup_slub_min_order(c
__setup("slub_min_order=", setup_slub_min_order);
+static int __init setup_slub_min_objects(char *str)
+{
+ get_option (&str, &slub_min_objects);
+
+ return 1;
+}
+
+__setup("slub_min_objects=", setup_slub_min_objects);
+
static int __init setup_slub_nomerge(char *str)
{
slub_nomerge = 1;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 2:35 ` [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs Christoph Lameter
@ 2007-03-07 2:40 ` Matt Mackall
2007-03-07 3:22 ` Christoph Lameter
2007-03-07 9:01 ` Peter Zijlstra
1 sibling, 1 reply; 21+ messages in thread
From: Matt Mackall @ 2007-03-07 2:40 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, Manfred Spraul
On Tue, Mar 06, 2007 at 06:35:16PM -0800, Christoph Lameter wrote:
> Unlimited kmalloc size and removal of general caches >=4.
>
> We can directly use the page allocator for all allocations 4K and larger. This
> means that no general slabs are necessary and the size of the allocation passed
> to kmalloc() can be arbitrarily large. Remove the useless general caches over 4k.
I've been meaning to do this in SLOB as well. Perhaps it warrants
doing in stock kmalloc? I've got a grand total of 18 of these objects
here.
The downside is this makes them suddenly disappear off the slabinfo
radar.
--
Mathematics is the supreme nostalgia of our time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 2:40 ` Matt Mackall
@ 2007-03-07 3:22 ` Christoph Lameter
0 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 3:22 UTC (permalink / raw)
To: Matt Mackall
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, Manfred Spraul
On Tue, 6 Mar 2007, Matt Mackall wrote:
> I've been meaning to do this in SLOB as well. Perhaps it warrants
> doing in stock kmalloc? I've got a grand total of 18 of these objects
> here.
The number increases with the number numa nodes. We have had trouble with
the maximum kmalloc size before and this will get rid of it for good.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 2:35 ` [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs Christoph Lameter
2007-03-07 2:40 ` Matt Mackall
@ 2007-03-07 9:01 ` Peter Zijlstra
2007-03-07 15:34 ` Christoph Lameter
1 sibling, 1 reply; 21+ messages in thread
From: Peter Zijlstra @ 2007-03-07 9:01 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Tue, 2007-03-06 at 18:35 -0800, Christoph Lameter wrote:
> Unlimited kmalloc size and removal of general caches >=4.
>
> We can directly use the page allocator for all allocations 4K and larger. This
> means that no general slabs are necessary and the size of the allocation passed
> to kmalloc() can be arbitrarily large. Remove the useless general caches over 4k.
>
> Index: linux-2.6.21-rc2-mm1/include/linux/slub_def.h
> ===================================================================
> --- linux-2.6.21-rc2-mm1.orig/include/linux/slub_def.h 2007-03-06 17:56:14.000000000 -0800
> +++ linux-2.6.21-rc2-mm1/include/linux/slub_def.h 2007-03-06 17:57:11.000000000 -0800
> @@ -55,7 +55,7 @@ struct kmem_cache {
> */
> #define KMALLOC_SHIFT_LOW 3
>
> -#define KMALLOC_SHIFT_HIGH 18
> +#define KMALLOC_SHIFT_HIGH 11
>
> #if L1_CACHE_BYTES <= 64
> #define KMALLOC_EXTRAS 2
> @@ -93,13 +93,6 @@ static inline int kmalloc_index(int size
> if (size <= 512) return 9;
> if (size <= 1024) return 10;
> if (size <= 2048) return 11;
> - if (size <= 4096) return 12;
> - if (size <= 8 * 1024) return 13;
> - if (size <= 16 * 1024) return 14;
> - if (size <= 32 * 1024) return 15;
> - if (size <= 64 * 1024) return 16;
> - if (size <= 128 * 1024) return 17;
> - if (size <= 256 * 1024) return 18;
> return -1;
> }
Perhaps so something with PAGE_SIZE here, as you know there are
platforms/configs where PAGE_SIZE != 4k :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 9:01 ` Peter Zijlstra
@ 2007-03-07 15:34 ` Christoph Lameter
2007-03-07 18:03 ` Matt Mackall
0 siblings, 1 reply; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 15:34 UTC (permalink / raw)
To: Peter Zijlstra
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Wed, 7 Mar 2007, Peter Zijlstra wrote:
> > return -1;
> > }
>
> Perhaps so something with PAGE_SIZE here, as you know there are
> platforms/configs where PAGE_SIZE != 4k :-)
Any allocation > 2k just uses a regular allocation which will waste space.
I have a patch here to make this dependent on page size using a loop. The
problem is that it does not work with some versions of gcc. On the
other hand we really need this since one arch can
actually have an order 22 page size!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 15:34 ` Christoph Lameter
@ 2007-03-07 18:03 ` Matt Mackall
2007-03-07 18:23 ` Christoph Lameter
0 siblings, 1 reply; 21+ messages in thread
From: Matt Mackall @ 2007-03-07 18:03 UTC (permalink / raw)
To: Christoph Lameter
Cc: Peter Zijlstra, akpm, Marcelo Tosatti, linux-kernel, linux-mm,
Manfred Spraul
On Wed, Mar 07, 2007 at 07:34:38AM -0800, Christoph Lameter wrote:
> On Wed, 7 Mar 2007, Peter Zijlstra wrote:
>
> > > return -1;
> > > }
> >
> > Perhaps so something with PAGE_SIZE here, as you know there are
> > platforms/configs where PAGE_SIZE != 4k :-)
>
> Any allocation > 2k just uses a regular allocation which will waste space.
>
> I have a patch here to make this dependent on page size using a loop. The
> problem is that it does not work with some versions of gcc. On the
> other hand we really need this since one arch can
> actually have an order 22 page size!
You don't need a loop, you need an if (s >= PAGE_SIZE) at the head of
your static list.
--
Mathematics is the supreme nostalgia of our time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs
2007-03-07 18:03 ` Matt Mackall
@ 2007-03-07 18:23 ` Christoph Lameter
0 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-07 18:23 UTC (permalink / raw)
To: Matt Mackall
Cc: Peter Zijlstra, akpm, Marcelo Tosatti, linux-kernel, linux-mm,
Manfred Spraul
On Wed, 7 Mar 2007, Matt Mackall wrote:
> > I have a patch here to make this dependent on page size using a loop. The
> > problem is that it does not work with some versions of gcc. On the
> > other hand we really need this since one arch can
> > actually have an order 22 page size!
>
> You don't need a loop, you need an if (s >= PAGE_SIZE) at the head of
> your static list.
As I just said: PAGE_SIZE may be quite high. So I would need a looong
static list. We already check for the size being bigger than 2048 which is
half the usual page size. Anything larger will get passed through.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-07 2:35 [SLUB 0/3] SLUB: The unqueued slab allocator V4 Christoph Lameter
` (2 preceding siblings ...)
2007-03-07 2:35 ` [SLUB 3/3] Guarantee minimum number of objects in a slab Christoph Lameter
@ 2007-03-08 10:54 ` Mel Gorman
2007-03-08 16:48 ` Christoph Lameter
2007-03-08 17:46 ` Christoph Lameter
3 siblings, 2 replies; 21+ messages in thread
From: Mel Gorman @ 2007-03-08 10:54 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Tue, 6 Mar 2007, Christoph Lameter wrote:
> [PATCH] SLUB The unqueued slab allocator v4
>
Hi Christoph,
I shoved these patches through a few tests on x86, x86_64, ia64 and ppc64
last night to see how they got on. I enabled slub_debug to catch any
suprises that may be creeping about.
The results are mixed.
On x86_64, it completed successfully and looked reliable. There was a 5%
performance loss on kernbench and aim9 figures were way down. However,
with slub_debug enabled, I would expect that so it's not a fair comparison
performance wise. I'll rerun the tests without debug and see what it looks
like if you're interested and do not think it's too early to worry about
performance instead of clarity. This is what I have for bl6-13 (machine
appears on test.kernel.org so additional details are there).
KernBench Comparison
--------------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-list-based
%diff
User CPU time 84.32 86.03 -2.03%
System CPU time 32.97 38.21 -15.89%
Total CPU time 117.29 124.24 -5.93%
Elapsed time 34.95 37.31 -6.75%
AIM9 Comparison
---------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-list-based
1 creat-clo 160706.55 62918.54 -97788.01 -60.85% File Creations and Closes/second
2 page_test 190371.67 204050.99 13679.32 7.19% System Allocations & Pages/second
3 brk_test 2320679.89 1923512.75 -397167.14 -17.11% System Memory Allocations/second
4 jmp_test 16391869.38 16380353.27 -11516.11 -0.07% Non-local gotos/second
5 signal_test 492234.63 235710.71 -256523.92 -52.11% Signal Traps/second
6 exec_test 232.26 220.88 -11.38 -4.90% Program Loads/second
7 fork_test 4514.25 3609.40 -904.85 -20.04% Task Creations/second
8 link_test 53639.76 26925.91 -26713.85 -49.80% Link/Unlink Pairs/second
IA64 (machine not visible on TKO) curiously did not exhibit the same
problems on kernbench for Total CPU time which is very unexpected but you
can see the System CPU times. The AIM9 figures were a bit of an upset but
again, I blame slub_debug being enabled
KernBench Comparison
--------------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-list-based %diff
User CPU time 1084.64 1033.46 4.72%
System CPU time 73.38 84.14 -14.66%
Total CPU time 1158.02 1117.6 3.49%
Elapsed time 307.00 291.29 5.12%
AIM9 Comparison
---------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-list-based
1 creat-clo 425460.75 137709.84 -287750.91 -67.63% File Creations and Closes/second
2 page_test 2097119.26 2373083.49 275964.23 13.16% System Allocations & Pages/second
3 brk_test 7008395.33 3787961.51 -3220433.82 -45.95% System Memory Allocations/second
4 jmp_test 12226295.31 12254744.03 28448.72 0.23% Non-local gotos/second
5 signal_test 1271126.28 334357.29 -936768.99 -73.70% Signal Traps/second
6 exec_test 395.54 349.00 -46.54 -11.77% Program Loads/second
7 fork_test 13218.23 8822.93 -4395.30 -33.25% Task Creations/second
8 link_test 64776.04 7410.75 -57365.29 -88.56% Link/Unlink Pairs/second
(as an aside, the succes rates for high-order allocations are lower with
SLUB. Again, I blame slub_debug. I know that enabling SLAB_DEBUG has
similar effects because of red-zoning and the like)
Now, the bad news. This exploded on ppc64. It started going wrong early in
the boot process and got worse. I haven't looked closely as to why yet as
there is other stuff on my plate but I've included a console log that
might be some use to you. If you think you have a fix for it, feel free to
send it on and I'll give it a test.
Config file read, 1024 bytes
Welcome
Welcome to yaboot version 1.3.12
Enter "help" to get some basic usage information
boot: autobench
Please wait, loading kernel...
Elf64 kernel loaded...
Loading ramdisk...
ramdisk loaded at 02400000, size: 1648 Kbytes
OF stdout device is: /vdevice/vty@30000000
Hypertas detected, assuming LPAR !
command line: ro console=hvc0 autobench_args: root=/dev/sda6 ABAT:1173335344 loglevel=8 slub_debug
memory layout at init:
alloc_bottom : 000000000259c000
alloc_top : 0000000008000000
alloc_top_hi : 0000000100000000
rmo_top : 0000000008000000
ram_top : 0000000100000000
Looking for displays
instantiating rtas at 0x00000000077d9000 ... done
0000000000000000 : boot cpu 0000000000000000
0000000000000002 : starting cpu hw idx 0000000000000002... done
copying OF device tree ...
Building dt strings...
Building dt structure...
Device tree strings 0x000000000269d000 -> 0x000000000269e1d9
Device tree struct 0x000000000269f000 -> 0x00000000026a7000
Calling quiesce ...
returning from prom_init
Partition configured for 4 cpus.
Starting Linux PPC64 #1 SMP Wed Mar 7 22:23:06 PST 2007
-----------------------------------------------------
ppc64_pft_size = 0x1a
physicalMemorySize = 0x100000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address = 0x0000000000000000
htab_hash_mask = 0x7ffff
-----------------------------------------------------
Linux version 2.6.21-rc2-mm2-autokern1 (root@gekko-lp1) (gcc version 4.0.3 20051201 (prerelease) (Debian 4.0.2-5)) #1 SMP Wed Mar 7 22:23:06 PST 2007
[boot]0012 Setup Arch
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 7168 bytes
Zone PFN ranges:
DMA 0 -> 1048576
Normal 1048576 -> 1048576
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0 -> 1048576
[boot]0015 Setup Done
Built 1 zonelists. Total pages: 1034240
Kernel command line: ro console=hvc0 autobench_args: root=/dev/sda6 ABAT:1173335344 loglevel=8 slub_debug
[boot]0020 XICS Init
xics: no ISA interrupt controller
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
time_init: decrementer frequency = 238.059000 MHz
time_init: processor frequency = 1904.472000 MHz
Using pSeries machine description
Page orders: linear mapping = 24, virtual = 12, io = 12
Found initrd at 0xc000000002400000:0xc00000000259c000
Partition configured for 4 cpus.
Starting Linux PPC64 #1 SMP Wed Mar 7 22:23:06 PST 2007
-----------------------------------------------------
ppc64_pft_size = 0x1a
physicalMemorySize = 0x100000000
ppc64_caches.dcache_line_size = 0x80
ppc64_caches.icache_line_size = 0x80
htab_address = 0x0000000000000000
htab_hash_mask = 0x7ffff
-----------------------------------------------------
Linux version 2.6.21-rc2-mm2-autokern1 (root@gekko-lp1) (gcc version 4.0.3 20051201 (prerelease) (Debian 4.0.2-5)) #1 SMP Wed Mar 7 22:23:06 PST 2007
[boot]0012 Setup Arch
Entering add_active_range(0, 0, 32768) 0 entries of 256 used
Entering add_active_range(0, 32768, 65536) 1 entries of 256 used
Entering add_active_range(0, 65536, 98304) 1 entries of 256 used
Entering add_active_range(0, 98304, 131072) 1 entries of 256 used
Entering add_active_range(0, 131072, 163840) 1 entries of 256 used
Entering add_active_range(0, 163840, 196608) 1 entries of 256 used
Entering add_active_range(0, 196608, 229376) 1 entries of 256 used
Entering add_active_range(0, 229376, 262144) 1 entries of 256 used
Entering add_active_range(0, 262144, 294912) 1 entries of 256 used
Entering add_active_range(0, 294912, 327680) 1 entries of 256 used
Entering add_active_range(0, 327680, 360448) 1 entries of 256 used
Entering add_active_range(0, 360448, 393216) 1 entries of 256 used
Entering add_active_range(0, 393216, 425984) 1 entries of 256 used
Entering add_active_range(0, 425984, 458752) 1 entries of 256 used
Entering add_active_range(0, 458752, 491520) 1 entries of 256 used
Entering add_active_range(0, 491520, 524288) 1 entries of 256 used
Entering add_active_range(0, 524288, 557056) 1 entries of 256 used
Entering add_active_range(0, 557056, 589824) 1 entries of 256 used
Entering add_active_range(0, 589824, 622592) 1 entries of 256 used
Entering add_active_range(0, 622592, 655360) 1 entries of 256 used
Entering add_active_range(0, 655360, 688128) 1 entries of 256 used
Entering add_active_range(0, 688128, 720896) 1 entries of 256 used
Entering add_active_range(0, 720896, 753664) 1 entries of 256 used
Entering add_active_range(0, 753664, 786432) 1 entries of 256 used
Entering add_active_range(0, 786432, 819200) 1 entries of 256 used
Entering add_active_range(0, 819200, 851968) 1 entries of 256 used
Entering add_active_range(0, 851968, 884736) 1 entries of 256 used
Entering add_active_range(0, 884736, 917504) 1 entries of 256 used
Entering add_active_range(0, 917504, 950272) 1 entries of 256 used
Entering add_active_range(0, 950272, 983040) 1 entries of 256 used
Entering add_active_range(0, 983040, 1015808) 1 entries of 256 used
Entering add_active_range(0, 1015808, 1048576) 1 entries of 256 used
Node 0 Memory: 0x0-0x100000000
EEH: PCI Enhanced I/O Error Handling Enabled
PPC64 nvram contains 7168 bytes
Using dedicated idle loop
sizeof(struct page) = 56
Zone PFN ranges:
DMA 0 -> 1048576
Normal 1048576 -> 1048576
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
0: 0 -> 1048576
On node 0 totalpages: 1048576
DMA zone: 14336 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 1034240 pages, LIFO batch:31
Normal zone: 0 pages used for memmap
Movable zone: 0 pages used for memmap
[boot]0015 Setup Done
Built 1 zonelists. Total pages: 1034240
Kernel command line: ro console=hvc0 autobench_args: root=/dev/sda6 ABAT:1173335344 loglevel=8 slub_debug
[boot]0020 XICS Init
xics: no ISA interrupt controller
[boot]0021 XICS Done
PID hash table entries: 4096 (order: 12, 32768 bytes)
time_init: decrementer frequency = 238.059000 MHz
time_init: processor frequency = 1904.472000 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
freeing bootmem node 0
Memory: 4113864k/4194304k available (4672k kernel code, 80440k reserved, 988k data, 576k bss, 252k init)
SLUB V4: General Slabs=9, HW alignment=128, Processors=4, Nodes=16
Calibrating delay loop... 475.13 BogoMIPS (lpj=950272)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
Processor 1 found.
Processor 2 found.
Processor 3 found.
Brought up 4 CPUs
Node 0 CPUs: 0-3
mm/memory.c:111: bad pud c0000000050e4480.
could not vmalloc 20971520 bytes for cache!
migration_cost=0,1000
*** SLUB: Redzone Inactive check fails in kmalloc-64@c0000000050de0f0 Slab c000000000756090
offset=240 flags=5000000000c7 inuse=3 freelist=c0000000050de0f0
Bytes b4 c0000000050de0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object c0000000050de0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object c0000000050de100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object c0000000050de110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Object c0000000050de120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
Redzone c0000000050de130: 00 00 00 00 00 00 00 00 ........
FreePointer c0000000050de138: 0000000000000000
Call Trace:
[C00000000506B9D0] [C000000000011188] .show_stack+0x6c/0x1a0 (unreliable)
[C00000000506BA70] [C0000000000CB9BC] .object_err+0x1bc/0x1e8
[C00000000506BB10] [C0000000000CBB3C] .check_object+0x154/0x23c
[C00000000506BBB0] [C0000000000CCFB0] .alloc_object_checks+0xc0/0x154
[C00000000506BC40] [C0000000000CD600] .kmem_cache_alloc+0xc8/0x4a8
[C00000000506BD00] [C0000000000CD9FC] .kmem_cache_zalloc+0x1c/0x50
[C00000000506BD90] [C000000000070334] .__create_workqueue+0x48/0x1b8
[C00000000506BE40] [C00000000046C36C] .helper_init+0x24/0x54
[C00000000506BEC0] [C000000000451B7C] .init+0x1c4/0x2f8
[C00000000506BF90] [C0000000000275D0] .kernel_thread+0x4c/0x68
NET: Registered protocol family 16
PCI: Probing PCI hardware
IOMMU table initialized, virtual merging enabled
mapping IO 3fe00600000 -> d000080000000000, size: 100000
PCI: Probing PCI hardware done
Registering pmac pic with sysfs...
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 524288 (order: 11, 12582912 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 524288 bind 65536)
TCP reno registered
checking if image is initramfs...it isn't (bad gzip magic numbers); looks like an initrd
Freeing initrd memory: 1648k freed
vio_bus_init: processing c0000000ffffe3a0
vio_bus_init: processing c0000000ffffe558
vio_bus_init: processing c0000000ffffe9f8
vio_bus_init: processing c0000000ffffeb30
vio_bus_init: processing c0000000ffffec88
scan-log-dump not implemented on this system
RTAS daemon started
RTAS: event: 1, Type: Platform Error, Severity: 2
audit: initializing netlink socket (disabled)
audit(1173335571.256:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
JFS: nTxBlock = 8192, nTxLock = 65536
SELinux: Registering netfilter hooks
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
rpaphp: Slot [0000:00:02.2](PCI location=U7879.001.DQD0T7T-P1-C4) registered
vio_register_driver: driver hvc_console registering
------------[ cut here ]------------
Badness at mm/slub.c:1701
Call Trace:
[C00000000506B730] [C000000000011188] .show_stack+0x6c/0x1a0 (unreliable)
[C00000000506B7D0] [C0000000001EE9F4] .report_bug+0x94/0xe8
[C00000000506B860] [C00000000038C85C] .program_check_exception+0x16c/0x5f4
[C00000000506B930] [C0000000000046F4] program_check_common+0xf4/0x100
--- Exception: 700 at .get_slab+0xbc/0x18c
LR = .__kmalloc+0x28/0x104
[C00000000506BC20] [C00000000506BCC0] 0xc00000000506bcc0 (unreliable)
[C00000000506BCD0] [C0000000000CE2EC] .__kmalloc+0x28/0x104
[C00000000506BD60] [C00000000022E724] .tty_register_driver+0x5c/0x23c
[C00000000506BE10] [C000000000477910] .hvsi_init+0x154/0x1b4
[C00000000506BEC0] [C000000000451B7C] .init+0x1c4/0x2f8
[C00000000506BF90] [C0000000000275D0] .kernel_thread+0x4c/0x68
HVSI: registered 0 devices
Generic RTC Driver v1.07
[drm] Initialized drm 1.1.0 20060810
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ide-floppy driver 0.99.newide
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
mice: PS/2 mouse device common for all mice
async_tx: api initialized (sync-only)
xor: measuring software checksumming speed
8regs : 6524.000 MB/sec
8regs_prefetch: 4997.000 MB/sec
32regs : 6994.000 MB/sec
32regs_prefetch: 4985.000 MB/sec
xor: using function: 32regs (6994.000 MB/sec)
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
RAMDISK: cramfs filesystem found at block 0
RAMDISK: Loading 1648KiB [1 disk] into ram disk... |/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-\|/-done.
VFS: Mounted root (cramfs filesystem) readonly.
mm/memory.c:111: bad pud c000000005a0ad80.
mm/memory.c:111: bad pud c000000005a0b200.
mm/memory.c:111: bad pud c000000005a0a480.
mm/memory.c:111: bad pud c000000005a0b680.
mm/memory.c:111: bad pud c000000005a0a900.
mm/memory.c:111: bad pud c000000005a0bb00.
mm/memory.c:111: bad pud c000000005762900.
mm/memory.c:111: bad pud c000000005762480.
------------[ cut here ]------------
kernel BUG at mm/mmap.c:1999!
cpu 0x3: Vector: 700 (Program Check) at [c00000000576b430]
pc: c0000000000b37d4: .exit_mmap+0x150/0x178
lr: c0000000000b37b8: .exit_mmap+0x134/0x178
sp: c00000000576b6b0
msr: 8000000000029032
current = 0xc000000005177680
paca = 0xc0000000004a5280
pid = 235, comm = linuxrc
kernel BUG at mm/mmap.c:1999!
------------[ cut here ]------------
Badness at arch/powerpc/kernel/entry_64.S:651
Call Trace:
[C00000000576A720] [C000000000011188] .show_stack+0x6c/0x1a0 (unreliable)
[C00000000576A7C0] [C0000000001EE9F4] .report_bug+0x94/0xe8
[C00000000576A850] [C00000000038C85C] .program_check_exception+0x16c/0x5f4
[C00000000576A920] [C0000000000046F4] program_check_common+0xf4/0x100
--- Exception: 700 at .enter_rtas+0xa0/0x10c
LR = .xmon_core+0x580/0x920
[C00000000576AC10] [C00000000004DCD4] .xmon_printf+0x64/0x7c (unreliable)
[C00000000576ADF0] [C00000000004D118] .xmon_core+0x580/0x920
[C00000000576AF80] [C00000000004D700] .xmon+0x30/0x40
[C00000000576B150] [C000000000025D0C] .die+0x50/0x1b8
[C00000000576B1E0] [C0000000000260AC] ._exception+0x40/0x134
[C00000000576B2F0] [C00000000038CCA8] .program_check_exception+0x5b8/0x5f4
[C00000000576B3C0] [C0000000000046F4] program_check_common+0xf4/0x100
--- Exception: 700 at .exit_mmap+0x150/0x178
LR = .exit_mmap+0x134/0x178
[C00000000576B760] [C0000000000574FC] .mmput+0x78/0x170
[C00000000576B800] [C00000000005C968] .exit_mm+0x128/0x148
[C00000000576B890] [C00000000005E8E4] .do_exit+0x274/0x958
[C00000000576B940] [C00000000005F09C] .sys_exit_group+0x0/0x8
[C00000000576B9D0] [C00000000006AC0C] .get_signal_to_deliver+0x678/0x700
[C00000000576BB60] [C00000000000F2E0] .do_signal32+0x7c/0x818
[C00000000576BCD0] [C000000000017BB8] .do_signal+0x4c/0x8b8
[C00000000576BE30] [C000000000008CE8] do_work+0x28/0x2c
enter ? for help
[c00000000576b760] c0000000000574fc .mmput+0x78/0x170
[c00000000576b800] c00000000005c968 .exit_mm+0x128/0x148
[c00000000576b890] c00000000005e8e4 .do_exit+0x274/0x958
[c00000000576b940] c00000000005f09c .sys_exit_group+0x0/0x8
[c00000000576b9d0] c00000000006ac0c .get_signal_to_deliver+0x678/0x700
[c00000000576bb60] c00000000000f2e0 .do_signal32+0x7c/0x818
[c00000000576bcd0] c000000000017bb8 .do_signal+0x4c/0x8b8
[c00000000576be30] c000000000008ce8 do_work+0x28/0x2c
--- Exception: 300 (Data Access) at 000000000ff36f8c
SP (ffb5f8a0) is in userspace
3:mon>-- 0:conmux-control -- time-stamp -- Mar/07/07 22:35:07 --
-- 0:conmux-control -- time-stamp -- Mar/07/07 22:49:34 --
(bot:conmon-payload) disconnected
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 10:54 ` [SLUB 0/3] SLUB: The unqueued slab allocator V4 Mel Gorman
@ 2007-03-08 16:48 ` Christoph Lameter
2007-03-08 17:40 ` Mel Gorman
2007-03-08 17:46 ` Christoph Lameter
1 sibling, 1 reply; 21+ messages in thread
From: Christoph Lameter @ 2007-03-08 16:48 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Thu, 8 Mar 2007, Mel Gorman wrote:
> On x86_64, it completed successfully and looked reliable. There was a 5%
> performance loss on kernbench and aim9 figures were way down. However, with
> slub_debug enabled, I would expect that so it's not a fair comparison
> performance wise. I'll rerun the tests without debug and see what it looks
> like if you're interested and do not think it's too early to worry about
> performance instead of clarity. This is what I have for bl6-13 (machine
> appears on test.kernel.org so additional details are there).
No its good to start worrying about performance now. There are still some
performance issues to be ironed out in particular on NUMA. I am not sure
f.e. how the reduction of partial lists affect performance.
> IA64 (machine not visible on TKO) curiously did not exhibit the same problems
> on kernbench for Total CPU time which is very unexpected but you can see the
> System CPU times. The AIM9 figures were a bit of an upset but again, I blame
> slub_debug being enabled
This was a single node box? Note that the 16kb page size has a major
impact on SLUB performance. On IA64 slub will use only 1/4th the locking
overhead as on 4kb platforms.
> (as an aside, the succes rates for high-order allocations are lower with SLUB.
> Again, I blame slub_debug. I know that enabling SLAB_DEBUG has similar effects
> because of red-zoning and the like)
We have some additional patches here that reduce the max order for some
allocs. I believe the task_struct gets to be an order 2 alloc with V4,
> Now, the bad news. This exploded on ppc64. It started going wrong early in the
> boot process and got worse. I haven't looked closely as to why yet as there is
> other stuff on my plate but I've included a console log that might be some use
> to you. If you think you have a fix for it, feel free to send it on and I'll
> give it a test.
Hmmm... Looks like something is zapping an object. Try to rerun with
a kernel compiled with CONFIG_SLAB_DEBUG. I would expect similar results.
> Brought up 4 CPUs
> Node 0 CPUs: 0-3
> mm/memory.c:111: bad pud c0000000050e4480.
> could not vmalloc 20971520 bytes for cache!
Hmmm... a bad pud? I need to look at how the puds are managed on power.
> migration_cost=0,1000
> *** SLUB: Redzone Inactive check fails in kmalloc-64@c0000000050de0f0 Slab
An object was overwritten with zeros after it was freed.
> RTAS daemon started
> RTAS: event: 1, Type: Platform Error, Severity: 2
> audit: initializing netlink socket (disabled)
> audit(1173335571.256:1): initialized
> Total HugeTLB memory allocated, 0
> VFS: Disk quotas dquot_6.5.1
> Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> JFS: nTxBlock = 8192, nTxLock = 65536
> SELinux: Registering netfilter hooks
> io scheduler noop registered
> io scheduler anticipatory registered (default)
> io scheduler deadline registered
> io scheduler cfq registered
> pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
> rpaphp: Slot [0000:00:02.2](PCI location=U7879.001.DQD0T7T-P1-C4) registered
> vio_register_driver: driver hvc_console registering
> ------------[ cut here ]------------
> Badness at mm/slub.c:1701
Someone did a kmalloc(0, ...). Zero sized allocation are not flagged
by SLAB but SLUB does.
> Call Trace:
> [C00000000506B730] [C000000000011188] .show_stack+0x6c/0x1a0 (unreliable)
> [C00000000506B7D0] [C0000000001EE9F4] .report_bug+0x94/0xe8
> [C00000000506B860] [C00000000038C85C] .program_check_exception+0x16c/0x5f4
> [C00000000506B930] [C0000000000046F4] program_check_common+0xf4/0x100
> --- Exception: 700 at .get_slab+0xbc/0x18c
> LR = .__kmalloc+0x28/0x104
> [C00000000506BC20] [C00000000506BCC0] 0xc00000000506bcc0 (unreliable)
> [C00000000506BCD0] [C0000000000CE2EC] .__kmalloc+0x28/0x104
> [C00000000506BD60] [C00000000022E724] .tty_register_driver+0x5c/0x23c
> [C00000000506BE10] [C000000000477910] .hvsi_init+0x154/0x1b4
> [C00000000506BEC0] [C000000000451B7C] .init+0x1c4/0x2f8
> [C00000000506BF90] [C0000000000275D0] .kernel_thread+0x4c/0x68
> mm/memory.c:111: bad pud c000000005762900.
> mm/memory.c:111: bad pud c000000005762480.
> ------------[ cut here ]------------
> kernel BUG at mm/mmap.c:1999!
More page table trouble.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 16:48 ` Christoph Lameter
@ 2007-03-08 17:40 ` Mel Gorman
2007-03-08 18:16 ` Christoph Lameter
2007-03-08 21:54 ` Christoph Lameter
0 siblings, 2 replies; 21+ messages in thread
From: Mel Gorman @ 2007-03-08 17:40 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On (08/03/07 08:48), Christoph Lameter didst pronounce:
> On Thu, 8 Mar 2007, Mel Gorman wrote:
>
> > On x86_64, it completed successfully and looked reliable. There was a 5%
> > performance loss on kernbench and aim9 figures were way down. However, with
> > slub_debug enabled, I would expect that so it's not a fair comparison
> > performance wise. I'll rerun the tests without debug and see what it looks
> > like if you're interested and do not think it's too early to worry about
> > performance instead of clarity. This is what I have for bl6-13 (machine
> > appears on test.kernel.org so additional details are there).
>
> No its good to start worrying about performance now. There are still some
> performance issues to be ironed out in particular on NUMA. I am not sure
> f.e. how the reduction of partial lists affect performance.
>
Ok, I've sent off a bunch of tests - two of which are on NUMA (numaq and
x86_64). It'll take them a long time to complete though as there is a
lot of testing going on right now.
> > IA64 (machine not visible on TKO) curiously did not exhibit the same problems
> > on kernbench for Total CPU time which is very unexpected but you can see the
> > System CPU times. The AIM9 figures were a bit of an upset but again, I blame
> > slub_debug being enabled
>
> This was a single node box?
Yes, memory looks like this;
Zone PFN ranges:
DMA 1024 -> 262144
Normal 262144 -> 262144
Movable zone start PFN for each node
early_node_map[3] active PFN ranges
0: 1024 -> 30719
0: 32768 -> 65413
0: 65440 -> 65505
On node 0 totalpages: 62405
Node 0 memmap at 0xe000000001126000 size 3670016 first pfn 0xe000000001134000
DMA zone: 220 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 62185 pages, LIFO batch:7
Normal zone: 0 pages used for memmap
Movable zone: 0 pages used for memmap
> Note that the 16kb page size has a major
> impact on SLUB performance. On IA64 slub will use only 1/4th the locking
> overhead as on 4kb platforms.
>
It'll be interesting to see the kernbench tests then with debugging
disabled.
> > (as an aside, the succes rates for high-order allocations are lower with SLUB.
> > Again, I blame slub_debug. I know that enabling SLAB_DEBUG has similar effects
> > because of red-zoning and the like)
>
> We have some additional patches here that reduce the max order for some
> allocs. I believe the task_struct gets to be an order 2 alloc with V4,
>
Should make a difference for slab fragmentation
> > Now, the bad news. This exploded on ppc64. It started going wrong early in the
> > boot process and got worse. I haven't looked closely as to why yet as there is
> > other stuff on my plate but I've included a console log that might be some use
> > to you. If you think you have a fix for it, feel free to send it on and I'll
> > give it a test.
>
> Hmmm... Looks like something is zapping an object. Try to rerun with
> a kernel compiled with CONFIG_SLAB_DEBUG. I would expect similar results.
>
I've queued up a few tests. One completed as I wrote this and it didn't
explode with SLAB_DEBUG set. Maybe the others will be different. I'll
kick it around for a bit.
It could be a real bug that slab is just not catuching.
> > Brought up 4 CPUs
> > Node 0 CPUs: 0-3
> > mm/memory.c:111: bad pud c0000000050e4480.
> > could not vmalloc 20971520 bytes for cache!
>
> Hmmm... a bad pud? I need to look at how the puds are managed on power.
>
> > migration_cost=0,1000
> > *** SLUB: Redzone Inactive check fails in kmalloc-64@c0000000050de0f0 Slab
>
> An object was overwritten with zeros after it was freed.
> > RTAS daemon started
> > RTAS: event: 1, Type: Platform Error, Severity: 2
> > audit: initializing netlink socket (disabled)
> > audit(1173335571.256:1): initialized
> > Total HugeTLB memory allocated, 0
> > VFS: Disk quotas dquot_6.5.1
> > Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
> > JFS: nTxBlock = 8192, nTxLock = 65536
> > SELinux: Registering netfilter hooks
> > io scheduler noop registered
> > io scheduler anticipatory registered (default)
> > io scheduler deadline registered
> > io scheduler cfq registered
> > pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> > rpaphp: RPA HOT Plug PCI Controller Driver version: 0.1
> > rpaphp: Slot [0000:00:02.2](PCI location=U7879.001.DQD0T7T-P1-C4) registered
> > vio_register_driver: driver hvc_console registering
> > ------------[ cut here ]------------
> > Badness at mm/slub.c:1701
>
> Someone did a kmalloc(0, ...). Zero sized allocation are not flagged
> by SLAB but SLUB does.
>
I'll chase up what's happening here. It will be "reproducable" independent
of SLUB by adding a similar check.
> > Call Trace:
> > [C00000000506B730] [C000000000011188] .show_stack+0x6c/0x1a0 (unreliable)
> > [C00000000506B7D0] [C0000000001EE9F4] .report_bug+0x94/0xe8
> > [C00000000506B860] [C00000000038C85C] .program_check_exception+0x16c/0x5f4
> > [C00000000506B930] [C0000000000046F4] program_check_common+0xf4/0x100
> > --- Exception: 700 at .get_slab+0xbc/0x18c
> > LR = .__kmalloc+0x28/0x104
> > [C00000000506BC20] [C00000000506BCC0] 0xc00000000506bcc0 (unreliable)
> > [C00000000506BCD0] [C0000000000CE2EC] .__kmalloc+0x28/0x104
> > [C00000000506BD60] [C00000000022E724] .tty_register_driver+0x5c/0x23c
> > [C00000000506BE10] [C000000000477910] .hvsi_init+0x154/0x1b4
> > [C00000000506BEC0] [C000000000451B7C] .init+0x1c4/0x2f8
> > [C00000000506BF90] [C0000000000275D0] .kernel_thread+0x4c/0x68
> > mm/memory.c:111: bad pud c000000005762900.
> > mm/memory.c:111: bad pud c000000005762480.
> > ------------[ cut here ]------------
> > kernel BUG at mm/mmap.c:1999!
>
> More page table trouble.
--
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 10:54 ` [SLUB 0/3] SLUB: The unqueued slab allocator V4 Mel Gorman
2007-03-08 16:48 ` Christoph Lameter
@ 2007-03-08 17:46 ` Christoph Lameter
1 sibling, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-08 17:46 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Thu, 8 Mar 2007, Mel Gorman wrote:
> Brought up 4 CPUs
> Node 0 CPUs: 0-3
> mm/memory.c:111: bad pud c0000000050e4480.
Lower bits must be clear right? Looks like the pud was released
and then reused for a 64 byte cache or so. This is likely a freelist
pointer that slub put there after allocating the page for the 64 byte
cache. Then we tried to use the pud.
> migration_cost=0,1000
> *** SLUB: Redzone Inactive check fails in kmalloc-64@c0000000050de0f0 Slab
> c000000000756090
> offset=240 flags=5000000000c7 inuse=3 freelist=c0000000050de0f0
> Bytes b4 c0000000050de0e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Object c0000000050de0f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Object c0000000050de100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Object c0000000050de110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Object c0000000050de120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> Redzone c0000000050de130: 00 00 00 00 00 00 00 00
> ........ FreePointer c0000000050de138: 0000000000000000
Data overwritten after free or after slab was allocated. So this may be
the same issue. pud was zapped after it was freed destroying the poison
of another object in the 64 byte cache.
Hmmm.. Maybe I should put the pad checks before the object checks.
That way we detect that the whole slab was corrupted and do not flag just
a single object.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 17:40 ` Mel Gorman
@ 2007-03-08 18:16 ` Christoph Lameter
2007-03-09 13:55 ` Mel Gorman
2007-03-08 21:54 ` Christoph Lameter
1 sibling, 1 reply; 21+ messages in thread
From: Christoph Lameter @ 2007-03-08 18:16 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
On Thu, 8 Mar 2007, Mel Gorman wrote:
> > Note that the 16kb page size has a major
> > impact on SLUB performance. On IA64 slub will use only 1/4th the locking
> > overhead as on 4kb platforms.
> It'll be interesting to see the kernbench tests then with debugging
> disabled.
You can get a similar effect on 4kb platforms by specifying slub_min_order=2 on bootup.
This means that we have to rely on your patches to allow higher order
allocs to work reliably though. The higher the order of slub the less
locking overhead. So the better your patches deal with fragmentation the
more we can reduce locking overhead in slub.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 17:40 ` Mel Gorman
2007-03-08 18:16 ` Christoph Lameter
@ 2007-03-08 21:54 ` Christoph Lameter
2007-03-09 14:00 ` Mel Gorman
2007-03-09 15:06 ` Mel Gorman
1 sibling, 2 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-08 21:54 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, linux-kernel, linux-mm, mpm, Manfred Spraul
Note that I am amazed that the kernbench even worked. On small machine I
seem to be getting into trouble with order 1 allocations. SLAB seems to be
able to avoid the situation by keeping higher order pages on a freelist
and reduce the alloc/frees of higher order pages that the page allocator
has to deal with. Maybe we need per order queues in the page allocator?
There must be something fundamentally wrong in the page allocator if the
SLAB queues fix this issue. I was able to fix the issue in V5 by forcing
SLUB to keep a mininum number of objects around regardless of the fit to
a page order page. Pass through is deadly since the crappy page allocator
cannot handle it.
Higher order page allocation failures can be avoided by using kmalloc.
Yuck! Hopefully your patches fix that fundamental problem.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 18:16 ` Christoph Lameter
@ 2007-03-09 13:55 ` Mel Gorman
0 siblings, 0 replies; 21+ messages in thread
From: Mel Gorman @ 2007-03-09 13:55 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, Linux Kernel Mailing List,
Linux Memory Management List, mpm, Manfred Spraul
On Thu, 8 Mar 2007, Christoph Lameter wrote:
> On Thu, 8 Mar 2007, Mel Gorman wrote:
>
>>> Note that the 16kb page size has a major
>>> impact on SLUB performance. On IA64 slub will use only 1/4th the locking
>>> overhead as on 4kb platforms.
>> It'll be interesting to see the kernbench tests then with debugging
>> disabled.
>
> You can get a similar effect on 4kb platforms by specifying slub_min_order=2 on bootup.
> This means that we have to rely on your patches to allow higher order
> allocs to work reliably though.
It should work out because of the way buddy always selects the minimum
page size will tend to cluster the slab allocations together whether they
are reclaimable or not. It's something I can investigate when slub has
stabilised a bit.
However, in general, high order kernel allocations remain a bad idea.
Depending on high order allocations that do not group could potentially
lead to a situation where the movable areas are used more and more by
kernel allocations. I cannot think of a workload that would actually break
everything, but it's a possibility.
> The higher the order of slub the less
> locking overhead. So the better your patches deal with fragmentation the
> more we can reduce locking overhead in slub.
>
I can certainly kick it around a lot and see what happen. It's best that
slub_min_order=2 remain an optional performance enhancing switch though.
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 21:54 ` Christoph Lameter
@ 2007-03-09 14:00 ` Mel Gorman
2007-03-09 16:40 ` Christoph Lameter
2007-03-09 15:06 ` Mel Gorman
1 sibling, 1 reply; 21+ messages in thread
From: Mel Gorman @ 2007-03-09 14:00 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, Linux Kernel Mailing List,
Linux Memory Management List, mpm, Manfred Spraul
On Thu, 8 Mar 2007, Christoph Lameter wrote:
> Note that I am amazed that the kernbench even worked. On small machine
How small? The machines I am testing on aren't "big" but they aren't
misterable either.
> I
> seem to be getting into trouble with order 1 allocations.
That in itself is pretty incredible. From what I see, allocations up to 3
generally work unless they are atomic even with the vanilla kernel. That
said, it could be because slab is holding onto the high order pages for
itself.
> SLAB seems to be
> able to avoid the situation by keeping higher order pages on a freelist
> and reduce the alloc/frees of higher order pages that the page allocator
> has to deal with. Maybe we need per order queues in the page allocator?
>
I'm not sure what you mean by per-order queues. The buddy allocator
already has per-order lists.
> There must be something fundamentally wrong in the page allocator if the
> SLAB queues fix this issue. I was able to fix the issue in V5 by forcing
> SLUB to keep a mininum number of objects around regardless of the fit to
> a page order page. Pass through is deadly since the crappy page allocator
> cannot handle it.
>
> Higher order page allocation failures can be avoided by using kmalloc.
> Yuck! Hopefully your patches fix that fundamental problem.
>
One way to find out for sure.
--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-08 21:54 ` Christoph Lameter
2007-03-09 14:00 ` Mel Gorman
@ 2007-03-09 15:06 ` Mel Gorman
2007-03-09 17:21 ` Christoph Lameter
1 sibling, 1 reply; 21+ messages in thread
From: Mel Gorman @ 2007-03-09 15:06 UTC (permalink / raw)
To: Christoph Lameter
Cc: akpm, Marcelo Tosatti, Linux Kernel Mailing List,
Linux Memory Management List, mpm, Manfred Spraul
> Note that I am amazed that the kernbench even worked.
The results without slub_debug were not good except for IA64. x86_64 and
ppc64 both blew up for a variety of reasons. The IA64 results were
KernBench Comparison
--------------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-slub %diff
User CPU time 1084.64 1032.93 4.77%
System CPU time 73.38 63.14 13.95%
Total CPU time 1158.02 1096.07 5.35%
Elapsed time 307.00 285.62 6.96%
AIM9 Comparison
---------------
2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-slub
1 creat-clo 425460.75 438809.64 13348.89 3.14% File Creations and Closes/second
2 page_test 2097119.26 3398259.27 1301140.01 62.04% System Allocations & Pages/second
3 brk_test 7008395.33 6728755.72 -279639.61 -3.99% System Memory Allocations/second
4 jmp_test 12226295.31 12254966.21 28670.90 0.23% Non-local gotos/second
5 signal_test 1271126.28 1235510.96 -35615.32 -2.80% Signal Traps/second
6 exec_test 395.54 381.18 -14.36 -3.63% Program Loads/second
7 fork_test 13218.23 13211.41 -6.82 -0.05% Task Creations/second
8 link_test 64776.04 7488.13 -57287.91 -88.44% Link/Unlink Pairs/second
An example console log from x86_64 is below. It's not particular clear why
it went blamo and I haven't had a chance all day to kick it around for a
bit due to a variety of other hilarity floating around.
Linux version 2.6.21-rc2-mm2-autokern1 (root@bl6-13.ltc.austin.ibm.com) (gcc version 4.1.1 20060525 (Red Hat 4.1.1-1)) #1 SMP Thu Mar 8 12:13:27 CST 2007
Command line: ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS1,19200 selinux=no autobench_args: root=30726124 ABAT:1173378546 loglevel=8
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009d400 (usable)
BIOS-e820: 000000000009d400 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003ffcddc0 (usable)
BIOS-e820: 000000003ffcddc0 - 000000003ffd0000 (ACPI data)
BIOS-e820: 000000003ffd0000 - 0000000040000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
Entering add_active_range(0, 0, 157) 0 entries of 3200 used
Entering add_active_range(0, 256, 262093) 1 entries of 3200 used
end_pfn_map = 1048576
DMI 2.3 present.
ACPI: RSDP 000FDFC0, 0014 (r0 IBM )
ACPI: RSDT 3FFCFF80, 0034 (r1 IBM SERBLADE 1000 IBM 45444F43)
ACPI: FACP 3FFCFEC0, 0084 (r2 IBM SERBLADE 1000 IBM 45444F43)
ACPI: DSDT 3FFCDDC0, 1EA6 (r1 IBM SERBLADE 1000 INTL 2002025)
ACPI: FACS 3FFCFCC0, 0040
ACPI: APIC 3FFCFE00, 009C (r1 IBM SERBLADE 1000 IBM 45444F43)
ACPI: SRAT 3FFCFD40, 0098 (r1 IBM SERBLADE 1000 IBM 45444F43)
ACPI: HPET 3FFCFD00, 0038 (r1 IBM SERBLADE 1000 IBM 45444F43)
SRAT: PXM 0 -> APIC 0 -> Node 0
SRAT: PXM 0 -> APIC 1 -> Node 0
SRAT: PXM 1 -> APIC 2 -> Node 1
SRAT: PXM 1 -> APIC 3 -> Node 1
SRAT: Node 0 PXM 0 0-40000000
Entering add_active_range(0, 0, 157) 0 entries of 3200 used
Entering add_active_range(0, 256, 262093) 1 entries of 3200 used
NUMA: Using 63 for the hash shift.
Bootmem setup node 0 0000000000000000-000000003ffcd000
Node 0 memmap at 0xffff81003efcd000 size 16773952 first pfn 0xffff81003efcd000
sizeof(struct page) = 64
Zone PFN ranges:
DMA 0 -> 4096
DMA32 4096 -> 1048576
Normal 1048576 -> 1048576
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
0: 0 -> 157
0: 256 -> 262093
On node 0 totalpages: 261994
DMA zone: 64 pages used for memmap
DMA zone: 2017 pages reserved
DMA zone: 1916 pages, LIFO batch:0
DMA32 zone: 4031 pages used for memmap
DMA32 zone: 253966 pages, LIFO batch:31
Normal zone: 0 pages used for memmap
Movable zone: 0 pages used for memmap
ACPI: PM-Timer IO Port: 0x2208
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 (Bootup-CPU)
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
Processor #2
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] enabled)
Processor #3
ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
ACPI: IOAPIC (id[0x0e] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 14, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x0d] address[0xfec10000] gsi_base[24])
IOAPIC[1]: apic_id 13, address 0xfec10000, GSI 24-27
ACPI: IOAPIC (id[0x0c] address[0xfec20000] gsi_base[48])
IOAPIC[2]: apic_id 12, address 0xfec20000, GSI 48-51
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ11 used by override.
Setting APIC routing to flat
ACPI: HPET id: 0x10228203 base: 0xfecff000
Using ACPI (MADT) for SMP configuration information
Nosave address range: 000000000009d000 - 000000000009e000
Nosave address range: 000000000009e000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000e0000
Nosave address range: 00000000000e0000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:bec00000)
SMP: Allowing 4 CPUs, 0 hotplug CPUs
PERCPU: Allocating 66368 bytes of per cpu data
Built 1 zonelists. Total pages: 255882
Kernel command line: ro root=/dev/VolGroup00/LogVol00 rhgb console=tty0 console=ttyS1,19200 selinux=no autobench_args: root=30726124 ABAT:1173378546 loglevel=8
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Marking TSC unstable due to TSCs unsynchronized
time.c: Detected 1993.781 MHz processor.
Console: colour VGA+ 80x25
Checking aperture...
CPU 0: aperture @ dc000000 size 64 MB
CPU 1: aperture @ dc000000 size 64 MB
Memory: 1021548k/1048372k available (2878k kernel code, 26428k reserved, 1472k data, 340k init)
SLUB V4: General Slabs=11, HW alignment=64, Processors=4, Nodes=64
Calibrating delay using timer specific routine.. 3991.49 BogoMIPS (lpj=7982991)
Security Framework v1.0.0 initialized
SELinux: Disabled at boot.
Capability LSM initialized
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 0/0 -> Node 0
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
SMP alternatives: switching to UP code
ACPI: Core revision 20070126
Using local APIC timer interrupts.
result 12461150
Detected 12.461 MHz APIC timer.
SMP alternatives: switching to SMP code
Booting processor 1/4 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 3987.64 BogoMIPS (lpj=7975295)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 1/1 -> Node 0
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
Dual Core AMD Opteron(tm) Processor 270 stepping 02
SMP alternatives: switching to SMP code
Booting processor 2/4 APIC 0x2
Initializing CPU#2
Calibrating delay using timer specific routine.. 3987.64 BogoMIPS (lpj=7975291)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 2/2 -> Node 0
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 0
Dual Core AMD Opteron(tm) Processor 270 stepping 02
SMP alternatives: switching to SMP code
Booting processor 3/4 APIC 0x3
Initializing CPU#3
Calibrating delay using timer specific routine.. 3987.64 BogoMIPS (lpj=7975292)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
CPU 3/3 -> Node 0
CPU: Physical Processor ID: 1
CPU: Processor Core ID: 1
Dual Core AMD Opteron(tm) Processor 270 stepping 02
Brought up 4 CPUs
migration_cost=413
PM: Adding info for No Bus:platform
NET: Registered protocol family 16
PM: Adding info for No Bus:vtcon0
ACPI: bus type pci registered
PCI: Using configuration type 1
ACPI: Interpreter enabled
ACPI: (supports S0 S1 S4 S5)
ACPI: Using IOAPIC for interrupt routing
PM: Adding info for acpi:acpi_system:00
PM: Adding info for acpi:button_power:00
PM: Adding info for acpi:ACPI0007:00
PM: Adding info for acpi:ACPI0007:01
PM: Adding info for acpi:ACPI0007:02
PM: Adding info for acpi:ACPI0007:03
PM: Adding info for acpi:device:00
PM: Adding info for acpi:PNP0A03:00
PM: Adding info for acpi:device:01
PM: Adding info for acpi:device:02
PM: Adding info for acpi:device:03
PM: Adding info for acpi:device:04
PM: Adding info for acpi:PNP0C02:00
PM: Adding info for acpi:PNP0501:00
PM: Adding info for acpi:PNP0501:01
PM: Adding info for acpi:PNP0000:00
PM: Adding info for acpi:PNP0003:00
PM: Adding info for acpi:PNP0003:01
PM: Adding info for acpi:PNP0003:02
PM: Adding info for acpi:PNP0200:00
PM: Adding info for acpi:PNP0100:00
PM: Adding info for acpi:PNP0B00:00
PM: Adding info for acpi:PNP0800:00
PM: Adding info for acpi:PNP0C04:00
PM: Adding info for acpi:PNP0C02:01
PM: Adding info for acpi:device:05
PM: Adding info for acpi:device:06
PM: Adding info for acpi:device:07
PM: Adding info for acpi:PNP0103:00
PM: Adding info for acpi:device:08
PM: Adding info for acpi:device:09
PM: Adding info for acpi:device:0a
PM: Adding info for acpi:device:0b
PM: Adding info for acpi:device:0c
PM: Adding info for acpi:device:0d
PM: Adding info for acpi:thermal:00
PM: Adding info for acpi:PNP0C0F:00
PM: Adding info for acpi:PNP0C0F:01
PM: Adding info for acpi:PNP0C0F:02
PM: Adding info for acpi:PNP0C0F:03
ACPI: PCI Root Bridge [PCI0] (0000:00)
PM: Adding info for No Bus:pci0000:00
Boot video device is 0000:01:04.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI2._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI3._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT]
PM: Adding info for pci:0000:00:06.0
PM: Adding info for pci:0000:00:07.0
PM: Adding info for pci:0000:00:07.3
PM: Adding info for pci:0000:00:0a.0
PM: Adding info for pci:0000:00:0a.1
PM: Adding info for pci:0000:00:0b.0
PM: Adding info for pci:0000:00:0b.1
PM: Adding info for pci:0000:00:18.0
PM: Adding info for pci:0000:00:18.1
PM: Adding info for pci:0000:00:18.2
PM: Adding info for pci:0000:00:18.3
PM: Adding info for pci:0000:00:19.0
PM: Adding info for pci:0000:00:19.1
PM: Adding info for pci:0000:00:19.2
PM: Adding info for pci:0000:00:19.3
PM: Adding info for pci:0000:01:00.0
PM: Adding info for pci:0000:01:00.1
PM: Adding info for pci:0000:01:04.0
PM: Adding info for pci:0000:02:01.0
PM: Adding info for pci:0000:02:01.1
PM: Adding info for pci:0000:02:02.0
ACPI: PCI Interrupt Link [LP00] (IRQs *10)
ACPI: PCI Interrupt Link [LP01] (IRQs *7)
ACPI: PCI Interrupt Link [LP02] (IRQs *9)
ACPI: PCI Interrupt Link [LP03] (IRQs *5)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
PM: Adding info for No Bus:pnp0
PM: Adding info for pnp:00:00
PM: Adding info for pnp:00:01
PM: Adding info for pnp:00:02
PM: Adding info for pnp:00:03
PM: Adding info for pnp:00:04
PM: Adding info for pnp:00:05
PM: Adding info for pnp:00:06
PM: Adding info for pnp:00:07
PM: Adding info for pnp:00:08
PM: Adding info for pnp:00:09
PM: Adding info for pnp:00:0a
PM: Adding info for pnp:00:0b
PM: Adding info for pnp:00:0c
pnp: PnP ACPI: found 13 devices
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
hpet0: at MMIO 0xfecff000, IRQs 2, 8, 0
hpet0: 3 32-bit timers, 14318180 Hz
pnp: 00:01: ioport range 0x510-0x517 has been reserved
Time: hpet clocksource has been installed.
pnp: 00:01: ioport range 0x504-0x507 has been reserved
pnp: 00:01: ioport range 0x500-0x503 has been reserved
pnp: 00:01: ioport range 0x520-0x53f has been reserved
pnp: 00:01: ioport range 0x540-0x547 has been reserved
pnp: 00:01: ioport range 0x460-0x461 has been reserved
pnp: 00:0b: iomem range 0xfec00000-0xffffffff has been reserved
PM: Adding info for No Bus:mem
PM: Adding info for No Bus:kmem
PM: Adding info for No Bus:null
PM: Adding info for No Bus:port
PM: Adding info for No Bus:zero
PM: Adding info for No Bus:full
PM: Adding info for No Bus:random
PM: Adding info for No Bus:urandom
PM: Adding info for No Bus:kmsg
PCI: Bridge: 0000:00:06.0
IO window: 3000-3fff
MEM window: fd000000-feafffff
PREFETCH window: f0000000-fcffffff
PCI: Bridge: 0000:00:0a.0
IO window: 4000-4fff
MEM window: ee000000-efffffff
PREFETCH window: 50000000-500fffff
PCI: Bridge: 0000:00:0b.0
IO window: 5000-ffff
MEM window: disabled.
PREFETCH window: disabled.
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 10, 5242880 bytes)
TCP bind hash table entries: 65536 (order: 9, 2097152 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
checking if image is initramfs... it is
Freeing initrd memory: 1602k freed
PM: Adding info for No Bus:mcelog
PM: Adding info for No Bus:msr0
PM: Adding info for No Bus:msr1
PM: Adding info for No Bus:msr2
PM: Adding info for No Bus:msr3
PM: Adding info for No Bus:cpu0
PM: Adding info for No Bus:cpu1
PM: Adding info for No Bus:cpu2
PM: Adding info for No Bus:cpu3
PM: Adding info for platform:pcspkr
audit: initializing netlink socket (disabled)
audit(1173357216.800:1): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
PM: Adding info for platform:vesafb.0
ACPI: Processor [CPU3] (supports 8 throttling states)
ACPI: Processor [CPU2] (supports 8 throttling states)
ACPI: Processor [CPU1] (supports 8 throttling states)
ACPI: Processor [CPU0] (supports 8 throttling states)
PM: Adding info for No Bus:tty
PM: Adding info for No Bus:console
PM: Adding info for No Bus:ptmx
PM: Adding info for No Bus:tty0
PM: Adding info for No Bus:vcs
PM: Adding info for No Bus:vcsa
PM: Adding info for No Bus:tty1
PM: Adding info for No Bus:tty2
PM: Adding info for No Bus:tty3
PM: Adding info for No Bus:tty4
PM: Adding info for No Bus:tty5
PM: Adding info for No Bus:tty6
PM: Adding info for No Bus:tty7
PM: Adding info for No Bus:tty8
PM: Adding info for No Bus:tty9
PM: Adding info for No Bus:tty10
PM: Adding info for No Bus:tty11
PM: Adding info for No Bus:tty12
PM: Adding info for No Bus:tty13
PM: Adding info for No Bus:tty14
PM: Adding info for No Bus:tty15
PM: Adding info for No Bus:tty16
PM: Adding info for No Bus:tty17
PM: Adding info for No Bus:tty18
PM: Adding info for No Bus:tty19
PM: Adding info for No Bus:tty20
PM: Adding info for No Bus:tty21
PM: Adding info for No Bus:tty22
PM: Adding info for No Bus:tty23
PM: Adding info for No Bus:tty24
PM: Adding info for No Bus:tty25
PM: Adding info for No Bus:tty26
PM: Adding info for No Bus:tty27
PM: Adding info for No Bus:tty28
PM: Adding info for No Bus:tty29
PM: Adding info for No Bus:tty30
PM: Adding info for No Bus:tty31
PM: Adding info for No Bus:tty32
PM: Adding info for No Bus:tty33
PM: Adding info for No Bus:tty34
PM: Adding info for No Bus:tty35
PM: Adding info for No Bus:tty36
PM: Adding info for No Bus:tty37
PM: Adding info for No Bus:tty38
PM: Adding info for No Bus:tty39
PM: Adding info for No Bus:tty40
PM: Adding info for No Bus:tty41
PM: Adding info for No Bus:tty42
PM: Adding info for No Bus:tty43
PM: Adding info for No Bus:tty44
PM: Adding info for No Bus:tty45
PM: Adding info for No Bus:tty46
PM: Adding info for No Bus:tty47
PM: Adding info for No Bus:tty48
PM: Adding info for No Bus:tty49
PM: Adding info for No Bus:tty50
PM: Adding info for No Bus:tty51
PM: Adding info for No Bus:tty52
PM: Adding info for No Bus:tty53
PM: Adding info for No Bus:tty54
PM: Adding info for No Bus:tty55
PM: Adding info for No Bus:tty56
PM: Adding info for No Bus:tty57
PM: Adding info for No Bus:tty58
PM: Adding info for No Bus:tty59
PM: Adding info for No Bus:tty60
PM: Adding info for No Bus:tty61
PM: Adding info for No Bus:tty62
PM: Adding info for No Bus:tty63
PM: Adding info for No Bus:rtc
Real Time Clock Driver v1.12ac
PM: Adding info for No Bus:hpet
hpet_resources: 0xfecff000 is busy
Linux agpgart interface v0.102 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
PM: Adding info for platform:serial8250
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
PM: Adding info for No Bus:ttyS0
erial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
PM: Adding info for No Bus:ttyS1
PM: Adding info for No Bus:ttyS2
PM: Adding info for No Bus:ttyS3
PM: Removing info for No Bus:ttyS0
00:02: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
PM: Adding info for No Bus:ttyS0
PM: Removing info for No Bus:ttyS1
00:03: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
PM: Adding info for No Bus:ttyS1
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
tg3.c:v3.74 (February 20, 2007)
ACPI: PCI Interrupt 0000:02:01.0[A] -> GSI 24 (level, low) -> IRQ 24
PM: Adding info for No Bus:eth0
eth0: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:100MHz:64-bit) 1000Base-SX Ethernet 00:11:25:75:af:6e
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[0] TSOcap[0]
eth0: dma_rwctrl[769f4000] dma_mask[64-bit]
ACPI: PCI Interrupt 0000:02:01.1[B] -> GSI 25 (level, low) -> IRQ 25
PM: Adding info for No Bus:eth1
eth1: Tigon3 [partno(BCM95704A41) rev 2100 PHY(serdes)] (PCIX:100MHz:64-bit) 1000Base-SX Ethernet 00:11:25:75:af:6f
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[0] TSOcap[1]
eth1: dma_rwctrl[769f4000] dma_mask[64-bit]
PM: Adding info for No Bus:lo
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
Fusion MPT base driver 3.04.04
Copyright (c) 1999-2007 LSI Logic Corporation
Fusion MPT SPI Host driver 3.04.04
ACPI: PCI Interrupt 0000:02:02.0[A] -> GSI 26 (level, low) -> IRQ 26
mptbase: Initiating ioc0 bringup
ioc0: 53C1030: Capabilities={Initiator}
scsi0 : ioc0: LSI53C1030, FwRev=01032700h, Ports=1, MaxQ=222, IRQ=26
PM: Adding info for No Bus:host0
PM: Adding info for No Bus:target0:0:0
scsi 0:0:0:0: Direct-Access IBM-ESXS ST973401LC FN B41D PQ: 0 ANSI: 4
target0:0:0: Beginning Domain Validation
target0:0:0: Ending Domain Validation
target0:0:0: FAST-160 WIDE SCSI 320.0 MB/s DT IU RTI WRFLOW PCOMP (6.25 ns, offset 63)
PM: Adding info for scsi:0:0:0:0
SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
sda: Write Protect is off
sda: Mode Sense: b3 00 10 08
SCSI device sda: write cache: disabled, read cache: enabled, supports DPO and FUA
SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
sda: Write Protect is off
sda: Mode Sense: b3 00 10 08
SCSI device sda: write cache: disabled, read cache: enabled, supports DPO and FUA
sda: sda1 sda2
sd 0:0:0:0: Attached scsi disk sda
sd 0:0:0:0: Attached scsi generic sg0 type 0
PM: Adding info for No Bus:target0:0:1
PM: Removing info for No Bus:target0:0:1
PM: Adding info for No Bus:target0:0:2
PM: Removing info for No Bus:target0:0:2
PM: Adding info for No Bus:target0:0:3
PM: Removing info for No Bus:target0:0:3
PM: Adding info for No Bus:target0:0:4
PM: Removing info for No Bus:target0:0:4
PM: Adding info for No Bus:target0:0:5
PM: Removing info for No Bus:target0:0:5
PM: Adding info for No Bus:target0:0:6
PM: Removing info for No Bus:target0:0:6
PM: Adding info for No Bus:target0:0:8
PM: Removing info for No Bus:target0:0:8
PM: Adding info for No Bus:target0:0:9
PM: Removing info for No Bus:target0:0:9
PM: Adding info for No Bus:target0:0:10
PM: Removing info for No Bus:target0:0:10
PM: Adding info for No Bus:target0:0:11
PM: Removing info for No Bus:target0:0:11
PM: Adding info for No Bus:target0:0:12
PM: Removing info for No Bus:target0:0:12
PM: Adding info for No Bus:target0:0:13
PM: Removing info for No Bus:target0:0:13
PM: Adding info for No Bus:target0:0:14
PM: Removing info for No Bus:target0:0:14
PM: Adding info for No Bus:target0:0:15
PM: Removing info for No Bus:target0:0:15
PM: Adding info for No Bus:target0:1:0
PM: Removing info for No Bus:target0:1:0
PM: Adding info for No Bus:target0:1:1
PM: Removing info for No Bus:target0:1:1
PM: Adding info for No Bus:target0:1:2
PM: Removing info for No Bus:target0:1:2
PM: Adding info for No Bus:target0:1:3
PM: Removing info for No Bus:target0:1:3
PM: Adding info for No Bus:target0:1:4
PM: Removing info for No Bus:target0:1:4
PM: Adding info for No Bus:target0:1:5
PM: Removing info for No Bus:target0:1:5
PM: Adding info for No Bus:target0:1:6
PM: Removing info for No Bus:target0:1:6
PM: Adding info for No Bus:target0:1:8
PM: Removing info for No Bus:target0:1:8
PM: Adding info for No Bus:target0:1:9
PM: Removing info for No Bus:target0:1:9
PM: Adding info for No Bus:target0:1:10
PM: Removing info for No Bus:target0:1:10
PM: Adding info for No Bus:target0:1:11
PM: Removing info for No Bus:target0:1:11
PM: Adding info for No Bus:target0:1:12
PM: Removing info for No Bus:target0:1:12
PM: Adding info for No Bus:target0:1:13
PM: Removing info for No Bus:target0:1:13
PM: Adding info for No Bus:target0:1:14
PM: Removing info for No Bus:target0:1:14
PM: Adding info for No Bus:target0:1:15
PM: Removing info for No Bus:target0:1:15
Fusion MPT FC Host driver 3.04.04
Fusion MPT SAS Host driver 3.04.04
Fusion MPT misc device (ioctl) driver 3.04.04
PM: Adding info for No Bus:mptctl
mptctl: Registered with Fusion MPT base driver
mptctl: /dev/mptctl @ (major,minor=10,220)
Fusion MPT LAN driver 3.04.04
mptlan: ioc0: PortNum=0, ProtocolFlags=08h (Itlb)
mptlan: ioc0: Hmmm... LAN protocol seems to be disabled on this adapter port!
ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
ACPI: PCI Interrupt 0000:01:00.0[D] -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:01:00.0: OHCI Host Controller
ohci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 1
ohci_hcd 0000:01:00.0: irq 19, io mem 0xfeaff000
usb usb1: new device found, idVendor=0000, idProduct=0000
usb usb1: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: OHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.21-rc2-mm2-autokern1 ohci_hcd
usb usb1: SerialNumber: 0000:01:00.0
PM: Adding info for usb:usb1
PM: Adding info for No Bus:usbdev1.1_ep00
usb usb1: configuration #1 chosen from 1 choice
PM: Adding info for usb:1-0:1.0
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 3 ports detected
PM: Adding info for No Bus:usbdev1.1_ep81
PM: Adding info for No Bus:usbdev1.1
ACPI: PCI Interrupt 0000:01:00.1[D] -> GSI 19 (level, low) -> IRQ 19
ohci_hcd 0000:01:00.1: OHCI Host Controller
ohci_hcd 0000:01:00.1: new USB bus registered, assigned bus number 2
ohci_hcd 0000:01:00.1: irq 19, io mem 0xfeafe000
usb usb2: new device found, idVendor=0000, idProduct=0000
usb usb2: new device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: OHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.21-rc2-mm2-autokern1 ohci_hcd
usb usb2: SerialNumber: 0000:01:00.1
PM: Adding info for usb:usb2
PM: Adding info for No Bus:usbdev2.1_ep00
usb usb2: configuration #1 chosen from 1 choice
PM: Adding info for usb:2-0:1.0
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 3 ports detected
PM: Adding info for No Bus:usbdev2.1_ep81
PM: Adding info for No Bus:usbdev2.1
USB Universal Host Controller Interface driver v3.0
Initializing USB Mass Storage driver...
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usbcore: registered new interface driver libusual
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: No PS/2 controller found. Probing ports directly.
PM: Adding info for platform:i8042
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
PM: Adding info for serio:serio0
PM: Adding info for serio:serio1
mice: PS/2 mouse device common for all mice
async_tx: api initialized (sync-only)
xor: automatically using best checksumming function: generic_sse
generic_sse: 6105.000 MB/sec
xor: using function: generic_sse (6105.000 MB/sec)
PM: Adding info for No Bus:device-mapper
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
powernow-k8: Found 4 Dual Core AMD Opteron(tm) Processor 270 processors (version 2.00.00)
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
powernow-k8: MP systems not supported by PSB BIOS structure
Freeing unused kernel memory: 340k freed
Write protecting the kernel read-only data: 984k
Red Hat nash version 5.0.32 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Making device-mapper control node
Scanning logical volumes
Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2
Activating logical volumes
4 logical volume(s) in volume group "VolGroup00" now active
Creating root device.
Mounting root filesystem.
kjournald starting. Commit interval 5 seconds
Setting up otherEXT3-fs: mounted filesystem with ordered data mode.
filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
PM: Adding info for No Bus:vcs1
PM: Adding info for No Bus:vcsa1
PM: Removing info for No Bus:vcs1
PM: Removing info for No Bus:vcsa1
INIT: version 2.86 booting
Welcome to Fedora Core
Press 'I' to enter interactive startup.
Setting clock (localtime): Thu Mar 8 12:33:58 CST 2007 [ OK ]
Starting udev: [ OK ]
Setting hostname bl6-13.ltc.austin.ibm.com: [ OK ]
Setting up Logical Volume Management: 4 logical volume(s) in volume group "VolGroup00" now active
[ OK ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/VolGroup00/LogVol00
/dev/VolGroup00/LogVol00: clean, 363818/7929856 files, 3638923/7929856 blocks
[/sbin/fsck.ext3 (1) -- /boot] fsck.ext3 -a /dev/sda1
/boot: clean, 85/512512 files, 81422/512064 blocks
[ OK ]
Remounting root filesystem in read-write mode: [ OK ]
Mounting local filesystems: [ OK ]
Enabling local filesystem quotas: [ OK ]
Enabling swap space: [ OK ]
INIT: Entering runlevel: 3
Entering non-interactive startup
Starting readahead_early: Starting background readahead: [ OK ]
[ OK ]
FATAL: Error inserting acpi_cpufreq (/lib/modules/2.6.21-rc2-mm2-autokern1/kernel/arch/x86_64/kernel/cpufreq/acpi-cpufreq.ko): No such device
Bringing up loopback interface: [ OK ]
Bringing up interface eth1: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Starting irqbalance: [ OK ]
Starting portmap: [ OK ]
Starting NFS statd: [ OK ]
Starting RPC idmapd: [ OK ]
Starting kdump: No kdump kernel image found.[WARNING]
Tried to locate /boot/vmlinux--kdump[ OK ]
Starting system message bus: [ OK ]
Starting Bluetooth services:[ OK ][ OK ]
Mounting other filesystems: [ OK ]
Starting hidd: [ OK ]
Starting automount: [ OK ]
Starting smartd: [ OK ]
Starting acpi daemon: [ OK ]
Starting hpiod: [ OK ]
Starting hpssd: [ OK ]
Starting cups: [ OK ]
Starting sshd: [ OK ]
Starting sendmail: [ OK ]
Starting sm-client: [ OK ]
Starting console mouse services: [ OK ]
Starting crond: [ OK ]
Starting xfs: [ OK ]
Starting anacron: [ OK ]
Starting atd: [ OK ]
Starting Avahi daemon: general protection fault: 0000 [1] SMP
last sysfs file: class/net/eth1/address
CPU 3
Modules linked in: ipv6 hidp rfcomm l2cap bluetooth sunrpc video button battery asus_acpi ac lp parport_pc parport nvram pcspkr amd_rng rng_core i2c_amd756 i2c_core
Pid: 0, comm: swapper Not tainted 2.6.21-rc2-mm2-autokern1 #1
RIP: 0010:[<ffffffff80483dbc>] [<ffffffff80483dbc>] inet_putpeer+0x10/0x53
RSP: 0018:ffff810001813ec0 EFLAGS: 00010202
RAX: ffff810001623500 RBX: e805b300000003bf RCX: ffff8100039c8438
RDX: ffff81000219c280 RSI: ffff81000104bf80 RDI: ffffffff80622970
RBP: ffff81000219c280 R08: 0000000000000003 R09: ffff810002f54100
R10: 00000000fc000106 R11: ffff8100039c83c0 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
FS: 00002ab1c24786f0(0000) GS:ffff810001401680(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 00002ab1c2217000 CR3: 0000000003be8000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff81000162e000, task ffff810001623500)
Stack: ffff810002eb3600 ffffffff80483bfe ffff810001628000 ffff810002e88200
ffff81000219c280 ffffffff8046c03b ffff81000219c500 ffff81000104bf80
0000000000000001 ffffffff804801a8 0000000000000001 ffffffff8025d8e2
Call Trace:
<IRQ> [<ffffffff80483bfe>] ipv4_dst_destroy+0x2c/0x58
[<ffffffff8046c03b>] dst_destroy+0x85/0xdf
[<ffffffff804801a8>] dst_rcu_free+0x19/0x29
[<ffffffff8025d8e2>] __rcu_process_callbacks+0x122/0x18a
[<ffffffff8023722b>] __do_softirq+0x55/0xc3
[<ffffffff8020acfc>] call_softirq+0x1c/0x28
[<ffffffff8020c095>] do_softirq+0x2c/0x7d
[<ffffffff80218ae6>] smp_apic_timer_interrupt+0x49/0x5f
[<ffffffff80208ca4>] default_idle+0x0/0x3d
[<ffffffff8020a7a6>] apic_timer_interrupt+0x66/0x70
<EOI> [<ffffffff80208ccd>] default_idle+0x29/0x3d
[<ffffffff80208d6c>] cpu_idle+0x8b/0xae
Code: f0 ff 4b 2c 0f 94 c0 84 c0 74 2b 48 8b 05 ba eb 19 00 48 c7
RIP [<ffffffff80483dbc>] inet_putpeer+0x10/0x53
RSP <ffff810001813ec0>
Kernel panic - not syncing: Aiee, killing interrupt handler!
-- 0:conmux-control -- time-stamp -- Mar/08/07 10:34:35 --
-- 0:conmux-control -- time-stamp -- Mar/08/07 10:47:37 --
(bot:conmon-payload) disconnected
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-09 14:00 ` Mel Gorman
@ 2007-03-09 16:40 ` Christoph Lameter
0 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-09 16:40 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, Linux Kernel Mailing List,
Linux Memory Management List, mpm, Manfred Spraul
On Fri, 9 Mar 2007, Mel Gorman wrote:
> I'm not sure what you mean by per-order queues. The buddy allocator already
> has per-order lists.
Somehow they do not seem to work right. SLAB (and now SLUB too) can avoid
(or defer) fragmentation by keeping its own queues.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [SLUB 0/3] SLUB: The unqueued slab allocator V4
2007-03-09 15:06 ` Mel Gorman
@ 2007-03-09 17:21 ` Christoph Lameter
0 siblings, 0 replies; 21+ messages in thread
From: Christoph Lameter @ 2007-03-09 17:21 UTC (permalink / raw)
To: Mel Gorman
Cc: akpm, Marcelo Tosatti, Linux Kernel Mailing List,
Linux Memory Management List, mpm, Manfred Spraul
On Fri, 9 Mar 2007, Mel Gorman wrote:
> The results without slub_debug were not good except for IA64. x86_64 and ppc64
> both blew up for a variety of reasons. The IA64 results were
Yuck that is the dst issue that Adrian is also looking at. Likely an issue
with slab merging and RCU frees.
> KernBench Comparison
> --------------------
> 2.6.21-rc2-mm2-clean 2.6.21-rc2-mm2-slub
> %diff
> User CPU time 1084.64 1032.93 4.77%
> System CPU time 73.38 63.14 13.95%
> Total CPU time 1158.02 1096.07 5.35%
> Elapsed time 307.00 285.62 6.96%
Wow! The first indication that we are on the right track with this.
> AIM9 Comparison
> 2 page_test 2097119.26 3398259.27 1301140.01 62.04% System Allocations & Pages/second
Wow! Must have all stayed within slab boundaries.
> 8 link_test 64776.04 7488.13 -57287.91 -88.44% Link/Unlink Pairs/second
Crap. Maybe we straddled a slab boundary here?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2007-03-09 17:21 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-03-07 2:35 [SLUB 0/3] SLUB: The unqueued slab allocator V4 Christoph Lameter
2007-03-07 2:35 ` [SLUB 1/3] SLUB core Christoph Lameter
2007-03-07 2:35 ` [SLUB 2/3] Large kmalloc pass through. Removal of large general slabs Christoph Lameter
2007-03-07 2:40 ` Matt Mackall
2007-03-07 3:22 ` Christoph Lameter
2007-03-07 9:01 ` Peter Zijlstra
2007-03-07 15:34 ` Christoph Lameter
2007-03-07 18:03 ` Matt Mackall
2007-03-07 18:23 ` Christoph Lameter
2007-03-07 2:35 ` [SLUB 3/3] Guarantee minimum number of objects in a slab Christoph Lameter
2007-03-08 10:54 ` [SLUB 0/3] SLUB: The unqueued slab allocator V4 Mel Gorman
2007-03-08 16:48 ` Christoph Lameter
2007-03-08 17:40 ` Mel Gorman
2007-03-08 18:16 ` Christoph Lameter
2007-03-09 13:55 ` Mel Gorman
2007-03-08 21:54 ` Christoph Lameter
2007-03-09 14:00 ` Mel Gorman
2007-03-09 16:40 ` Christoph Lameter
2007-03-09 15:06 ` Mel Gorman
2007-03-09 17:21 ` Christoph Lameter
2007-03-08 17:46 ` Christoph Lameter
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox