[PATCH] 0/2 swap token tuning

[PATCH] 0/2 swap token tuning

[Rik Van Riel]

A while ago the swap token (aka token based thrashing control)
mechanism was introduced into Linux.  This code improves performance
under heavy VM loads, but can reduce performance under very light
VM loads.

The cause turns out to be me overlooking something in the original
token based thrashing control paper: the swap token is only supposed
to be enforced while the task holding the swap token is paging data
in, not while the task is running (and referencing its working set).

The temporary solution in Linux was to disable the swap token code
and have users turn it on again via /proc.  The following patch
instead approximates the "only enforce the swap token if the task
holding it is swapping something in" idea.  This should make sure
the swap token is effectively disabled when the VM load is low.

I have not benchmarked these patches yet; instead, I'm posting
them before the weekend is over, hoping to catch a bit of test
time from others while my own tests are being run ;)

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

[PATCH] 1/2 swap token tuning

[Rik Van Riel]

Add sem_is_read/write_locked functions to the read/write semaphores,
along the same lines of the *_is_locked spinlock functions.  The swap
token tuning patch uses sem_is_read_locked; sem_is_write_locked is
added for completeness.

Signed-off-by: Rik van Riel <riel@redhat.com>

 asm-alpha/rwsem.h      |   10 ++++++++++
 asm-i386/rwsem.h       |   10 ++++++++++
 asm-ia64/rwsem.h       |   10 ++++++++++
 asm-ppc/rwsem.h        |   10 ++++++++++
 asm-ppc64/rwsem.h      |   10 ++++++++++
 asm-s390/rwsem.h       |   10 ++++++++++
 asm-sh/rwsem.h         |   10 ++++++++++
 asm-sparc64/rwsem.h    |   10 ++++++++++
 asm-x86_64/rwsem.h     |   10 ++++++++++
 linux/rwsem-spinlock.h |   10 ++++++++++
 10 files changed, 100 insertions(+)

diff -uNrp vanilla/include/asm-alpha/rwsem.h 2.6.12-rwsem/include/asm-alpha/rwsem.h
--- vanilla/include/asm-alpha/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-alpha/rwsem.h	2005-06-26 14:25:34.000000000 -0400
@@ -262,5 +262,15 @@ static inline long rwsem_atomic_update(l
 #endif
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _ALPHA_RWSEM_H */
diff -uNrp vanilla/include/asm-i386/rwsem.h 2.6.12-rwsem/include/asm-i386/rwsem.h
--- vanilla/include/asm-i386/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-i386/rwsem.h	2005-06-26 14:20:35.000000000 -0400
@@ -284,5 +284,15 @@ LOCK_PREFIX	"xadd %0,(%2)"
 	return tmp+delta;
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _I386_RWSEM_H */
diff -uNrp vanilla/include/asm-ia64/rwsem.h 2.6.12-rwsem/include/asm-ia64/rwsem.h
--- vanilla/include/asm-ia64/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-ia64/rwsem.h	2005-06-26 15:36:40.000000000 -0400
@@ -185,4 +185,14 @@ __downgrade_write (struct rw_semaphore *
 #define rwsem_atomic_add(delta, sem)	atomic_add(delta, (atomic_t *)(&(sem)->count))
 #define rwsem_atomic_update(delta, sem)	atomic_add_return(delta, (atomic_t *)(&(sem)->count))
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* _ASM_IA64_RWSEM_H */
diff -uNrp vanilla/include/asm-ppc/rwsem.h 2.6.12-rwsem/include/asm-ppc/rwsem.h
--- vanilla/include/asm-ppc/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-ppc/rwsem.h	2005-06-26 16:00:10.000000000 -0400
@@ -168,5 +168,15 @@ static inline int rwsem_atomic_update(in
 	return atomic_add_return(delta, (atomic_t *)(&sem->count));
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _PPC_RWSEM_XADD_H */
diff -uNrp vanilla/include/asm-ppc64/rwsem.h 2.6.12-rwsem/include/asm-ppc64/rwsem.h
--- vanilla/include/asm-ppc64/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-ppc64/rwsem.h	2005-06-26 15:59:48.000000000 -0400
@@ -163,5 +163,15 @@ static inline int rwsem_atomic_update(in
 	return atomic_add_return(delta, (atomic_t *)(&sem->count));
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _PPC_RWSEM_XADD_H */
diff -uNrp vanilla/include/asm-s390/rwsem.h 2.6.12-rwsem/include/asm-s390/rwsem.h
--- vanilla/include/asm-s390/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-s390/rwsem.h	2005-06-26 16:33:04.000000000 -0400
@@ -351,5 +351,15 @@ static inline long rwsem_atomic_update(l
 	return new;
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _S390_RWSEM_H */
diff -uNrp vanilla/include/asm-sh/rwsem.h 2.6.12-rwsem/include/asm-sh/rwsem.h
--- vanilla/include/asm-sh/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-sh/rwsem.h	2005-06-26 16:33:20.000000000 -0400
@@ -166,5 +166,15 @@ static inline int rwsem_atomic_update(in
 	return atomic_add_return(delta, (atomic_t *)(&sem->count));
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _ASM_SH_RWSEM_H */
diff -uNrp vanilla/include/asm-sparc64/rwsem.h 2.6.12-rwsem/include/asm-sparc64/rwsem.h
--- vanilla/include/asm-sparc64/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-sparc64/rwsem.h	2005-06-26 16:43:50.000000000 -0400
@@ -95,6 +95,16 @@ static __inline__ signed long rwsem_cmpx
 	return cmpxchg(&sem->count,old,new);
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 
 #endif /* _SPARC64_RWSEM_H */
diff -uNrp vanilla/include/asm-x86_64/rwsem.h 2.6.12-rwsem/include/asm-x86_64/rwsem.h
--- vanilla/include/asm-x86_64/rwsem.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/asm-x86_64/rwsem.h	2005-06-26 16:36:35.000000000 -0400
@@ -274,5 +274,15 @@ LOCK_PREFIX	"xaddl %0,(%2)"
 	return tmp+delta;
 }
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->count > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->count < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _X8664_RWSEM_H */
diff -uNrp vanilla/include/linux/rwsem-spinlock.h 2.6.12-rwsem/include/linux/rwsem-spinlock.h
--- vanilla/include/linux/rwsem-spinlock.h	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-rwsem/include/linux/rwsem-spinlock.h	2005-06-26 16:37:56.000000000 -0400
@@ -61,5 +61,15 @@ extern void FASTCALL(__up_read(struct rw
 extern void FASTCALL(__up_write(struct rw_semaphore *sem));
 extern void FASTCALL(__downgrade_write(struct rw_semaphore *sem));
 
+static inline int sem_is_read_locked(struct rw_semaphore *sem)
+{
+	return (sem->activity > 0);
+}
+
+static inline int sem_is_write_locked(struct rw_semaphore *sem)
+{
+	return (sem->activity < 0);
+}
+
 #endif /* __KERNEL__ */
 #endif /* _LINUX_RWSEM_SPINLOCK_H */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

[PATCH] 2/2 swap token tuning

[Rik Van Riel]

It turns out that the original swap token implementation, by Song
Jiang, only enforced the swap token while the task holding the
token is handling a page fault.  This patch approximates that,
without adding an additional flag to the mm_struct, by checking
whether the mm->mmap_sem is held for reading, like the page
fault code does.

This patch should have the effect of automatically, and gradually,
disabling the enforcement of the swap token when there is little
or no paging going on, and "turning up" the intensity of the swap
token code the more the task holding the token is thrashing.

Thanks to Song Jiang for pointing out this aspect of the token
based thrashing control concept.

Signed-off-by: Rik van Riel

--- 2.6.12-swaptoken/mm/thrash.c.orig	2005-06-17 15:48:29.000000000 -0400
+++ 2.6.12-swaptoken/mm/thrash.c	2005-06-26 17:17:04.000000000 -0400
@@ -19,7 +19,7 @@ static unsigned long swap_token_check;
 struct mm_struct * swap_token_mm = &init_mm;
 
 #define SWAP_TOKEN_CHECK_INTERVAL (HZ * 2)
-#define SWAP_TOKEN_TIMEOUT	0
+#define SWAP_TOKEN_TIMEOUT	300
 /*
  * Currently disabled; Needs further code to work at HZ * 300.
  */
--- 2.6.12-swaptoken/mm/rmap.c.orig	2005-06-26 17:16:50.000000000 -0400
+++ 2.6.12-swaptoken/mm/rmap.c	2005-06-26 17:13:17.000000000 -0400
@@ -301,7 +301,11 @@ static int page_referenced_one(struct pa
 		if (ptep_clear_flush_young(vma, address, pte))
 			referenced++;
 
-		if (mm != current->mm && !ignore_token && has_swap_token(mm))
+		/* Pretend the page is referenced if the task has the
+		   swap token and is in the middle of a page fault. */
+		if (mm != current->mm && !ignore_token &&
+				has_swap_token(mm) &&
+				sem_is_read_locked(mm->mmap_sem))
 			referenced++;
 
 		(*mapcount)--;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] 2/2 swap token tuning

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Sun, 2005-06-26 at 18:35 -0400, Rik Van Riel wrote:

<snip>

> This patch should have the effect of automatically, and gradually,
> disabling the enforcement of the swap token when there is little
> or no paging going on, and "turning up" the intensity of the swap
> token code the more the task holding the token is thrashing.
> 
> Thanks to Song Jiang for pointing out this aspect of the token
> based thrashing control concept.
> 
> Signed-off-by: Rik van Riel
> 

<snip>

I might be missing something here, but shouldn't this rather be:

> --- 2.6.12-swaptoken/mm/rmap.c.orig	2005-06-26 17:16:50.000000000 -0400
> +++ 2.6.12-swaptoken/mm/rmap.c	2005-06-26 17:13:17.000000000 -0400
> @@ -301,7 +301,11 @@ static int page_referenced_one(struct pa
>  		if (ptep_clear_flush_young(vma, address, pte))
>  			referenced++;
>  
> -		if (mm != current->mm && !ignore_token && has_swap_token(mm))
> +		/* Pretend the page is referenced if the task has the
> +		   swap token and is in the middle of a page fault. */
> +		if (mm != current->mm && !ignore_token &&
> +				has_swap_token(mm) &&
-+				sem_is_read_locked(mm->mmap_sem))
+                               sem_is_read_locked(&mm->mmap_sem))

As mm is a pointer, but mm->mmap_sem is not a pointer?  This is with
2.6.12-mm2 though ...


Thanks,

-- 
Martin Schlemmer


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] 2/2 swap token tuning

[Rik Van Riel]

On Mon, 27 Jun 2005, Martin Schlemmer wrote:

> -+				sem_is_read_locked(mm->mmap_sem))
> +                               sem_is_read_locked(&mm->mmap_sem))

Yes, you are right.  I sent out the patch before the weekend
was over, before having tested it locally ;)

My compile hit the error a few minutes after I sent out the
mail, doh ;)

Andrew has a fixed version of the patch already.

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] 2/2 swap token tuning

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 611 bytes --]

On Mon, 2005-06-27 at 09:08 -0400, Rik Van Riel wrote:
> On Mon, 27 Jun 2005, Martin Schlemmer wrote:
> 
> > -+				sem_is_read_locked(mm->mmap_sem))
> > +                               sem_is_read_locked(&mm->mmap_sem))
> 
> Yes, you are right.  I sent out the patch before the weekend
> was over, before having tested it locally ;)
> 
> My compile hit the error a few minutes after I sent out the
> mail, doh ;)
> 
> Andrew has a fixed version of the patch already.
> 

Ok, thanks - wanted to test it, and just wanted to verify that my
changes are OK.


Regards,

-- 
Martin Schlemmer


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH] 0/2 swap token tuning

[Ed Tomlinson]

On Sunday 26 June 2005 18:34, Rik Van Riel wrote:
> A while ago the swap token (aka token based thrashing control)
> mechanism was introduced into Linux.  This code improves performance
> under heavy VM loads, but can reduce performance under very light
> VM loads.
> 
> The cause turns out to be me overlooking something in the original
> token based thrashing control paper: the swap token is only supposed
> to be enforced while the task holding the swap token is paging data
> in, not while the task is running (and referencing its working set).
> 
> The temporary solution in Linux was to disable the swap token code
> and have users turn it on again via /proc.  The following patch
> instead approximates the "only enforce the swap token if the task
> holding it is swapping something in" idea.  This should make sure
> the swap token is effectively disabled when the VM load is low.
> 
> I have not benchmarked these patches yet; instead, I'm posting
> them before the weekend is over, hoping to catch a bit of test
> time from others while my own tests are being run ;)

Rik,

What are the suggested  values to put into /proc/sys/vm/swap_token_timeout ?
The docs are not at all clear about this (proc/filesystems.txt).

TIA,
Ed Tomlinson
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] 0/2 swap token tuning

[Rik Van Riel]

On Mon, 27 Jun 2005, Ed Tomlinson wrote:

> What are the suggested  values to put into /proc/sys/vm/swap_token_timeout ?
> The docs are not at all clear about this (proc/filesystems.txt).

Beats me ;)

I tried a number of values in the original implementation, and
300 seconds turned out to work fine...

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] 0/2 swap token tuning

[Rik Van Riel]

On Mon, 27 Jun 2005, Rik Van Riel wrote:
> On Mon, 27 Jun 2005, Ed Tomlinson wrote:
> 
> > What are the suggested  values to put into /proc/sys/vm/swap_token_timeout ?
> > The docs are not at all clear about this (proc/filesystems.txt).
> 
> Beats me ;)
> 
> I tried a number of values in the original implementation, and
> 300 seconds turned out to work fine...

Hmmmm, strange.   That means I ran it with an effective
timeout of only 2 seconds in my tests yesterday, and it
still had an effect !

Interesting ;)

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: [PATCH] 0/2 swap token tuning

[Rik Van Riel]

On Mon, 27 Jun 2005, Rik Van Riel wrote:
> On Mon, 27 Jun 2005, Rik Van Riel wrote:
> > On Mon, 27 Jun 2005, Ed Tomlinson wrote:
> > 
> > > What are the suggested  values to put into /proc/sys/vm/swap_token_timeout ?
> > > The docs are not at all clear about this (proc/filesystems.txt).
> > 
> > Beats me ;)
> > 
> > I tried a number of values in the original implementation, and
> > 300 seconds turned out to work fine...

Never mind my previous mail - after looking at kernel/sysctl.c
it turns out that the sysctl code compensates for the jiffies
vs. seconds difference.

Just the default in mm/thrash.c should be changed from "300"
to "(300 * HZ)"...

-- 
The Theory of Escalating Commitment: "The cost of continuing mistakes is
borne by others, while the cost of admitting mistakes is borne by yourself."
  -- Joseph Stiglitz, Nobel Laureate in Economics
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>