Might refill_inactive_zone () be too aggressive?

Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On my target board, an ARM cpu with 32MiB or RAM, I'm finding that the
performance is quite poor once RAM fills with IO pages and the code
pages of the program being executed are evicted.  

In my test setup, rootfs is mounted over NFS.  The degenerate example
is a simple program that copies a 40MiB file over NFS using a
read/write loop.  As it runs and as memory fills with NFS cached
pages, I can watched the VM evict the code that is executing the loop.
Since there are no other programs running (no TLB flushes from context
switching), there is nothing to stop the VM from aging the code pages.
During the copy, it may evict this same page a dozen times.  While I
understand that this setup by design, I wonder if there isn't
something that can (or should) be done to reduce this behavior.

There are a couple of other things to keep in mind.  

  1) This is an embedded system.  
  2) The root filesystem will not be NFS mounted in production.  The
     root is most likely to be stored in bootflash.  
  3) Some of these systems may perform significant amounts of IO, but
     almost none will be filesystem IO.  Thus, there is unlikely to be
     much hanging about the page cache.
  4) Performance in my test scenarios is quite poor.  Once I've copied
     the 40MiB file, executing an 'ls' command may take several
     seconds while the machine reloads libraries from the NFS server.
     The cached IO pages hang about in RAM for some time such
     that any programs executed will experience code page evictions.
  5) Removing the reclaim_mapped=1 line improves system response
     dramatically...just as I'd expect.

So, is this something to worry about?  Should it be a tunable feature?
Should this be something addressed in the platform specific VM code?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Fri, Apr 16, 2004 at 11:09:20PM -0700, Marc Singer wrote:
>   5) Removing the reclaim_mapped=1 line improves system response
>      dramatically...just as I'd expect.
> So, is this something to worry about?  Should it be a tunable feature?
> Should this be something addressed in the platform specific VM code?

A very interesting point there. The tendency to set reclaim_mapped = 1
is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve
your performance or behave closer to how the case you cited where vmscan.c
never sets reclaim_mapped = 1 improved performance.

The default value is 60, which begins unmapping mapped memory about
when 40% of memory is mapped by userspace.


-- wli

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On Fri, Apr 16, 2004 at 11:18:47PM -0700, William Lee Irwin III wrote:
> On Fri, Apr 16, 2004 at 11:09:20PM -0700, Marc Singer wrote:
> >   5) Removing the reclaim_mapped=1 line improves system response
> >      dramatically...just as I'd expect.
> > So, is this something to worry about?  Should it be a tunable feature?
> > Should this be something addressed in the platform specific VM code?
> 
> A very interesting point there. The tendency to set reclaim_mapped = 1
> is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve
> your performance or behave closer to how the case you cited where vmscan.c
> never sets reclaim_mapped = 1 improved performance.
> 
> The default value is 60, which begins unmapping mapped memory about
> when 40% of memory is mapped by userspace.

I did a little more looking at when reclaim_mapped is set to one.  In
my case, I don't think that very much memory is mapped.  I've got one
program running that has one or two code pages, there may be some
libraries.  The system has 28MiB of free RAM.  I don't see how I could
be getting more than 20% of RAM mapped.
 
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Fri, Apr 16, 2004 at 11:18:47PM -0700, William Lee Irwin III wrote:
>> A very interesting point there. The tendency to set reclaim_mapped = 1
>> is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve
>> your performance or behave closer to how the case you cited where vmscan.c
>> never sets reclaim_mapped = 1 improved performance.
>> The default value is 60, which begins unmapping mapped memory about
>> when 40% of memory is mapped by userspace.

On Sat, Apr 17, 2004 at 07:08:12AM -0700, Marc Singer wrote:
> I did a little more looking at when reclaim_mapped is set to one.  In
> my case, I don't think that very much memory is mapped.  I've got one
> program running that has one or two code pages, there may be some
> libraries.  The system has 28MiB of free RAM.  I don't see how I could
> be getting more than 20% of RAM mapped.

So adjusting swappiness didn't help? Does adjusting this help?
(uncompiled, untested)


-- wli


Index: singer-2.6.5-mm6/include/linux/sysctl.h
===================================================================
--- singer-2.6.5-mm6.orig/include/linux/sysctl.h	2004-04-14 23:21:18.000000000 -0700
+++ singer-2.6.5-mm6/include/linux/sysctl.h	2004-04-17 07:16:25.000000000 -0700
@@ -161,6 +161,7 @@
 	VM_MAX_MAP_COUNT=22,	/* int: Maximum number of mmaps/address-space */
 	VM_LAPTOP_MODE=23,	/* vm laptop mode */
 	VM_BLOCK_DUMP=24,	/* block dump mode */
+	VM_MAPPED_SCALE_FACTOR=25, /* scale factor for mapped ratio */
 };
 
 
Index: singer-2.6.5-mm6/include/linux/swap.h
===================================================================
--- singer-2.6.5-mm6.orig/include/linux/swap.h	2004-04-14 23:21:18.000000000 -0700
+++ singer-2.6.5-mm6/include/linux/swap.h	2004-04-17 07:17:05.000000000 -0700
@@ -174,7 +174,7 @@
 /* linux/mm/vmscan.c */
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
-extern int vm_swappiness;
+extern int vm_swappiness, vm_mapped_scale_factor;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */
Index: singer-2.6.5-mm6/kernel/sysctl.c
===================================================================
--- singer-2.6.5-mm6.orig/kernel/sysctl.c	2004-04-14 23:21:19.000000000 -0700
+++ singer-2.6.5-mm6/kernel/sysctl.c	2004-04-17 07:16:10.000000000 -0700
@@ -766,6 +766,17 @@
 		.strategy	= &sysctl_intvec,
 		.extra1		= &zero,
 	},
+	{
+		.ctl_name	= VM_MAPPED_SCALE_FACTOR,
+		.procname	= "mapped_scale_factor",
+		.data		= &vm_mapped_scale_factor,
+		.maxlen		= sizeof(vm_mapped_scale_factor),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_minmax,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+		.extra2		= &one_hundred,
+	},
 	{ .ctl_name = 0 }
 };
 
Index: singer-2.6.5-mm6/mm/vmscan.c
===================================================================
--- singer-2.6.5-mm6.orig/mm/vmscan.c	2004-04-14 23:21:19.000000000 -0700
+++ singer-2.6.5-mm6/mm/vmscan.c	2004-04-17 07:18:33.000000000 -0700
@@ -43,6 +43,7 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
+int vm_mapped_scale_factor = 50;
 static long total_memory;
 
 #define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))
@@ -625,7 +626,7 @@
 	 * mapped memory instead of just pagecache.  Work out how much memory
 	 * is mapped.
 	 */
-	mapped_ratio = (ps->nr_mapped * 100) / total_memory;
+	mapped_ratio = (ps->nr_mapped * vm_mapped_scale_factor) / total_memory;
 
 	/*
 	 * Now decide how much we really want to unmap some pages.  The mapped
@@ -636,7 +637,7 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swap_tendency = mapped_ratio + distress + vm_swappiness;
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Fri, Apr 16, 2004 at 11:18:47PM -0700, William Lee Irwin III wrote:
>> A very interesting point there. The tendency to set reclaim_mapped = 1
>> is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve
>> your performance or behave closer to how the case you cited where vmscan.c
>> never sets reclaim_mapped = 1 improved performance.
>> The default value is 60, which begins unmapping mapped memory about
>> when 40% of memory is mapped by userspace.

On Sat, Apr 17, 2004 at 07:08:12AM -0700, Marc Singer wrote:
> I did a little more looking at when reclaim_mapped is set to one.  In
> my case, I don't think that very much memory is mapped.  I've got one
> program running that has one or two code pages, there may be some
> libraries.  The system has 28MiB of free RAM.  I don't see how I could
> be getting more than 20% of RAM mapped.

Also, it would be helpful to log periodic snapshots of /proc/meminfo and
/proc/vmstat to see what's being fed to the various heuristics.


-- wli

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On Fri, Apr 16, 2004 at 11:18:47PM -0700, William Lee Irwin III wrote:
> On Fri, Apr 16, 2004 at 11:09:20PM -0700, Marc Singer wrote:
> >   5) Removing the reclaim_mapped=1 line improves system response
> >      dramatically...just as I'd expect.
> > So, is this something to worry about?  Should it be a tunable feature?
> > Should this be something addressed in the platform specific VM code?
> 
> A very interesting point there. The tendency to set reclaim_mapped = 1
> is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve
> your performance or behave closer to how the case you cited where vmscan.c
> never sets reclaim_mapped = 1 improved performance.
> 
> The default value is 60, which begins unmapping mapped memory about
> when 40% of memory is mapped by userspace.

I don't think that's the whole story.  I printed distress,
mapped_ratio, and swappiness when vmscan starts trying to reclaim
mapped pages.

reclaim_mapped: distress 50  mapped_ratio 0  swappiness 60 

  50 + 60 > 100 

So, part of the problem is swappiness.  I could set that value to 25,
for example, to stop the machine from swapping.

I'd be fine stopping here, except for you comment about what
swappiness means.  In my case, nearly none of memory is mapped.  It is
zone priority which has dropped to 1 that is precipitating the
eviction.  Is this what you expect and want?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Sat, Apr 17, 2004 at 10:57:24AM -0700, Marc Singer wrote:
> I don't think that's the whole story.  I printed distress,
> mapped_ratio, and swappiness when vmscan starts trying to reclaim
> mapped pages.
> reclaim_mapped: distress 50  mapped_ratio 0  swappiness 60 
>   50 + 60 > 100 
> So, part of the problem is swappiness.  I could set that value to 25,
> for example, to stop the machine from swapping.
> I'd be fine stopping here, except for you comment about what
> swappiness means.  In my case, nearly none of memory is mapped.  It is
> zone priority which has dropped to 1 that is precipitating the
> eviction.  Is this what you expect and want?

I'm not sure it's expected. Maybe this patch fares better?


-- wli


Index: singer-2.6.5-mm6/mm/vmscan.c
===================================================================
--- singer-2.6.5-mm6.orig/mm/vmscan.c	2004-04-14 23:21:19.000000000 -0700
+++ singer-2.6.5-mm6/mm/vmscan.c	2004-04-17 11:09:35.000000000 -0700
@@ -636,7 +636,7 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swap_tendency = mapped_ratio / 2 + max(distress, vm_swappiness);
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On Sat, Apr 17, 2004 at 11:10:42AM -0700, William Lee Irwin III wrote:
> On Sat, Apr 17, 2004 at 10:57:24AM -0700, Marc Singer wrote:
> > I don't think that's the whole story.  I printed distress,
> > mapped_ratio, and swappiness when vmscan starts trying to reclaim
> > mapped pages.
> > reclaim_mapped: distress 50  mapped_ratio 0  swappiness 60 
> >   50 + 60 > 100 
> > So, part of the problem is swappiness.  I could set that value to 25,
> > for example, to stop the machine from swapping.
> > I'd be fine stopping here, except for you comment about what
> > swappiness means.  In my case, nearly none of memory is mapped.  It is
> > zone priority which has dropped to 1 that is precipitating the
> > eviction.  Is this what you expect and want?
> 
> I'm not sure it's expected. Maybe this patch fares better?

Ah, that's a much different thing.  That works for me.  Is that
something you'd want to put into the kernel?

> 
> 
> -- wli
> 
> 
> Index: singer-2.6.5-mm6/mm/vmscan.c
> ===================================================================
> --- singer-2.6.5-mm6.orig/mm/vmscan.c	2004-04-14 23:21:19.000000000 -0700
> +++ singer-2.6.5-mm6/mm/vmscan.c	2004-04-17 11:09:35.000000000 -0700
> @@ -636,7 +636,7 @@
>  	 *
>  	 * A 100% value of vm_swappiness overrides this algorithm altogether.
>  	 */
> -	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
> +	swap_tendency = mapped_ratio / 2 + max(distress, vm_swappiness);
>  
>  	/*
>  	 * Now use this metric to decide whether to start moving mapped memory
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Sat, Apr 17, 2004 at 11:10:42AM -0700, William Lee Irwin III wrote:
>> I'm not sure it's expected. Maybe this patch fares better?

On Sat, Apr 17, 2004 at 11:28:38AM -0700, Marc Singer wrote:
> Ah, that's a much different thing.  That works for me.  Is that
> something you'd want to put into the kernel?

Since we have a coherent story about this working for you, I think
we should probably send it upstream for review. I don't have a
particular opinion about it being the right thing to do, as since it's
a policy decision, it's rather arbitrary.

If this is important to you, it may help to numerically quantify your
results, e.g. some before/after benchmark/throughput/whatever numbers.


-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On Sat, Apr 17, 2004 at 11:33:25AM -0700, William Lee Irwin III wrote:
> On Sat, Apr 17, 2004 at 11:10:42AM -0700, William Lee Irwin III wrote:
> >> I'm not sure it's expected. Maybe this patch fares better?
> 
> On Sat, Apr 17, 2004 at 11:28:38AM -0700, Marc Singer wrote:
> > Ah, that's a much different thing.  That works for me.  Is that
> > something you'd want to put into the kernel?
> 
> Since we have a coherent story about this working for you, I think
> we should probably send it upstream for review. I don't have a
> particular opinion about it being the right thing to do, as since it's
> a policy decision, it's rather arbitrary.
> 
> If this is important to you, it may help to numerically quantify your
> results, e.g. some before/after benchmark/throughput/whatever numbers.

That's a difficult thing to do.  My test setup uses an NFS root and
the IO is over NFS.  Due to some oddities in the NFS code, performance
is variable to a degree that does not make for good timing
comparisons.  I'm looking for a way to enable TCP nfsroot mounts.
Once this is working, I may be able to get some reliable numbers.

It's your call about waiting for performance numbers.  As soon as I
have better data, I'll post it.  Setting the swappiness flag does
work, so I can give my users something for now.  It is possible that
this will work for all cases that I'm ever going to see.  Setting
swappiness to zero and the user mapping 50% of RAM will once again
cause reclaim_mapped to go into action.  The difference is that with
swappiness of 60, I'm not allowed to keep any mapped pages in RAM.  At
swappiness of 0, I'm allows to keep half of total RAM mapped.

Cheers.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Sat, Apr 17, 2004 at 11:44:24AM -0700, Marc Singer wrote:
> That's a difficult thing to do.  My test setup uses an NFS root and
> the IO is over NFS.  Due to some oddities in the NFS code, performance
> is variable to a degree that does not make for good timing
> comparisons.  I'm looking for a way to enable TCP nfsroot mounts.
> Once this is working, I may be able to get some reliable numbers.
> It's your call about waiting for performance numbers.  As soon as I
> have better data, I'll post it.  Setting the swappiness flag does
> work, so I can give my users something for now.  It is possible that
> this will work for all cases that I'm ever going to see.  Setting
> swappiness to zero and the user mapping 50% of RAM will once again
> cause reclaim_mapped to go into action.  The difference is that with
> swappiness of 60, I'm not allowed to keep any mapped pages in RAM.  At
> swappiness of 0, I'm allows to keep half of total RAM mapped.

That's something of a normative question about the heuristics, and I
try to steer clear of those, though I'm not entirely sure that's how I
would interpret it the tunings for your descriptive parts.

In the absence of "hard" numbers, you might still be able to use things
like wall clock timings. Another thing that would help is to expose the
thing to a variety of workloads/etc. For that, I guess I post to lkml.


-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[Marc Singer]

On Sat, Apr 17, 2004 at 12:19:55PM -0700, William Lee Irwin III wrote:
> On Sat, Apr 17, 2004 at 11:44:24AM -0700, Marc Singer wrote:
> > That's a difficult thing to do.  My test setup uses an NFS root and
> > the IO is over NFS.  Due to some oddities in the NFS code, performance
> > is variable to a degree that does not make for good timing
> > comparisons.  I'm looking for a way to enable TCP nfsroot mounts.
> > Once this is working, I may be able to get some reliable numbers.
> > It's your call about waiting for performance numbers.  As soon as I
> > have better data, I'll post it.  Setting the swappiness flag does
> > work, so I can give my users something for now.  It is possible that
> > this will work for all cases that I'm ever going to see.  Setting
> > swappiness to zero and the user mapping 50% of RAM will once again
> > cause reclaim_mapped to go into action.  The difference is that with
> > swappiness of 60, I'm not allowed to keep any mapped pages in RAM.  At
> > swappiness of 0, I'm allows to keep half of total RAM mapped.
> 
> That's something of a normative question about the heuristics, and I
> try to steer clear of those, though I'm not entirely sure that's how I
> would interpret it the tunings for your descriptive parts.

The more I think about it, the more I think that there is something
awry.  Once distress reaches 50, swappiness is going to rule the
ability of the system to keep pages mapped.  If swappiness is then 50
or more, vmscan is going age and then purge every mapped page.

> In the absence of "hard" numbers, you might still be able to use things
> like wall clock timings. Another thing that would help is to expose the
> thing to a variety of workloads/etc. For that, I guess I post to lkml.

Are you suggesting that I post to LKML to get some ideas about other
workloads?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>

Re: Might refill_inactive_zone () be too aggressive?

[William Lee Irwin III]

On Sat, Apr 17, 2004 at 12:19:55PM -0700, William Lee Irwin III wrote:
>> That's something of a normative question about the heuristics, and I
>> try to steer clear of those, though I'm not entirely sure that's how I
>> would interpret it the tunings for your descriptive parts.

On Sat, Apr 17, 2004 at 12:25:47PM -0700, Marc Singer wrote:
> The more I think about it, the more I think that there is something
> awry.  Once distress reaches 50, swappiness is going to rule the
> ability of the system to keep pages mapped.  If swappiness is then 50
> or more, vmscan is going age and then purge every mapped page.

There's not any a priori reason to believe this is wrong that I know of,
though it looks like things do better for you when slightly rearranged.


On Sat, Apr 17, 2004 at 12:19:55PM -0700, William Lee Irwin III wrote:
>> In the absence of "hard" numbers, you might still be able to use things
>> like wall clock timings. Another thing that would help is to expose the
>> thing to a variety of workloads/etc. For that, I guess I post to lkml.

On Sat, Apr 17, 2004 at 12:25:47PM -0700, Marc Singer wrote:
> Are you suggesting that I post to LKML to get some ideas about other
> workloads?

Well, yes, but I just did so myself.


-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>