swapoff throttling and speedup?

swapoff throttling and speedup?

[Joel Krauska]

On occasion we need to unswap a system that's gotten unruly.

Scenario: Some leaky app eats up way more RAM than it should, and pushes
a few gigs of the running system in to swap.  The leaky app is killed, 
but there's still lots of good stuff sitting in swap that we need to tidy
up to get the system back to normal performance levels.


The normal recourse is to run
 swapoff -a ; swapon -a


I have two related questions about the swap tools and how they work.


1. Has anyone tried making a nicer swapoff?
Right now swapoff can be pretty aggressive if the system is otherwise
heavily loaded.  On systems that I need to leave running other jobs,
swapoff compounds the slowness of the system overall by burning up
a single CPU and lots of IO

I wrote a perl wrapper that briefly runs swapoff 
and then kills it, but it would seem more reasonable to have a knob
to make swapoff less aggressive. (max kb/s, etc)  

It looked to me like the swapoff code was immediately hitting kernel 
internals instead of doing more lifting itself (and making it 
obvious where I could insert some sleeps)

Has anyone found better options here?



2. A faster(multithreaded?) swapoff?
>From what I can tell, swapoff is single threaded, which seems to make 
unswapping a CPU bound activity.  

In the opposite use case of my first question, on systems that I /can/
halt all the running code (assuming if they've gone off the deep end and have
several gigs in SWAP) it can take quite a long time for unswap to 
tidy up the mess.  

Has anyone considered improvements to swapoff to speed it up?
(multiple threads?)


I'm hoping others have been down this road before.

As a rule, we try to avoid swapping when possible, but using:
vm.swappiness = 1

But it does still happen on occasion and that lead to this mail.

Cheers,

Joel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: swapoff throttling and speedup?

[KAMEZAWA Hiroyuki]

On Wed, 03 Jun 2009 10:01:39 -0700
Joel Krauska <jkrauska@gmail.com> wrote:

> On occasion we need to unswap a system that's gotten unruly.
> 
> Scenario: Some leaky app eats up way more RAM than it should, and pushes
> a few gigs of the running system in to swap.  The leaky app is killed, 
> but there's still lots of good stuff sitting in swap that we need to tidy
> up to get the system back to normal performance levels.
> 
> 
> The normal recourse is to run
>  swapoff -a ; swapon -a
> 
> 
> I have two related questions about the swap tools and how they work.
> 
> 
> 1. Has anyone tried making a nicer swapoff?
> Right now swapoff can be pretty aggressive if the system is otherwise
> heavily loaded.  On systems that I need to leave running other jobs,
> swapoff compounds the slowness of the system overall by burning up
> a single CPU and lots of IO
> 
> I wrote a perl wrapper that briefly runs swapoff 
> and then kills it, but it would seem more reasonable to have a knob
> to make swapoff less aggressive. (max kb/s, etc)  
> 
> It looked to me like the swapoff code was immediately hitting kernel 
> internals instead of doing more lifting itself (and making it 
> obvious where I could insert some sleeps)
> 
I haven't heard this swapoff issue for years.

Hmm, swapoff -a is proper operation ? (I don't think so..)
I think most of people just want "fast" swapoff.

> Has anyone found better options here?
> 
> 
If you know what are the leaky apps, memory cgroup may be a help for
avoiding unnecessary swap-out.
 
How about throttling swapoff's cpu usage by cpu scheduler cgroup ?
No help ?

> 
> 2. A faster(multithreaded?) swapoff?
> From what I can tell, swapoff is single threaded, which seems to make 
> unswapping a CPU bound activity.  
> 
> In the opposite use case of my first question, on systems that I /can/
> halt all the running code (assuming if they've gone off the deep end and have
> several gigs in SWAP) it can take quite a long time for unswap to 
> tidy up the mess.  
> 
> Has anyone considered improvements to swapoff to speed it up?
> (multiple threads?)
> 
not heared of in this mailing list.

But I think swapoff() is a system-call and making it as multithreaded is
not easy. (And we have to take care of complex racy cases...)

> 
> I'm hoping others have been down this road before.
> 
> As a rule, we try to avoid swapping when possible, but using:
> vm.swappiness = 1
> 
> But it does still happen on occasion and that lead to this mail.
> 


Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: swapoff throttling and speedup?

[Joel Krauska]

KAMEZAWA Hiroyuki wrote:
>> 1. Has anyone tried making a nicer swapoff?
>> Right now swapoff can be pretty aggressive if the system is otherwise
>> heavily loaded.  On systems that I need to leave running other jobs,
>> swapoff compounds the slowness of the system overall by burning up
>> a single CPU and lots of IO
>>
>> I wrote a perl wrapper that briefly runs swapoff 
>> and then kills it, but it would seem more reasonable to have a knob
>> to make swapoff less aggressive. (max kb/s, etc)  
>>
>> It looked to me like the swapoff code was immediately hitting kernel 
>> internals instead of doing more lifting itself (and making it 
>> obvious where I could insert some sleeps)
>>

I find I need a slower swapoff when a system that's already running very hot
needs to be recovered from lots of swapping without overly impacting the other
running processes.

The bulk of the work is still being done in normal RAM, and the overhead
of consuming an entire CPU just for swapoff degrades my other running processes.

> How about throttling swapoff's cpu usage by cpu scheduler cgroup ?
> No help ?

I think swapoff is all done as systemcalls, not in userspace, so I'm not
sure that cgroups would apply here.  (granted I had never heard of control
groups until just now...)

My initial analogy and insight for this was the MD RAID rebuild throttle toggles.
/proc/sys/dev/raid/speed_limit_max

Which I've had to tune down on occasion to reduce impact to other running processes.
(aside: MD RAID rebuilds do seem to be multi-threaded?)

Thanks Kame,

Joel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: swapoff throttling and speedup?

[KAMEZAWA Hiroyuki]

On Wed, 03 Jun 2009 19:43:06 -0700
Joel Krauska <jkrauska@gmail.com> wrote:

> KAMEZAWA Hiroyuki wrote:
> >> 1. Has anyone tried making a nicer swapoff?
> >> Right now swapoff can be pretty aggressive if the system is otherwise
> >> heavily loaded.  On systems that I need to leave running other jobs,
> >> swapoff compounds the slowness of the system overall by burning up
> >> a single CPU and lots of IO
> >>
> >> I wrote a perl wrapper that briefly runs swapoff 
> >> and then kills it, but it would seem more reasonable to have a knob
> >> to make swapoff less aggressive. (max kb/s, etc)  
> >>
> >> It looked to me like the swapoff code was immediately hitting kernel 
> >> internals instead of doing more lifting itself (and making it 
> >> obvious where I could insert some sleeps)
> >>
> 
> I find I need a slower swapoff when a system that's already running very hot
> needs to be recovered from lots of swapping without overly impacting the other
> running processes.
> 
> The bulk of the work is still being done in normal RAM, and the overhead
> of consuming an entire CPU just for swapoff degrades my other running processes.
> 
> > How about throttling swapoff's cpu usage by cpu scheduler cgroup ?
> > No help ?
> 
> I think swapoff is all done as systemcalls, not in userspace, so I'm not
> sure that cgroups would apply here.  (granted I had never heard of control
> groups until just now...)
> 
IIUC, some "cond_resched()" , means "reschedule if necessary", are inserted to
swapoff's main loop. Then, limiting usage of cpu may have effects, I think.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: swapoff throttling and speedup?

[Hugh Dickins]

On Wed, 3 Jun 2009, Joel Krauska wrote:

> On occasion we need to unswap a system that's gotten unruly.
> 
> Scenario: Some leaky app eats up way more RAM than it should, and pushes
> a few gigs of the running system in to swap.  The leaky app is killed, but
> there's still lots of good stuff sitting in swap that we need to tidy
> up to get the system back to normal performance levels.
> 
> 
> The normal recourse is to run
> swapoff -a ; swapon -a
> 
> 
> I have two related questions about the swap tools and how they work.
> 
> 
> 1. Has anyone tried making a nicer swapoff?
> Right now swapoff can be pretty aggressive if the system is otherwise
> heavily loaded.  On systems that I need to leave running other jobs,
> swapoff compounds the slowness of the system overall by burning up
> a single CPU and lots of IO
> 
> I wrote a perl wrapper that briefly runs swapoff and then kills it, but it
> would seem more reasonable to have a knob
> to make swapoff less aggressive. (max kb/s, etc)  

Though what you're doing is inelegant, it does seem a good solution to me:
why add fancy kernel tunables, for what is (I think) a rare use case?

> It looked to me like the swapoff code was immediately hitting kernel internals
> instead of doing more lifting itself (and making it obvious where I could
> insert some sleeps)

Yes, all the hard work is in the one swapoff(2) system call.
You could add an option to swapoff(8), to alarm and sleep and retry;
but why bother when you've already got your perl script?

> 
> Has anyone found better options here?

Could the answer be in your question: "a nicer swapoff?"
Does running "nice swapoff -a" behave more as you want?
Won't help on the IO side, I suppose.

> 
> 2. A faster(multithreaded?) swapoff?
> From what I can tell, swapoff is single threaded, which seems to make
> unswapping a CPU bound activity.  
> In the opposite use case of my first question, on systems that I /can/
> halt all the running code (assuming if they've gone off the deep end and have
> several gigs in SWAP) it can take quite a long time for unswap to tidy up the
> mess.  
> Has anyone considered improvements to swapoff to speed it up?
> (multiple threads?)

Until there's some unforeseen revolution in swap usage, swapoff will
always be nasty.

It's hugely inefficient, but you're one of the very few people to be
hurt by that: for most people, it's something that only gets run at
shutdown time, when there's very little work left for it to do, and
nothing left to mind anyway.

Certainly there are ways in which swapoff could be made much nicer;
but at the cost of maintaining additional pointers which use more
memory and slow down codepaths critical to performance, when most
critical processes will never go to swap in the first place.

So, in view of the tradeoffs, swapoff has remained a nasty backwater:
it goes about its business in that hugely inefficient way, to save
the rest of mm from having to support it throughout normal usage.

Multiple threads?  Hmm, never pondered on that.  It would certainly
make your first case worse, occupying all CPUs instead of just one.

If you can muster a chorus of other users upset by swapoff's
behaviour, then it would be fun to improve it somewhat; but I'd
much rather cut down its CPU usage, than spread the same across CPUs.

I've often thought that the 16 bits of the swap_map are poorly used
(rarely can more than 2 be set): we could make better use of them
with a hash to reduce the amount of blind scanning by an order or
two of magnitude.  But that's always seemed more a bloat of kernel
text than a good use of a developer's time.

And it's a very long time since I tried swapin_readahead() instead
of the read_swap_cache_async() in try_to_unuse(): more likely to
help with your first use case (when you've competing IO) than your
second (when the disk is likely to be caching the readahead anyway).

> 
> I'm hoping others have been down this road before.
> 
> As a rule, we try to avoid swapping when possible, but using:
> vm.swappiness = 1
> 
> But it does still happen on occasion and that lead to this mail.

Thanks for taking the trouble to write: opinions, anyone?

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: swapoff throttling and speedup?

[KAMEZAWA Hiroyuki]

Hugh Dickins wrote:
> On Wed, 3 Jun 2009, Joel Krauska wrote:
>> I'm hoping others have been down this road before.
>>
>> As a rule, we try to avoid swapping when possible, but using:
>> vm.swappiness = 1
>>
>> But it does still happen on occasion and that lead to this mail.
>
> Thanks for taking the trouble to write: opinions, anyone?
>

Is there anyone who wants a system call like this ?

  int mem_swapin(int pid, start-addr, size)
  - try to swap in pages from range [addr, addr+size) of pid.
    we can do this force-pagein against file caches and shmem now.
    this is for swap.

I doubts there are no one who can make use of this in sane way. But I'm
sometimes surprised to find that there are people make use of swap
intentionally...

Thanks,
-Kame



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>