Swap Memory

Swap Memory

[William Chan]

Hi All,

According to my understanding of the kernel mm, swap pages are
allocated in order of priority.

For example, I have the follow swap devices: FlashDevice1 with
priority 1 and DiskDevice2 with priority 2 and DiskDevice3 with
priority3. FlashDevice1 will get filled up, then DsikDevice2 and
DiskDevice3.

To allocate a page of memroy in swap, the kernel will call
get_swap_page to find the first device with available swap slots and
then pass that device to scan_swap_map to allocate a page.

I see a "problem" with this: The kernel does not take advantage of
available bandwidth. For example: my system has 2 swap
devices...DiskDevice2 and DiskDevice3, they are both identical 20 GB
7200rpm drives. If we need 4 GB worth of swap pages, only DiskDevice2
will be filled up. We have available free bandwidth on DiskDevice3
that is never used. If we were to split the swap pages into the two
drives, 2 GB of swap on each drive - we can potentially double our
bandwidth (latency is another issue).

Another problem that I am working on is what if one device is Flash
and the second device is Rotational. Does the kernel mm employ a
scheme to evict LRU pages in Priority1 swap to Priority2 swap?



Regards,
will

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap Memory

[Hugh Dickins]

On Tue, 3 Feb 2009, William Chan wrote:
> 
> According to my understanding of the kernel mm, swap pages are
> allocated in order of priority.
> 
> For example, I have the follow swap devices: FlashDevice1 with
> priority 1 and DiskDevice2 with priority 2 and DiskDevice3 with
> priority3. FlashDevice1 will get filled up, then DsikDevice2 and
> DiskDevice3.

Correct (or you can have several at the same priority,
and it rotates around them before going down to the next priority).

> 
> To allocate a page of memroy in swap, the kernel will call
> get_swap_page to find the first device with available swap slots and
> then pass that device to scan_swap_map to allocate a page.
> 
> I see a "problem" with this: The kernel does not take advantage of
> available bandwidth. For example: my system has 2 swap
> devices...DiskDevice2 and DiskDevice3, they are both identical 20 GB
> 7200rpm drives. If we need 4 GB worth of swap pages, only DiskDevice2
> will be filled up. We have available free bandwidth on DiskDevice3
> that is never used. If we were to split the swap pages into the two
> drives, 2 GB of swap on each drive - we can potentially double our
> bandwidth (latency is another issue).

True.  But wouldn't you use MD/DM for that, say, RAID 0 swap?
The priority scheme in swap is rather ancient, but is there any
point in fiddling with that, when there's already a logical
volume management layer which could do it better for you?

Though googling for "RAID 0 swap" doesn't inspire confidence.

> 
> Another problem that I am working on is what if one device is Flash
> and the second device is Rotational. Does the kernel mm employ a
> scheme to evict LRU pages in Priority1 swap to Priority2 swap?

No, it has no such scheme.

I'm accustomed to answering that you should be adding RAM rather
than worrying about the speed of swap.  But it could well be that
SSDs will change the game, and deserve more attention to tiered swap.

However, I don't get what you're proposing.  You write of evicting
LRU pages in priority 1 swap to priority 2 swap.  But if those pages
are still on an LRU in memory, doesn't that imply that they're useful
pages, which we're more likely to want to delete from swap, than copy
to slower storage?

I can imagine wanting to move long-forgotten pages from fast swap to
slower swap; but the overhead of such housekeeping rather puts me off.
It sounds like swap prefetch, but for those pages which we least want
to have in memory rather than those which we're likely to want.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap Memory

[William Chan]

> Correct (or you can have several at the same priority,
> and it rotates around them before going down to the next priority).

Where does it rotate the priorities? I am looking at mm/swapfile.c and
swp_entry_t get_swap_page(void), I can not find where it rotates.

> True.  But wouldn't you use MD/DM for that, say, RAID 0 swap?
> The priority scheme in swap is rather ancient, but is there any
> point in fiddling with that, when there's already a logical
> volume management layer which could do it better for you?
>
> Though googling for "RAID 0 swap" doesn't inspire confidence.

There are many cases where RAID 0 may not be applicable. What if my
user is cost conscience and can't afford a RAID chip? Or fFor example,
what if I have uneven drives - I have 1 drive that is 20 GB and 5400
rpm and 40 GB at 7200 rpm. It would still be advantageous to take
advantage of the additional bandwidth - I mean if the system already
has two swap drives - why not take advantage of it?

> I'm accustomed to answering that you should be adding RAM rather
> than worrying about the speed of swap.  But it could well be that
> SSDs will change the game, and deserve more attention to tiered swap.

Sure, for PCs yes - adding more RAM is probably the way to go. However
adding RAM may not always be the solution - sure RAM is a lot cheaper
nowadays - but there are still many applications that need swap. For
example, what if I am doing protein folding and I need 1 TB of RAM,
but after I finish a set of calculations I just need to set it aside,
but I won't touch that piece of data until 5 gazillion cpu cycles
later for a 2nd pass.

> However, I don't get what you're proposing.  You write of evicting
> LRU pages in priority 1 swap to priority 2 swap.  But if those pages
> are still on an LRU in memory, doesn't that imply that they're useful
> pages, which we're more likely to want to delete from swap, than copy
> to slower storage?

I am saying - there may be other pages that need to be evicted to swap
and are more used than the LRU page in priority 1 swap. IE. I have a
page I want to evict to swap, but Swap1 is full - I want to evict some
of the LRU pages on Swap1 to Swap2 to make room for the new pages I
want to evict.

> I can imagine wanting to move long-forgotten pages from fast swap to
> slower swap; but the overhead of such housekeeping rather puts me off.
> It sounds like swap prefetch, but for those pages which we least want
> to have in memory rather than those which we're likely to want.

I think this is an area that is definitely worth exploring - I agree
tho, for some systems, the overhead may be big enough to make it not
worth it. If we use a linked list, the overhead would be linearly
proportional to the number of pages in the swap. We would need to
update an LRU linked list for each memory access into swap. We may or
may not want a daemon which is responsible for evicting pages from
high priority swap into low priority (or vice versa if pages in the
2nd priority swap becomes used a lot). I have not done any
benchmarking or intensive research to measure the overhead - but
doesn't the kernel mm already do an LRU list for pages in physical
memory to evict them to swap?


will


On Thu, Feb 5, 2009 at 10:33 AM, Hugh Dickins <hugh@veritas.com> wrote:
> On Tue, 3 Feb 2009, William Chan wrote:
>>
>> According to my understanding of the kernel mm, swap pages are
>> allocated in order of priority.
>>
>> For example, I have the follow swap devices: FlashDevice1 with
>> priority 1 and DiskDevice2 with priority 2 and DiskDevice3 with
>> priority3. FlashDevice1 will get filled up, then DsikDevice2 and
>> DiskDevice3.
>
> Correct (or you can have several at the same priority,
> and it rotates around them before going down to the next priority).
>
>>
>> To allocate a page of memroy in swap, the kernel will call
>> get_swap_page to find the first device with available swap slots and
>> then pass that device to scan_swap_map to allocate a page.
>>
>> I see a "problem" with this: The kernel does not take advantage of
>> available bandwidth. For example: my system has 2 swap
>> devices...DiskDevice2 and DiskDevice3, they are both identical 20 GB
>> 7200rpm drives. If we need 4 GB worth of swap pages, only DiskDevice2
>> will be filled up. We have available free bandwidth on DiskDevice3
>> that is never used. If we were to split the swap pages into the two
>> drives, 2 GB of swap on each drive - we can potentially double our
>> bandwidth (latency is another issue).
>
> True.  But wouldn't you use MD/DM for that, say, RAID 0 swap?
> The priority scheme in swap is rather ancient, but is there any
> point in fiddling with that, when there's already a logical
> volume management layer which could do it better for you?
>
> Though googling for "RAID 0 swap" doesn't inspire confidence.
>
>>
>> Another problem that I am working on is what if one device is Flash
>> and the second device is Rotational. Does the kernel mm employ a
>> scheme to evict LRU pages in Priority1 swap to Priority2 swap?
>
> No, it has no such scheme.
>
> I'm accustomed to answering that you should be adding RAM rather
> than worrying about the speed of swap.  But it could well be that
> SSDs will change the game, and deserve more attention to tiered swap.
>
> However, I don't get what you're proposing.  You write of evicting
> LRU pages in priority 1 swap to priority 2 swap.  But if those pages
> are still on an LRU in memory, doesn't that imply that they're useful
> pages, which we're more likely to want to delete from swap, than copy
> to slower storage?
>
> I can imagine wanting to move long-forgotten pages from fast swap to
> slower swap; but the overhead of such housekeeping rather puts me off.
> It sounds like swap prefetch, but for those pages which we least want
> to have in memory rather than those which we're likely to want.
>
> Hugh
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap Memory

[Hugh Dickins]

On Thu, 5 Feb 2009, William Chan wrote:
> > Correct (or you can have several at the same priority,
> > and it rotates around them before going down to the next priority).
> 
> Where does it rotate the priorities? I am looking at mm/swapfile.c and
> swp_entry_t get_swap_page(void), I can not find where it rotates.

Sorry to confuse: I meant that get_swap_page() cycles around swap
areas of the same priority before going on to the next priority,
I didn't mean that it "rotates the priorities".

> 
> > True.  But wouldn't you use MD/DM for that, say, RAID 0 swap?
> > The priority scheme in swap is rather ancient, but is there any
> > point in fiddling with that, when there's already a logical
> > volume management layer which could do it better for you?
> >
> > Though googling for "RAID 0 swap" doesn't inspire confidence.
> 
> There are many cases where RAID 0 may not be applicable. What if my
> user is cost conscience and can't afford a RAID chip?

I'm not a good person to discuss such matters with;
but I thought MD/DM was perfectly capable of software RAID?

> Or fFor example,
> what if I have uneven drives - I have 1 drive that is 20 GB and 5400
> rpm and 40 GB at 7200 rpm. It would still be advantageous to take
> advantage of the additional bandwidth - I mean if the system already
> has two swap drives - why not take advantage of it?

Would MD/DM prevent that?  As I see it, you're asking for striping,
and we already have a layer that specializes in that and more,
so why add such features in at the swap end.

> > However, I don't get what you're proposing.  You write of evicting
> > LRU pages in priority 1 swap to priority 2 swap.  But if those pages
> > are still on an LRU in memory, doesn't that imply that they're useful
> > pages, which we're more likely to want to delete from swap, than copy
> > to slower storage?
> 
> I am saying - there may be other pages that need to be evicted to swap
> and are more used than the LRU page in priority 1 swap. IE. I have a
> page I want to evict to swap, but Swap1 is full - I want to evict some
> of the LRU pages on Swap1 to Swap2 to make room for the new pages I
> want to evict.

I'm confused by your use of "LRU".  We have LRUs for pages in memory,
and sometimes a page is in memory on LRU and also has a copy on swap;
but in general the copies on swap are not on any LRU, they're on swap.

> > I can imagine wanting to move long-forgotten pages from fast swap to
> > slower swap; but the overhead of such housekeeping rather puts me off.
> > It sounds like swap prefetch, but for those pages which we least want
> > to have in memory rather than those which we're likely to want.
> 
> I think this is an area that is definitely worth exploring - I agree
> tho, for some systems, the overhead may be big enough to make it not
> worth it. If we use a linked list, the overhead would be linearly
> proportional to the number of pages in the swap. We would need to
> update an LRU linked list for each memory access into swap. We may or
> may not want a daemon which is responsible for evicting pages from
> high priority swap into low priority (or vice versa if pages in the
> 2nd priority swap becomes used a lot). I have not done any
> benchmarking or intensive research to measure the overhead - but
> doesn't the kernel mm already do an LRU list for pages in physical
> memory to evict them to swap?

Yes, but the LRU in memory is for pages in memory: once they're out
to swap, and freed from memory, there is no LRU for them.

That could be changed, yes: but would multiply the amount of memory
needed for recording pages out of swap.  The present design is to
minimize the memory needed by what's out on swap.

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap Memory

[William Chan]

> Sorry to confuse: I meant that get_swap_page() cycles around swap
> areas of the same priority before going on to the next priority,
> I didn't mean that it "rotates the priorities".

I think we should at a minimum do a rotation around same priority.
This is easy to implement with minimal overhead. I can send a patch in
for this later.

> I'm not a good person to discuss such matters with;
> but I thought MD/DM was perfectly capable of software RAID?

Software raid consumes lots of CPU cycles - it is not ideal. Even the
cheap hardware RAIDs that come with a lot of desktop mobos nowadays,
they consume a lot of CPU cycles as well.

> Would MD/DM prevent that?  As I see it, you're asking for striping,
> and we already have a layer that specializes in that and more,
> so why add such features in at the swap end.

I do not think MD/DM would take "full" advantage. And it is very hard
to take full advantage. By full advantage I mean correct load
balancing, put Z % of pages on the 7200 rpm drive and (1 - Z) % of
pages on 5400 rpm drive to maximize the average E(x) of bandwidth and
minimize latency.

> I'm confused by your use of "LRU".  We have LRUs for pages in memory,
> and sometimes a page is in memory on LRU and also has a copy on swap;
> but in general the copies on swap are not on any LRU, they're on swap.

Sorry for the confusion - What I mean by LRU is Least Recently Used.
ie. Least Recently used page of memory in Swap1 or least recently used
page of memory in system memory.

> That could be changed, yes: but would multiply the amount of memory
> needed for recording pages out of swap.  The present design is to
> minimize the memory needed by what's out on swap.

Hopefully there will be less pages in swap than in system memory. If
this is true - the overhead introduced should be minimal relative to
the overhead the kernel already has for manging system memory pages.

will



On Thu, Feb 5, 2009 at 11:57 AM, Hugh Dickins <hugh@veritas.com> wrote:
> On Thu, 5 Feb 2009, William Chan wrote:
>> > Correct (or you can have several at the same priority,
>> > and it rotates around them before going down to the next priority).
>>
>> Where does it rotate the priorities? I am looking at mm/swapfile.c and
>> swp_entry_t get_swap_page(void), I can not find where it rotates.
>
> Sorry to confuse: I meant that get_swap_page() cycles around swap
> areas of the same priority before going on to the next priority,
> I didn't mean that it "rotates the priorities".
>
>>
>> > True.  But wouldn't you use MD/DM for that, say, RAID 0 swap?
>> > The priority scheme in swap is rather ancient, but is there any
>> > point in fiddling with that, when there's already a logical
>> > volume management layer which could do it better for you?
>> >
>> > Though googling for "RAID 0 swap" doesn't inspire confidence.
>>
>> There are many cases where RAID 0 may not be applicable. What if my
>> user is cost conscience and can't afford a RAID chip?
>
> I'm not a good person to discuss such matters with;
> but I thought MD/DM was perfectly capable of software RAID?
>
>> Or fFor example,
>> what if I have uneven drives - I have 1 drive that is 20 GB and 5400
>> rpm and 40 GB at 7200 rpm. It would still be advantageous to take
>> advantage of the additional bandwidth - I mean if the system already
>> has two swap drives - why not take advantage of it?
>
> Would MD/DM prevent that?  As I see it, you're asking for striping,
> and we already have a layer that specializes in that and more,
> so why add such features in at the swap end.
>
>> > However, I don't get what you're proposing.  You write of evicting
>> > LRU pages in priority 1 swap to priority 2 swap.  But if those pages
>> > are still on an LRU in memory, doesn't that imply that they're useful
>> > pages, which we're more likely to want to delete from swap, than copy
>> > to slower storage?
>>
>> I am saying - there may be other pages that need to be evicted to swap
>> and are more used than the LRU page in priority 1 swap. IE. I have a
>> page I want to evict to swap, but Swap1 is full - I want to evict some
>> of the LRU pages on Swap1 to Swap2 to make room for the new pages I
>> want to evict.
>
> I'm confused by your use of "LRU".  We have LRUs for pages in memory,
> and sometimes a page is in memory on LRU and also has a copy on swap;
> but in general the copies on swap are not on any LRU, they're on swap.
>
>> > I can imagine wanting to move long-forgotten pages from fast swap to
>> > slower swap; but the overhead of such housekeeping rather puts me off.
>> > It sounds like swap prefetch, but for those pages which we least want
>> > to have in memory rather than those which we're likely to want.
>>
>> I think this is an area that is definitely worth exploring - I agree
>> tho, for some systems, the overhead may be big enough to make it not
>> worth it. If we use a linked list, the overhead would be linearly
>> proportional to the number of pages in the swap. We would need to
>> update an LRU linked list for each memory access into swap. We may or
>> may not want a daemon which is responsible for evicting pages from
>> high priority swap into low priority (or vice versa if pages in the
>> 2nd priority swap becomes used a lot). I have not done any
>> benchmarking or intensive research to measure the overhead - but
>> doesn't the kernel mm already do an LRU list for pages in physical
>> memory to evict them to swap?
>
> Yes, but the LRU in memory is for pages in memory: once they're out
> to swap, and freed from memory, there is no LRU for them.
>
> That could be changed, yes: but would multiply the amount of memory
> needed for recording pages out of swap.  The present design is to
> minimize the memory needed by what's out on swap.
>
> Hugh
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap Memory

[KAMEZAWA Hiroyuki]

On Thu, 5 Feb 2009 12:40:31 -0800
William Chan <williamchan@google.com> wrote:
> > That could be changed, yes: but would multiply the amount of memory
> > needed for recording pages out of swap.  The present design is to
> > minimize the memory needed by what's out on swap.
> 
> Hopefully there will be less pages in swap than in system memory. If
> this is true - the overhead introduced should be minimal relative to
> the overhead the kernel already has for manging system memory pages.
> 
In my experience, you can't assume that ;)

BTW, if you want to do that, changing device layer is much easier than changing
memory management layer. 

Maybe adding device mapper for good-scheduled-swap(but not Raid0) is enough.
Preparing device-mapper layer which does
  1. It can tie several devices of different size.
  2. It chases each block's usage by some logic (LRU) and do block migration
     if necessary.
  3. priority between devices can be set by dm's user-land tools.

Hmm? But I'm not sure this is worth tring.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>