page migration: Fail with error if swap not setup

page migration: Fail with error if swap not setup

[Christoph Lameter]

Currently the migration of anonymous pages will silently fail if no swap 
is setup. This patch makes page migration functions check for available 
swap and fail with -ENODEV if no swap space is available.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.16-rc6/mm/mempolicy.c
===================================================================
--- linux-2.6.16-rc6.orig/mm/mempolicy.c	2006-03-14 16:31:15.000000000 -0800
+++ linux-2.6.16-rc6/mm/mempolicy.c	2006-03-14 17:25:09.000000000 -0800
@@ -330,9 +330,14 @@ check_range(struct mm_struct *mm, unsign
 	int err;
 	struct vm_area_struct *first, *vma, *prev;
 
-	/* Clear the LRU lists so pages can be isolated */
-	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
+	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
+		/* Must have available swap entries for migration */
+		if (nr_swap_pages <=0)
+			return ERR_PTR(-ENODEV);
+
+		/* Clear the LRU lists so pages can be isolated */
 		lru_add_drain_all();
+	}
 
 	first = find_vma(mm, start);
 	if (!first)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Andrew Morton]

Christoph Lameter <clameter@sgi.com> wrote:
>
> Currently the migration of anonymous pages will silently fail if no swap 
> is setup.

Why?

I mean, if something tries to allocate a swap page and that fails then the
error should be propagated back.  That's race-free.

> This patch makes page migration functions check for available 
> swap and fail with -ENODEV if no swap space is available.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.16-rc6/mm/mempolicy.c
> ===================================================================
> --- linux-2.6.16-rc6.orig/mm/mempolicy.c	2006-03-14 16:31:15.000000000 -0800
> +++ linux-2.6.16-rc6/mm/mempolicy.c	2006-03-14 17:25:09.000000000 -0800
> @@ -330,9 +330,14 @@ check_range(struct mm_struct *mm, unsign
>  	int err;
>  	struct vm_area_struct *first, *vma, *prev;
>  
> -	/* Clear the LRU lists so pages can be isolated */
> -	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
> +	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
> +		/* Must have available swap entries for migration */
> +		if (nr_swap_pages <=0)

(ObCodingStyleWhine)

> +			return ERR_PTR(-ENODEV);
> +
> +		/* Clear the LRU lists so pages can be isolated */
>  		lru_add_drain_all();
> +	}
>  

Whereas this appears to be racy...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Tue, 14 Mar 2006, Andrew Morton wrote:

> Christoph Lameter <clameter@sgi.com> wrote:
> >
> > Currently the migration of anonymous pages will silently fail if no swap 
> > is setup.
> 
> Why?

The allocation of the swap page will fail in migrate_pages() and then the 
page is going on the permant failure list. Hmm... This is not a real 
total failure of page migration since file backed pages can be migrated 
without having swap and page migration will continue for those. However, 
all anonymous pages will end up on the failed list. At the end of page 
migration these will be returned to the LRU. Thus they stay where they 
were.

> I mean, if something tries to allocate a swap page and that fails then the
> error should be propagated back.  That's race-free.

It is propaged back in the form of a list of pages that failed to migrate. 
Its just no clear at the end what the reasons for the individual failures
were. Its better just to check for swap availability before migration.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Andrew Morton]

Christoph Lameter <clameter@sgi.com> wrote:
>
> On Tue, 14 Mar 2006, Andrew Morton wrote:
> 
> > Christoph Lameter <clameter@sgi.com> wrote:
> > >
> > > Currently the migration of anonymous pages will silently fail if no swap 
> > > is setup.
> > 
> > Why?
> 
> The allocation of the swap page will fail in migrate_pages() and then the 
> page is going on the permant failure list. Hmm... This is not a real 
> total failure of page migration since file backed pages can be migrated 
> without having swap and page migration will continue for those. However, 
> all anonymous pages will end up on the failed list. At the end of page 
> migration these will be returned to the LRU. Thus they stay where they 
> were.
> 
> > I mean, if something tries to allocate a swap page and that fails then the
> > error should be propagated back.  That's race-free.
> 
> It is propaged back in the form of a list of pages that failed to migrate. 
> Its just no clear at the end what the reasons for the individual failures
> were. Its better just to check for swap availability before migration.

But the operation can still fail if we run out of swapspace partway through
- so this problem can still occur.  The patch just makes it (much) less
frequent.

Surely it's possible to communicate -ENOSWAP correctly and reliably?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Tue, 14 Mar 2006, Andrew Morton wrote:

> >  		lru_add_drain_all();
> > +	}
> >  
> 
> Whereas this appears to be racy...

Migration just makes the best effort. Page that are moved off the LRU 
after the draining end up on the failed migration list and will not be 
migrated.

Sorry about the blank. New patch with more explanations?



page migration: Fail with error if swap not setup

Currently the migration of anonymous pages will silently fail if no
swap is setup. This patch makes page migration functions to check
for available swap and fail with -ENODEV if no swap space is available.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.16-rc6/mm/mempolicy.c
===================================================================
--- linux-2.6.16-rc6.orig/mm/mempolicy.c	2006-03-14 16:31:15.000000000 -0800
+++ linux-2.6.16-rc6/mm/mempolicy.c	2006-03-14 19:52:25.000000000 -0800
@@ -330,9 +330,19 @@ check_range(struct mm_struct *mm, unsign
 	int err;
 	struct vm_area_struct *first, *vma, *prev;
 
-	/* Clear the LRU lists so pages can be isolated */
-	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL))
+	if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
+		/* Must have swap device for migration */
+		if (nr_swap_pages <= 0)
+			return ERR_PTR(-ENODEV);
+
+		/*
+		 * Clear the LRU lists so pages can be isolated.
+		 * Note that pages may be moved off the LRU after we have
+		 * drained them. Those pages will fail to migrate like other
+		 * pages that may be busy.
+		 */
 		lru_add_drain_all();
+	}
 
 	first = find_vma(mm, start);
 	if (!first)
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Tue, 14 Mar 2006, Andrew Morton wrote:

> But the operation can still fail if we run out of swapspace partway through
> - so this problem can still occur.  The patch just makes it (much) less
> frequent.
> 
> Surely it's possible to communicate -ENOSWAP correctly and reliably?

There are a number of possible failure conditions. The strategy of the 
migration function is to migrate as much as possible and return the rest 
without giving any reason. migrate_pages() returns the number of leftover 
pages not the reasons they failed.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Nick Piggin]

Christoph Lameter wrote:
> On Tue, 14 Mar 2006, Andrew Morton wrote:
> 
> 
>>But the operation can still fail if we run out of swapspace partway through
>>- so this problem can still occur.  The patch just makes it (much) less
>>frequent.
>>
>>Surely it's possible to communicate -ENOSWAP correctly and reliably?
> 
> 
> There are a number of possible failure conditions. The strategy of the 
> migration function is to migrate as much as possible and return the rest 
> without giving any reason. migrate_pages() returns the number of leftover 
> pages not the reasons they failed.
> 

Could you return the reason the first failing page failed. At least then
the caller can have some idea about what is needed to make further progress.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Lee Schermerhorn]

On Tue, 2006-03-14 at 19:05 -0800, Christoph Lameter wrote:
> Currently the migration of anonymous pages will silently fail if no swap 
> is setup. This patch makes page migration functions check for available 
> swap and fail with -ENODEV if no swap space is available.

Migration Cache, anyone?  ;-)



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Wed, 15 Mar 2006, Nick Piggin wrote:

> > There are a number of possible failure conditions. The strategy of the
> > migration function is to migrate as much as possible and return the rest
> > without giving any reason. migrate_pages() returns the number of leftover
> > pages not the reasons they failed.
> Could you return the reason the first failing page failed. At least then
> the caller can have some idea about what is needed to make further progress.

The return value of migrate_pages() is the number of pages that were not 
migrated. It is up to the caller to figure out why a page was not 
migrated. We could change that in the future but that would be a big 
change to the code. Migrate_pages() makes the best effort at 
migration and categorizes failing pages into those who with permanent 
failures and those which may be retriable. Currently page migration simply 
skips over any soft or hard failures to migrate pages and leaves them in 
place. The current page migration code is intentionally designed to only 
make a reasonable attempt on a group of pages. Earlier code attempted to 
guarantee migration but that never worked the right way and introduced 
unacceptable delays while holding locks.

The calling program may go through the list of failing pages and 
investigate the reasons by inspecting page count, mapping, swap etc. I 
guess we could add some sort of a callback in the future that determines 
what to do on failure. Or add some flags to return immediately if 
migration fails.

But I think the current code is just fine.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Wed, 15 Mar 2006, Lee Schermerhorn wrote:

> On Tue, 2006-03-14 at 19:05 -0800, Christoph Lameter wrote:
> > Currently the migration of anonymous pages will silently fail if no swap 
> > is setup. This patch makes page migration functions check for available 
> > swap and fail with -ENODEV if no swap space is available.
> 
> Migration Cache, anyone?  ;-)

Yes, but please cleanly integrated into the way swap works.

At that point we can also follow Marcelo's suggestion and move the 
migration code into mm/mmigrate.c because it then becomes easier to 
separate the migration code from swap.

There are a couple of other pending things that are also listed in 
the todo list in Documentation/vm/page-migration

1. Somehow safely track the prior mm_structs that a pte was mapped to 
(increase mm refcount?) and restore those mappings to avoid faults to 
restore ptes after a page was moved.

2. Avoid dirty bit faults for dirty pages.

More things to consider:

- Add migration support for more filesystems.

- Lazy migration in the fault paths (seems to depend on first implementing 
proper policy support for file backed pages).

- Support migration of VM_LOCKED pages (First question is if we want to 
  have that at all. Does VM_LOCKED imply that a page is fixed at a 
  specific location in memory?).

- Think about how to realize migration of kernel pages (some arches have
  page table for kernel space, one could potentially remap the address 
  instead of going through all the twists and turns of the existing 
  hotplug approach. See also what virtual iron has done about this.).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Wed, 15 Mar 2006, Marcelo Tosatti wrote:

> > At that point we can also follow Marcelo's suggestion and move the 
> > migration code into mm/mmigrate.c because it then becomes easier to 
> > separate the migration code from swap. 
> 
> Please - the migration code really does not belong to mm/vmscan.c.

It performs scanning and has lots of overlapping functionality with 
swap. Migration was developed based on the swap code in vmscan.c.

> On the assumption that those page mappings are going to be used, which
> is questionable.
> 
> Lazily faulting the page mappings instead of "pre-faulting" really
> depends on the load (tradeoff) - might be interesting to make it 
> selectable.

If the ptes are removed then the mapcount of the pages also sinks which 
makes it likely that the swapper will evict these.

> > - Support migration of VM_LOCKED pages (First question is if we want to 
> >   have that at all. Does VM_LOCKED imply that a page is fixed at a 
> >   specific location in memory?).
> Cryptographic  security  software often handles critical bytes like passwords
> or secret keys as data structures. As a result of paging, these secrets
> could  be  transferred  onto a persistent swap store medium, where they
> might be accessible to the enemy long after the security  software  has
> erased  the secrets in RAM and terminated. 

That does not answer the question if VM_LOCKED pages should be 
migratable. We all agree that they should not show up on swap.

> > - Think about how to realize migration of kernel pages (some arches have
> >   page table for kernel space, one could potentially remap the address 
> >   instead of going through all the twists and turns of the existing 
> >   hotplug approach. See also what virtual iron has done about this.).
> 
> Locking sounds tricky, how do you guarantee that nobody is going to
> access such kernel virtual addresses (and their TLB-cached entries)
> while they're physical address is being changed ?

I guess this could be done by having a very simple fault handler for 
kernel memory that would simply wait on a valid pte.

Then invalidate pte, move the page and reinstall pte.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Christoph Lameter]

On Wed, 15 Mar 2006, Marcelo Tosatti wrote:

> > That does not answer the question if VM_LOCKED pages should be 
> > migratable. We all agree that they should not show up on swap.
> 
> I guess you missed the first part of the man page:
> 
> All pages which contain a part of the specified memory range are
> guaranteed be resident in RAM when the mlock system call returns
> successfully and they are guaranteed to stay in RAM until the pages are
> unlocked by munlock or munlockall, until the pages are unmapped via
> munmap, or until the process terminates or starts another program with
> exec. Child processes do not inherit page locks across a fork.
> 
> That is, mlock() only guarantees that pages are kept in RAM and not
> swapped. It does seem to refer to physical placing of pages.

If VM_LOCKED is not pinning memory then how does one pin memory? There are 
likely applications / drivers that require memory not to move. Increase 
pagecount?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Marcelo Tosatti]

On Wed, Mar 15, 2006 at 09:11:07AM -0800, Christoph Lameter wrote:
> On Wed, 15 Mar 2006, Lee Schermerhorn wrote:
> 
> > On Tue, 2006-03-14 at 19:05 -0800, Christoph Lameter wrote:
> > > Currently the migration of anonymous pages will silently fail if no swap 
> > > is setup. This patch makes page migration functions check for available 
> > > swap and fail with -ENODEV if no swap space is available.
> > 
> > Migration Cache, anyone?  ;-)
> 
> Yes, but please cleanly integrated into the way swap works.
> 
> At that point we can also follow Marcelo's suggestion and move the 
> migration code into mm/mmigrate.c because it then becomes easier to 
> separate the migration code from swap. 

Please - the migration code really does not belong to mm/vmscan.c.

> There are a couple of other pending things that are also listed in 
> the todo list in Documentation/vm/page-migration
> 
> 1. Somehow safely track the prior mm_structs that a pte was mapped to 
> (increase mm refcount?) and restore those mappings to avoid faults to 
> restore ptes after a page was moved.

On the assumption that those page mappings are going to be used, which
is questionable.

Lazily faulting the page mappings instead of "pre-faulting" really
depends on the load (tradeoff) - might be interesting to make it 
selectable.

> 2. Avoid dirty bit faults for dirty pages.

When prefaulting as you suggest, yep...

> More things to consider:
> 
> - Add migration support for more filesystems.
> 
> - Lazy migration in the fault paths (seems to depend on first implementing 
> proper policy support for file backed pages).
> 
> - Support migration of VM_LOCKED pages (First question is if we want to 
>   have that at all. Does VM_LOCKED imply that a page is fixed at a 
>   specific location in memory?).

No, mlock(2):

mlock disables paging for the memory in the range starting at addr with
length len bytes. All pages which contain a part of the specified  mem-
ory  range are guaranteed be resident in RAM when the mlock system call
returns successfully and they are guaranteed to stay in RAM  until  the
pages  are  unlocked  by  munlock  or  munlockall,  until the pages are
unmapped via munmap, or until the process terminates or starts  another
program  with exec.  Child processes do not inherit page locks across a
fork.

...

Cryptographic  security  software often handles critical bytes like passwords
or secret keys as data structures. As a result of paging, these secrets
could  be  transferred  onto a persistent swap store medium, where they
might be accessible to the enemy long after the security  software  has
erased  the secrets in RAM and terminated. 

> - Think about how to realize migration of kernel pages (some arches have
>   page table for kernel space, one could potentially remap the address 
>   instead of going through all the twists and turns of the existing 
>   hotplug approach. See also what virtual iron has done about this.).

Locking sounds tricky, how do you guarantee that nobody is going to
access such kernel virtual addresses (and their TLB-cached entries)
while they're physical address is being changed ?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Marcelo Tosatti]

On Wed, Mar 15, 2006 at 10:08:34AM -0800, Christoph Lameter wrote:
> On Wed, 15 Mar 2006, Marcelo Tosatti wrote:
> 
> > > At that point we can also follow Marcelo's suggestion and move the 
> > > migration code into mm/mmigrate.c because it then becomes easier to 
> > > separate the migration code from swap. 
> > 
> > Please - the migration code really does not belong to mm/vmscan.c.
> 
> It performs scanning and has lots of overlapping functionality with 
> swap. Migration was developed based on the swap code in vmscan.c.
> 
> > On the assumption that those page mappings are going to be used, which
> > is questionable.
> > 
> > Lazily faulting the page mappings instead of "pre-faulting" really
> > depends on the load (tradeoff) - might be interesting to make it 
> > selectable.
> 
> If the ptes are removed then the mapcount of the pages also sinks which 
> makes it likely that the swapper will evict these.
> 
> > > - Support migration of VM_LOCKED pages (First question is if we want to 
> > >   have that at all. Does VM_LOCKED imply that a page is fixed at a 
> > >   specific location in memory?).
> > Cryptographic  security  software often handles critical bytes like passwords
> > or secret keys as data structures. As a result of paging, these secrets
> > could  be  transferred  onto a persistent swap store medium, where they
> > might be accessible to the enemy long after the security  software  has
> > erased  the secrets in RAM and terminated. 
> 
> That does not answer the question if VM_LOCKED pages should be 
> migratable. We all agree that they should not show up on swap.

I guess you missed the first part of the man page:

All pages which contain a part of the specified memory range are
guaranteed be resident in RAM when the mlock system call returns
successfully and they are guaranteed to stay in RAM until the pages are
unlocked by munlock or munlockall, until the pages are unmapped via
munmap, or until the process terminates or starts another program with
exec. Child processes do not inherit page locks across a fork.

That is, mlock() only guarantees that pages are kept in RAM and not
swapped. It does seem to refer to physical placing of pages.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: page migration: Fail with error if swap not setup

[Marcelo Tosatti]

On Wed, Mar 15, 2006 at 11:00:31AM -0800, Christoph Lameter wrote:
> On Wed, 15 Mar 2006, Marcelo Tosatti wrote:
> 
> > > That does not answer the question if VM_LOCKED pages should be 
> > > migratable. We all agree that they should not show up on swap.
> > 
> > I guess you missed the first part of the man page:
> > 
> > All pages which contain a part of the specified memory range are
> > guaranteed be resident in RAM when the mlock system call returns
> > successfully and they are guaranteed to stay in RAM until the pages are
> > unlocked by munlock or munlockall, until the pages are unmapped via
> > munmap, or until the process terminates or starts another program with
> > exec. Child processes do not inherit page locks across a fork.
> > 
> > That is, mlock() only guarantees that pages are kept in RAM and not
> > swapped. It does seem to refer to physical placing of pages.
> 
> If VM_LOCKED is not pinning memory then how does one pin memory? There are 
> likely applications / drivers that require memory not to move. Increase 
> pagecount?

Err, I meant that mlock() does _not_ refer to physical placing of pages, 
it only refers to guaranteed availability of page in RAM (as can be read
in the man page).

Now drivers using VM_LOCKED is another history... 

In the end my comments haven't been useful at all, oh well.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>