[PATCH 0/2] Synchronous Lumpy Reclaim V3

[PATCH 0/2] Synchronous Lumpy Reclaim V3

[Andy Whitcroft]

[This is a re-spin based on feedback from akpm.]

As pointed out by Mel when reclaim is applied at higher orders a
significant amount of IO may be started.  As this takes finite time
to drain reclaim will consider more areas than ultimatly needed
to satisfy the request.  This leads to more reclaim than strictly
required and reduced success rates.

I was able to confirm Mel's test results on systems locally.
These show that even under light load the success rates drop off far
more than expected.  Testing with a modified version of his patch
(which follows) I was able to allocate almost all of ZONE_MOVABLE
with a near idle system.  I ran 5 test passes sequentially following
system boot (the system has 29 hugepages in ZONE_MOVABLE):

  2.6.23-rc1              11  8  6  7  7
  sync_lumpy              28 28 29 29 26

These show that although hugely better than the near 0% success
normally expected we can only allocate about a 1/4 of the zone.
Using synchronous reclaim for these allocations we get close to 100%
as expected.

I have also run our standard high order tests and these show no
regressions in allocation success rates at rest, and some significant
improvements under load.

Following this email are two patches, both should be considered as
bug fixes to lumpy reclaim for 2.6.23:

ensure-we-count-pages-transitioning-inactive-via-clear_active_flags:
  this a bug fix for Lumpy Reclaim fixing up a bug in VM Event
  accounting when it marks pages inactive, and

Wait-for-page-writeback-when-directly-reclaiming-contiguous-areas:
  updates reclaim making direct reclaim synchronous when applied
  at orders above PAGE_ALLOC_COSTLY_ORDER.

Patches against 2.6.23-rc1.  Andrew please consider for -mm and
for pushing to mainline.

-apw

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 1/2] ensure we count pages transitioning inactive via clear_active_flags

[Andy Whitcroft]

We are transitioning pages from active to inactive in
clear_active_flags, those need counting as PGDEACTIVATE vm events.

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
Acked-by: Mel Gorman <mel@csn.ul.ie>
---
 mm/vmscan.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d419e10..99ec7fa 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -777,6 +777,7 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
 			     (sc->order > PAGE_ALLOC_COSTLY_ORDER)?
 					     ISOLATE_BOTH : ISOLATE_INACTIVE);
 		nr_active = clear_active_flags(&page_list);
+		__count_vm_events(PGDEACTIVATE, nr_active);
 
 		__mod_zone_page_state(zone, NR_ACTIVE, -nr_active);
 		__mod_zone_page_state(zone, NR_INACTIVE,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 2/2] wait for page writeback when directly reclaiming contiguous areas

[Andy Whitcroft]

Lumpy reclaim works by selecting a lead page from the LRU list and
then selecting pages for reclaim from the order-aligned area of
pages. In the situation were all pages in that region are inactive
and not referenced by any process over time, it works well.

In the situation where there is even light load on the system, the
pages may not free quickly. Out of a area of 1024 pages, maybe only
950 of them are freed when the allocation attempt occurs because
lumpy reclaim returned early.  This patch alters the behaviour of
direct reclaim for large contiguous blocks.

The first attempt to call shrink_page_list() is asynchronous but
if it fails, the pages are submitted a second time and the calling
process waits for the IO to complete. This may stall allocators
waiting for contiguous memory but that should be expected behaviour
for high-order users. It is preferable behaviour to potentially
queueing unnecessary areas for IO. Note that kswapd will not stall
in this fashion.

Changelog:

Changes in V3:
 - remove the typdedef from the enum pageout_io.
 - if we cannot enter fs do not wait for io completion
 - fix spelling in commentary

Changes in V2:
 - remove retry loop
 - fix up active accounting (count deactivate events correctly)
 - use our own sync/async flag type

[apw@shadowen.org: update to version 2]
[apw@shadowen.org: update to version 3]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---
 mm/vmscan.c |   60 +++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 52 insertions(+), 8 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 99ec7fa..b1e9291 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -271,6 +271,12 @@ static void handle_write_error(struct address_space *mapping,
 	unlock_page(page);
 }
 
+/* Request for sync pageout. */
+enum pageout_io {
+	PAGEOUT_IO_ASYNC,
+	PAGEOUT_IO_SYNC,
+};
+
 /* possible outcome of pageout() */
 typedef enum {
 	/* failed to write page out, page is locked */
@@ -287,7 +293,8 @@ typedef enum {
  * pageout is called by shrink_page_list() for each dirty page.
  * Calls ->writepage().
  */
-static pageout_t pageout(struct page *page, struct address_space *mapping)
+static pageout_t pageout(struct page *page, struct address_space *mapping,
+						enum pageout_io sync_writeback)
 {
 	/*
 	 * If the page is dirty, only perform writeback if that write
@@ -346,6 +353,15 @@ static pageout_t pageout(struct page *page, struct address_space *mapping)
 			ClearPageReclaim(page);
 			return PAGE_ACTIVATE;
 		}
+
+		/*
+		 * Wait on writeback if requested to. This happens when
+		 * direct reclaiming a large contiguous area and the
+		 * first attempt to free a range of pages fails.
+		 */
+		if (PageWriteback(page) && sync_writeback == PAGEOUT_IO_SYNC)
+			wait_on_page_writeback(page);
+
 		if (!PageWriteback(page)) {
 			/* synchronous write or broken a_ops? */
 			ClearPageReclaim(page);
@@ -423,7 +439,8 @@ cannot_free:
  * shrink_page_list() returns the number of reclaimed pages
  */
 static unsigned long shrink_page_list(struct list_head *page_list,
-					struct scan_control *sc)
+					struct scan_control *sc,
+					enum pageout_io sync_writeback)
 {
 	LIST_HEAD(ret_pages);
 	struct pagevec freed_pvec;
@@ -458,8 +475,15 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		if (page_mapped(page) || PageSwapCache(page))
 			sc->nr_scanned++;
 
-		if (PageWriteback(page))
-			goto keep_locked;
+		may_enter_fs = (sc->gfp_mask & __GFP_FS) ||
+			(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
+
+		if (PageWriteback(page)) {
+			if (sync_writeback == PAGEOUT_IO_SYNC && may_enter_fs)
+				wait_on_page_writeback(page);
+			else
+				goto keep_locked;
+		}
 
 		referenced = page_referenced(page, 1);
 		/* In active use or really unfreeable?  Activate it. */
@@ -478,8 +502,6 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 #endif /* CONFIG_SWAP */
 
 		mapping = page_mapping(page);
-		may_enter_fs = (sc->gfp_mask & __GFP_FS) ||
-			(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
 
 		/*
 		 * The page is mapped into the page tables of one or more
@@ -505,7 +527,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */
-			switch(pageout(page, mapping)) {
+			switch (pageout(page, mapping, sync_writeback)) {
 			case PAGE_KEEP:
 				goto keep_locked;
 			case PAGE_ACTIVATE:
@@ -786,7 +808,29 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
 		spin_unlock_irq(&zone->lru_lock);
 
 		nr_scanned += nr_scan;
-		nr_freed = shrink_page_list(&page_list, sc);
+		nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);
+
+		/*
+		 * If we are direct reclaiming for contiguous pages and we do
+		 * not reclaim everything in the list, try again and wait
+		 * for IO to complete. This will stall high-order allocations
+		 * but that should be acceptable to the caller
+		 */
+		if (nr_freed < nr_taken && !current_is_kswapd() &&
+					sc->order > PAGE_ALLOC_COSTLY_ORDER) {
+			congestion_wait(WRITE, HZ/10);
+
+			/*
+			 * The attempt at page out may have made some
+			 * of the pages active, mark them inactive again.
+			 */
+			nr_active = clear_active_flags(&page_list);
+			count_vm_events(PGDEACTIVATE, nr_active);
+
+			nr_freed += shrink_page_list(&page_list, sc,
+							PAGEOUT_IO_SYNC);
+		}
+
 		nr_reclaimed += nr_freed;
 		local_irq_disable();
 		if (current_is_kswapd()) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 0/2] Synchronous Lumpy Reclaim V3

[Peter Zijlstra]

On Thu, 2007-08-02 at 19:17 +0100, Andy Whitcroft wrote:
> [This is a re-spin based on feedback from akpm.]
> 
> As pointed out by Mel when reclaim is applied at higher orders a
> significant amount of IO may be started.  As this takes finite time
> to drain reclaim will consider more areas than ultimatly needed
> to satisfy the request.  This leads to more reclaim than strictly
> required and reduced success rates.
> 
> I was able to confirm Mel's test results on systems locally.
> These show that even under light load the success rates drop off far
> more than expected.  Testing with a modified version of his patch
> (which follows) I was able to allocate almost all of ZONE_MOVABLE
> with a near idle system.  I ran 5 test passes sequentially following
> system boot (the system has 29 hugepages in ZONE_MOVABLE):
> 
>   2.6.23-rc1              11  8  6  7  7
>   sync_lumpy              28 28 29 29 26
> 
> These show that although hugely better than the near 0% success
> normally expected we can only allocate about a 1/4 of the zone.
> Using synchronous reclaim for these allocations we get close to 100%
> as expected.
> 
> I have also run our standard high order tests and these show no
> regressions in allocation success rates at rest, and some significant
> improvements under load.
> 
> Following this email are two patches, both should be considered as
> bug fixes to lumpy reclaim for 2.6.23:
> 
> ensure-we-count-pages-transitioning-inactive-via-clear_active_flags:
>   this a bug fix for Lumpy Reclaim fixing up a bug in VM Event
>   accounting when it marks pages inactive, and
> 
> Wait-for-page-writeback-when-directly-reclaiming-contiguous-areas:
>   updates reclaim making direct reclaim synchronous when applied
>   at orders above PAGE_ALLOC_COSTLY_ORDER.
> 
> Patches against 2.6.23-rc1.  Andrew please consider for -mm and
> for pushing to mainline.

Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 2/2] wait for page writeback when directly reclaiming contiguous areas

[Andrew Morton]

On Thu, 02 Aug 2007 19:18:43 +0100 Andy Whitcroft <apw@shadowen.org> wrote:

> @@ -458,8 +475,15 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  		if (page_mapped(page) || PageSwapCache(page))
>  			sc->nr_scanned++;
>  
> -		if (PageWriteback(page))
> -			goto keep_locked;
> +		may_enter_fs = (sc->gfp_mask & __GFP_FS) ||
> +			(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
> +
> +		if (PageWriteback(page)) {
> +			if (sync_writeback == PAGEOUT_IO_SYNC && may_enter_fs)
> +				wait_on_page_writeback(page);
> +			else
> +				goto keep_locked;
> +		}

this bit could do with a comment explaining the design decisions, please.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH] synchronous lumpy: improve commentary on writeback wait

[Andy Whitcroft]

Improve code commentary on the initial writeback wait in synchronous
reclaim mode.

Signed-off-by: Andy Whitcroft <apw@shadowen.org>
---
 mm/vmscan.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index b1e9291..a6e65d0 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -479,6 +479,14 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 			(PageSwapCache(page) && (sc->gfp_mask & __GFP_IO));
 
 		if (PageWriteback(page)) {
+			/*
+			 * Synchronous reclaim is performed in two passes,
+			 * first an asynchronous pass over the list to
+			 * start parallel writeback, and a second synchronous
+			 * pass to wait for the IO to complete.  Wait here
+			 * for any page for which writeback has already
+			 * started.
+			 */
 			if (sync_writeback == PAGEOUT_IO_SYNC && may_enter_fs)
 				wait_on_page_writeback(page);
 			else

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [PATCH 2/2] Wait for page writeback when directly reclaiming contiguous areas

[Andrew Morton]

On Sat, 28 Jul 2007 23:52:30 +0100
Andy Whitcroft <apw@shadowen.org> wrote:

> 
> From: Mel Gorman <mel@csn.ul.ie>
> 
> Lumpy reclaim works by selecting a lead page from the LRU list and then
> selecting pages for reclaim from the order-aligned area of pages. In the
> situation were all pages in that region are inactive and not referenced by
> any process over time, it works well.
> 
> In the situation where there is even light load on the system, the pages may
> not free quickly. Out of a area of 1024 pages, maybe only 950 of them are
> freed when the allocation attempt occurs because lumpy reclaim returned early.
> This patch alters the behaviour of direct reclaim for large contiguous blocks.
> 
> The first attempt to call shrink_page_list() is asynchronous but if it
> fails, the pages are submitted a second time and the calling process waits
> for the IO to complete. It'll retry up to 5 times for the pages to be
> fully freed. This may stall allocators waiting for contiguous memory but
> that should be expected behaviour for high-order users. It is preferable
> behaviour to potentially queueing unnecessary areas for IO. Note that kswapd
> will not stall in this fashion.

I agree with the intent.

> +/* Request for sync pageout. */
> +typedef enum {
> +	PAGEOUT_IO_ASYNC,
> +	PAGEOUT_IO_SYNC,
> +} pageout_io_t;

no typedefs.

(checkpatch.pl knew that ;))

>  /* possible outcome of pageout() */
>  typedef enum {
>  	/* failed to write page out, page is locked */
> @@ -287,7 +293,8 @@ typedef enum {
>   * pageout is called by shrink_page_list() for each dirty page.
>   * Calls ->writepage().
>   */
> -static pageout_t pageout(struct page *page, struct address_space *mapping)
> +static pageout_t pageout(struct page *page, struct address_space *mapping,
> +						pageout_io_t sync_writeback)
>  {
>  	/*
>  	 * If the page is dirty, only perform writeback if that write
> @@ -346,6 +353,15 @@ static pageout_t pageout(struct page *page, struct address_space *mapping)
>  			ClearPageReclaim(page);
>  			return PAGE_ACTIVATE;
>  		}
> +
> +		/*
> +		 * Wait on writeback if requested to. This happens when
> +		 * direct reclaiming a large contiguous area and the
> +		 * first attempt to free a ranage of pages fails

cnat tpye.

> +		 */
> +		if (PageWriteback(page) && sync_writeback == PAGEOUT_IO_SYNC)
> +			wait_on_page_writeback(page);
> +
>
>  		if (!PageWriteback(page)) {
>  			/* synchronous write or broken a_ops? */
>  			ClearPageReclaim(page);
> @@ -423,7 +439,8 @@ cannot_free:
>   * shrink_page_list() returns the number of reclaimed pages
>   */
>  static unsigned long shrink_page_list(struct list_head *page_list,
> -					struct scan_control *sc)
> +					struct scan_control *sc,
> +					pageout_io_t sync_writeback)
>  {
>  	LIST_HEAD(ret_pages);
>  	struct pagevec freed_pvec;
> @@ -458,8 +475,12 @@ static unsigned long shrink_page_list(struct list_head *page_list,
>  		if (page_mapped(page) || PageSwapCache(page))
>  			sc->nr_scanned++;
>  
> -		if (PageWriteback(page))
> -			goto keep_locked;
> +		if (PageWriteback(page)) {
> +			if (sync_writeback == PAGEOUT_IO_SYNC)
> +				wait_on_page_writeback(page);
> +			else
> +				goto keep_locked;
> +		}

This is unneeded and conceivably deadlocky for !__GFP_FS allocations. 
Probably we avoid doing all this if the test which may_enter_fs uses is
false.

It's unlikely that any very-high-order allocators are using GFP_NOIO or
whatever, but still...


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[PATCH 2/2] Wait for page writeback when directly reclaiming contiguous areas

[Andy Whitcroft]

From: Mel Gorman <mel@csn.ul.ie>

Lumpy reclaim works by selecting a lead page from the LRU list and then
selecting pages for reclaim from the order-aligned area of pages. In the
situation were all pages in that region are inactive and not referenced by
any process over time, it works well.

In the situation where there is even light load on the system, the pages may
not free quickly. Out of a area of 1024 pages, maybe only 950 of them are
freed when the allocation attempt occurs because lumpy reclaim returned early.
This patch alters the behaviour of direct reclaim for large contiguous blocks.

The first attempt to call shrink_page_list() is asynchronous but if it
fails, the pages are submitted a second time and the calling process waits
for the IO to complete. It'll retry up to 5 times for the pages to be
fully freed. This may stall allocators waiting for contiguous memory but
that should be expected behaviour for high-order users. It is preferable
behaviour to potentially queueing unnecessary areas for IO. Note that kswapd
will not stall in this fashion.

[apw@shadowen.org: update to version 2]
Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Signed-off-by: Andy Whitcroft <apw@shadowen.org>

Changelog:

Changes in V2:
 - remove retry loop
 - fix up active accounting (count deactivate events correctly)
 - use our own sync/async flag type
---
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 99ec7fa..1c21714 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -271,6 +271,12 @@ static void handle_write_error(struct address_space *mapping,
 	unlock_page(page);
 }
 
+/* Request for sync pageout. */
+typedef enum {
+	PAGEOUT_IO_ASYNC,
+	PAGEOUT_IO_SYNC,
+} pageout_io_t;
+
 /* possible outcome of pageout() */
 typedef enum {
 	/* failed to write page out, page is locked */
@@ -287,7 +293,8 @@ typedef enum {
  * pageout is called by shrink_page_list() for each dirty page.
  * Calls ->writepage().
  */
-static pageout_t pageout(struct page *page, struct address_space *mapping)
+static pageout_t pageout(struct page *page, struct address_space *mapping,
+						pageout_io_t sync_writeback)
 {
 	/*
 	 * If the page is dirty, only perform writeback if that write
@@ -346,6 +353,15 @@ static pageout_t pageout(struct page *page, struct address_space *mapping)
 			ClearPageReclaim(page);
 			return PAGE_ACTIVATE;
 		}
+
+		/*
+		 * Wait on writeback if requested to. This happens when
+		 * direct reclaiming a large contiguous area and the
+		 * first attempt to free a ranage of pages fails
+		 */
+		if (PageWriteback(page) && sync_writeback == PAGEOUT_IO_SYNC)
+			wait_on_page_writeback(page);
+
 		if (!PageWriteback(page)) {
 			/* synchronous write or broken a_ops? */
 			ClearPageReclaim(page);
@@ -423,7 +439,8 @@ cannot_free:
  * shrink_page_list() returns the number of reclaimed pages
  */
 static unsigned long shrink_page_list(struct list_head *page_list,
-					struct scan_control *sc)
+					struct scan_control *sc,
+					pageout_io_t sync_writeback)
 {
 	LIST_HEAD(ret_pages);
 	struct pagevec freed_pvec;
@@ -458,8 +475,12 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 		if (page_mapped(page) || PageSwapCache(page))
 			sc->nr_scanned++;
 
-		if (PageWriteback(page))
-			goto keep_locked;
+		if (PageWriteback(page)) {
+			if (sync_writeback == PAGEOUT_IO_SYNC)
+				wait_on_page_writeback(page);
+			else
+				goto keep_locked;
+		}
 
 		referenced = page_referenced(page, 1);
 		/* In active use or really unfreeable?  Activate it. */
@@ -505,7 +526,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
 				goto keep_locked;
 
 			/* Page is dirty, try to write it out here */
-			switch(pageout(page, mapping)) {
+			switch (pageout(page, mapping, sync_writeback)) {
 			case PAGE_KEEP:
 				goto keep_locked;
 			case PAGE_ACTIVATE:
@@ -786,7 +807,29 @@ static unsigned long shrink_inactive_list(unsigned long max_scan,
 		spin_unlock_irq(&zone->lru_lock);
 
 		nr_scanned += nr_scan;
-		nr_freed = shrink_page_list(&page_list, sc);
+		nr_freed = shrink_page_list(&page_list, sc, PAGEOUT_IO_ASYNC);
+
+		/*
+		 * If we are direct reclaiming for contiguous pages and we do
+		 * not reclaim everything in the list, try again and wait
+		 * for IO to complete. This will stall high-order allocations
+		 * but that should be acceptable to the caller
+		 */
+		if (nr_freed < nr_taken && !current_is_kswapd() &&
+					sc->order > PAGE_ALLOC_COSTLY_ORDER) {
+			congestion_wait(WRITE, HZ/10);
+
+			/*
+			 * The attempt at page out may have made some
+			 * of the pages active, mark them inactive again.
+			 */
+			nr_active = clear_active_flags(&page_list);
+			count_vm_events(PGDEACTIVATE, nr_active);
+
+			nr_freed += shrink_page_list(&page_list, sc,
+							PAGEOUT_IO_SYNC);
+		}
+
 		nr_reclaimed += nr_freed;
 		local_irq_disable();
 		if (current_is_kswapd()) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>