Re: zone state overhead

[Mel Gorman <mel@csn.ul.ie>]

On Thu, Oct 14, 2010 at 12:07:29PM +0900, KOSAKI Motohiro wrote:
> Hi
> 
> > > > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > > > index c5dfabf..47ba29e 100644
> > > > --- a/mm/vmscan.c
> > > > +++ b/mm/vmscan.c
> > > > @@ -2378,7 +2378,9 @@ static int kswapd(void *p)
> > > >  				 */
> > > >  				if (!sleeping_prematurely(pgdat, order, remaining)) {
> > > >  					trace_mm_vmscan_kswapd_sleep(pgdat->node_id);
> > > > +					enable_pgdat_percpu_threshold(pgdat);
> > > >  					schedule();
> > > > +					disable_pgdat_percpu_threshold(pgdat);
> > > 
> > > If we have 4096 cpus, max drift = 125x4096x4096 ~= 2GB. It is higher than zone watermark.
> > > Then, such sysmtem can makes memory exshost before kswap call disable_pgdat_percpu_threshold().
> > > 
> > 
> > I don't *think* so but lets explore that possibility. For this to occur, all
> > CPUs would have to be allocating all of their memory from the one node (4096
> > CPUs is not going to be UMA) which is not going to happen. But allocations
> > from one node could be falling over to others of course.
> > 
> > Lets take an early condition that has to occur for a 4096 CPU machine to
> > get into trouble - node 0 exhausted and moving to node 1 and counter drift
> > makes us think everything is fine.
> > 
> > __alloc_pages_nodemask
> >   -> get_page_from_freelist
> >     -> zone_watermark_ok == true (because we are drifting)
> >     -> buffered_rmqueue
> >       -> __rmqueue (fails eventually, no pages despite watermark_ok)
> >   -> __alloc_pages_slowpath
> >     -> wake_all_kswapd()
> > ...
> >
> > kswapd wakes
> >   -> disable_pgdat_percpu_threshold()
> > 
> > i.e. as each node becomes exhausted in reality, kswapd will wake up, disable
> > the thresholds until the high watermark is back and go back to sleep. I'm
> > not seeing how we'd get into a situation where all kswapds are asleep at the
> > same time while each allocator allocates all of memory without managing to
> > wake kswapd. Even GFP_ATOMIC allocations will wakeup kswapd.
> > 
> > Hence, I think the current patch of disabling thresholds while kswapd is
> > awake to be sufficient to avoid livelock due to memory exhaustion and
> > counter drift.
> > 
> 
> In this case, wakeup_kswapd() don't wake kswapd because
> 
> ---------------------------------------------------------------------------------
> void wakeup_kswapd(struct zone *zone, int order)
> {
>         pg_data_t *pgdat;
> 
>         if (!populated_zone(zone))
>                 return;
> 
>         pgdat = zone->zone_pgdat;
>         if (zone_watermark_ok(zone, order, low_wmark_pages(zone), 0, 0))
>                 return;                          // HERE
> ---------------------------------------------------------------------------------
> 
> So, if we take your approach, we need to know exact free pages in this.

Good point!

> But, zone_page_state_snapshot() is slow. that's dilemma.
> 

Very true. I'm prototyping a version of the patch that keeps
zone_page_state_snapshot but only uses is in wakeup_kswapd and
sleeping_prematurely.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>