From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-la0-f41.google.com (mail-la0-f41.google.com [209.85.215.41]) by kanga.kvack.org (Postfix) with ESMTP id 7D34C6B0038 for ; Thu, 16 Apr 2015 03:51:24 -0400 (EDT) Received: by labbd9 with SMTP id bd9so50555782lab.2 for ; Thu, 16 Apr 2015 00:51:23 -0700 (PDT) Received: from numascale.com (numascale.com. [213.162.240.84]) by mx.google.com with ESMTPS id r2si5814196lar.102.2015.04.16.00.51.20 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Apr 2015 00:51:21 -0700 (PDT) Date: Thu, 16 Apr 2015 15:51:05 +0800 From: Daniel J Blueman Subject: Re: [RFC PATCH 0/14] Parallel memory initialisation Message-Id: <1429170665.19274.0@cpanel21.proisp.no> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=-8GnILWyjjcoTFyBHINwl" Sender: owner-linux-mm@kvack.org List-ID: To: Mel Gorman Cc: Steffen Persvold , Linux-MM , Robin Holt , Nathan Zimmer , Daniel Rahn , Davidlohr Bueso , Dave Hansen , Tom Vaden , Scott Norton , LKML --=-8GnILWyjjcoTFyBHINwl Content-Type: text/plain; charset=utf-8; format=flowed On Monday, April 13, 2015 at 6:20:05 PM UTC+8, Mel Gorman wrote: > Memory initialisation had been identified as one of the reasons why large > machines take a long time to boot. Patches were posted a long time ago > that attempted to move deferred initialisation into the page allocator > paths. This was rejected on the grounds it should not be necessary to hurt > the fast paths to parallelise initialisation. This series reuses much of > the work from that time but defers the initialisation of memory to kswapd > so that one thread per node initialises memory local to that node. The > issue is that on the machines I tested with, memory initialisation was not > a major contributor to boot times. I'm posting the RFC to both review the > series and see if it actually helps users of very large machines. > > After applying the series and setting the appropriate Kconfig variable I > see this in the boot log on a 64G machine > > [ 7.383764] kswapd 0 initialised deferred memory in 188ms > [ 7.404253] kswapd 1 initialised deferred memory in 208ms > [ 7.411044] kswapd 3 initialised deferred memory in 216ms > [ 7.411551] kswapd 2 initialised deferred memory in 216ms > > On a 1TB machine, I see > > [ 11.913324] kswapd 0 initialised deferred memory in 1168ms > [ 12.220011] kswapd 2 initialised deferred memory in 1476ms > [ 12.245369] kswapd 3 initialised deferred memory in 1500ms > [ 12.271680] kswapd 1 initialised deferred memory in 1528ms > > Once booted the machine appears to work as normal. Boot times were measured > from the time shutdown was called until ssh was available again. In the > 64G case, the boot time savings are negligible. On the 1TB machine, the > savings were 10 seconds (about 8% improvement on kernel times but 1-2% > overall as POST takes so long). > > It would be nice if the people that have access to really large machines > would test this series and report back if the complexity is justified. Nice work! On an older Numascale system with 1TB memory and 256 cores/32 NUMA nodes, platform init takes 52s (cold boot), firmware takes 84s (includes one warm reboot), stock linux 4.0 then takes 732s to boot [1] (due to the 700ns roundtrip, RMW cache-coherent cycles due to the temporal writes for pagetable init and per-core store queue limits), so there is huge potential. Alas I ran into crashing during list manipulation [2] which list debugging detects [3]; I had started adding some debug [4], but need to look a bit deeper into it. I annotated the time of the output from cold power on. Thanks, Daniel [1] https://resources.numascale.com/telemetry/defermem/console-stock.txt [2] https://resources.numascale.com/telemetry/defermem/console-patched.txt [3] https://resources.numascale.com/telemetry/defermem/console-patched-debug.txt -- [4] static void free_pcppages_bulk(struct zone *zone, int count, struct per_cpu_pages *pcp) ... pr_err("migrate_type=%d\n", migratetype); /* This is the only non-empty list. Free them all. */ if (batch_free == MIGRATE_PCPTYPES) batch_free = to_free; do { int mt; /* migratetype of the to-be-freed page */ pr_err("list_empty=%d\n", list_empty(list)); --=-8GnILWyjjcoTFyBHINwl Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
On Monday, April 13, 2015 at 6:20:05 PM UTC+8, Mel Gorman wrote:
=
> Memory initialisation had been identified as one of the reasons w= hy large
> machines take a long time to boot. Patches were pos= ted a long time ago
> that attempted to move deferred initiali= sation into the page allocator
> paths. This was rejected on t= he grounds it should not be necessary to hurt
> the fast paths= to parallelise initialisation. This series reuses much of
> t= he work from that time but defers the initialisation of memory to kswapd
> so that one thread per node initialises memory local to that n= ode. The
> issue is that on the machines I tested with, memory= initialisation was not
> a major contributor to boot times. I= 'm posting the RFC to both review the
> series and see if it a= ctually helps users of very large machines.
> 
= > After applying the series and setting the appropriate Kconfig variable= I
> see this in the boot log on a 64G machine
>&= nbsp;
> [    7.383764] kswapd 0 initialised deferred= memory in 188ms
> [    7.404253] kswapd 1 initialis= ed deferred memory in 208ms
> [    7.411044] kswapd = 3 initialised deferred memory in 216ms
> [    7.4115= 51] kswapd 2 initialised deferred memory in 216ms
> 
> On a 1TB machine, I see
> 
> [ &n= bsp; 11.913324] kswapd 0 initialised deferred memory in 1168ms
&g= t; [   12.220011] kswapd 2 initialised deferred memory in 1476ms
=
> [   12.245369] kswapd 3 initialised deferred memory in 1500m= s
> [   12.271680] kswapd 1 initialised deferred memory i= n 1528ms
> 
> Once booted the machine appear= s to work as normal. Boot times were measured
> from the time = shutdown was called until ssh was available again.  In the
&= gt; 64G case, the boot time savings are negligible. On the 1TB machine, the=
> savings were 10 seconds (about 8% improvement on kernel tim= es but 1-2%
> overall as POST takes so long).
>&n= bsp;
> It would be nice if the people that have access to real= ly large machines
> would test this series and report back if = the complexity is justified.

Nice work!
=
On an older Numascale system with 1TB memory and 256 cores/3= 2 NUMA nodes, platform init takes 52s (cold boot), firmware takes 84s (incl= udes one warm reboot), stock linux 4.0 then takes 732s to boot [1] (due to = the 700ns roundtrip, RMW cache-coherent cycles due to the temporal writes f= or pagetable init and per-core store queue limits), so there is huge potent= ial.

Alas I ran into crashing during list manipula= tion [2] which list debugging detects [3]; I had started adding some debug = [4], but need to look a bit deeper into it. I annotated the time of the out= put from cold power on.

Thanks,
  D= aniel


-- [4]
static void free_pcppages_bulk(struct zone *zone, int cou= nt,
= struct per_cpu_pages *pcp)
...
pr_err("migrate_type=3D= %d\n", migratetype);

/* This is the only non-empty list. Fr= ee them all. */
if (batch_free =3D=3D MIGRATE_PCPTYPES)
batch_free =3D t= o_free;

do {
int mt; /* migratetype of the to-be-freed page */

pr_err("list_empty=3D%d\n", list_empty(list));
= --=-8GnILWyjjcoTFyBHINwl-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org