From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D152C4727C for ; Tue, 22 Sep 2020 14:37:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BC2432395C for ; Tue, 22 Sep 2020 14:37:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BC2432395C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6DEF590009B; Tue, 22 Sep 2020 10:37:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5553490009E; Tue, 22 Sep 2020 10:37:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 420E090009B; Tue, 22 Sep 2020 10:37:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 18A6990009E for ; Tue, 22 Sep 2020 10:37:26 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D292A180AD804 for ; Tue, 22 Sep 2020 14:37:25 +0000 (UTC) X-FDA: 77290950450.13.house38_54097702714e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 6515F18140630 for ; Tue, 22 Sep 2020 14:37:23 +0000 (UTC) X-HE-Tag: house38_54097702714e X-Filterd-Recvd-Size: 4431 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Tue, 22 Sep 2020 14:37:22 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 6CB8BB12C; Tue, 22 Sep 2020 14:37:57 +0000 (UTC) From: Vlastimil Babka To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Michal Hocko , Pavel Tatashin , David Hildenbrand , Oscar Salvador , Joonsoo Kim , Vlastimil Babka Subject: [PATCH 8/9] mm, page_alloc: drain all pcplists during memory offline Date: Tue, 22 Sep 2020 16:37:11 +0200 Message-Id: <20200922143712.12048-9-vbabka@suse.cz> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200922143712.12048-1-vbabka@suse.cz> References: <20200922143712.12048-1-vbabka@suse.cz> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: drain_all_pages() is optimized to only execute on cpus where pcplists are= not empty. The check can however race with a free to pcplist that has not yet increased the pcp->count from 0 to 1. Make the drain optionally skip the = racy check and drain on all cpus, and use it in memory offline context, where = we want to make sure no isolated pages are left behind on pcplists. Signed-off-by: Vlastimil Babka --- include/linux/gfp.h | 1 + mm/memory_hotplug.c | 4 ++-- mm/page_alloc.c | 29 ++++++++++++++++++++--------- 3 files changed, 23 insertions(+), 11 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 67a0774e080b..cc52c5cc9fab 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -592,6 +592,7 @@ extern void page_frag_free(void *addr); =20 void page_alloc_init(void); void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp); +void __drain_all_pages(struct zone *zone, bool page_isolation); void drain_all_pages(struct zone *zone); void drain_local_pages(struct zone *zone); =20 diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 08f729922e18..bbde415b558b 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1524,7 +1524,7 @@ int __ref offline_pages(unsigned long start_pfn, un= signed long nr_pages) goto failed_removal; } =20 - drain_all_pages(zone); + __drain_all_pages(zone, true); =20 arg.start_pfn =3D start_pfn; arg.nr_pages =3D nr_pages; @@ -1588,7 +1588,7 @@ int __ref offline_pages(unsigned long start_pfn, un= signed long nr_pages) */ ret =3D test_pages_isolated(start_pfn, end_pfn, MEMORY_OFFLINE); if (ret) - drain_all_pages(zone); + __drain_all_pages(zone, true); } while (ret); =20 /* Mark all sections offline and remove free pages from the buddy. */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4e37bc3f6077..33cc35d152b1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2960,14 +2960,7 @@ static void drain_local_pages_wq(struct work_struc= t *work) preempt_enable(); } =20 -/* - * Spill all the per-cpu pages from all CPUs back into the buddy allocat= or. - * - * When zone parameter is non-NULL, spill just the single zone's pages. - * - * Note that this can be extremely slow as the draining happens in a wor= kqueue. - */ -void drain_all_pages(struct zone *zone) +void __drain_all_pages(struct zone *zone, bool force_all_cpus) { int cpu; =20 @@ -3006,7 +2999,13 @@ void drain_all_pages(struct zone *zone) struct zone *z; bool has_pcps =3D false; =20 - if (zone) { + if (force_all_cpus) { + /* + * The pcp.count check is racy, some callers need a + * guarantee that no cpu is missed. + */ + has_pcps =3D true; + } else if (zone) { pcp =3D per_cpu_ptr(zone->pageset, cpu); if (pcp->pcp.count) has_pcps =3D true; @@ -3039,6 +3038,18 @@ void drain_all_pages(struct zone *zone) mutex_unlock(&pcpu_drain_mutex); } =20 +/* + * Spill all the per-cpu pages from all CPUs back into the buddy allocat= or. + * + * When zone parameter is non-NULL, spill just the single zone's pages. + * + * Note that this can be extremely slow as the draining happens in a wor= kqueue. + */ +void drain_all_pages(struct zone *zone) +{ + __drain_all_pages(zone, false); +} + #ifdef CONFIG_HIBERNATION =20 /* --=20 2.28.0