From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB522C433E4 for ; Fri, 14 Aug 2020 17:31:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A62B120838 for ; Fri, 14 Aug 2020 17:31:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="C6ftLx9s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A62B120838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BE5738D0002; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6C7F8D0003; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BD908D0002; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id 749AA8D0003 for ; Fri, 14 Aug 2020 13:31:45 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2E013181AEF23 for ; Fri, 14 Aug 2020 17:31:45 +0000 (UTC) X-FDA: 77149866570.09.jam52_531080126ffe Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 89454180AD81A for ; Fri, 14 Aug 2020 17:31:44 +0000 (UTC) X-HE-Tag: jam52_531080126ffe X-Filterd-Recvd-Size: 8417 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Fri, 14 Aug 2020 17:31:43 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id m8so4885744pfh.3 for ; Fri, 14 Aug 2020 10:31:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EkTZgMvscD2KwIBX0vZD343fn5ehrZxMsNsNzYcucec=; b=C6ftLx9sbYiyyYKgmA04wWOsnc7KvbG16gcWcNOv/hcRQw8ybQMpUwoyUXTR4dmtd0 uZe72diH9C0c9xNNh6MC0rK+V4NcOpIoVZm+yvCAf/fkIUYY5jyh5BlDOaVauQoDwEKI 4ctGErOacT8LiOg7a5hP/unoZeRiLFafHJ/dU6lGeosk0jyLsGZRcVtrzAVQglt2rg9r yPa3M8DbpvIadtJ79ZQ5Ku0NOXf9GljegFXH30kKLB2U3hAl2QXgC1un7EzyYX8sNB0A d21DpyE9mccr1JS1m/roezO+OUJaqx3ip7oPJxx216u6kZgUxQQy6tFEhkoYqKAYB8k/ G+Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=EkTZgMvscD2KwIBX0vZD343fn5ehrZxMsNsNzYcucec=; b=ip+L71GTlT2Z9stLPEmJ5UTXPmKJ7jvQ/xFC4Xr0pBXPf8hi8aPx6+vuvR1NcjB1xx WXZWZdriva8wUg/hwzofX4qEeJq0I6DWuYkQtXFIdLFIRsVTanvmlXokD3N9PU1UuVBq hNdzVE2xrI0cqajxzaI4Vw8x77S0SeehueDwAmBmIdcmYBzpKw2s7eX5ozUGiolDRkzR +SVXPemRnClqObK2G0QGi66agAKYMbHR7JP6+BkPUpbnGFDmvG8g5ZboXO9n19L8PazQ OCnH2SK2mpCadG2EcN2W3lMmPParXRHhEVzgPV4XFg7VkD/dcCCdeP2jsbRKMrru5fi0 N4aA== X-Gm-Message-State: AOAM530MRPjRT+8BWORO2wzzjehwwGSyCpkbbukpPdN/SBoEgf13ATjY oMPjDYAIpfKie4yban55JSM= X-Google-Smtp-Source: ABdhPJzay7FypJUmNSI99B+mNjGzIjNonb2GXNqZn8e1q9by06s3lfGRmPkPuA29rxIOZolhWT6ktA== X-Received: by 2002:a65:62c3:: with SMTP id m3mr2407190pgv.338.1597426302983; Fri, 14 Aug 2020 10:31:42 -0700 (PDT) Received: from bbox-1.mtv.corp.google.com ([2620:15c:211:1:7220:84ff:fe09:5e58]) by smtp.gmail.com with ESMTPSA id n22sm8522973pjq.25.2020.08.14.10.31.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Aug 2020 10:31:41 -0700 (PDT) From: Minchan Kim To: Andrew Morton Cc: linux-mm , Joonsoo Kim , Vlastimil Babka , John Dias , Suren Baghdasaryan , pullip.cho@samsung.com, Minchan Kim Subject: [RFC 3/7] mm: compaction: deal with upcoming high-order page splitting Date: Fri, 14 Aug 2020 10:31:27 -0700 Message-Id: <20200814173131.2803002-4-minchan@kernel.org> X-Mailer: git-send-email 2.28.0.220.ged08abb693-goog In-Reply-To: <20200814173131.2803002-1-minchan@kernel.org> References: <20200814173131.2803002-1-minchan@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 89454180AD81A X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When compaction isolates free pages, it needs to consider freed pages's order and sub-page splitting to support upcoming high order page bulk allocation. Since we have primitive functions to deal with high order page splitting, this patch introduces cc->isolate_order to indicate what order pages the API user want to allocate. It isolates free pages with order greater or equal to cc->isolate_order. After isolating it splits them into sub pages of cc->isolate_order order. Signed-off-by: Minchan Kim --- mm/compaction.c | 42 ++++++++++++++++++++++++++++-------------- mm/internal.h | 1 + 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index f31799a841f2..76f380cb801d 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -68,7 +68,8 @@ static const unsigned int HPAGE_FRAG_CHECK_INTERVAL_MSE= C =3D 500; #define COMPACTION_HPAGE_ORDER (PMD_SHIFT - PAGE_SHIFT) #endif =20 -static unsigned long release_freepages(struct list_head *freelist) +static unsigned long release_freepages(struct list_head *freelist, + unsigned int order) { struct page *page, *next; unsigned long high_pfn =3D 0; @@ -76,7 +77,7 @@ static unsigned long release_freepages(struct list_head= *freelist) list_for_each_entry_safe(page, next, freelist, lru) { unsigned long pfn =3D page_to_pfn(page); list_del(&page->lru); - __free_page(page); + __free_pages(page, order); if (pfn > high_pfn) high_pfn =3D pfn; } @@ -84,7 +85,7 @@ static unsigned long release_freepages(struct list_head= *freelist) return high_pfn; } =20 -static void split_map_pages(struct list_head *list) +static void split_map_pages(struct list_head *list, unsigned int split_o= rder) { unsigned int i, order, nr_pages; struct page *page, *next; @@ -94,15 +95,15 @@ static void split_map_pages(struct list_head *list) list_del(&page->lru); =20 order =3D page_private(page); - nr_pages =3D 1 << order; + nr_pages =3D 1 << (order - split_order); =20 post_alloc_hook(page, order, __GFP_MOVABLE); - if (order) - split_page_by_order(page, order, 0); + if (order > split_order) + split_page_by_order(page, order, split_order); =20 for (i =3D 0; i < nr_pages; i++) { list_add(&page->lru, &tmp_list); - page++; + page +=3D 1 << split_order; } } =20 @@ -547,8 +548,10 @@ static bool compact_unlock_should_abort(spinlock_t *= lock, } =20 /* - * Isolate free pages onto a private freelist. If @strict is true, will = abort - * returning 0 on any invalid PFNs or non-free pages inside of the pageb= lock + * Isolate free pages onto a private freelist if order of page is greate= r + * or equal to cc->isolate_order. If @strict is true, will abort + * returning 0 on any invalid PFNs, pages with order lower than + * cc->isolate_order or non-free pages inside of the pageblock * (even though it may still end up isolating some pages). */ static unsigned long isolate_freepages_block(struct compact_control *cc, @@ -625,8 +628,19 @@ static unsigned long isolate_freepages_block(struct = compact_control *cc, goto isolate_fail; } =20 - /* Found a free page, will break it into order-0 pages */ + /* + * Found a free page. will isolate and possibly split the page + * into isolate_order sub pages if the page's order is greater + * than or equal to the isolate_order. Otherwise, it will keep + * going with further pages to isolate them unless strict is + * true. + */ order =3D page_order(page); + if (order < cc->isolate_order) { + blockpfn +=3D (1UL << order) - 1; + cursor +=3D (1UL << order) - 1; + goto isolate_fail; + } isolated =3D __isolate_free_page(page, order); if (!isolated) break; @@ -752,11 +766,11 @@ isolate_freepages_range(struct compact_control *cc, } =20 /* __isolate_free_page() does not map the pages */ - split_map_pages(&freelist); + split_map_pages(&freelist, cc->isolate_order); =20 if (pfn < end_pfn) { /* Loop terminated early, cleanup. */ - release_freepages(&freelist); + release_freepages(&freelist, cc->isolate_order); return 0; } =20 @@ -1564,7 +1578,7 @@ static void isolate_freepages(struct compact_contro= l *cc) =20 splitmap: /* __isolate_free_page() does not map the pages */ - split_map_pages(freelist); + split_map_pages(freelist, 0); } =20 /* @@ -2376,7 +2390,7 @@ compact_zone(struct compact_control *cc, struct cap= ture_control *capc) * so we don't leave any returned pages behind in the next attempt. */ if (cc->nr_freepages > 0) { - unsigned long free_pfn =3D release_freepages(&cc->freepages); + unsigned long free_pfn =3D release_freepages(&cc->freepages, 0); =20 cc->nr_freepages =3D 0; VM_BUG_ON(free_pfn =3D=3D 0); diff --git a/mm/internal.h b/mm/internal.h index 10c677655912..5f1e9d76a623 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -244,6 +244,7 @@ struct compact_control { bool contended; /* Signal lock or sched contention */ bool rescan; /* Rescanning the same pageblock */ bool alloc_contig; /* alloc_contig_range allocation */ + int isolate_order; /* minimum order isolated from buddy */ }; =20 /* --=20 2.28.0.220.ged08abb693-goog