From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30DF0C43461 for ; Thu, 15 Apr 2021 10:36:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CFF766137D for ; Thu, 15 Apr 2021 10:36:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CFF766137D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8E8296B0074; Thu, 15 Apr 2021 06:35:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BE506B0075; Thu, 15 Apr 2021 06:35:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7374A6B0078; Thu, 15 Apr 2021 06:35:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id 56E146B0074 for ; Thu, 15 Apr 2021 06:35:55 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1A1505923C0F for ; Thu, 15 Apr 2021 10:35:55 +0000 (UTC) X-FDA: 78034245870.24.EE7D869 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf06.hostedemail.com (Postfix) with ESMTP id ACAE6C0007D3 for ; Thu, 15 Apr 2021 10:35:56 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 857B9AF0D; Thu, 15 Apr 2021 10:35:53 +0000 (UTC) From: Oscar Salvador To: Andrew Morton Cc: Mike Kravetz , Vlastimil Babka , David Hildenbrand , Michal Hocko , Muchun Song , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador , Michal Hocko Subject: [PATCH v8 6/7] mm: Make alloc_contig_range handle in-use hugetlb pages Date: Thu, 15 Apr 2021 12:35:43 +0200 Message-Id: <20210415103544.6791-7-osalvador@suse.de> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20210415103544.6791-1-osalvador@suse.de> References: <20210415103544.6791-1-osalvador@suse.de> MIME-Version: 1.0 X-Stat-Signature: yp6xpzt9jhuctxo5pdub4xm7m9741jhs X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ACAE6C0007D3 Received-SPF: none (suse.de>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1618482956-379419 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: alloc_contig_range() will fail if it finds a HugeTLB page within the rang= e, without a chance to handle them. Since HugeTLB pages can be migrated as a= ny LRU or Movable page, it does not make sense to bail out without trying. Enable the interface to recognize in-use HugeTLB pages so we can migrate them, and have much better chances to succeed the call. Signed-off-by: Oscar Salvador Reviewed-by: Mike Kravetz Acked-by: Michal Hocko --- include/linux/hugetlb.h | 5 +++-- mm/compaction.c | 12 +++++++++++- mm/hugetlb.c | 24 ++++++++++++++++++------ mm/vmscan.c | 5 +++-- 4 files changed, 35 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index b2d2118bfd1a..b92f25ccef58 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -595,7 +595,7 @@ struct huge_bootmem_page { struct hstate *hstate; }; =20 -int isolate_or_dissolve_huge_page(struct page *page); +int isolate_or_dissolve_huge_page(struct page *page, struct list_head *l= ist); struct page *alloc_huge_page(struct vm_area_struct *vma, unsigned long addr, int avoid_reserve); struct page *alloc_huge_page_nodemask(struct hstate *h, int preferred_ni= d, @@ -878,7 +878,8 @@ static inline void huge_ptep_modify_prot_commit(struc= t vm_area_struct *vma, #else /* CONFIG_HUGETLB_PAGE */ struct hstate {}; =20 -static inline int isolate_or_dissolve_huge_page(struct page *page) +static inline int isolate_or_dissolve_huge_page(struct page *page, + struct list_head *list) { return -ENOMEM; } diff --git a/mm/compaction.c b/mm/compaction.c index 73f00f36b6e7..8149ffc6bccd 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -907,7 +907,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, } =20 if (PageHuge(page) && cc->alloc_contig) { - ret =3D isolate_or_dissolve_huge_page(page); + ret =3D isolate_or_dissolve_huge_page(page, &cc->migratepages); =20 /* * Fail isolation in case isolate_or_dissolve_huge_page() @@ -923,6 +923,15 @@ isolate_migratepages_block(struct compact_control *c= c, unsigned long low_pfn, goto isolate_fail; } =20 + if (PageHuge(page)) { + /* + * Hugepage was successfully isolated and placed + * on the cc->migratepages list. + */ + low_pfn +=3D compound_nr(page) - 1; + goto isolate_success_no_list; + } + /* * Ok, the hugepage was dissolved. Now these pages are * Buddy and cannot be re-allocated because they are @@ -1064,6 +1073,7 @@ isolate_migratepages_block(struct compact_control *= cc, unsigned long low_pfn, =20 isolate_success: list_add(&page->lru, &cc->migratepages); +isolate_success_no_list: cc->nr_migratepages +=3D compound_nr(page); nr_isolated +=3D compound_nr(page); =20 diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4abd4c4565dd..efe2f1fed74e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2270,10 +2270,12 @@ static void restore_reserve_on_error(struct hstat= e *h, * alloc_and_dissolve_huge_page - Allocate a new page and dissolve the o= ld one * @h: struct hstate old page belongs to * @old_page: Old page to dissolve + * @list: List to isolate the page in case we need to * Returns 0 on success, otherwise negated error. */ =20 -static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *o= ld_page) +static int alloc_and_dissolve_huge_page(struct hstate *h, struct page *o= ld_page, + struct list_head *list) { gfp_t gfp_mask =3D htlb_alloc_mask(h) | __GFP_THISNODE; int nid =3D page_to_nid(old_page); @@ -2300,9 +2302,13 @@ static int alloc_and_dissolve_huge_page(struct hst= ate *h, struct page *old_page) goto free_new; } else if (page_count(old_page)) { /* - * Someone has grabbed the page, fail for now. + * Someone has grabbed the page, try to isolate it here. + * Fail with -EBUSY if not possible. */ - ret =3D -EBUSY; + spin_unlock_irq(&hugetlb_lock); + if (!isolate_huge_page(old_page, list)) + ret =3D -EBUSY; + spin_lock_irq(&hugetlb_lock); goto free_new; } else if (!HPageFreed(old_page)) { /* @@ -2352,10 +2358,11 @@ static int alloc_and_dissolve_huge_page(struct hs= tate *h, struct page *old_page) return ret; } =20 -int isolate_or_dissolve_huge_page(struct page *page) +int isolate_or_dissolve_huge_page(struct page *page, struct list_head *l= ist) { struct hstate *h; struct page *head; + int ret =3D -EBUSY; =20 /* * The page might have been dissolved from under our feet, so make sure @@ -2374,13 +2381,18 @@ int isolate_or_dissolve_huge_page(struct page *pa= ge) =20 /* * Fence off gigantic pages as there is a cyclic dependency between - * alloc_contig_range and them. Return -ENOME as this has the effect + * alloc_contig_range and them. Return -ENOMEM as this has the effect * of bailing out right away without further retrying. */ if (hstate_is_gigantic(h)) return -ENOMEM; =20 - return alloc_and_dissolve_huge_page(h, head); + if (page_count(head) && isolate_huge_page(head, list)) + ret =3D 0; + else if (!page_count(head)) + ret =3D alloc_and_dissolve_huge_page(h, head, list); + + return ret; } =20 struct page *alloc_huge_page(struct vm_area_struct *vma, diff --git a/mm/vmscan.c b/mm/vmscan.c index bb8321026c0c..5199b9696bab 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1703,8 +1703,9 @@ unsigned int reclaim_clean_pages_from_list(struct z= one *zone, LIST_HEAD(clean_pages); =20 list_for_each_entry_safe(page, next, page_list, lru) { - if (page_is_file_lru(page) && !PageDirty(page) && - !__PageMovable(page) && !PageUnevictable(page)) { + if (!PageHuge(page) && page_is_file_lru(page) && + !PageDirty(page) && !__PageMovable(page) && + !PageUnevictable(page)) { ClearPageActive(page); list_move(&page->lru, &clean_pages); } --=20 2.16.3