From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7973C2B9F4 for ; Sat, 26 Jun 2021 00:21:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7CAF861469 for ; Sat, 26 Jun 2021 00:21:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7CAF861469 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 552576B0088; Fri, 25 Jun 2021 20:21:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 500B86B0092; Fri, 25 Jun 2021 20:21:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37D0E8D0003; Fri, 25 Jun 2021 20:21:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id 023056B0088 for ; Fri, 25 Jun 2021 20:21:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 15227180ACF8B for ; Sat, 26 Jun 2021 00:21:52 +0000 (UTC) X-FDA: 78293972064.27.EF78A66 Received: from mail-il1-f169.google.com (mail-il1-f169.google.com [209.85.166.169]) by imf11.hostedemail.com (Postfix) with ESMTP id B39E72001060 for ; Sat, 26 Jun 2021 00:21:51 +0000 (UTC) Received: by mail-il1-f169.google.com with SMTP id t12so2525060ile.13 for ; Fri, 25 Jun 2021 17:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FexPTJAXmVOLFTUUDErwm13SfJ4gYBZiX3PTNyQHySA=; b=ZjlrepTBA5dF0/huE1OtVBc8xjl84O+dHOupI5JRw74PUcMZvliaKjtMMNCa2aIZDC UA0rH2bJD9fEp+E03T/aBykZUy+pRNSGmXH9JqSiM2CHIudaItoKlbY7KGqYWfoEs8n9 iNmOSWqXwzoSbrXrly53luyOe/s70XSp/ZjzbT1MJZXbeCg49xxcot4A1Dw959cpSZGe zMeIhWDO9ljW2xh13UgGAIYolNQ+SQtnRvg9Xwk2zwMMmUw4ejOs8rTLLyvSnqOgPy0c fDv0vNFwfW+EMGiPEnDu/RpRzNkTDLp4evJ9Bq6T7NFJwfat0Y7RELlOPn6z7iRWlh6b HZHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FexPTJAXmVOLFTUUDErwm13SfJ4gYBZiX3PTNyQHySA=; b=F/Ri/t2oy5WGuJNVLh6r6MqtpA3wVbHxDeMYixiWnVRHG4mpy0mW3jVBdnatsBhKwn wr/MLjl7hPUjEhTjjvDLnmbWTjnKCzAanfms8Kx1pFyrih1GRn9KRBYRCEfufZ3VLebW 7uxvJ+xg4/TkUYh1xeGsTP/2PiN0T7CgbEUK5Xl4c50LBHXb6SzO6l5biffu/PVAU9B5 yasUH7tT5iUQVMEm34tMu7qoPfWpPMQez9r4JgrAZCYzQXxOgY9W7/6gRQIH38a2lcup NDTj/90Qu/7jarGGoeq0Zrty6rZkWrUupYl1faoYD7PiZ62EWgR4Vfqqw/VRn7pEEkTz opCg== X-Gm-Message-State: AOAM532sTN4uSQp85OM6RjAaUimhRmd16l2qTJSiMI43+xRG8foEM6Hr Q/ESzgpaX3jR1vBQwZ8VUAlzS6g6fji4wGy+2lVYkQ== X-Google-Smtp-Source: ABdhPJy+5BXiSrdIt1yeun2S1r5wlZ7iB/7NF6TDYn41n9012gVHaEtv8/9tGUyjX8eQpQ+otS1Y4PBH2i9CbVLo4sE= X-Received: by 2002:a92:290f:: with SMTP id l15mr9328392ilg.220.1624666910978; Fri, 25 Jun 2021 17:21:50 -0700 (PDT) MIME-Version: 1.0 References: <20210625073204.1005986-1-ying.huang@intel.com> <20210625073204.1005986-5-ying.huang@intel.com> In-Reply-To: <20210625073204.1005986-5-ying.huang@intel.com> From: Wei Xu Date: Fri, 25 Jun 2021 17:21:40 -0700 Message-ID: Subject: Re: [PATCH -V9 4/9] mm/migrate: demote pages during reclaim To: Huang Ying Cc: Linux MM , Linux Kernel Mailing List , Dave Hansen , Yang Shi , Oscar Salvador , Michal Hocko , Zi Yan , David Rientjes , Dan Williams Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=ZjlrepTB; spf=pass (imf11.hostedemail.com: domain of weixugc@google.com designates 209.85.166.169 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: qh77azsrgwm9kp1khzz4wbi9q63rjxk4 X-Rspamd-Queue-Id: B39E72001060 X-Rspamd-Server: rspam06 X-HE-Tag: 1624666911-851178 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jun 25, 2021 at 12:33 AM Huang Ying wrote: > > From: Dave Hansen > > This is mostly derived from a patch from Yang Shi: > > https://lore.kernel.org/linux-mm/1560468577-101178-10-git-send-email-yang.shi@linux.alibaba.com/ > > Add code to the reclaim path (shrink_page_list()) to "demote" data > to another NUMA node instead of discarding the data. This always > avoids the cost of I/O needed to read the page back in and sometimes > avoids the writeout cost when the page is dirty. > > A second pass through shrink_page_list() will be made if any demotions > fail. This essentially falls back to normal reclaim behavior in the > case that demotions fail. Previous versions of this patch may have > simply failed to reclaim pages which were eligible for demotion but > were unable to be demoted in practice. > > For some cases, for example, MADV_PAGEOUT, the pages are always > discarded instead of demoted to follow the kernel API definition. > Because MADV_PAGEOUT is defined as freeing specified pages regardless > in which tier they are. > > Note: This just adds the start of infrastructure for migration. It is > actually disabled next to the FIXME in migrate_demote_page_ok(). > > Signed-off-by: Dave Hansen > Signed-off-by: "Huang, Ying" > Reviewed-by: Yang Shi > Reviewed-by: Wei Xu > Reviewed-by: Oscar Salvador > Cc: Michal Hocko > Cc: Zi Yan > Cc: David Rientjes > Cc: Dan Williams > > -- > changes from 20210618: > * Fix some typos in comments and patch description. > * Fix MADV_PAGEOUT behavior per Zi's comments. > > changes from 20210122: > * move from GFP_HIGHUSER -> GFP_HIGHUSER_MOVABLE (Ying) > > changes from 202010: > * add MR_NUMA_MISPLACED to trace MIGRATE_REASON define > * make migrate_demote_page_ok() static, remove 'sc' arg until > later patch > * remove unnecessary alloc_demote_page() hugetlb warning > * Simplify alloc_demote_page() gfp mask. Depend on > __GFP_NORETRY to make it lightweight instead of fancier > stuff like leaving out __GFP_IO/FS. > * Allocate migration page with alloc_migration_target() > instead of allocating directly. > changes from 20200730: > * Add another pass through shrink_page_list() when demotion > fails. > changes from 20210302: > * Use __GFP_THISNODE and revise the comment explaining the > GFP mask constructionn > --- > include/linux/migrate.h | 9 ++++ > include/trace/events/migrate.h | 3 +- > mm/vmscan.c | 93 ++++++++++++++++++++++++++++++++++ > 3 files changed, 104 insertions(+), 1 deletion(-) > > diff --git a/include/linux/migrate.h b/include/linux/migrate.h > index 166dc1fe4eee..d678e02a27ff 100644 > --- a/include/linux/migrate.h > +++ b/include/linux/migrate.h > @@ -28,6 +28,7 @@ enum migrate_reason { > MR_NUMA_MISPLACED, > MR_CONTIG_RANGE, > MR_LONGTERM_PIN, > + MR_DEMOTION, > MR_TYPES > }; > > @@ -191,6 +192,14 @@ struct migrate_vma { > int migrate_vma_setup(struct migrate_vma *args); > void migrate_vma_pages(struct migrate_vma *migrate); > void migrate_vma_finalize(struct migrate_vma *migrate); > +int next_demotion_node(int node); > + > +#else /* CONFIG_MIGRATION disabled: */ > + > +static inline int next_demotion_node(int node) > +{ > + return NUMA_NO_NODE; > +} > > #endif /* CONFIG_MIGRATION */ > > diff --git a/include/trace/events/migrate.h b/include/trace/events/migrate.h > index 9fb2a3bbcdfb..779f3fad9ecd 100644 > --- a/include/trace/events/migrate.h > +++ b/include/trace/events/migrate.h > @@ -21,7 +21,8 @@ > EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ > EM( MR_NUMA_MISPLACED, "numa_misplaced") \ > EM( MR_CONTIG_RANGE, "contig_range") \ > - EMe(MR_LONGTERM_PIN, "longterm_pin") > + EM( MR_LONGTERM_PIN, "longterm_pin") \ > + EMe(MR_DEMOTION, "demotion") > > /* > * First define the enums in the above macros to be exported to userspace > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 5199b9696bab..badaf3d4360d 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -41,6 +41,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -118,6 +119,9 @@ struct scan_control { > /* The file pages on the current node are dangerously low */ > unsigned int file_is_tiny:1; > > + /* Always discard instead of demoting to lower tier memory */ > + unsigned int no_demotion:1; > + > /* Allocation order */ > s8 order; > > @@ -1231,6 +1235,27 @@ static enum page_references page_check_references(struct page *page, > return PAGEREF_RECLAIM; > } > > +static bool migrate_demote_page_ok(struct page *page, > + struct scan_control *sc) > +{ > + int next_nid; > + > + VM_BUG_ON_PAGE(!PageLocked(page), page); > + VM_BUG_ON_PAGE(PageHuge(page), page); > + VM_BUG_ON_PAGE(PageLRU(page), page); > + > + if (sc->no_demotion) > + return false; > + next_nid = next_demotion_node(page_to_nid(page)); > + if (next_nid == NUMA_NO_NODE) > + return false; The above sc and node checks are not specific to the page. They can be moved into shrink_page_list() to set do_demote_pass accordingly. We then don't need to repeat these checks for every page. > + if (PageTransHuge(page) && !thp_migration_supported()) > + return false; > + > + // FIXME: actually enable this later in the series > + return false; > +} > + > /* Check if a page is dirty or under writeback */ > static void page_check_dirty_writeback(struct page *page, > bool *dirty, bool *writeback) > @@ -1261,6 +1286,49 @@ static void page_check_dirty_writeback(struct page *page, > mapping->a_ops->is_dirty_writeback(page, dirty, writeback); > } > > +static struct page *alloc_demote_page(struct page *page, unsigned long node) > +{ > + struct migration_target_control mtc = { > + /* > + * Allocate from 'node', or fail quickly and quietly. > + * When this happens, 'page' will likely just be discarded > + * instead of migrated. > + */ > + .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | > + __GFP_THISNODE | __GFP_NOWARN | > + __GFP_NOMEMALLOC | GFP_NOWAIT, > + .nid = node > + }; > + > + return alloc_migration_target(page, (unsigned long)&mtc); > +} > + > +/* > + * Take pages on @demote_list and attempt to demote them to > + * another node. Pages which are not demoted are left on > + * @demote_pages. > + */ > +static unsigned int demote_page_list(struct list_head *demote_pages, > + struct pglist_data *pgdat) > +{ > + int target_nid = next_demotion_node(pgdat->node_id); > + unsigned int nr_succeeded; > + int err; > + > + if (list_empty(demote_pages)) > + return 0; > + > + if (target_nid == NUMA_NO_NODE) > + return 0; > + > + /* Demotion ignores all cpuset and mempolicy settings */ > + err = migrate_pages(demote_pages, alloc_demote_page, NULL, > + target_nid, MIGRATE_ASYNC, MR_DEMOTION, > + &nr_succeeded); > + > + return nr_succeeded; > +} > + > /* > * shrink_page_list() returns the number of reclaimed pages > */ > @@ -1272,12 +1340,15 @@ static unsigned int shrink_page_list(struct list_head *page_list, > { > LIST_HEAD(ret_pages); > LIST_HEAD(free_pages); > + LIST_HEAD(demote_pages); > unsigned int nr_reclaimed = 0; > unsigned int pgactivate = 0; > + bool do_demote_pass = true; > > memset(stat, 0, sizeof(*stat)); > cond_resched(); > > +retry: > while (!list_empty(page_list)) { > struct address_space *mapping; > struct page *page; > @@ -1426,6 +1497,16 @@ static unsigned int shrink_page_list(struct list_head *page_list, > ; /* try to reclaim the page below */ > } > > + /* > + * Before reclaiming the page, try to relocate > + * its contents to another node. > + */ > + if (do_demote_pass && migrate_demote_page_ok(page, sc)) { > + list_add(&page->lru, &demote_pages); > + unlock_page(page); > + continue; > + } > + > /* > * Anonymous process memory has backing store? > * Try to allocate it some swap space here. > @@ -1676,6 +1757,17 @@ static unsigned int shrink_page_list(struct list_head *page_list, > list_add(&page->lru, &ret_pages); > VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page); > } > + /* 'page_list' is always empty here */ > + > + /* Migrate pages selected for demotion */ > + nr_reclaimed += demote_page_list(&demote_pages, pgdat); > + /* Pages that could not be demoted are still in @demote_pages */ > + if (!list_empty(&demote_pages)) { > + /* Pages which failed to demoted go back on @page_list for retry: */ > + list_splice_init(&demote_pages, page_list); > + do_demote_pass = false; > + goto retry; > + } > > pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; > > @@ -2312,6 +2404,7 @@ unsigned long reclaim_pages(struct list_head *page_list) > .may_writepage = 1, > .may_unmap = 1, > .may_swap = 1, > + .no_demotion = 1, > }; > > while (!list_empty(page_list)) { > -- > 2.30.2 >