From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1D0BC4361B for ; Tue, 15 Dec 2020 20:34:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9670A222BB for ; Tue, 15 Dec 2020 20:34:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9670A222BB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3B95F8D0013; Tue, 15 Dec 2020 15:34:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 340D68D0012; Tue, 15 Dec 2020 15:34:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E3DB8D0013; Tue, 15 Dec 2020 15:34:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id 010E48D0012 for ; Tue, 15 Dec 2020 15:34:36 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B999C629 for ; Tue, 15 Dec 2020 20:34:36 +0000 (UTC) X-FDA: 77596669752.25.magic09_3f0812427426 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 903571804E3B7 for ; Tue, 15 Dec 2020 20:34:36 +0000 (UTC) X-HE-Tag: magic09_3f0812427426 X-Filterd-Recvd-Size: 10185 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Dec 2020 20:34:35 +0000 (UTC) Date: Tue, 15 Dec 2020 12:34:33 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1608064475; bh=mpOsvdVNyVZwb8pCK2adVRJD4Osq5c6wcgfx2B+scWc=; h=From:To:Subject:In-Reply-To:From; b=SDGlEwz83pKvHoRNaltzN1vRwQ6XDPp/QOEc6E7+lcEpssaoFxSCBHPUghlYm7wqX MSP4p7NVU9GTuvjy2YGAJi28oZPFGg7BgTlQsnapKJfTwiFFMVF0rGCAEVRg3RIvOK TkNpKvM6BymIMWQsW5RyPRbH42V6VvFrnJU9zVv4= From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, alex.shi@linux.alibaba.com, alexander.h.duyck@linux.intel.com, aryabinin@virtuozzo.com, daniel.m.jordan@oracle.com, hannes@cmpxchg.org, hughd@google.com, iamjoonsoo.kim@lge.com, jannh@google.com, khlebnikov@yandex-team.ru, kirill.shutemov@linux.intel.com, kirill@shutemov.name, linux-mm@kvack.org, mgorman@techsingularity.net, mhocko@kernel.org, mhocko@suse.com, mika.penttila@nextfour.com, minchan@kernel.org, mm-commits@vger.kernel.org, richard.weiyang@gmail.com, rong.a.chen@intel.com, shakeelb@google.com, tglx@linutronix.de, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz, vdavydov.dev@gmail.com, willy@infradead.org, yang.shi@linux.alibaba.com, ying.huang@intel.com Subject: [patch 18/19] mm/lru: introduce relock_page_lruvec() Message-ID: <20201215203433.gd0RZE7Fb%akpm@linux-foundation.org> In-Reply-To: <20201215123253.954eca9a5ef4c0d52fd381fa@linux-foundation.org> User-Agent: s-nail v14.8.16 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: =46rom: Alexander Duyck Subject: mm/lru: introduce relock_page_lruvec() Add relock_page_lruvec() to replace repeated same code, no functional change. When testing for relock we can avoid the need for RCU locking if we simply compare the page pgdat and memcg pointers versus those that the lruvec is holding. By doing this we can avoid the extra pointer walks and accesses of the memory cgroup. In addition we can avoid the checks entirely if lruvec is currently NULL. [alex.shi@linux.alibaba.com: use page_memcg()] Link: https://lkml.kernel.org/r/66d8e79d-7ec6-bfbc-1c82-bf32db3ae5b7@linu= x.alibaba.com Link: https://lkml.kernel.org/r/1604566549-62481-19-git-send-email-alex.shi= @linux.alibaba.com Signed-off-by: Alexander Duyck Signed-off-by: Alex Shi Acked-by: Hugh Dickins Acked-by: Johannes Weiner Acked-by: Vlastimil Babka Cc: Thomas Gleixner Cc: Andrey Ryabinin Cc: Matthew Wilcox Cc: Mel Gorman Cc: Konstantin Khlebnikov Cc: Tejun Heo Cc: Andrea Arcangeli Cc: "Chen, Rong A" Cc: Daniel Jordan Cc: "Huang, Ying" Cc: Jann Horn Cc: Joonsoo Kim Cc: Kirill A. Shutemov Cc: Kirill A. Shutemov Cc: Michal Hocko Cc: Michal Hocko Cc: Mika Penttil=C3=A4 Cc: Minchan Kim Cc: Shakeel Butt Cc: Vladimir Davydov Cc: Wei Yang Cc: Yang Shi Signed-off-by: Andrew Morton --- include/linux/memcontrol.h | 52 +++++++++++++++++++++++++++++++++++ mm/mlock.c | 11 ------- mm/swap.c | 31 ++++---------------- mm/vmscan.c | 12 +------- 4 files changed, 61 insertions(+), 45 deletions(-) --- a/include/linux/memcontrol.h~mm-lru-introduce-the-relock_page_lruvec-fu= nction +++ a/include/linux/memcontrol.h @@ -485,6 +485,22 @@ out: =20 struct lruvec *mem_cgroup_page_lruvec(struct page *, struct pglist_data *); =20 +static inline bool lruvec_holds_page_lru_lock(struct page *page, + struct lruvec *lruvec) +{ + pg_data_t *pgdat =3D page_pgdat(page); + const struct mem_cgroup *memcg; + struct mem_cgroup_per_node *mz; + + if (mem_cgroup_disabled()) + return lruvec =3D=3D &pgdat->__lruvec; + + mz =3D container_of(lruvec, struct mem_cgroup_per_node, lruvec); + memcg =3D page_memcg(page) ? : root_mem_cgroup; + + return lruvec->pgdat =3D=3D pgdat && mz->memcg =3D=3D memcg; +} + struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); =20 struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm); @@ -984,6 +1000,14 @@ static inline struct lruvec *mem_cgroup_ return &pgdat->__lruvec; } =20 +static inline bool lruvec_holds_page_lru_lock(struct page *page, + struct lruvec *lruvec) +{ + pg_data_t *pgdat =3D page_pgdat(page); + + return lruvec =3D=3D &pgdat->__lruvec; +} + static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memc= g) { return NULL; @@ -1354,6 +1378,34 @@ static inline void unlock_page_lruvec_ir spin_unlock_irqrestore(&lruvec->lru_lock, flags); } =20 +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irq(struct page *page, + struct lruvec *locked_lruvec) +{ + if (locked_lruvec) { + if (lruvec_holds_page_lru_lock(page, locked_lruvec)) + return locked_lruvec; + + unlock_page_lruvec_irq(locked_lruvec); + } + + return lock_page_lruvec_irq(page); +} + +/* Don't lock again iff page's lruvec locked */ +static inline struct lruvec *relock_page_lruvec_irqsave(struct page *page, + struct lruvec *locked_lruvec, unsigned long *flags) +{ + if (locked_lruvec) { + if (lruvec_holds_page_lru_lock(page, locked_lruvec)) + return locked_lruvec; + + unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + } + + return lock_page_lruvec_irqsave(page, flags); +} + #ifdef CONFIG_CGROUP_WRITEBACK =20 struct wb_domain *mem_cgroup_wb_domain(struct bdi_writeback *wb); --- a/mm/mlock.c~mm-lru-introduce-the-relock_page_lruvec-function +++ a/mm/mlock.c @@ -277,16 +277,7 @@ static void __munlock_pagevec(struct pag * so we can spare the get_page() here. */ if (TestClearPageLRU(page)) { - struct lruvec *new_lruvec; - - new_lruvec =3D mem_cgroup_page_lruvec(page, - page_pgdat(page)); - if (new_lruvec !=3D lruvec) { - if (lruvec) - unlock_page_lruvec_irq(lruvec); - lruvec =3D lock_page_lruvec_irq(page); - } - + lruvec =3D relock_page_lruvec_irq(page, lruvec); del_page_from_lru_list(page, lruvec, page_lru(page)); continue; --- a/mm/swap.c~mm-lru-introduce-the-relock_page_lruvec-function +++ a/mm/swap.c @@ -210,19 +210,12 @@ static void pagevec_lru_move_fn(struct p =20 for (i =3D 0; i < pagevec_count(pvec); i++) { struct page *page =3D pvec->pages[i]; - struct lruvec *new_lruvec; =20 /* block memcg migration during page moving between lru */ if (!TestClearPageLRU(page)) continue; =20 - new_lruvec =3D mem_cgroup_page_lruvec(page, page_pgdat(page)); - if (lruvec !=3D new_lruvec) { - if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec =3D lock_page_lruvec_irqsave(page, &flags); - } - + lruvec =3D relock_page_lruvec_irqsave(page, lruvec, &flags); (*move_fn)(page, lruvec); =20 SetPageLRU(page); @@ -918,17 +911,12 @@ void release_pages(struct page **pages, } =20 if (PageLRU(page)) { - struct lruvec *new_lruvec; + struct lruvec *prev_lruvec =3D lruvec; =20 - new_lruvec =3D mem_cgroup_page_lruvec(page, - page_pgdat(page)); - if (new_lruvec !=3D lruvec) { - if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, - flags); + lruvec =3D relock_page_lruvec_irqsave(page, lruvec, + &flags); + if (prev_lruvec !=3D lruvec) lock_batch =3D 0; - lruvec =3D lock_page_lruvec_irqsave(page, &flags); - } =20 VM_BUG_ON_PAGE(!PageLRU(page), page); __ClearPageLRU(page); @@ -1033,15 +1021,8 @@ void __pagevec_lru_add(struct pagevec *p =20 for (i =3D 0; i < pagevec_count(pvec); i++) { struct page *page =3D pvec->pages[i]; - struct lruvec *new_lruvec; - - new_lruvec =3D mem_cgroup_page_lruvec(page, page_pgdat(page)); - if (lruvec !=3D new_lruvec) { - if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); - lruvec =3D lock_page_lruvec_irqsave(page, &flags); - } =20 + lruvec =3D relock_page_lruvec_irqsave(page, lruvec, &flags); __pagevec_lru_add_fn(page, lruvec); } if (lruvec) --- a/mm/vmscan.c~mm-lru-introduce-the-relock_page_lruvec-function +++ a/mm/vmscan.c @@ -1883,8 +1883,7 @@ static unsigned noinline_for_stack move_ * All pages were isolated from the same lruvec (and isolation * inhibits memcg migration). */ - VM_BUG_ON_PAGE(mem_cgroup_page_lruvec(page, page_pgdat(page)) - !=3D lruvec, page); + VM_BUG_ON_PAGE(!lruvec_holds_page_lru_lock(page, lruvec), page); lru =3D page_lru(page); nr_pages =3D thp_nr_pages(page); =20 @@ -4273,7 +4272,6 @@ void check_move_unevictable_pages(struct for (i =3D 0; i < pvec->nr; i++) { struct page *page =3D pvec->pages[i]; int nr_pages; - struct lruvec *new_lruvec; =20 if (PageTransTail(page)) continue; @@ -4285,13 +4283,7 @@ void check_move_unevictable_pages(struct if (!TestClearPageLRU(page)) continue; =20 - new_lruvec =3D mem_cgroup_page_lruvec(page, page_pgdat(page)); - if (lruvec !=3D new_lruvec) { - if (lruvec) - unlock_page_lruvec_irq(lruvec); - lruvec =3D lock_page_lruvec_irq(page); - } - + lruvec =3D relock_page_lruvec_irq(page, lruvec); if (page_evictable(page) && PageUnevictable(page)) { enum lru_list lru =3D page_lru_base_type(page); =20 _