From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A961C4338F for ; Sat, 14 Aug 2021 05:25:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1109460F51 for ; Sat, 14 Aug 2021 05:25:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 1109460F51 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id ADB948D0005; Sat, 14 Aug 2021 01:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8C598D0002; Sat, 14 Aug 2021 01:25:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 953BA8D0005; Sat, 14 Aug 2021 01:25:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id 7B6358D0002 for ; Sat, 14 Aug 2021 01:25:52 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 23D611BCBF for ; Sat, 14 Aug 2021 05:25:52 +0000 (UTC) X-FDA: 78472549344.10.A7A7664 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf09.hostedemail.com (Postfix) with ESMTP id C34463001DAF for ; Sat, 14 Aug 2021 05:25:51 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id m24-20020a17090a7f98b0290178b1a81700so19050653pjl.4 for ; Fri, 13 Aug 2021 22:25:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=srhzdC6dpfDSqfVf+RD/0prfJUSgfxeDUXo+j1ykPgc=; b=Ebm1NQwKcXSarqSFsJgcWgom5FuU3zoKjjle2EaDp0vepGN+D/VQaW+WlMGe/fb+YX W1dNsip3hRd0esRbu3h7yKdAKg9oMyfbkGSx1bc6A17BLSlfMzVDB+4Gdy4GeLC+g+55 NQZeSiXlj+XoKf/vy3gz9YDJXiwK/0t7gToJFBS/E9+aKkSruTNkM3QlmzUKDr0ccgOQ CJcouMtFk64wxIeykIxrsdyVl0oor/KH6kc6Le1QQPIOTmERu2+oN4M6rA05eMcypoWP z5MCehzKD3ylHYwl130/jajV+RTJ5HG6gX8QkHoXOL3fcURdA7rhwsr6aZwvTppr4yka 8TFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=srhzdC6dpfDSqfVf+RD/0prfJUSgfxeDUXo+j1ykPgc=; b=UE6GrgUx2hMBh7Fd59clmmsouAwVweWiycmCZtJ5rH+2yW8GPU/OGvaEA0HNojusAD FbTUEWqwJodnNiY/PMq7conEWllxlX+QRdb5LlaVHE5kPhSxuN4IvftGrL6Co3pGe1tn 4p+s5Zwex91WmSRikONdjmjkGvyZGmiD78TcDclTt0Dzepcwj6smu0czixej6OKuqSjq knASDHk+iyKeYuz0n5/f/lJ6a5JX/UCzKq8FvVZyWyAn7Vue72PpP5Sb6l7Z9dNFwGc3 oMh919QEgslQVJugmGJu03N/AtPat1wuw9kK3u5CLx5SOdlBVL0fTTIGQlVW7TddJOrO dktQ== X-Gm-Message-State: AOAM531e8FOTpFnHpCnGOgjcYkCIUEtJgPys6ihpU5osF7otxhdN9LOf vt1yN7bnHCCBlL7sTdKQbrtoPA== X-Google-Smtp-Source: ABdhPJyuYgFZ3sES+0HS/OxwmUQF9hXLqPNkVQOQj2IFtDyG3xKC//DtY1+36hzn68Qec11oagpZZg== X-Received: by 2002:a63:5fd4:: with SMTP id t203mr5486107pgb.141.1628918750893; Fri, 13 Aug 2021 22:25:50 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id s5sm4783133pgp.81.2021.08.13.22.25.45 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Aug 2021 22:25:50 -0700 (PDT) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alexs@kernel.org, smuchun@gmail.com, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH v1 03/12] mm: memcontrol: make lruvec lock safe when LRU pages are reparented Date: Sat, 14 Aug 2021 13:25:10 +0800 Message-Id: <20210814052519.86679-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210814052519.86679-1-songmuchun@bytedance.com> References: <20210814052519.86679-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=Ebm1NQwK; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf09.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: hbd4o17d1mt8c6m8qkj4rhox3wjm3ajq X-Rspamd-Queue-Id: C34463001DAF X-Rspamd-Server: rspam05 X-HE-Tag: 1628918751-429766 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The diagram below shows how to make the folio lruvec lock safe when LRU pages are reparented. folio_lruvec_lock(folio) retry: lruvec =3D folio_lruvec(folio); // The folio is reparented at this time. spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) // Acquired the wrong lruvec lock and need to retry. // Because this folio is on the parent memcg lruvec list. goto retry; // If we reach here, it means that folio_memcg(folio) is stable. memcg_reparent_objcgs(memcg) // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); // Move all the pages from the lruvec list to the parent lruvec list. spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After we acquire the lruvec lock, we need to check whether the folio is reparented. If so, we need to reacquire the new lruvec lock. On the routine of the LRU pages reparenting, we will also acquire the lruvec lock (will be implemented in the later patch). So folio_memcg() cannot be changed when we hold the lruvec lock. Since lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after we hold the lruvec lock, lruvec_memcg_debug() check is pointless. So remove it. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 18 +++----------- mm/compaction.c | 10 +++++++- mm/memcontrol.c | 62 +++++++++++++++++++++++++++++-----------= ------ mm/swap.c | 4 +++ 4 files changed, 55 insertions(+), 39 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 41a35de93d75..5a8f85bd9bbf 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -769,7 +769,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct= mem_cgroup *memcg, * folio_lruvec - return lruvec for isolating/putting an LRU folio * @folio: Pointer to the folio. * - * This function relies on folio->mem_cgroup being stable. + * The lruvec can be changed to its parent lruvec when the page reparent= ed. + * The caller need to recheck if it cares about this changes (just like + * folio_lruvec_lock() does). */ static inline struct lruvec *folio_lruvec(struct folio *folio) { @@ -788,15 +790,6 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *f= olio); struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags); =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio); -#else -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} -#endif - static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -1243,11 +1236,6 @@ static inline struct lruvec *folio_lruvec(struct f= olio *folio) return &pgdat->__lruvec; } =20 -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} - static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *me= mcg) { return NULL; diff --git a/mm/compaction.c b/mm/compaction.c index 3131558a7c31..97d72f6b3dd5 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -515,6 +515,8 @@ compact_folio_lruvec_lock_irqsave(struct folio *folio= , unsigned long *flags, { struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: lruvec =3D folio_lruvec(folio); =20 /* Track if the lock is contended in async mode */ @@ -527,7 +529,13 @@ compact_folio_lruvec_lock_irqsave(struct folio *foli= o, unsigned long *flags, =20 spin_lock_irqsave(&lruvec->lru_lock, *flags); out: - lruvec_memcg_debug(lruvec, folio); + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7df2176e4f02..7955da38e385 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1146,23 +1146,6 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg= , return ret; } =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ - struct mem_cgroup *memcg; - - if (mem_cgroup_disabled()) - return; - - memcg =3D folio_memcg(folio); - - if (!memcg) - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D root_mem_cgroup, folio); - else - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D memcg, folio); -} -#endif - /** * folio_lruvec_lock - lock and return lruvec for a given folio. * @folio: Pointer to the folio. @@ -1175,20 +1158,43 @@ void lruvec_memcg_debug(struct lruvec *lruvec, st= ruct folio *folio) */ struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock(&lruvec->lru_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); =20 return lruvec; } =20 struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } @@ -1196,10 +1202,20 @@ struct lruvec *folio_lruvec_lock_irq(struct folio= *folio) struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } diff --git a/mm/swap.c b/mm/swap.c index 2bca632b73df..9554ff008fe6 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -295,6 +295,10 @@ void lru_note_cost(struct lruvec *lruvec, bool file,= unsigned int nr_pages) =20 void lru_note_cost_folio(struct folio *folio) { + /* + * The rcu read lock is held by the caller, so we do not need to + * care about the lruvec returned by folio_lruvec() being released. + */ lru_note_cost(folio_lruvec(folio), folio_is_file_lru(folio), folio_nr_pages(folio)); } --=20 2.11.0