From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4AF3C433F5 for ; Wed, 16 Feb 2022 11:52:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 586296B0082; Wed, 16 Feb 2022 06:52:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 50F3C6B0083; Wed, 16 Feb 2022 06:52:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AFF76B0085; Wed, 16 Feb 2022 06:52:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 2C66A6B0082 for ; Wed, 16 Feb 2022 06:52:51 -0500 (EST) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E742E8249980 for ; Wed, 16 Feb 2022 11:52:50 +0000 (UTC) X-FDA: 79148481300.08.5CD8AF5 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf25.hostedemail.com (Postfix) with ESMTP id 786E7A0002 for ; Wed, 16 Feb 2022 11:52:50 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id i6so1929746pfc.9 for ; Wed, 16 Feb 2022 03:52:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=35RlTAsUx3yejoEebytGRZ97Bh68Tl3UTQMeyrDekNI=; b=moRystZEHt66uvzmqSShBkGW+R2V6RH38qwB8xS4uNWWO9+kZx/5LLX6Q70OvVONti vslfKpv5q4ggM2+6oYPcXECgDEeMDjLXaqiXEnvyMCkcwWQs9TrQFf0yqliG+xu3h/G7 i/hpaJyUaAeN0zFzNJDRgtYG4oGSUW5MdPaJbpXYanP90+ohvdiR23D7bLCjoo/gdSAV BYB8GOnTlqICbtj6kyQnFfTDwm/9MnySMsLHwOa69XQLlBg6Eo6JwezYXMMjcNiajB5i jOTxdvt5LXbumEnI8vA5mahfALOtx9DnYod3Je86BPoFIdqglsjq87OTSdlzYmv1P4Ea F8Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=35RlTAsUx3yejoEebytGRZ97Bh68Tl3UTQMeyrDekNI=; b=yuyIuEuI6EOpvq106ZzpXYP2VylaB8+EDK8u2gtin/69N7liqBEb4MgkTdeXAutR8j zJwU6QQCap03sWkV/K6fJMMt3y4HfoiPN+OCxK5vsJNp3poLTCX21KrpK3lwBS2pA7zl jUCo4vD/Q0zSfRSHBfk6rdFg/7/IfB1NRpS/rvDXafXoW6yxXT1Ql06ynyla1GbbdhSi TFJs8vLlWX21ceP3iQ95+tgTma2yAH8dWssIoqVkCRHev8QDCZauQy4hEN1gze/TGY3U 9ja+2+BOATbRi4MNFC9HsRKGeQdZ2TUCa4VTTEAyQdkUMAfpzxUMS0bFJGYGpxlffJEK vw2A== X-Gm-Message-State: AOAM532fTW8+o14oGpYjNZ2a3f6I5ZbU4Pv6h4ixAfey6UBRKErtNisO vgGq/GVo5mg6FrBwflpvb8EYGQ== X-Google-Smtp-Source: ABdhPJwvmMrCkqNlkgisvdhcfgXRCaZkt9on70NqNBs9//l0VrAzUX8Dbh/M/8lvPW2VFtZFdTuf6g== X-Received: by 2002:a63:5f97:0:b0:372:f7dc:6cf6 with SMTP id t145-20020a635f97000000b00372f7dc6cf6mr1949242pgb.315.1645012369596; Wed, 16 Feb 2022 03:52:49 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.249]) by smtp.gmail.com with ESMTPSA id m16sm14790221pfc.156.2022.02.16.03.52.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:52:49 -0800 (PST) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alexs@kernel.org, smuchun@gmail.com, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH v3 08/12] mm: memcontrol: introduce memcg_reparent_ops Date: Wed, 16 Feb 2022 19:51:28 +0800 Message-Id: <20220216115132.52602-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220216115132.52602-1-songmuchun@bytedance.com> References: <20220216115132.52602-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 786E7A0002 X-Stat-Signature: xtgffqzy4bbmbs99kmm57ao4h36fohxo X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=moRystZE; spf=pass (imf25.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1645012370-844302 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent me= mcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) do reparent // Move all the pages from the lruvec list to the parent lruvec l= ist. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three ste= ps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) reparent memcg_reparent_ops->reparent(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 7 +++++++ mm/memcontrol.c | 39 +++++++++++++++++++++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 6e0f7104f2fa..3c841c155f0d 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -346,6 +346,13 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; =20 +struct memcg_reparent_ops { + /* Irq is disabled before calling those callbacks. */ + void (*lock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*unlock)(struct mem_cgroup *memcg, struct mem_cgroup *parent); + void (*reparent)(struct mem_cgroup *memcg, struct mem_cgroup *parent); +}; + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dd2602149ef3..6a393fe8e589 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -336,6 +336,35 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 +static const struct memcg_reparent_ops *memcg_reparent_ops[] =3D {}; + +static void memcg_reparent_lock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->lock(memcg, parent); +} + +static void memcg_reparent_unlock(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->unlock(memcg, parent); +} + +static void memcg_do_reparent(struct mem_cgroup *memcg, + struct mem_cgroup *parent) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) + memcg_reparent_ops[i]->reparent(memcg, parent); +} + static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; @@ -345,9 +374,11 @@ static void memcg_reparent_objcgs(struct mem_cgroup = *memcg) if (!parent) parent =3D root_mem_cgroup; =20 + local_irq_disable(); + memcg_reparent_lock(memcg, parent); objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); =20 - spin_lock_irq(&objcg_lock); + spin_lock(&objcg_lock); =20 /* 1) Ready to reparent active objcg. */ list_add(&objcg->list, &memcg->objcg_list); @@ -357,7 +388,11 @@ static void memcg_reparent_objcgs(struct mem_cgroup = *memcg) /* 3) Move already reparented objcgs to the parent's list */ list_splice(&memcg->objcg_list, &parent->objcg_list); =20 - spin_unlock_irq(&objcg_lock); + spin_unlock(&objcg_lock); + + memcg_do_reparent(memcg, parent); + memcg_reparent_unlock(memcg, parent); + local_irq_enable(); =20 percpu_ref_kill(&objcg->refcnt); } --=20 2.11.0