From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17904EB64DD for ; Tue, 8 Aug 2023 02:44:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FD026B0071; Mon, 7 Aug 2023 22:44:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5AD278D0002; Mon, 7 Aug 2023 22:44:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49BF28D0001; Mon, 7 Aug 2023 22:44:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 392526B0071 for ; Mon, 7 Aug 2023 22:44:18 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id ED80280769 for ; Tue, 8 Aug 2023 02:44:17 +0000 (UTC) X-FDA: 81099393354.05.1EB7F01 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf06.hostedemail.com (Postfix) with ESMTP id EBE12180005 for ; Tue, 8 Aug 2023 02:44:15 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=fromorbit-com.20221208.gappssmtp.com header.s=20221208 header.b=c9iOPpy4; spf=pass (imf06.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=david@fromorbit.com; dmarc=pass (policy=quarantine) header.from=fromorbit.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691462656; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZZ3n364wJBBn5i6olC7X8xOphz5aoEwi3PPn9Humxy4=; b=cM0cLcP7uc6GlIs8rQSMSPd8PrxHzHS5NbT1MxUnRG8IHUNjH1QbfLklp12O6sGciBgl77 XLk5Hb8qvjPwOd3ZvKSkyURMzFJJqr9MMc3oI9tRR94aifVmQLBUzbBWF8BoR/SWqOW2rt UmESTBDRjbq/hibLqDGhupFI8ph22Ks= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691462656; a=rsa-sha256; cv=none; b=GPaE7itmax5b6o1qTQGIQYXJQS9IkPySTPrXRdp6wbOIsR5T5/84CteAmWf9ZA90UUZ1n+ naiYW4ot7GuDvlb1QeLUtqc+G2YLEY0ZdjEjSkXOPGtwpPnNreGus3G61XRD88yCxN0voG Wso5wNRoSOItlamKwnd670WKEf+pP24= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=fromorbit-com.20221208.gappssmtp.com header.s=20221208 header.b=c9iOPpy4; spf=pass (imf06.hostedemail.com: domain of david@fromorbit.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=david@fromorbit.com; dmarc=pass (policy=quarantine) header.from=fromorbit.com Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1bc8045e09dso1996795ad.0 for ; Mon, 07 Aug 2023 19:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fromorbit-com.20221208.gappssmtp.com; s=20221208; t=1691462655; x=1692067455; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZZ3n364wJBBn5i6olC7X8xOphz5aoEwi3PPn9Humxy4=; b=c9iOPpy41VC6QSzv5vLbmCPsxZB2K7hAItf8MEtryT/BAlfvdA0UPM4PAg3DtN7rr+ GFW2nm1AoLFEe5zmSSiqyIhn92P6iecybeuExkqa7jbnj+LEVM1LioV7SZGiR/8Kzy2+ mJYrJW8XMi2+dbRvP7Xg6QuSzUnp/yHkq29M7vbAYNrF1NokfShgkpn7COiSBb38dkF0 MWE0ZNG9ZS4LzaWgVeAy0Pmh7etHcKYaOYLW+oCVP3Q2zlcEbOnfLG0lI+GjqQwBJGqv OKDqkP3u3mDYRaZI2hDArPl64kIP09oRYh791qhymi4aMBqJthtqlGcAIAw7e+Im4Q2d AiTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691462655; x=1692067455; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZZ3n364wJBBn5i6olC7X8xOphz5aoEwi3PPn9Humxy4=; b=i2cuMgQ9lZoKaljj8dsYzOT0VbWAz05t16/7tYxm4ZnoIC6h9s9qeORzVv24xNwL5u ZBCNHAMkgoi63ZJRkm9gUYZ0KR5yR1yWYaALv8WZrZFKp7gh+Hjh8GcJ5QlK8VAddO4B Rk+DMVN20h0bglBiQrigE9z+gRenkm7g9L2MuErc0Hu+LLSxIJbFBdNUrfRaPA7HGvci gYBLJzbGvjUyukwC5BF/w0TydL1gGZ9xsXaUDQM5u94TmR5escVnEqk6xMw5TYfC4ebq a1DsU1joyr4RJXA6uKrjywHdowEFH/dJLqX3psH7NcgwwRMSvIlyofWA4Rk1zXxQkYR0 qLYQ== X-Gm-Message-State: AOJu0YxV2dbsCK4y8TDgfAXa2LwgoxRWEqhUimvPsa3+/D20vZBfMNZ2 FhiE5IbLzddLODDjITSuKvA9XQ== X-Google-Smtp-Source: AGHT+IEZT2gUmOSYrkIbMjP5ySF6z1UEt5Q+o+GMN8zU98oO9pgzqqi+i8QK+swPthTlPTk4WQ36Qw== X-Received: by 2002:a17:902:7287:b0:1b3:f5c7:4e75 with SMTP id d7-20020a170902728700b001b3f5c74e75mr9699662pll.58.1691462654669; Mon, 07 Aug 2023 19:44:14 -0700 (PDT) Received: from dread.disaster.area (pa49-180-166-213.pa.nsw.optusnet.com.au. [49.180.166.213]) by smtp.gmail.com with ESMTPSA id u2-20020a170902e80200b001b893b689a0sm7632067plg.84.2023.08.07.19.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Aug 2023 19:44:14 -0700 (PDT) Received: from dave by dread.disaster.area with local (Exim 4.96) (envelope-from ) id 1qTChf-002XBg-27; Tue, 08 Aug 2023 12:44:11 +1000 Date: Tue, 8 Aug 2023 12:44:11 +1000 From: Dave Chinner To: Qi Zheng Cc: akpm@linux-foundation.org, tkhai@ya.ru, vbabka@suse.cz, roman.gushchin@linux.dev, djwong@kernel.org, brauner@kernel.org, paulmck@kernel.org, tytso@mit.edu, steven.price@arm.com, cel@kernel.org, senozhatsky@chromium.org, yujie.liu@intel.com, gregkh@linuxfoundation.org, muchun.song@linux.dev, simon.horman@corigine.com, dlemoal@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org, kvm@vger.kernel.org, xen-devel@lists.xenproject.org, linux-erofs@lists.ozlabs.org, linux-f2fs-devel@lists.sourceforge.net, cluster-devel@redhat.com, linux-nfs@vger.kernel.org, linux-mtd@lists.infradead.org, rcu@vger.kernel.org, netdev@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-arm-msm@vger.kernel.org, dm-devel@redhat.com, linux-raid@vger.kernel.org, linux-bcache@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org Subject: Re: [PATCH v4 46/48] mm: shrinker: make memcg slab shrink lockless Message-ID: References: <20230807110936.21819-1-zhengqi.arch@bytedance.com> <20230807110936.21819-47-zhengqi.arch@bytedance.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230807110936.21819-47-zhengqi.arch@bytedance.com> X-Stat-Signature: db1h7dkex8gw5qmtkgmhpp8d9asyn9xg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EBE12180005 X-Rspam-User: X-HE-Tag: 1691462655-249308 X-HE-Meta: U2FsdGVkX1+OVqTw+Jg0W8cv4gDQgreRsfS+S0S8V8Wnlx4fp9hpuI5P24RacBQv9I56rFJQMeemxhUqOu4eOmEmLX6lPuslpvuB3nilFaFaB2wGzqVfztAGQGHc4H8CB6a1GVLuD2zh+5mDI+tQZ4e7GRABkKN1l/VCltcaj2nQADbtNvpEiEpEmTeLQrJ+oJHSvJKKZXLDc6kxSOeK6Fd8fcnTce3K20efH0IkcZ5IR+XoZc4PKq3MrCm0nOM0R5aoQFEeff06VQfP5Ng7uBVP4xM5yFPVYKC2ozHN8z5loUDNa8BPX61iBt5qERKaBXlKSoY1+Q6W+OhBLyywsWtR3BgforCsIjy6Rh6uGTg2DhxqBMjF0+Yu49xq2e0zHbVY2xSUWfh56o4MRd95QkHAV2fokTl2NRVcMEBJXLe0gmERAIUVIOtvu1gPzYaa2v3EaKdPLzI6Uirgv+zMHhRcUTQw7f1HVu0MU5lNZDjTATfQfbOgJ4d+JTQJPwJ5z0bsKokswxCOL/E/V5S45vxiLmkJkFsQob49TBI5aQzvLJ5b62JZ7fvpW89+ePxFbbBNZ4U4jjDsTs7iYwkDKrllMVeazbm50WI13xpcSGfv0+bWFXe2JCWDHryydPEnJfRUxrcKRH5Zj3uph6oZLmzlfmG2cs8uOIOHBm0jNQLcQFiiP1OAKNr+rPWRMuoMGJBwm/v22ne0meVV9OnUkPFkBZ6pYZz1wsqEkrp54nGtzIdIDxHwHs3SK6K+RqfewmnWjd3T5bH7A8c+ixmow7rMInLeNo/OU/oMu8HcBFxkhu0fPmr6lXgwOogc63xc9d+a24x0bOyyYXE1FI3PPXic7cPOSrUFYQNyhGvAnHGhWkB3uA10M1QcJk75ZsOfsy7Y1rUJHkCbEddyRJ3v2zAj9hE3sk+kBSKLu+rAy0WGqoza4HFE3v7L7WfYtP1DMCBQHHTJoUhOwrjqDXC Oa7kJcjP v+VsvYMRthzLb1rkELnsa9/e5TW7EfK7EzwpG6BgpzMSWtMFQfx6b8V/prL9+qd3TERVi6etWIofrJ5q0nniNGufzQQHsnHAh+2HVApJ2zOzW9gnsmkHXdotjh8uz0xvVXywiDp9txMySVf9C8arOWtxuvsEig2Ixjgiit0sdUVo9ovJKE08otWP/ewuNmLW6KtXUPQ03jrH2yDtIzlDYIMlaeGIV5obtAOX1sDoWLfCyASpd4RFqMFjV5PjVr1CqXzBkJpoI1GqqizL+0vdao3vbKVVBh8RXYcj33dfMtVhQbqRuRxpM1HKWPMskA6USsc8uilyxWkrYUIwcugHt+hFk91zMT1Te51sZMQgOviG81yOyY9psdtQvbHfXY8I78yh8YD23xaFCSSHOoDEfoKAT9lOGPVObqDQFgfqllWY5peuh10DldgsRRQ9kxfqeJhm6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Aug 07, 2023 at 07:09:34PM +0800, Qi Zheng wrote: > Like global slab shrink, this commit also uses refcount+RCU method to make > memcg slab shrink lockless. This patch does random code cleanups amongst the actual RCU changes. Can you please move the cleanups to a spearate patch to reduce the noise in this one? > diff --git a/mm/shrinker.c b/mm/shrinker.c > index d318f5621862..fee6f62904fb 100644 > --- a/mm/shrinker.c > +++ b/mm/shrinker.c > @@ -107,6 +107,12 @@ static struct shrinker_info *shrinker_info_protected(struct mem_cgroup *memcg, > lockdep_is_held(&shrinker_rwsem)); > } > > +static struct shrinker_info *shrinker_info_rcu(struct mem_cgroup *memcg, > + int nid) > +{ > + return rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); > +} This helper doesn't add value. It doesn't tell me that rcu_read_lock() needs to be held when it is called, for one.... > static int expand_one_shrinker_info(struct mem_cgroup *memcg, int new_size, > int old_size, int new_nr_max) > { > @@ -198,7 +204,7 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) > struct shrinker_info_unit *unit; > > rcu_read_lock(); > - info = rcu_dereference(memcg->nodeinfo[nid]->shrinker_info); > + info = shrinker_info_rcu(memcg, nid); ... whilst the original code here was obviously correct. > unit = info->unit[shriner_id_to_index(shrinker_id)]; > if (!WARN_ON_ONCE(shrinker_id >= info->map_nr_max)) { > /* Pairs with smp mb in shrink_slab() */ > @@ -211,7 +217,7 @@ void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id) > > static DEFINE_IDR(shrinker_idr); > > -static int prealloc_memcg_shrinker(struct shrinker *shrinker) > +static int shrinker_memcg_alloc(struct shrinker *shrinker) Cleanups in a separate patch. > @@ -253,10 +258,15 @@ static long xchg_nr_deferred_memcg(int nid, struct shrinker *shrinker, > { > struct shrinker_info *info; > struct shrinker_info_unit *unit; > + long nr_deferred; > > - info = shrinker_info_protected(memcg, nid); > + rcu_read_lock(); > + info = shrinker_info_rcu(memcg, nid); > unit = info->unit[shriner_id_to_index(shrinker->id)]; > - return atomic_long_xchg(&unit->nr_deferred[shriner_id_to_offset(shrinker->id)], 0); > + nr_deferred = atomic_long_xchg(&unit->nr_deferred[shriner_id_to_offset(shrinker->id)], 0); > + rcu_read_unlock(); > + > + return nr_deferred; > } This adds two rcu_read_lock() sections to every call to do_shrink_slab(). It's not at all clear ifrom any of the other code that do_shrink_slab() now has internal rcu_read_lock() sections.... > @@ -464,18 +480,23 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, > if (!mem_cgroup_online(memcg)) > return 0; > > - if (!down_read_trylock(&shrinker_rwsem)) > - return 0; > - > - info = shrinker_info_protected(memcg, nid); > +again: > + rcu_read_lock(); > + info = shrinker_info_rcu(memcg, nid); > if (unlikely(!info)) > goto unlock; > > - for (; index < shriner_id_to_index(info->map_nr_max); index++) { > + if (index < shriner_id_to_index(info->map_nr_max)) { > struct shrinker_info_unit *unit; > > unit = info->unit[index]; > > + /* > + * The shrinker_info_unit will not be freed, so we can > + * safely release the RCU lock here. > + */ > + rcu_read_unlock(); Why - what guarantees that the shrinker_info_unit exists at this point? We hold no reference to it, we hold no reference to any shrinker, etc. What provides this existence guarantee? > + > for_each_set_bit(offset, unit->map, SHRINKER_UNIT_BITS) { > struct shrink_control sc = { > .gfp_mask = gfp_mask, > @@ -485,12 +506,14 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, > struct shrinker *shrinker; > int shrinker_id = calc_shrinker_id(index, offset); > > + rcu_read_lock(); > shrinker = idr_find(&shrinker_idr, shrinker_id); > - if (unlikely(!shrinker || !(shrinker->flags & SHRINKER_REGISTERED))) { > - if (!shrinker) > - clear_bit(offset, unit->map); > + if (unlikely(!shrinker || !shrinker_try_get(shrinker))) { > + clear_bit(offset, unit->map); > + rcu_read_unlock(); > continue; > } > + rcu_read_unlock(); > > /* Call non-slab shrinkers even though kmem is disabled */ > if (!memcg_kmem_online() && > @@ -523,15 +546,20 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, > set_shrinker_bit(memcg, nid, shrinker_id); > } > freed += ret; > - > - if (rwsem_is_contended(&shrinker_rwsem)) { > - freed = freed ? : 1; > - goto unlock; > - } > + shrinker_put(shrinker); Ok, so why is this safe to call without holding the rcu read lock? The global shrinker has to hold the rcu_read_lock() whilst calling shrinker_put() to guarantee the validity of the list next pointer, but we don't hold off RCU here so what guarantees a racing global shrinker walk doesn't trip over this shrinker_put() call dropping the refcount to zero and freeing occuring in a different context... > + /* > + * We have already exited the read-side of rcu critical section > + * before calling do_shrink_slab(), the shrinker_info may be > + * released in expand_one_shrinker_info(), so reacquire the > + * shrinker_info. > + */ > + index++; > + goto again; With that, what makes the use of shrinker_info in xchg_nr_deferred_memcg() in do_shrink_slab() coherent and valid? -Dave. -- Dave Chinner david@fromorbit.com