From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAAB3C3ABCB for ; Tue, 13 May 2025 03:14:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E22646B0099; Mon, 12 May 2025 23:14:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DFB696B009A; Mon, 12 May 2025 23:14:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CBFE66B009B; Mon, 12 May 2025 23:14:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AC2536B0099 for ; Mon, 12 May 2025 23:14:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 000A0141334 for ; Tue, 13 May 2025 03:14:48 +0000 (UTC) X-FDA: 83436417456.20.C114239 Received: from out-182.mta1.migadu.com (out-182.mta1.migadu.com [95.215.58.182]) by imf16.hostedemail.com (Postfix) with ESMTP id 5EEBC180007 for ; Tue, 13 May 2025 03:14:47 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=pDrJPdFU; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747106087; a=rsa-sha256; cv=none; b=PRWnwePxOKy9LLEs3BakOgPb8yjyMABOvLjA0frEMWz5Wss+QarsFRVTeAZTTGnhZ2xNa8 OjYNoqbj5b0xIF+Yb8fyhk7UzfOwSHV4R5RaYlEDZZkIr50dmUKsm2K3nquPUTPARQRAfr /jo4NreIBdtLh7e8fAU3oBDC0DHaW04= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=pDrJPdFU; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf16.hostedemail.com: domain of shakeel.butt@linux.dev designates 95.215.58.182 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747106087; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fO3n7VreyxXtpgsjbDq9VlopfXabKVT3tylZkZXuQ8Q=; b=dsJ5aP9pKk1IrkK55JawksStUdQdZlI6NtKXrQlmbhAVlMvarokhzP0OomlVd5P3DDRLKu u7drjakzDB3gOKzzIsCQoajNLTQ+VHpOSQHS039ON6QBDw54B6furGWwPkyGIbPzAFgTN3 6K1lTgSRUS1FcoQE4d0y5PHnAIvMhD8= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1747106085; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fO3n7VreyxXtpgsjbDq9VlopfXabKVT3tylZkZXuQ8Q=; b=pDrJPdFUkledZkOpZSrVThmJnwlzIPPtE5A/fYKeqLewpsDPY157dmQoLr7OnQk7QY33XP 0HqvK4rK8E795Gp0RDxNXi9XpYhIZBQPB7f3h3rPmGJJelyscFEQACOESqw0iAv6FE3nV8 ZfRjYtHnABfCNVuo8b6DOnvJoM8P6ug= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , Harry Yoo , Yosry Ahmed , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [RFC PATCH 6/7] memcg: objcg stock trylock without irq disabling Date: Mon, 12 May 2025 20:13:15 -0700 Message-ID: <20250513031316.2147548-7-shakeel.butt@linux.dev> In-Reply-To: <20250513031316.2147548-1-shakeel.butt@linux.dev> References: <20250513031316.2147548-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 5EEBC180007 X-Stat-Signature: o4jmr5wbrfihdwawscsycij3nwf1obmd X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1747106087-304225 X-HE-Meta: U2FsdGVkX18AcWsflMCS04Hh3eeLIvkbF1vLmUMvKNTqNCw/yUaDYKmLJPZ8RLlKdg8sv0YiwVUfatqfxJ+Un23yEyVjTYzFAa8hVQQ0fbROw8gLx798uknYOgiFBohavRwx40sjiJ9SjiGJJORV8hzjiCvXG4VODjFOiJ8gueVD+NEX9Sd99TIN8+5l/iJlzP/XTuYSH4rhGDSxh0IcLFCNePIrKXnoe6dJOxjgfjiP/uUDe5UASXGwEjQKh3R9oyuXiy2Mj5tnqgT9/aNMVecGNAISGMXlOHCnzSoHiLfIWO6UM9aOGaefqBuZ3VH+JSOZa1AQypTVZSVvRMfMimHsjG89xkUG//T/pNOaY6IpU3Gysxu9Lu2fdP+aN5rI6Yuqkw5lHvLSKJ4Mly/Ixx5dIIXa1pZ6bsODX5WFbxd8JqQ2AkpUTAMXECKb7qrFurpNSiTVs/rz/XaBhUo322AVkdZmdu52avsGL0IbgAN/f1BLrwrTa18XKUDt6awv4rZ0Q5Gy3mfOAVt+4z0qQJi3n2AXxpzDvQk3i7SwWCUVKPVuUl3jbzOfDXCqZEXIFYvCZUPFWJLaDgPIRBuG3o94afjTXIUl8BlHBCoAY+CYQNhki7DkidXERiq0ZJdWVn474M7REv2hpbzC2GDCYAB+z8jSW86S1AcGer1+7GQIaMbyVw757bNoHywtNljgrUGpG5ItZgE/u+HeANHoIUW2zzQqwS7ab07zx6chLZhvZCJNF2d5HqspiBL6itB8DtK2NKb52esZWJWWSRq/1eeno5wjp+Oih0UvMuUnZCT+1dGmIglhkbAhJyvEXBu7aAgN05cwvJzGIamRLot7M06O7PbXZBQmZtEyVsWUb7zLccpGULBzc3+sXoxOgHp+pwU7gC/z77tP85Yy+dIUlmpdKhDVCaKO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no need to disable irqs to use objcg per-cpu stock, so let's just not do that but consume_obj_stock() and refill_obj_stock() will need to use trylock instead to keep per-cpu stock safe. One consequence of this change is that the charge request from irq context may take slowpath more often but it should be rare. Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index adf2f1922118..af7df675d733 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1918,18 +1918,17 @@ static void drain_local_memcg_stock(struct work_struct *dummy) static void drain_local_obj_stock(struct work_struct *dummy) { struct obj_stock_pcp *stock; - unsigned long flags; if (WARN_ONCE(!in_task(), "drain in non-task context")) return; - local_lock_irqsave(&obj_stock.lock, flags); + local_lock(&obj_stock.lock); stock = this_cpu_ptr(&obj_stock); drain_obj_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&obj_stock.lock, flags); + local_unlock(&obj_stock.lock); } static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) @@ -2062,14 +2061,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg) static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct obj_stock_pcp *obj_st; - unsigned long flags; obj_st = &per_cpu(obj_stock, cpu); /* drain_obj_stock requires objstock.lock */ - local_lock_irqsave(&obj_stock.lock, flags); + local_lock(&obj_stock.lock); drain_obj_stock(obj_st); - local_unlock_irqrestore(&obj_stock.lock, flags); + local_unlock(&obj_stock.lock); /* no need for the local lock */ drain_stock_fully(&per_cpu(memcg_stock, cpu)); @@ -2943,14 +2941,12 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, struct pglist_data *pgdat, enum node_stat_item idx) { struct obj_stock_pcp *stock; - unsigned long flags; bool ret = false; - if (unlikely(in_nmi())) + if (unlikely(in_nmi()) || + !local_trylock(&obj_stock.lock)) return ret; - local_lock_irqsave(&obj_stock.lock, flags); - stock = this_cpu_ptr(&obj_stock); if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; @@ -2960,7 +2956,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, __account_obj_stock(objcg, stock, nr_bytes, pgdat, idx); } - local_unlock_irqrestore(&obj_stock.lock, flags); + local_unlock(&obj_stock.lock); return ret; } @@ -3049,10 +3045,10 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, enum node_stat_item idx) { struct obj_stock_pcp *stock; - unsigned long flags; unsigned int nr_pages = 0; - if (unlikely(in_nmi())) { + if (unlikely(in_nmi()) || + !local_trylock(&obj_stock.lock)) { if (pgdat) mod_objcg_mlstate(objcg, pgdat, idx, nr_bytes); nr_pages = nr_bytes >> PAGE_SHIFT; @@ -3061,8 +3057,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, goto out; } - local_lock_irqsave(&obj_stock.lock, flags); - stock = this_cpu_ptr(&obj_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ drain_obj_stock(stock); @@ -3083,7 +3077,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&obj_stock.lock, flags); + local_unlock(&obj_stock.lock); out: if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); -- 2.47.1