From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DD93C3601A for ; Fri, 4 Apr 2025 01:40:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8DC4280004; Thu, 3 Apr 2025 21:40:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A411B280001; Thu, 3 Apr 2025 21:40:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B4C3280004; Thu, 3 Apr 2025 21:40:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 67F1E280001 for ; Thu, 3 Apr 2025 21:40:16 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 327021A1CD9 for ; Fri, 4 Apr 2025 01:40:17 +0000 (UTC) X-FDA: 83294656074.26.28C0980 Received: from out-178.mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) by imf13.hostedemail.com (Postfix) with ESMTP id 831592000E for ; Fri, 4 Apr 2025 01:40:15 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YVIdQTwo; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743730815; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BaL+YTe2+GRQTxvSOsA8F0slQZP0gsLtQVxtzdV+oik=; b=dIxGNwvJCbfBtiBPjrM8o6RD0A8n7ucJiTQOvBGZ84D6aKHlGu86zLXsbDpzpTtps26DOF FFnaqUSWucaKm+SYK66c/8pEmToqjRk4CoZHNzSpuyyMQX9uMj3q0/UIMgCFM1NbsZxjqt EjKcaevWiR9eqbTF4XwqjTKb+Y8lta4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YVIdQTwo; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf13.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743730815; a=rsa-sha256; cv=none; b=NNnf8HgmA5tmhG31Fe0oz+ILQQMQqA2lwJ8oHSxxweI2azWeIeZr/37b+J3tePe9cAyTyt fNZCfO6xdrHehXHmxO+ksMv8QAf4riHxZhC5TYeGP8xLcG1Nxf3hUmpBGskCIuOGb1Yn9w ObwqWFBIesadFoaksapUq4PnlSp5zpI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1743730814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BaL+YTe2+GRQTxvSOsA8F0slQZP0gsLtQVxtzdV+oik=; b=YVIdQTwo2vNEeKkxMFLZpTiRY9Tw49WcPNXtYb9sRqKpBiDZDA6NoG3Z2N2wJFJSP6T989 6fGtuqPuZcGBUzJ/81KT5TR2UWOQVfZDwQjl3oesbA1Zu4hGcO/aXcHyoqQ+2nfVWlgWya tDhVUf4+urqwi6rscCd66ZwobItacxM= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Sebastian Andrzej Siewior , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 9/9] memcg: manually inline replace_stock_objcg Date: Thu, 3 Apr 2025 18:39:13 -0700 Message-ID: <20250404013913.1663035-10-shakeel.butt@linux.dev> In-Reply-To: <20250404013913.1663035-1-shakeel.butt@linux.dev> References: <20250404013913.1663035-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam01 X-Stat-Signature: htq4xr3w3o9x4kk59nac43p3dc9c7z9x X-Rspam-User: X-Rspamd-Queue-Id: 831592000E X-HE-Tag: 1743730815-583767 X-HE-Meta: U2FsdGVkX18rBiyL70RFdxxHPKR67MMSyi/DdHnmGQcTcAIC9cCuCG05wO18YjWER4Vn/gfYtWF90aeidQbGZPu5PVOmsBpYutmyTrvIaqxDyAjrbKLrXg82mb1nzSiJiWD1nI+yDMGXEaGb1JMDZQ9dF9Ul0yEk8+9LL8J9m8iScotQIQfU5ly2ZoLPrlsx75lJSOD3KDDupuSJZUg2Du18C64wiGj38EOid26SGMpHGKaCH2NqeouR2wLAP4TfSIm69R6hRkOUkQ5t5Qyk3tIwSrW5OqDhXzwRFQn+IygE+r+f8BlDAh3/yj760xhyld7g33+zI5NleZ1pL9q9FDAl64GQT3h44CQd0AELlVT8rMFDxbCHVCO0eqXGbUR75yflmHYpnlTNSbfjenIXJx5qJxwHT2AoNdMxZxXoAQ6WbOqGxKHT3ylF7RvUxRyfRN+T+fMz5br4k/6xjdkaFiQT/atO4SacCZdPtk71zeDIojQWuW/XcE2GPHPC3fN7ZlLwcQypWBHggnoqm7lcuaaZV0MRl2DSCptStN+ZjVpXcHeLRBVtddvVZMBG8QqItbimNOhSTJxjx07xbdQsSdXUe//t8meAehTQBb8j1Yh4WN6nQNzWs8Jzob/O326CYbgwvihk89hPY099wK/qTmIDkke8fDLzMfhSjtx/GcwKMFKGi4QUZfrJy4kDOQAQ5Csc1kJlRPgZEC5+Vm1uZcGzVto8E4An5XdmuOx4j58gmYeMPF7o0Cvv76Rb76aYbimpfrPwel+S62aSgiU4p3/kcr3C/7D1+XqN4umJJyKnFmU6o7ccZTwYdBTJUcvZrHzqRKQ2D1aG8P32CQ/ChTQ5cShUJH/9j+eWTLmY97iuFTChizh8SyDK3PzisnkSLF+kEwbJXunWR2XlcTLRchvxv4pt9sazMKJwXgz0h5WRkgnFqdsbQcVakZN3a8SjnueR4C0vNLM5E5HnwbX tRi4t2gi phGEG2c2r0xV0mjoMly7Na/c19NkPnqVqA9Ri/LdGX2Y6pKMY0FNPIQX8+0tVzWUMimPLngMZ2tFKOoGEjBBoP/0WirGqYYH5D8LFoHjFjwrkNCGEV9Cxs+XAN2X8Xl0MYIli7I/k2PaEsM9Giny+JI1xbOi6FoooHXKIpfl4/oUgVZCHFibV+VWG+N6VWl3o920lp3ggtpwYB7+Cyu+twl+TazLnmYbUyT63 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The replace_stock_objcg() is being called by only refill_obj_stock, so manually inline it. Reviewed-by: Roman Gushchin Acked-by: Vlastimil Babka Signed-off-by: Shakeel Butt --- mm/memcontrol.c | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3bb02f672e39..aebb1f2c8657 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2763,17 +2763,6 @@ void __memcg_kmem_uncharge_page(struct page *page, int order) obj_cgroup_put(objcg); } -/* Replace the stock objcg with objcg, return the old objcg */ -static void replace_stock_objcg(struct memcg_stock_pcp *stock, - struct obj_cgroup *objcg) -{ - drain_obj_stock(stock); - obj_cgroup_get(objcg); - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes) - ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0; - WRITE_ONCE(stock->cached_objcg, objcg); -} - static void __account_obj_stock(struct obj_cgroup *objcg, struct memcg_stock_pcp *stock, int nr, struct pglist_data *pgdat, enum node_stat_item idx) @@ -2934,7 +2923,12 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ - replace_stock_objcg(stock, objcg); + drain_obj_stock(stock); + obj_cgroup_get(objcg); + stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes) + ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0; + WRITE_ONCE(stock->cached_objcg, objcg); + allow_uncharge = true; /* Allow uncharge when objcg changes */ } stock->nr_bytes += nr_bytes; -- 2.47.1