From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A10EDC05027 for ; Thu, 26 Jan 2023 23:12:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA2286B0071; Thu, 26 Jan 2023 18:12:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B52686B0072; Thu, 26 Jan 2023 18:12:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A19E68E0001; Thu, 26 Jan 2023 18:12:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 91B006B0071 for ; Thu, 26 Jan 2023 18:12:56 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 507CF1A03E0 for ; Thu, 26 Jan 2023 23:12:56 +0000 (UTC) X-FDA: 80398502352.25.448B5F0 Received: from out-193.mta0.migadu.com (out-193.mta0.migadu.com [91.218.175.193]) by imf19.hostedemail.com (Postfix) with ESMTP id 5E32B1A000F for ; Thu, 26 Jan 2023 23:12:54 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="paF0N/iK"; spf=pass (imf19.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.193 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674774774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n7M1bfJLC4Rcsy/Y/usaqBvuQAjlpdOFXty5eUdj7PU=; b=VGXXPh+WjmF/hNAMOS/ZsDQUZEFJfmmlfORcd34CRNOs81vo/lngnh6PLPMBALFdAZxDEe C2iFrVrfhPuZYAC76nI3yvye40QlJ3B7KqPq/ItyY6OdFH94ZdJHj4jACtqQdCcCXnZZ+V ZY8cNfWWBLCz130sc7YrhZbxWgjDA0A= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="paF0N/iK"; spf=pass (imf19.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.193 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674774774; a=rsa-sha256; cv=none; b=e/KjeUySoL1doXeyiJGSdXdiOw+8F0MTKekrsg6L8zBKONCZ7xR8bKYrB4waHX8w1CtZDs 8USYZrI12wiOANvrqFype4hfFRoTz7zlUSzFdcll4OY/CiRbdfz/hqrvBPDMar+TPl2lC7 nVHcc9tcPG7MQMCTjQinwqQP2K/f44A= Date: Thu, 26 Jan 2023 15:12:35 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1674774772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n7M1bfJLC4Rcsy/Y/usaqBvuQAjlpdOFXty5eUdj7PU=; b=paF0N/iKfvlcN9yEEM/b2TNNUDXisDdGcz0WAIRf6cjejS0WzLPSni7TNGvbneMeKyzDyj Lebr5WBZ2hHo/G7w6N+e4AuL7TZD6obbBxM+8Sp8zk0464GNq3oxk2qH50NdUF/jGCuI0o X0rTkDGfQwBBfQ8Dxmnb/f/xbQMWc8E= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Michal Hocko Cc: Marcelo Tosatti , Leonardo =?iso-8859-1?Q?Br=E1s?= , Johannes Weiner , Shakeel Butt , Muchun Song , Andrew Morton , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 0/5] Introduce memcg_stock_pcp remote draining Message-ID: References: <20230125073502.743446-1-leobras@redhat.com> <9e61ab53e1419a144f774b95230b789244895424.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5E32B1A000F X-Rspam-User: X-Stat-Signature: tb4pkjhjjoi1uk38azr4yhn984j93ayf X-HE-Tag: 1674774774-285308 X-HE-Meta: U2FsdGVkX1/PdH55dnNih04wAIXUEeN8Kg8N7oALKsK8apkyu6b1DtrIrtWw595y/1nBJOUlkyofOIT0HPBo/Z9ZFWLW2k+Ym/2d3M9etEKGuvLgXhwab+X1SHGGMjkrG70Y/KXcmDbFNFAlPy+VlV073ciUV5bDTu0Yer7cJOh1+O13X0gt2Fiuo+tDuOEgpGAABmMMsS7aO2EYrmTBIOzHV8bVcIwxQwXb7ZHMxK+4OAKdN2XKB1yYTaDqEnFWZOSAP0KlZUjLcjM0I96BY3LMMa5ixmHDHGuq+ZzeOSgVSfiKVUX+qdSDFEhHb1C63UbUjC4Hzd7JtI9/mL48z2JOY+FTilczrBwDEY7xBmssDaZjCKIls//X9JvyzXTCtoCTeYQOunN2OvsyRjAdR73zDDsCyYdHic1roa82HuBKIyJggPU2SDWEiKiUTNAoLv9tmUIGkMVxo42FaXw5EFsW23tb13a8F+lMA7LmsBFuaxKzfdmUJQqWsZeMbawtSZyhZf4f2oXVsgHo5G8HIBlPU/ehzKD1NVBGnNsPeMHkTJ5u8/3SS6kW9AKXG/0nYbwN2HkUL6DB2QQnC1lwUiKEaFS+wBQRLHo0jicaFtJzt5LxP5P9TsPyiJAHl0TCjdUCDY3wi7OhNn7hLbTiv78PUGoaGm3qL0tRPZPTyrqNEaEh0Q0JmHfH9CadgUVOMEGwhYQ04QreuSyR53EAhRxVpLHA+gLkNTjaFoZe502Lb5pe/O9abAcSKl4oQZEbSeb8yGDfeH/AzhxlELAWIas2KesRhNnV2rmWQtgGPS0RVFKbkHccqVEXAsbqzkOn5zFGK3Nweb9U1jfW2af8MB1pUV3peIsB3LtbEF7hTt6bs2MoJ3wvkBgRTnmIcHipl6tCljpfEXp8KMXktRBdYzzRkvadiFmJyHHbWMROCiAvsPAwgdwSN8HlBez02jcD5jIcXPvFTawZI3uxWVU DuB+9Ohb U7L5EJADqmGAwvmXcej9aHu+o4GuVRQsehZQBJ6rUzqv3hnuXirO8xV6OClTManp9QNxlzqGoaHJZDruw+8auMhvURBpsYZszhgQ+8CFSi8aLT5Iogv8RPSru6PMNWpBEBS6nuj7ZHZjacFCJSWlPrYqA69gKZNKDwRiXbmZPlLQg3CYiG1Fy+riw9s4jFKMfNiRxLKCqL1JQKBgMKGcUQRb/Ig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000052, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jan 26, 2023 at 08:41:34AM +0100, Michal Hocko wrote: > On Wed 25-01-23 15:14:48, Roman Gushchin wrote: > > On Wed, Jan 25, 2023 at 03:22:00PM -0300, Marcelo Tosatti wrote: > > > On Wed, Jan 25, 2023 at 08:06:46AM -0300, Leonardo Brás wrote: > > > > On Wed, 2023-01-25 at 09:33 +0100, Michal Hocko wrote: > > > > > On Wed 25-01-23 04:34:57, Leonardo Bras wrote: > > > > > > Disclaimer: > > > > > > a - The cover letter got bigger than expected, so I had to split it in > > > > > > sections to better organize myself. I am not very confortable with it. > > > > > > b - Performance numbers below did not include patch 5/5 (Remove flags > > > > > > from memcg_stock_pcp), which could further improve performance for > > > > > > drain_all_stock(), but I could only notice the optimization at the > > > > > > last minute. > > > > > > > > > > > > > > > > > > 0 - Motivation: > > > > > > On current codebase, when drain_all_stock() is ran, it will schedule a > > > > > > drain_local_stock() for each cpu that has a percpu stock associated with a > > > > > > descendant of a given root_memcg. > > > > Do you know what caused those drain_all_stock() calls? I wonder if we should look > > into why we have many of them and whether we really need them? > > > > It's either some user's actions (e.g. reducing memory.max), either some memcg > > is entering pre-oom conditions. In the latter case a lot of drain calls can be > > scheduled without a good reason (assuming the cgroup contain multiple tasks running > > on multiple cpus). > > I believe I've never got a specific answer to that. We > have discussed that in the previous version submission > (20221102020243.522358-1-leobras@redhat.com and specifically > Y2TQLavnLVd4qHMT@dhcp22.suse.cz). Leonardo has mentioned a mix of RT and > isolcpus. I was wondering about using memcgs in RT workloads because > that just sounds weird but let's say this is the case indeed. Then an RT > task or whatever task that is running on an isolated cpu can have pcp > charges. > > > Essentially each cpu will try to grab the remains of the memory quota > > and move it locally. I wonder in such circumstances if we need to disable the pcp-caching > > on per-cgroup basis. > > I think it would be more than sufficient to disable pcp charging on an > isolated cpu. It might have significant performance consequences. I'd rather opt out of stock draining for isolated cpus: it might slightly reduce the accuracy of memory limits and slightly increase the memory footprint (all those dying memcgs...), but the impact will be limited. Actually it is limited by the number of cpus. > This is not a per memcg property. Sure, my point was that in pre-oom condition several cpus might try to consolidate the remains of the memory quota, actually working against each other. Separate issue, which might be a reason why there are many flush attempts in the case we discuss. Thanks!