From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EDDAC433FE for ; Wed, 5 Oct 2022 18:22:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 490D96B0072; Wed, 5 Oct 2022 14:22:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 43F726B0073; Wed, 5 Oct 2022 14:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E02C6B0074; Wed, 5 Oct 2022 14:22:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1B0F66B0072 for ; Wed, 5 Oct 2022 14:22:30 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DC727140367 for ; Wed, 5 Oct 2022 18:22:29 +0000 (UTC) X-FDA: 79987716018.29.132A288 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf21.hostedemail.com (Postfix) with ESMTP id 8B9F21C000E for ; Wed, 5 Oct 2022 18:22:29 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id h10so13360687plb.2 for ; Wed, 05 Oct 2022 11:22:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=7kOunpUwcD5B3LDxquXvMevQ7y8E8fEOG4N0wZf8Bfk=; b=WpMwCgvgGsA0PuBsnwhYNtJ0CK98v4OtYMf88jz1gMNhlOCoJTZkzylvAsRYIFsECA moN8UIUNqopRtKF76bR2MQaZRo0AKL/m7neTmV/WJeyFe9i/sUGa9JMaAtDuNR/q+P1Y lrkxNcDnYdeF5DiFn7bvBC0lZpQbOGQ63z6xl7UuLV/LUJZUdSlE3s7yOih61aXEcGLO knTvvOAPH9/7p8jMvcn1wEMMlvTGN7iFuojaVGOMyvXFWoVvwlvKyKMWnnn8TM9ySBGd MyaIj7a82TPsH/h5af28QagxWU3CuwJoEx3GDOcNX8riHVd/ej6F9gnMKslv9VO8TT/T 3OVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7kOunpUwcD5B3LDxquXvMevQ7y8E8fEOG4N0wZf8Bfk=; b=4pcl002u71MNST2J8JNqdR4ecT+tKT6QuWpbqqvsW95nG1sFtB1rXj+Qx+OGQprtpF CBbsi3ZBXYZxyByE+Q35AoVub1u1oh1W42Cax91bZiP8UCCd7BTpqxL8QMqHJdSfPGA6 DTWxnf6kWG8SYzvJthx9vbaqrJDFslhltvJ+MtLZx9XCUdJ2adAk6feJAhM43G60dcsb vzl7hUhr17Daj1gF+3FQxUMmWuNbEUvN/LbHYxtnQM0Zq5n3ErLV/pEoRt3sW6gncFT8 iNloJeb2K+C8GaoB0Sl4N2Fpr3Q8enxUIiX+et21y9dmQcHHI4cMWkMtjdPL1KTOrivM xnfA== X-Gm-Message-State: ACrzQf0/B2V2dd7VJJiyii/hdwXKMH2ueaUrQXtuNmRv8rp9k95uiBMq MDmvnAaroJsCTa1f4R4okUzFaxJTbnFjww== X-Google-Smtp-Source: AMsMyM4YyitnMq+UVEGMY49eaAawysEwKS8u2EvRmUfqBnw8GZqWEmldTX6QcsACNpEDPEOFrSQZ7g== X-Received: by 2002:a17:90b:f02:b0:20a:9965:eeee with SMTP id br2-20020a17090b0f0200b0020a9965eeeemr1006936pjb.182.1664994148322; Wed, 05 Oct 2022 11:22:28 -0700 (PDT) Received: from localhost (2603-800c-1a02-1bae-a7fa-157f-969a-4cde.res6.spectrum.com. [2603:800c:1a02:1bae:a7fa:157f:969a:4cde]) by smtp.gmail.com with ESMTPSA id a15-20020a170902710f00b0017f8edd3d8asm53290pll.177.2022.10.05.11.22.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 11:22:27 -0700 (PDT) Date: Wed, 5 Oct 2022 08:22:26 -1000 From: Tejun Heo To: Yosry Ahmed Cc: Zefan Li , Johannes Weiner , Michal Hocko , Shakeel Butt , Roman Gushchin , Michal =?iso-8859-1?Q?Koutn=FD?= , Andrew Morton , Linux-MM , Cgroups , Greg Thelen Subject: Re: [RFC] memcg rstat flushing optimization Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664994149; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7kOunpUwcD5B3LDxquXvMevQ7y8E8fEOG4N0wZf8Bfk=; b=dRvUumnyn1fUSlVERgut7/q1BxIiNAY6ry2BwvhYigvu+Q07wzOUk7H33BhPiadN2abzj3 xQoVIwMqqNo+afDz+A8TjxBub9kwScQHzGb9DzKhBhyusX7Nn6RxvP2Bz/h5O8wOefHLIH Aotr8rTv7TAyPit8Erp8RjToE5HRnEQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WpMwCgvg; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=pass (imf21.hostedemail.com: domain of htejun@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=htejun@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664994149; a=rsa-sha256; cv=none; b=M5PjWMk6SBXYupZoYZ63G437gdDKp4srgY/FBrrZyjARaNL95VOquMcVbmVm/wlH9fkHzz N8WU73f3KuY5qo9OzLZu2lCkn/+Zs/JPbnciVaOPZQq7dWNjOkLnHEKapqdHDhuXUxijem kgd15Kh8PLkMC6JWPL5M3OMNtX3Nbmw= X-Rspamd-Queue-Id: 8B9F21C000E Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=WpMwCgvg; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=pass (imf21.hostedemail.com: domain of htejun@gmail.com designates 209.85.214.176 as permitted sender) smtp.mailfrom=htejun@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-Stat-Signature: sqfhau61qegnagoxu5t4637syeh4er7t X-HE-Tag: 1664994149-869636 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, On Wed, Oct 05, 2022 at 11:02:23AM -0700, Yosry Ahmed wrote: > > I was thinking more that being done inside the flush function. > > I think the flush function already does that in some sense if > might_sleep is true, right? The problem here is that we are using Oh I forgot about that. Right. ... > I took a couple of crashed machines kdumps and ran a script to > traverse updated memcgs and check how many cpus have updates and how > many updates are there on each cpu. I found that on average only a > couple of stats are updated per-cpu per-cgroup, and less than 25% of > cpus (but this is on a large machine, I expect the number to go higher > on smaller machines). Which is why I suggested a bitmask. I understand > though that this depends on whatever workloads were running on those > machines, and that in case where most stats are updated the bitmask > will actually make things slightly worse. One worry I have about selective flushing is that it's only gonna improve things by some multiples while we can reasonably increase the problem size by orders of magnitude. The only real ways out I can think of are: * Implement a periodic flusher which keeps the stats needed in irqsafe path acceptably uptodate to avoid flushing with irq disabled. We can make this adaptive too - no reason to do all this if the number to flush isn't huge. * Shift some work to the updaters. e.g. in many cases, propagating per-cpu updates a couple levels up from update path will significantly reduce the fanouts and thus the number of entries which need to be flushed later. It does add on-going overhead, so it prolly should adaptive or configurable, hopefully the former. Thanks. -- tejun