From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1360FC32772 for ; Mon, 22 Aug 2022 15:09:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B0E5B8D000E; Mon, 22 Aug 2022 11:09:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ABD288D0003; Mon, 22 Aug 2022 11:09:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 985E18D000E; Mon, 22 Aug 2022 11:09:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8A8498D0003 for ; Mon, 22 Aug 2022 11:09:27 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 63B3BAAC70 for ; Mon, 22 Aug 2022 15:09:27 +0000 (UTC) X-FDA: 79827562374.04.C1B8A94 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf20.hostedemail.com (Postfix) with ESMTP id D746A1C0288 for ; Mon, 22 Aug 2022 15:06:26 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id d71so9604379pgc.13 for ; Mon, 22 Aug 2022 08:06:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=rfnLOnYHfibKVs6Fb4CyKhxW8tJzjkgGwDzJoC+bA0Y=; b=WrEDmTe0u3FiZugr1PESRLE48S+q7XVC3UauDjAcW6HIAjvR0HdyDSvY4Yh8Xv56pD zPk2pcgIpSUnoaMXQGMFXbv7itfEgQxASZu3plwAPK03XvXa6lCQJD9NZXVJ5o1FBtB8 X3x7rF1QRsVh4MXlsOXYfj66zh30hHfyZ2JsCANksMqp18M7jGVX8W3hXJJrdalpRqiG suK5gjJfRKexqrQW/aKqFE+rYX7mOOZjjtfZCMzUoTVD8zZSvk9fiLVnIeZIGLgGzJaA dDSVJr3UUABWryrmO8Xl1VpIQ5ZWGOWL9XjGltxQGHICMF2l1LmnoW0euDwnBGjZS2e+ 1bmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=rfnLOnYHfibKVs6Fb4CyKhxW8tJzjkgGwDzJoC+bA0Y=; b=xNyM/SmEjJ4bNUNKwz4/s2pbnIeWkPh32fGT4ZkJNm+P2p8R3XfOgGhYXjS5npDZ3m nbsTbbUFgFa55OdY3NrxulNSUun9I7BfPH0zLTpobe21QQ1MVnS7EbVIgNw+z50rXrh+ KFrZ1gOTmtyj+2dXlxHPsgHJ0f6/I4vLeCxqFBF7TFGXV5j0weM0Qw96A41OrU2oaWMI EpcrxGvFNZCIH4SNmMRoR9eqOEHC56tG44kcXbXEq+niZDkLkCoLLOdsszi4iGf+8e21 u7IL++9W7MYvHvQF8rRoO+SZFKrmCUHoM5YdoZrfuBS5kuxSrgiiGNl/cVu+bIT9SzRn nTSg== X-Gm-Message-State: ACgBeo0dhwHouVs/ee7L4YWhoWDm+UZKdsbA/HDtb7dNHMW4zj7UGCMi BAnL/LyYKJsZDZxBzN0jnWYCGG3xlzfOpXWI55b2kw== X-Google-Smtp-Source: AA6agR5ItK2qWDc7jjr7fBTr2W7+SERWzFKsbKa/joz2WD3UuK5X9GUCKjlxWu+Hd97AdsMRT/n9+/7Fvy1T5C4iolI= X-Received: by 2002:a05:6a00:2392:b0:52e:b4fb:848 with SMTP id f18-20020a056a00239200b0052eb4fb0848mr21499685pfc.8.1661180785557; Mon, 22 Aug 2022 08:06:25 -0700 (PDT) MIME-Version: 1.0 References: <20220822001737.4120417-1-shakeelb@google.com> <20220822001737.4120417-3-shakeelb@google.com> In-Reply-To: From: Shakeel Butt Date: Mon, 22 Aug 2022 08:06:14 -0700 Message-ID: Subject: Re: [PATCH 2/3] mm: page_counter: rearrange struct page_counter fields To: Michal Hocko Cc: Johannes Weiner , Roman Gushchin , Muchun Song , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , Andrew Morton , lkp@lists.01.org, Cgroups , Linux MM , netdev , LKML Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WrEDmTe0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of shakeelb@google.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661180786; a=rsa-sha256; cv=none; b=Zglgnx/dC8sIr4CJP15ImYO8k5vkDyGmB1C/ohc8MSS8oVY0wUrqtuJrW9XOgZ01d+rkKY Kr9G0FMtJxLzI+JYUwxgP1AwYCX1M1l/GJAu4jpwi9BzN3IwQLIk4EV7lXN+kBa17GF6/5 Bj2rpWoZGdf+1KQb+BILLTBU0hG15Rc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661180786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rfnLOnYHfibKVs6Fb4CyKhxW8tJzjkgGwDzJoC+bA0Y=; b=WpbmLoubCirkEAK9yCMEAn7WY8grqmrRqmUCLH1KYOuRx/RStKpWnrCI/dvhSW+cWkP7HS 7ZyyoYZPbINYD4naGwGsst2WX8CBenL5jf4eQoQdMOwTHNJhEbdY8MYJjV5xRqn0aalfKg BCgc5i8rETU0LccEzsFH2qexY2DigWk= X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=WrEDmTe0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of shakeelb@google.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Stat-Signature: uer8t7pfrf3x76ph9u76fe3es8gzgst3 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D746A1C0288 X-HE-Tag: 1661180786-346064 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Aug 22, 2022 at 3:23 AM Michal Hocko wrote: > > On Mon 22-08-22 00:17:36, Shakeel Butt wrote: > > With memcg v2 enabled, memcg->memory.usage is a very hot member for > > the workloads doing memcg charging on multiple CPUs concurrently. > > Particularly the network intensive workloads. In addition, there is a > > false cache sharing between memory.usage and memory.high on the charge > > path. This patch moves the usage into a separate cacheline and move all > > the read most fields into separate cacheline. > > > > To evaluate the impact of this optimization, on a 72 CPUs machine, we > > ran the following workload in a three level of cgroup hierarchy with top > > level having min and low setup appropriately. More specifically > > memory.min equal to size of netperf binary and memory.low double of > > that. > > Again the workload description is not particularly useful. I guess the > only important aspect is the netserver part below and the number of CPUs > because min and low setup doesn't have much to do with this, right? At > least that is my reading of the memory.high mentioned above. > The experiment numbers below are for only this patch independently i.e. the unnecessary min/low atomic xchg() is still happening for both setups. I could run the experiment without setting min and low but I wanted to keep the setup exactly the same for all three optimizations. This patch and the following perf numbers shows only the impact of removing false sharing in struct page_counter for memcg->memory on the charging code path. > > $ netserver -6 > > # 36 instances of netperf with following params > > $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K > > > > Results (average throughput of netperf): > > Without (6.0-rc1) 10482.7 Mbps > > With patch 12413.7 Mbps (18.4% improvement) > > > > With the patch, the throughput improved by 18.4%. > > > > One side-effect of this patch is the increase in the size of struct > > mem_cgroup. However for the performance improvement, this additional > > size is worth it. In addition there are opportunities to reduce the size > > of struct mem_cgroup like deprecation of kmem and tcpmem page counters > > and better packing. > > > > Signed-off-by: Shakeel Butt > > Reported-by: kernel test robot > > --- > > include/linux/page_counter.h | 34 +++++++++++++++++++++++----------- > > 1 file changed, 23 insertions(+), 11 deletions(-) > > > > diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h > > index 679591301994..8ce99bde645f 100644 > > --- a/include/linux/page_counter.h > > +++ b/include/linux/page_counter.h > > @@ -3,15 +3,27 @@ > > #define _LINUX_PAGE_COUNTER_H > > > > #include > > +#include > > #include > > #include > > > > +#if defined(CONFIG_SMP) > > +struct pc_padding { > > + char x[0]; > > +} ____cacheline_internodealigned_in_smp; > > +#define PC_PADDING(name) struct pc_padding name > > +#else > > +#define PC_PADDING(name) > > +#endif > > + > > struct page_counter { > > + /* > > + * Make sure 'usage' does not share cacheline with any other field. The > > + * memcg->memory.usage is a hot member of struct mem_cgroup. > > + */ > > + PC_PADDING(_pad1_); > > Why don't you simply require alignment for the structure? I don't just want the alignment of the structure. I want different fields of this structure to not share the cache line. More specifically the 'high' and 'usage' fields. With this change the usage will be its own cache line, the read-most fields will be on separate cache line and the fields which sometimes get updated on charge path based on some condition will be a different cache line from the previous two.