From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BAF2C28D13 for ; Thu, 25 Aug 2022 04:41:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5B776B0074; Thu, 25 Aug 2022 00:41:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0B286B0075; Thu, 25 Aug 2022 00:41:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD40F940007; Thu, 25 Aug 2022 00:41:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9EA486B0074 for ; Thu, 25 Aug 2022 00:41:55 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6BF41AADDF for ; Thu, 25 Aug 2022 04:41:55 +0000 (UTC) X-FDA: 79836867390.13.74829A3 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf15.hostedemail.com (Postfix) with ESMTP id 0A01AA0027 for ; Thu, 25 Aug 2022 04:41:54 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id c13-20020a17090a4d0d00b001fb6921b42aso3638023pjg.2 for ; Wed, 24 Aug 2022 21:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=INrevp61jvgo3SuTdlwO6+Vd1zseXzdZ0ssScT78usc=; b=kAHRzLJrcnUVbYLKrg/c7FYPqutwG9JKgj8uA6KK3PV+CyzNUEY6/mtraAJVl9Ty5E zfi9COFC1b5FEzcFfHSUPx2evppPqH2TIgm5fh0j1Gf9PfpoXw6zUcDMmjRAjbNEpjmo UWWJEkMK3ebPfk5TIcE5PqztVYrSkkAKjsKEMEiHVKHfIPpYeNRabgnvsrRrs0z+DE45 TE08FVwAYgW6arUGrYUuDTWTdjQy8k/78moOETYNMTUHIvR7e3542FJmr+s5Yvtszs9T hgG+bNxFT3c/GJT05S/xY/IOvz4k+c8rXLJrJHiiVMt9QzfvL38envLwAT5cvVYbsmLR 2uBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=INrevp61jvgo3SuTdlwO6+Vd1zseXzdZ0ssScT78usc=; b=Q+yqMrM6xETuDurk3Cpn9ImT1zIqxf0WC62MP9eAfhlvQU3BkQdRx/vWfesiwC4R/7 ue5sx3d9HC0zQidvUKUv4XOBZcEWQv5QKJ7EVr/zqQ5MpIRuTyJUjIZ+xFJCZMK9rtSm rc9SIJmnCIevwwsbmiJYZcvWm/Mfg9/1x1nrL59CT9qVjjVdbf5YeMyncu0eG/7KeZqq nfPf/JnMbhqeH3dcdPoo1zwMx6AYFzo9FloftNAx+ZR7lvLS3zcUaKg6MqphGIyr8PMD qfhJ4+PZKRzXUAz9/anYxkBPIF5rdi2ktfPAMiZHOBYbFwFyu5PO1YIaH+Wt9Ok715OY Bdsg== X-Gm-Message-State: ACgBeo2hnAxmvz6jQMwkca3vOthkuxSmmtn8Q5ZLYaHzJMJVCiDlp+vg rBHzGsZ+a+icLJfB005s8Z718Mq3rosfdwDtZGEsfQ== X-Google-Smtp-Source: AA6agR6hGN7BmkhP+Mw6B4QHFjCHWsgU7DDwEvi0UkEo46L6L55OpTIH8uF2pm+kufP7PumEiqap+MCob1VMq/R3zgE= X-Received: by 2002:a17:902:b410:b0:172:c9d1:7501 with SMTP id x16-20020a170902b41000b00172c9d17501mr2099534plr.106.1661402513848; Wed, 24 Aug 2022 21:41:53 -0700 (PDT) MIME-Version: 1.0 References: <20220825000506.239406-1-shakeelb@google.com> <20220825000506.239406-3-shakeelb@google.com> <20220824173330.2a15bcda24d2c3c248bc43c7@linux-foundation.org> In-Reply-To: <20220824173330.2a15bcda24d2c3c248bc43c7@linux-foundation.org> From: Shakeel Butt Date: Wed, 24 Aug 2022 21:41:42 -0700 Message-ID: Subject: Re: [PATCH v2 2/3] mm: page_counter: rearrange struct page_counter fields To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Eric Dumazet , Soheil Hassas Yeganeh , Feng Tang , Oliver Sang , lkp@lists.01.org, Cgroups , Linux MM , netdev , LKML Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661402515; a=rsa-sha256; cv=none; b=KmJrJYIidof6RuLdGQtTHY5mCLpuelR9OfPYRbFnr6SEMUzyk00Hu8RZ0Gb3ZnQIiZ5lHq /3clGNfPcbYKlJ2TckoLqxvE0MOAW8bc2HolBrJIQF8VfoKzXxI8AkDguZsPqRBGRPaNnt iKsVleTX46M0hZotBd1e6bQxW5O7bGw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kAHRzLJr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of shakeelb@google.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661402515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=INrevp61jvgo3SuTdlwO6+Vd1zseXzdZ0ssScT78usc=; b=tOBgiq0cXQgxi2eLXR/H7FKxPX1BE2J845EgCq5O9oSeS/E4H2khvd8sCG6qX1vNioeZL0 DY4YxiAepvn9L81P5jqUvD0VZawC7HBmwwzVvkV72D8V9FhjKqd4V53xUzETQrHxO45Are O4FH555UXQpSUTjApi4Qmn0jXpz5J6Q= Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kAHRzLJr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of shakeelb@google.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Rspamd-Queue-Id: 0A01AA0027 X-Rspamd-Server: rspam02 X-Stat-Signature: zhs9kum7quxyrsf7f9yeuurgpry9eryh X-Rspam-User: X-HE-Tag: 1661402514-616973 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Aug 24, 2022 at 5:33 PM Andrew Morton wrote: > > On Thu, 25 Aug 2022 00:05:05 +0000 Shakeel Butt wrote: > > > With memcg v2 enabled, memcg->memory.usage is a very hot member for > > the workloads doing memcg charging on multiple CPUs concurrently. > > Particularly the network intensive workloads. In addition, there is a > > false cache sharing between memory.usage and memory.high on the charge > > path. This patch moves the usage into a separate cacheline and move all > > the read most fields into separate cacheline. > > > > To evaluate the impact of this optimization, on a 72 CPUs machine, we > > ran the following workload in a three level of cgroup hierarchy. > > > > $ netserver -6 > > # 36 instances of netperf with following params > > $ netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K > > > > Results (average throughput of netperf): > > Without (6.0-rc1) 10482.7 Mbps > > With patch 12413.7 Mbps (18.4% improvement) > > > > With the patch, the throughput improved by 18.4%. > > > > One side-effect of this patch is the increase in the size of struct > > mem_cgroup. For example with this patch on 64 bit build, the size of > > struct mem_cgroup increased from 4032 bytes to 4416 bytes. However for > > the performance improvement, this additional size is worth it. In > > addition there are opportunities to reduce the size of struct > > mem_cgroup like deprecation of kmem and tcpmem page counters and > > better packing. > > Did you evaluate the effects of using a per-cpu counter of some form? Do you mean per-cpu counter for usage or something else? The usage needs to be compared against the limits and accumulating per-cpu is costly particularly on larger machines, so, no easy way to make usage a per-cpu counter. Or maybe I misunderstood you and you meant something else.