From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BFF2C46CD2 for ; Wed, 24 Jan 2024 20:54:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DA4C8D0001; Wed, 24 Jan 2024 15:54:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 289836B0085; Wed, 24 Jan 2024 15:54:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12AC48D0001; Wed, 24 Jan 2024 15:54:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EEC506B0082 for ; Wed, 24 Jan 2024 15:54:33 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B94BEA0192 for ; Wed, 24 Jan 2024 20:54:33 +0000 (UTC) X-FDA: 81715408026.06.F6F8F3E Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf12.hostedemail.com (Postfix) with ESMTP id E6DAA40003 for ; Wed, 24 Jan 2024 20:54:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q0d7MqSg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706129672; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8PpHpDGTvd15z1lyjwFnCzu8+TWE5IPpOUUlHpqfM8g=; b=5nC8OW3JpYwREIgeLvd1392r+MeYUBltl8f+sa4km0MDG2QN20aEYFCuthowyy17da2TkW XVrLXuiVBZqg2wspcKMQDbJsl81Ubp+/MF5SeYD/pGWLKiOh2iEJ66K3CwlUniLwMGwBRD H8wERcnlP0qFy5Uihng2Nh9M+6Fwh74= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q0d7MqSg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706129672; a=rsa-sha256; cv=none; b=n3mNdcfkNhychw7Ac2WrwURTeHJPHrSCxHR3ND+9nkY1kreWMlbcW9vSuuLGQM8bhmLQxx 3jtpp2i8PgV1qTpQoKugBQBfCEoLg/5mGjb6cI85fBluov2VVh1RiI6ANd/UjClYWf7w9X RhVln4JwccgH+LccDDnnGHdOzrHaKoY= Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a26f73732c5so669908066b.3 for ; Wed, 24 Jan 2024 12:54:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706129670; x=1706734470; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8PpHpDGTvd15z1lyjwFnCzu8+TWE5IPpOUUlHpqfM8g=; b=Q0d7MqSgXWnZ4VQtYSaKysTkOfr3q8X+3MF8ZVvaD7WYF3hrDpt5A5W0cqaARG4OEI fHXckBD2ZMu+2Qz87aTRf/bxMtIxCYvh7XdExPcmCbzEABSiNC/CbEigJ0UKEuC32iLn Ku4AK07HtW+99K6aXavFv8kC103eFi4qczLV3OTONYrgVUmu+h0zEW344ypBFuLQP9Qe wuD9uzt1fVLCgLcCgdbu8Vkq6swD2uJQp+B9FPQD432MxtvpZz50DBfUWGrzDtvpCgPi UPlIDjJga/xgt32TQTPLPf2PlKEBLW27DHKj++0c3Xy3jRmvJO7zaUYeC15DotJW5AWC T2dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706129670; x=1706734470; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8PpHpDGTvd15z1lyjwFnCzu8+TWE5IPpOUUlHpqfM8g=; b=LWOR7qj6JUmpKR7w7BWlBwD7GFKNLWxF+8HePdm6BblJImYhmIagZ0KoPDJoUGg7U1 uAcl3aIkffUfY4fCSlL8pPRz4H7/QocIRMtXBndHBIl3DmHjfpuslop2S8J7Fw/z5ce8 GyE9q5TKMfTdM4FAnH0EUDvGv0GBNDDB/W4i+kGdhcwL+jCUlRL/BzMTzwlFb59wVRFG bBzvPD9J3a9GBbH1pUPDcm4GRHpfKqSaDBJr/4sDlHjWzRYrPx8zPKbOewyF0/iH6gOw 8k4ansFsiwb6udxnmy6cjP4dcrPNypus96LC/Cthpc077gZ7XOBdQTU+l7Xcggt8Al4Y HeHQ== X-Gm-Message-State: AOJu0Yy6kp213fUkGj60OpxVBHeDE98VwGNw5SW8yQjkm/Av89/usJjo p5MORVitQnMxHsVdPThOmWNel5TIOtM0mDm6dvudwbkolNsGWd7S/hVl38CY9qjBHz6ZO8pz9Rn nC906G4AdPAb5aueqqqNc3CVFiN93M0xafQuL X-Google-Smtp-Source: AGHT+IFQWuIM/i4iz3prv5tuO7tWXlBOX7HrLWHwiZ4AjwTkkZqPJOMU3T7di+dK4PngjAfTyy/+XdKG5uJUYqGzGjE= X-Received: by 2002:a17:907:c5c4:b0:a31:524f:60e4 with SMTP id ts4-20020a170907c5c400b00a31524f60e4mr333052ejc.38.1706129670335; Wed, 24 Jan 2024 12:54:30 -0800 (PST) MIME-Version: 1.0 References: <20240124100023.660032-1-yosryahmed@google.com> In-Reply-To: From: Yosry Ahmed Date: Wed, 24 Jan 2024 12:53:52 -0800 Message-ID: Subject: Re: [PATCH] mm: memcg: optimize parent iteration in memcg_rstat_updated() To: Shakeel Butt Cc: Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel test robot Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: E6DAA40003 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: sc8xdchbnbpr1wm8pxfjyrggt98hb6mr X-HE-Tag: 1706129671-942039 X-HE-Meta: U2FsdGVkX187VJk1aOnCJ9pd8H/i77yVzxx5EnrgLTH5Ix5BrvAHsZGmP/X6vXCmSk2m6Zuyb7Rvdoeruw0ZVJb8TY7G/4VYfhJiMESJZWm8ofSzDR5KO/ZePdMlTDhIyHQFe8o3GknT3rS4Zr7XPWlFVFHG4jk8JJyj0AJ8XZn3ionxUsusS0bEV45mxhjKp6wh/E7BtEjWJiu/pPehjmkU8wWU3T3GhwXAV1SYDY7TyPrfex1V0EH/zJyznn3lqP0vLypub2IoKSSaFFloxcUA9P0qh0ieCvYfZjNOOqQV99+SYK1k4NgFDpG2FEypmEm4bC7/7Ki1n6gbHrtYeAp26PgEdtTCInAKmHPqrLuHNSN6QVGY+0NJzEuI4DrpYYY6z8oG2zxxZ/j6BOI8yly1kqoF3QcdVUkL4P6upFU8/6BjKxZ6YIsXqsNVTNcM0uh/39G4W/k80a/Kr6RPcbA3zcV6b+xrYsH9Dl+/vRFjHhxH76j5682TZvJOwrKLwg5Uy7iaMXMJzhIbU0XxWhqHiG4UyVYIZkUZhxnlS0KSO4G5m9u0895Lq3Nz1JuuJUR4CXKeM8IbBOXDMw0WiRqYpOPGutAeqjjLFMSMtJ1zaN1+tWQALqz+3JjCRzhCbveHXCYP4F2pyfSyV9U0WaxVyIAMi69gBRWmxnvR9gDeDKqi3CsDWJ+Cr/wEyC+T4gQ+R6hoN3ECeygZywBLPMbB40KVqSgMNUjUtt8ZRv6rL4upf87gAN8CnRs8FWHcBZVz2L6kQmCk+e3eArbcPM5CBESw4ho/inq984dt9sMr6h7lH0MBt4Dnc8K0yChVYXXHWfJodhyGH29tznyj2RIicRKWaukZDcJhiK9gjqCoOYExDcQl9WMarsTqWDOYIZSj3tUaqsTwxb2Nf6Lb9CE8lgGttg7NuyRocwc+q+/pzIRjjdVbBhcRr9SkrFXPPFrXJtqfEQttBwOwj9V Q9bGLkUo 9dY5PQpZMaTEnWyvtV+L/vd/udXs/ourBgKpWHs1FhPsz/VmzHW9v2qI6XVJ32esUmAZtIyKsDEtOGxb2ZNJ4wzIu1nKHC86tSIqy0W9mkFUuGDtXC0byPkm/sTYql9EnSUD+k41ZWTqvtrhWTxOs8vhrN485dhPad+yUo08XHZtP5IHGfd7TVir6t9IzYjnSpA4K8Z9uzwFazPGMZXLQvH1TIFE8yPACnk1PBTuy7y3jWqfx2ZQ/KAfZUxAQxapAPIqJMp4UyVhMbt5krCjYCaVwdehRXxusum8RZ8PVb5EATMorsnNGbCmWTJcNRhKBQGu7rK+55WQ5XqfLqH0skiDQMfrJRWsbeb8w/S/Beg3gdFzv2wrisIn+wPqJ3pT+p+81 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 24, 2024 at 9:38=E2=80=AFAM Shakeel Butt = wrote: > > On Wed, Jan 24, 2024 at 2:00=E2=80=AFAM Yosry Ahmed wrote: > > > > In memcg_rstat_updated(), we iterate the memcg being updated and its > > parents to update memcg->vmstats_percpu->stats_updates in the fast path > > (i.e. no atomic updates). According to my math, this is 3 memory loads > > (and potentially 3 cache misses) per memcg: > > - Load the address of memcg->vmstats_percpu. > > - Load vmstats_percpu->stats_updates (based on some percpu calculation)= . > > - Load the address of the parent memcg. > > > > Avoid most of the cache misses by caching a pointer from each struct > > memcg_vmstats_percpu to its parent on the corresponding CPU. In this > > case, for the first memcg we have 2 memory loads (same as above): > > - Load the address of memcg->vmstats_percpu. > > - Load vmstats_percpu->stats_updates (based on some percpu calculation)= . > > > > Then for each additional memcg, we need a single load to get the > > parent's stats_updates directly. This reduces the number of loads from > > O(3N) to O(2+N) -- where N is the number of memcgs we need to iterate. This is actually O(1+N) not O(2+N). Every memcg needs one load, and the first one needs an extra load. > > > > Additionally, stash a pointer to memcg->vmstats in each struct > > memcg_vmstats_percpu such that we can access the atomic counter that al= l > > CPUs fold into, memcg->vmstats->stats_updates. > > memcg_should_flush_stats() is changed to memcg_vmstats_needs_flush() to > > accept a struct memcg_vmstats pointer accordingly. > > > > In struct memcg_vmstats_percpu, make sure both pointers together with > > stats_updates live on the same cacheline. Finally, update > > mem_cgroup_alloc() to take in a parent pointer and initialize the new > > cache pointers on each CPU. The percpu loop in mem_cgroup_alloc() may > > look concerning, but there are multiple similar loops in the cgroup > > creation path (e.g. cgroup_rstat_init()), most of which are hidden > > within alloc_percpu(). > > > > According to Oliver's testing [1], this fixes multiple 30-38% > > regressions in vm-scalability, will-it-scale-tlb_flush2, and > > will-it-scale-fallocate1. This comes at a cost of 2 more pointers per > > CPU (<2KB on a machine with 128 CPUs). > > > > [1] https://lore.kernel.org/lkml/ZbDJsfsZt2ITyo61@xsang-OptiPlex-9020/ > > > > Fixes: 8d59d2214c23 ("mm: memcg: make stats flushing threshold per-memc= g") > > Tested-by: kernel test robot > > Reported-by: kernel test robot > > Closes: https://lore.kernel.org/oe-lkp/202401221624.cb53a8ca-oliver.san= g@intel.com > > Signed-off-by: Yosry Ahmed > > --- > > Nice work. > > Acked-by: Shakeel Butt Thanks!