From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D034EB64DC for ; Fri, 30 Jun 2023 23:22:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F1288E005C; Fri, 30 Jun 2023 19:22:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A13F8E0059; Fri, 30 Jun 2023 19:22:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28FC88E005C; Fri, 30 Jun 2023 19:22:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 15EA68E0059 for ; Fri, 30 Jun 2023 19:22:44 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AD7C1A0497 for ; Fri, 30 Jun 2023 23:22:43 +0000 (UTC) X-FDA: 80960991006.13.11A11F8 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf01.hostedemail.com (Postfix) with ESMTP id DA1F54001A for ; Fri, 30 Jun 2023 23:22:41 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=JIyBKNtR; spf=pass (imf01.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=ivan@cloudflare.com; dmarc=pass (policy=reject) header.from=cloudflare.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688167362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=VK1/OtAJXK8/kraYa6/U+8C0ZuqhuzcqNzn9O4FJGas=; b=vE2vgnK1NMM1K2bjQyQEnf07Qf0Bp47M/l2RE6KJVl/D5Qdj1oXI6HvhptWxCEsHpIIn6O kt7GxiaGAbMdg+Ch18QnYuZyc8C3+EgIANKalCyEBPkyD+W6z3ki7bCB5S8G5CqjlvQ6vi J54uWi5zq1DrZJ4Yn6hqYRzCOIpETYE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688167362; a=rsa-sha256; cv=none; b=WYUyJPB+bh496qekR8t9PFmkqunPnzlaZfvmk9hvArcH8YziswtNCYTbqcwP69rFniWyNm miNVXbNsSUpqhTJc6x++2kgMtM+mvRZ/H+AgLUOjH7EddusjyQ1E1zzTjtxOQ0Q38oKbt6 0A5h3RQ5FMnhbmxrlm/cCjpjqiSu1wg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=JIyBKNtR; spf=pass (imf01.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=ivan@cloudflare.com; dmarc=pass (policy=reject) header.from=cloudflare.com Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-3fbc587febfso16222385e9.2 for ; Fri, 30 Jun 2023 16:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google; t=1688167360; x=1690759360; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=VK1/OtAJXK8/kraYa6/U+8C0ZuqhuzcqNzn9O4FJGas=; b=JIyBKNtRrCvh1/J0Aia88RunCnBe34DcvcRR/xqkto9DKiP0CoJsMe9AEsMrcocQXa KXzU0xeE8Ynn4vsjVq6S/1cSMppdCM5vHTv6cycRYhKMbtS8osvCknVpybopDo8HlGmS 2aY3f2gRBCcrD1RClb7o73Cvr+nsmVkDEnhIo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688167360; x=1690759360; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=VK1/OtAJXK8/kraYa6/U+8C0ZuqhuzcqNzn9O4FJGas=; b=ItDhjU3PDVg10r19eT/debKtqg2YbKiBYPzIZQJ4zO0EEEk1tiBgfYydnU6Q88A2Xz cO92nPJLL6vou8gN5LMIH52sZ00PP1lObIymzIWvYCNGu6fhvTIEZxvjxoUGO27JvhZq BZy4bq3d4jyaEEr0hRtL5Bf2WaqUQGWNX6mG5S6l/VwCMo53fomyKw91K/kRfOoA6/b/ SiP+uGW2j8x4LTiv/pORznchVQInHXd3ifT2IlTHprqPSNkFqLEf58PGl8eDtrDmEv5a 30uqbd32yw6DRo1FvDxi/s5FnhQsevaE+lfDJGVVTIsOwYALXFdReskppzZ3eT43nUiF s4xQ== X-Gm-Message-State: AC+VfDwkrCz+FiL5FxzqyhULMU2cT1I7WB22sYTm6n5gAUWmqUwAhwlI DpDIIfuwsQPZR+luUrnWZ2IdHTy8rDHQToM7KfYMcw== X-Google-Smtp-Source: ACHHUZ6SNvsiIRzbKttRMIOu9i80rwHHJPSkwI+4sREgWXjA5v7EnjfhxpCc0WUYy7+esb0YOLV26PhKyYmeg8Pfgl4= X-Received: by 2002:a05:600c:221a:b0:3f4:a09f:1877 with SMTP id z26-20020a05600c221a00b003f4a09f1877mr2789670wml.23.1688167360268; Fri, 30 Jun 2023 16:22:40 -0700 (PDT) MIME-Version: 1.0 From: Ivan Babrou Date: Fri, 30 Jun 2023 16:22:28 -0700 Message-ID: Subject: Expensive memory.stat + cpu.stat reads To: cgroups@vger.kernel.org Cc: Linux MM , kernel-team , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , linux-kernel Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: 1ec577grkqsm58drdnu7r7sibezn9fxk X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DA1F54001A X-Rspam-User: X-HE-Tag: 1688167361-931539 X-HE-Meta: U2FsdGVkX1/LlFBOMBpmS5bodkre3y9e2xV09LMZmxoMO4LvhrTQvBt4h1Hms5La1hjdmscX3kjXSs35KTH8X40IaD2UIpm28v5EC3kzcPwsScj5VKvWd5NHnA9l84fAHOuMsJ6+AXw6aBMKLSqsGgmNS8xgixyJUVYjVzR7HSoQMlNxQptyAJ2jKLyTf/vnoScVl3sm5XDnl9QY5gJjVFgiFbmXRK4bVYNInZNBkLxmsyDhr2lPU1JGrPJi713ksVlf8d0sIIFZLU5IegkFcUomo/77HO0LxBwmD6Az0xhYJ0x3muCji0X0jj6dP7/A2fEZdKutsR+u8p+2ZlJ/ouz2BD/TyFioM3pjknM65On8/b5pf1vIIgE+5x1kZe5TiPB09anKat7/mKD/14UAetUzQbjGHXkkUXIs4qFYsu1H5c/pbmw0ovqNXnNtLLcw1S1p9Nv9Ug+6ZZe3Jhb9tS51e6vh1wvpIs+CpgT/gxw0bPhqE491uAnOTh5dOtCcDA9Q4HblprZbYIrvU/DtWKS+w8FzDJcfLfYUiL6tymeJs2y1WJF5J4OMp3ON46YOHY60KNStY+3lbkJQRQHsB2obmFAozur0KwsALSpFNdU7HXwYwI42fT7AgQegyh9oP+GlcDqkvRinR29KV2MulCuRO3WdZ5Z5u09PDKTUYPBxRI3JRm5qz7nHW6EbSXNBskRu6o/QUILDWPYf7zLOLu0GlINU143ulrLI3Q4W8trb8ixYXszvNWLkNKM5oju1mp3I0558DuIOA+TNcv5oHEBn5eubP93fT5Cf9zNi9hiCN4OKOtSosdiIEMfluK0De8+xrvtu44hqDj5uTtezyievosBpnYRqm9rN7TEf9qNHgqwWPB6n0LAGm3FsdFI/1/w58yrgx5YnmuJnghP11cMHFFdNxG15LpgRXf61ogBnyQFnly7t1Il3wid1tzebe0a/Xl8WvUwb2I5qG5z X82Kizv1 IrXXW94YzNG05yAa75x2TnRHb7S/ww5j8gmdl0KQTKgipPXRp5OkVHD3zby91hH1MIwqL+fMr8cjA7aezphr/wDQy35VJAfh3WipI+YReJvs/bwkiCIfpvyYK1k0FsV0FaIcCRXtF3WAF34dRBNjDZNSaKEsa5jVFlhyLvA2GltCVa7ZH1kBKKZ/ikwynA64IzLo8QssImM9Qt5Zft+1Fbs84yGcwQZuoLablqDOURSgDrTtQ6RHv9+JG/S+WNJZnLXi9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, We're seeing CPU load issues with cgroup stats retrieval. I made a public gist with all the details, including the repro code (which unfortunately requires heavily loaded hardware) and some flamegraphs: * https://gist.github.com/bobrik/5ba58fb75a48620a1965026ad30a0a13 I'll repeat the gist of that gist here. Our repro has the following output after a warm-up run: completed: 5.17s [manual / mem-stat + cpu-stat] completed: 5.59s [manual / cpu-stat + mem-stat] completed: 0.52s [manual / mem-stat] completed: 0.04s [manual / cpu-stat] The first two lines do effectively the following: for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null The latter two are the same thing, but via two loops: for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/cpu.stat > /dev/null; done for _ in $(seq 1 1000); do cat /sys/fs/cgroup/system.slice/memory.stat > /dev/null; done As you might've noticed from the output, splitting the loop into two makes the code run 10x faster. This isn't great, because most monitoring software likes to get all stats for one service before reading the stats for the next one, which maps to the slow and expensive way of doing this. We're running Linux v6.1 (the output is from v6.1.25) with no patches that touch the cgroup or mm subsystems, so you can assume vanilla kernel. >From the flamegraph it just looks like rstat flushing takes longer. I used the following flags on an AMD EPYC 7642 system (our usual pick cpu-clock was blaming spinlock irqrestore, which was questionable): perf -e cycles -g --call-graph fp -F 999 -- /tmp/repro Naturally, there are two questions that arise: * Is this expected (I guess not, but good to be sure)? * What can we do to make this better? I am happy to try out patches or to do some tracing to help understand this better.