From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 431E8C2BD09 for ; Sat, 6 Jul 2024 20:55:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 382716B007B; Sat, 6 Jul 2024 16:55:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 332266B0082; Sat, 6 Jul 2024 16:55:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F97F6B0083; Sat, 6 Jul 2024 16:55:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0256F6B007B for ; Sat, 6 Jul 2024 16:55:18 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 82706160964 for ; Sat, 6 Jul 2024 20:55:18 +0000 (UTC) X-FDA: 82310533116.13.BA0AC03 Received: from mail-il1-f180.google.com (mail-il1-f180.google.com [209.85.166.180]) by imf24.hostedemail.com (Postfix) with ESMTP id E70C2180019 for ; Sat, 6 Jul 2024 20:55:16 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FHXmbmqO; spf=pass (imf24.hostedemail.com: domain of rientjes@google.com designates 209.85.166.180 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720299289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=DB1R9Ub7uutVO5Cod0GAJLLCpf0xgoayjEQ3sFB7+04=; b=g9lgOje6UOSU154XmkM2/v2a4r84svBMBEzgZUdV1jdwAjAA3Ca21Msei4j83AiNyI/8No xdWTS3YjSzn+63yVJUrTKglyO3ApWvC1yA7r4Kt8pkzBgcTuCAxeqoEEluB0ZeJXWNk/u8 m0EyTqzWpuVUxbTw3gqFSRkUraz4VfI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720299289; a=rsa-sha256; cv=none; b=eebvj7VyJmmfkeebsDkEIhZv5RhO7KCnyQx4sgHe2RR8jWXvM45kugUaLR4sc3Koy3C3uf PmOu/6bWuGh+IWlMUDpExd2U2bbLa3vv7M0PsZRReNUQhXOyuQVl17BfvC1AEf9zDtHBEt dXI2f/Emrx2Keaa3A4yQ8bMhYtdXuPY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FHXmbmqO; spf=pass (imf24.hostedemail.com: domain of rientjes@google.com designates 209.85.166.180 as permitted sender) smtp.mailfrom=rientjes@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-il1-f180.google.com with SMTP id e9e14a558f8ab-3815ad8adc9so512065ab.0 for ; Sat, 06 Jul 2024 13:55:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720299316; x=1720904116; darn=kvack.org; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=DB1R9Ub7uutVO5Cod0GAJLLCpf0xgoayjEQ3sFB7+04=; b=FHXmbmqOpCe3kODEBjNbzcUw0XxZMmDKeUBUgimHNBgC/CUJGziKY4ecg4zeEMskbn TV6GrVxdG/UcMzvjgKymCZOI1DDbBs7Neqd7P1T9ipAfwaYSf5yyjpSsMAaOdOcrHX13 d4/l2fHJ4HDub6LfIMIIy/A1LzacvaMX4QEDkW3uheNRbYlivCAoKZTaxNrXo0qLxsYN 3brTifshDrYt8gIszxlAed1e8GphfbeAuBxqRvcPZMwBgCCUkdOSSE8L2ETRiWhrX9up qrBPORGjTwBJy+5H3OPtERIRqmvolsyK7FvNgMx0nh3cJ7vVqNDvRAAIao5miKSOxq0W Df8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720299316; x=1720904116; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=DB1R9Ub7uutVO5Cod0GAJLLCpf0xgoayjEQ3sFB7+04=; b=vnq1HtUvHG/jmnaEzargKPtGX4tFYE+Clq/smYqxshy5mPhH8lozM65MFoDFNADvLY YNZUZHUxXSqQkBkyLDIqcE13MPwkcqdG/85Ng75GKKH7uL5MB+QPki5Rs7A5vR5MmTvz cTo5B0miP/YO0nXZZZHs+zKu8cyKGqPcweBC+9pi/uXorYNWtn5lLoioNq1oTt+NBBA/ /6wx027dgBJLsnOu8g3DTwtmGf1Lt6oj7Sh++fVWsPTJKlmhCvox24n6GH0I9QTiMcz+ wd09Xd5SeoFa/CWwjaj5rnXZR/ME9eCy4yTseuxV3MzE6YNmkYBBPGSNJ/S8qHeeBrj0 cb3A== X-Gm-Message-State: AOJu0YyddtB3v3XL3rPUiZF0zUhzhSutQqIdAVRrGalXGANr7IgKxyCH EpkD48ornXPqy/Sewlkgol8nQoAqaEVX8V/ybwvQycXJdXtidEf0SSDD+rOb7w== X-Google-Smtp-Source: AGHT+IGtmFR1ap524GekeptaKH7QR+eVD/YEDHSgohb7k6fAop9jU4Oy63lp4zp/183JuQMh/qL9Vg== X-Received: by 2002:a92:c561:0:b0:376:3ad9:22c1 with SMTP id e9e14a558f8ab-383505406bcmr7144685ab.20.1720299315868; Sat, 06 Jul 2024 13:55:15 -0700 (PDT) Received: from [2620:0:1008:15:a970:1a5:7c64:a798] ([2620:0:1008:15:a970:1a5:7c64:a798]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fb1a3d25f1sm70443805ad.139.2024.07.06.13.55.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Jul 2024 13:55:12 -0700 (PDT) Date: Sat, 6 Jul 2024 13:55:11 -0700 (PDT) From: David Rientjes To: Michal Hocko , Andrew Morton , Mel Gorman , Balbir Singh , Peter Zijlstra cc: linux-mm@kvack.org Subject: Tools for explaining memory mappings/usage/pressure Message-ID: <29c27dab-a590-5df2-c840-279bf9dff090@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E70C2180019 X-Stat-Signature: b7ews3i888jttg3acz9cd7ej9yya4jgx X-HE-Tag: 1720299316-899283 X-HE-Meta: U2FsdGVkX1/oeVGJv35BdEZOZtGuHIA4k+vEu50Uxxsc8JCfKm7+EZ0GEPYHuAmTYfpeVk8ahG5/5fK/SpAEhrmdd+TUVSVkwDbb0OXcNbzjEH5Tgh4q/tKrQ32IYwfyMyKoXZONK2VT5dBVuf7aZUIfehy4s4AqPFJPfQj7rW+0Mt8ixmnIxVxfMeJAlnDn76COlTR/nDFn+Iav8tQ+yF01l/lPlw39x/Qede5/My+sJ60QWaGVt5RUOp7b4ew5sy2kSuQBOG0tJ9LISgYQBz8b62+kxhAfumcZu61A7fLFFD8sF5SyWQCQnnOIQr5ERA4PoWCpUawYIxCT+CrdIeCKb1qJAb2Sm0dqQpl1WFPoCIjFUYo416USvVClWaIC3ec4N2jTUV8LxCReB4chB5r7ijLASiCvg6O7TTwxnh+j1yLZsT0a+wslFXTy4CiAUabnTyIdtwShebGeEkqD0c10GNV44kKZPBXMUdPAp1kFM6hfPJDe/vOKc0j0fxDiYz30vzdm/hvxNcHb3WHVtRg5rRTeWuPejfv0rvuSPrshXzynrTcsJnPwB6dDbILnzYTcI4i0NLrWY/5WZl9MhU6D/CapghAuwOMLhKFvhh0iCNwutu3bk1Lp8F6IdQtBEhhKBqInXocaSbHkjO3jgyvLksCtZTqYGIiny1fDEWrz3FBjtWvk8EbUVr5f31A0VQLUxhXkpCaAJAlAmtx0tOooOz1iAURRj12IRYTpgvkCeZfjJVG+fOyN7i/Kz5wUPVWN7bQ+2h5sRPah/u1E4zuBoBIIZujw2mcNgy6c88jX1KzgEmA4Yl8DZ5MuZN+QobV3yv4C3PXoFPf6OU1j2ChIx9RTznY7J5PAjKnCI7GetCqkHy5PHORd1FSMzNs0ikqjp8OknDniNlqO+cujKWt92VWP7o1RL1pGBKyBbEgprv2NQHi6UxR4CK0655ALA7lDGtUg3OlAGaGQHXu Srwp3FVv T7DrrD0PSSzfnU8Gvuh1+S2/+XY+C770usFui1XWUxhsvilhZbiSi+WMtcB9tP34mYVxflgJfpq7QAU+d/qV2g+xfTkH89jX9UTUXmBlMDLp8nsvwbglyCY7xi9uczqEsUIE025FcdRtyRegieG3q8+9cNQIjfA1Ug5NXHKkJjCgiB/1go4ksdrGso8fLnnVr0aDtWjioZ2b2FyB1DUX9fAZr/lUEJhez6eyhOoon9/rVkTlPTaRTp9B668ZBE2f+6dn0t+THBW+I7qyK7ZRqVOK7Z6TVsq/WuDJA7+oLJ7ANFonCOTmaqW1pHSOzcHDI1wUDYohBMmQku24= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001107, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, I'm trying to crowdsource information on open source tools that can be used directly by customers to explain memory mappings, usage, pressure, etc. We encounter both internal and external users that are looking for this insight and it often requires significant engineering time to collect data to make any conclusions. A recent example is an external customer that recently upgraded their userspace and started to run into memcg constrained memory pressure that wasn't previously observed. After handing off a hacky script to run in the background, it was immediately obvious that the source of the direct reclaim was all of the MADV_FREE memory that was sitting around. Converting to MADV_DONTNEED solved their issue. A month ago, a different external customer was concerned about increased memory access latency in their guest on some instances although there were no issues observable on the host. After handing off a hacky script to run in the background, it was immediately obvious that memory fragmentation was resulting in a large disparity in the number of hugepages that were available on some instances. Rather than hacky scripts that collect things like vmstat, memory.stat, buddyinfo, etc, at regular intervals, it would be preferable to hand off something more complete. Idea is an open source tool that can be run in the background to collect metrics for the system, NUMA nodes, and memcg hierarchies, as well as potentially from subsystems in the kernel like delay accounting. IOW, I want to be able to say "install ${tool} and send over the log file." Are thre any open source tools that do a good job of this today that I can latch onto? If not, sounds like I'll be writing one from scratch. Let me know if there's interest in this as well. Thanks!