From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDF01C77B7F for ; Mon, 23 Jun 2025 13:45:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43EE96B00A4; Mon, 23 Jun 2025 09:45:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EE926B00A5; Mon, 23 Jun 2025 09:45:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B6066B00BB; Mon, 23 Jun 2025 09:45:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 10D456B00A4 for ; Mon, 23 Jun 2025 09:45:56 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BB62E803E8 for ; Mon, 23 Jun 2025 13:45:55 +0000 (UTC) X-FDA: 83586788670.17.4E557D7 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) by imf14.hostedemail.com (Postfix) with ESMTP id C3E01100004 for ; Mon, 23 Jun 2025 13:45:53 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=azzbdqMN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of joshua.hahnjy@gmail.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=joshua.hahnjy@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750686353; a=rsa-sha256; cv=none; b=XppwzZcvmdWOKiUs3ZdArRHxyPBCJ3wmzCC7eloAdEdBaZfPzGJGcQ3NbwbCcyiBit6Frh MsOYhgtNWZYcoDpS28swF4cTBRnG5tAPf7/7vDgscol7LNcigLlkylBprDMx1Gb3Fzlyip SwTc8w3WaFiH6jv4hFzrRYiTYnPqeXc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=azzbdqMN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of joshua.hahnjy@gmail.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=joshua.hahnjy@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750686353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zSfF7un8LIQsGUb2HSbfU97hICuhPSaYsldpijdfOsQ=; b=B8Hcj5RMD3NYIJfI+Vxco8iOClEH4N9PmQyZVAAtE6LX+mDQdIkblHBDZR1YrjD5tua6Q2 2sediQ+aOfhW6XnKH6bAqUJ12q9irMdGnkywB0LBlrwz2Ruez2wCWcUhlWPwu6FQYbCoRz SDx8FZ2fnFI2WCIL1WR7FvJ0S25k1rc= Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-70f147b5a52so30777217b3.3 for ; Mon, 23 Jun 2025 06:45:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750686353; x=1751291153; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zSfF7un8LIQsGUb2HSbfU97hICuhPSaYsldpijdfOsQ=; b=azzbdqMNjyO6h/0lwnAUPFOhyvwhxyOMU6UndesW4DDZHZhsM0m8ZcvmU5bbcAUqna ohCfUD4oi3X5PbKAkYzwURFxikDxzZ8d9vcolJSud1oQESw5IlJLxDhc2gUC75BJtwZU Y67i8cFjwDVelnDYFrGHsLR9Wq2ktCndzYRTmye/40Sw88Yo6fXk9+fdqAyk5nSkyDuf vA0b99txGPk+0fv5sJiz2v7ECFvs69DUPfyr3fZCENHp1GCLTmgkKT1F72b6hymCHmT2 KfXWWXPjwOK4NdwnF/KSfAZ9HwGPN9dttzT+NFoE+LZKOZteREUnxyofKyM8twpBUXMC NHjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750686353; x=1751291153; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zSfF7un8LIQsGUb2HSbfU97hICuhPSaYsldpijdfOsQ=; b=mvMkFQZNOekvP94OmmEgB/F617pjewWDMtb0r0aXa6sMP0vGnWCf+3QI1tso/JtUjW sLlhZyDUiroWASmh6B8WyIskRRJw3/w/Bka/6Zb+UYFZEUAVjBy0sM8F8LSyQcI2cXd4 TUAZhZtx8q4S2G3sJXrS8uM8pyVYd70EX1UYN1ZFbZtWL7P46ArMOkCyDaHMS0CxY5Ke 0Wtz3VVmUhqtu+hZf9A/nO6sMd6EbjAzBOEI4vAufw5rgHk8jGBOSCkFYAfx7vud1I79 ItqCbN8259n5e+vzpbPdc8c/j4R+CwN/fln4dJ/1g7GyMAUB8UImLt5JR2531w1uoSbG Oucw== X-Forwarded-Encrypted: i=1; AJvYcCU72eV8IydQS9Nmc+Aa7gXdOfUwoMEGqXORvLpXWgqs3vDqmQnWgADHl+J7lf23s4L/9oXZP21TPQ==@kvack.org X-Gm-Message-State: AOJu0Ywf/fF2DyOjAHc4LJvw1Wbm1fTW9rjU+P4oYi4BC9vL1uzST6BF KGd/T49QfGewf/hWdfTI2Ut0m8qnVxdgBNyETegNZclUQg8NcBQsndwG X-Gm-Gg: ASbGncuByJ9RUxy/uaDZIqW2dcCNPf64ZSBtXGYVjwrn5G/3P8Jw+lNGa+5CStt3pHE DwhWdqyOxDdI5WIpbuK2y7caDpOrHTwGkYxy9e35QOSH5nplNZLzlIPwmxVudoBTOjn6SwrnTvd xPHmXYxuNz+gLDAnFOTlUUT/wyb8AV147cPSyXs+8iZOB1jp3sEIKiu14N4uqbCImw6pXP0vK7k 0vKuIb8tDqbvHcm43KPHaGuV6LrpKPBgMfbVBziK6O2TEcv00oPEe7b4bGmZmMIHq73xoC4IvbX 4mdO8mUJVEMTuN6H43Ij4VzLDKW/V11Vik0XUg+WSZmhXGp+Ne6jDdUn5C3GsA== X-Google-Smtp-Source: AGHT+IG38eDF3uDJCo+9jSrUa+GIdCMuLehbCwzqrSlA2tsKSYXh5jQni+GVNFRrtzz7ofvR3E00YA== X-Received: by 2002:a05:690c:7345:b0:70c:a5c2:ceed with SMTP id 00721157ae682-712c65128a7mr173679777b3.25.1750686352703; Mon, 23 Jun 2025 06:45:52 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:50::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-712c4b96502sm15855337b3.79.2025.06.23.06.45.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jun 2025 06:45:52 -0700 (PDT) From: Joshua Hahn To: Bijan Tabatabai Cc: damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, sj@kernel.org, akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: Re: [RFC PATCH v2 0/2] mm/damon/paddr: Allow interleaving in migrate_{hot,cold} actions Date: Mon, 23 Jun 2025 06:45:49 -0700 Message-ID: <20250623134550.2367733-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250620180458.5041-1-bijan311@gmail.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: gdjxd9pyq9xmnxdfmjaywyzk5f13jmy7 X-Rspamd-Queue-Id: C3E01100004 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1750686353-626977 X-HE-Meta: U2FsdGVkX19Q41O57tMxkxCYDjOjGHQhJcGWhqY6WIWLjgapOTaoMCBnxe2SAeY+h5mijDtVx5ElqrmyW+OP1zaMChwJh4h7qcVRxQ5y42khPe8JOb/OkUGfm5loAqeQ35RPygcHwzfmiCxJ/Diwp0F0z95XfMYfEs2eBLkHUG+cw4wVRzNIfE+Y/BTLuxUpql7Gi4dPnAKQXAbA4nfSsevGSYfOrRrcT9pRsXgs+8xZirv3lYNs/AE6VQeXKRK32+rSsZBysUTAXQMXCmE5y2hTUufh6MaQ8sjKCsHnu3j6q6ErOLkbFgu0xPRFSQWq3uBWRwEoomFyVY+/awBenyYCtLEqOcY0tIOskcFF/HwXz0oAuyF/3M1oIUO9FZSYuSe2gMO/L1M20ZAUapn+oInHIUIvgdelQ1Ebz6nte4GjDFJNvO+uOtI7Ng3tpUzCe8C3g2uMe7VaP4HblzlEMzY2XIssmfAkJehRKuy1XjMiJ+4pvdWTcFJYe5T+i5DBgqNXqjXBMI8FkMXXSZd0kGnTeT7EEBXOdcI5/k2ZI7iP46H+fNmRXOUCNYCJhsu03tUo815pz5b3Q8oEjg/AMSoQ45078+7yQdRfAKXYh5jIW+hOyUwTlcWT6LhiTnqzFMRNUExEyZEnzMaeG4iycindRwQl12sproRIa+6lnh+1wPffx2w1pvl3eBVpP3Hrnd5stwKzMGVFDsi5gyKfNKRdmUpWuHKgbHxck0sPaMQdoE6qdSbvB4Ytp2kaBfofMNpu/h7mm+GJ/5SH2Xs/G21FsJV5lnN3x58MSp5fLcQsS9Yyd3nKPbgEPFSDerxGl4jKtKQAA0kE5uwA0V9EWwkeywKJkRImFCUUvOKwheKTQVr50oUV3Li4w4XHJU2LMmoreOoZcvcxOQgaofxhek457dYsG4kRu1ieWUQbfF4/o3fOPhPpVCwC8F1GIgQQ8ZsXI9EA7CS/0XLgkkn DOPtKN7y tU8sAkTKpfWyTsPhwQSe6ynAYCE1+IqoCjEU6nFOaJ2Ows6DKzw+RfOEkWCyJpipWpiUib0BZjuTz+GWkUDDpRi+E6Ab/kKebHNP/CxzhNp6FfPxnB1J/+CamM7mbY6HrUNG5+5ZrF8mB/UhhecIfxGD3+Nm4DeZ/RvrU8QS6ukGf7/MSTUBqoddOSeLnJCjT/e9ei4DJeYpl2vtA3Vj9NZDYQsY4rfg/aihTi61Pq8irufJhrGwroU0w6RgKpL2IlEmcjTlhEXfByRHxFrbTbBRd/exHfk9NEhgcVRo4iWHN0P4yuby8oYrLaVaTs2JVHHNSmpWV4IvKHTdVfmwhko/LCYVtCj/um2QLwAwapLnsyNz/3yqRzRFDlRTRmp/WsJIyL7JaziHGrj2WnFhQCxXFK5XprKQqx2jiaOoRhkNM0uwwOrS40ThGGLk1DZjAqy6IlkExuTj0i28oVYgnvG/gDMXN4Gb49b4PJ/mjG+XSLicv2EvGBakWRZEQcG8t03Y2698hsj+l3UIQ4Cf+bfreD320YyJX7KiwfZ1ofTUuih6sMo5fxo1q+TAimbRYJTgxY475TXokJy69UUpn1HtQUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, 20 Jun 2025 13:04:56 -0500 Bijan Tabatabai wrote: Hi Bijan, I hope you are doing well! Sorry for the late response. It seems like SJ already gave some great feedback already though, so I will just chime in with my 2c. [...snip...] > However, currently the interleave weights only are applied when data is > allocated. Migrating already allocated pages according to the dynamically > changing weights will better help balance the bandwidth utilization across > nodes. > > As a toy example, imagine some application that uses 75% of the local > bandwidth. Assuming sufficient capacity, when running alone, we want to > keep that application's data in local memory. However, if a second > instance of that application begins, using the same amount of bandwidth, > it would be best to interleave the data of both processes to alleviate the > bandwidth pressure from the local node. Likewise, when one of the processes > ends, the data should be moves back to local memory. I think the addition of this example helps illustrate the neccesity for interleaving, thank you for adding it in! > We imagine there would be a userspace application that would monitor system > performance characteristics, such as bandwidth utilization or memory access > latency, and uses that information to tune the interleave weights. Others > seem to have come to a similar conclusion in previous discussions [3]. > > Functionality Test > ================== [...snip...] > Performance Test > ================ > Below is a simple example showing that interleaving application data using > these patches can improve application performance. > To do this, we run a bandwidth intensive embedding reduction application > [5]. This workload is useful for this test because it reports the time it > takes each iteration to run and reuses its buffers between allocation, > allowing us to clearly see the benefits of the migration. > > We evaluate this a 128 core/256 thread AMD CPU, with 72 GB/s of local DDR > bandwidth and 26 GB/s of CXL memory. > > Before we start the workload, the system bandwidth utilization is low, so > we start with interleave weights biased as much as possible to the local > node. When the workload begins, it saturates the local bandwidth, making > the page placement suboptimal. To alleviate this, we modify the interleave > weights, triggering DAMON to migrate the workload's data. > > $ cd /sys/kernel/mm/damon/admin/kdamonds/0/ > $ sudo cat ./contexts/0/schemes/0/action > migrate_hot > $ sudo cat ./contexts/0/schemes/0/target_nid > 0-1 > $ echo 255 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node0 > $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node1 > $ /eval_baseline -d amazon_All -c 255 -r 100 > > Eval Phase 3: Running Baseline... > > REPEAT # 0 Baseline Total time : 9043.24 ms > REPEAT # 1 Baseline Total time : 7307.71 ms > REPEAT # 2 Baseline Total time : 7301.4 ms > REPEAT # 3 Baseline Total time : 7312.44 ms > REPEAT # 4 Baseline Total time : 7282.43 ms > # Interleave weights changed to 3:1 > REPEAT # 5 Baseline Total time : 6754.78 ms > REPEAT # 6 Baseline Total time : 5322.38 ms > REPEAT # 7 Baseline Total time : 5359.89 ms > REPEAT # 8 Baseline Total time : 5346.14 ms > REPEAT # 9 Baseline Total time : 5321.98 ms > > Updating the interleave weights, and having DAMON migrate the workload > data according to the weights resulted in an approximately 25% speedup. Thank you for sharing these very impressive results! So if I can understand correctly, this workload allocates once (mostly), and each iteration just re-uses the same allocation, meaning the effects of the weighted interleave change are isolated mostly to the migration portion. Based on that understanding, I'm wondering if a longer benchmark would help demonstrate the effects of this patch a bit better. That is, IIRC short-lived workloads should see most of its benefits come from correct allocation, while longer-lived workloads should see most of its benefits come from correct migration policies. I don't have a good idea of what the threshold is for characterizing short vs. long workloads, but I think this could be another prospective test you can use to demonstrate the gains of your patch. One last thing that I wanted to note is that it seems like iteration 5, where I imagine there is some additional work needed to balance the page placement from 255:0 to 3:1 *still* outperforms the normal case in the original benchmark. Really awesome!!! > Questions for Reviewers > ======================= > 1. Are you happy with the changes to the DAMON sysfs interface? > 2. Setting an interleave weight to 0 is currently not allowed. This makes > sense when the weights are only used for allocation. Does it make sense > to allow 0 weights now? If the goal of 0 weights is to prevent migration to that node, I think that we should try to re-use existing mechanisms. There was actually quite a bit of discussion on whether 0 weights should be allowed (the entire converstaion was split across multiple versions, but I think this is the first instance [1]). How about using nodemasks instead? I think that they serve a more explicit purpose of preventing certain nodes from being used. Please let me know if I'm missing something as to why we cannot use nodemasks here : -) [...snip...] One last thing that I wanted to note -- given that these weights now serve a dual purpose of setting allocation & migration weights, does it make sense to update the weighted interleave documentation with this information as well? Or, since it really only affects DAMON users, should we be ok with leaving it out? My preference is that we include it in weighted interleave documentation (Documentation/ABI/testing/sysfs-kernel-mm-mempolikcy-weighted-interleave) so that anyone who edits weighted interleave code in the future will at least be aware that the changes they make will have effects in other subsystems. Thank you for sharing these results! I hope you have a great day :):) Joshua [1] https://lore.kernel.org/all/87msfkh1ls.fsf@DESKTOP-5N7EMDA/ Sent using hkml (https://github.com/sjp38/hackermail)