From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 123D5C71136 for ; Fri, 13 Jun 2025 15:25:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99F546B007B; Fri, 13 Jun 2025 11:25:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 975E26B008C; Fri, 13 Jun 2025 11:25:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86B816B007B; Fri, 13 Jun 2025 11:25:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6763B6B007B for ; Fri, 13 Jun 2025 11:25:23 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 17BC2C0A83 for ; Fri, 13 Jun 2025 15:25:23 +0000 (UTC) X-FDA: 83550751326.13.FE5F586 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by imf17.hostedemail.com (Postfix) with ESMTP id 18B4B40017 for ; Fri, 13 Jun 2025 15:25:20 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MuUgf2QL; spf=pass (imf17.hostedemail.com: domain of joshua.hahnjy@gmail.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=joshua.hahnjy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749828321; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nLBwQeoFr9oCLfiw/yHROpbDBnE11hBXVxYow8WnAKg=; b=J0NsoPYMTJT2u5qCTErRfVCN002HG2VzA578hnChdjCvBdcH1J8gS2tsB4B5P0xslD2dU3 Rhausl+CnOr6wozl/zlwzfP0s0adSR0dkfU4x7ozxxzOf45n6x1zZVmJFJ65LdnMANH26R p9HRhBFKVjSjjIOfP029E839HbWRAhg= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=MuUgf2QL; spf=pass (imf17.hostedemail.com: domain of joshua.hahnjy@gmail.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=joshua.hahnjy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749828321; a=rsa-sha256; cv=none; b=goVA4qjZb0glytbslpeHdC708NhaoOTWwxWiFzgcWMwSEmHCw+dqLJ4zDqIQwQRxJ0jMBE q4HlAAqOMoT38cHYpgu9c/tLrq+ZgPsgWPmpvRnWzjNOC7txhuqYWt9yUgpk8t0sH6BgUi XRoVA0Bbt3nlhonTo6xTqIBR0SXHehQ= Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-e817b40d6e7so2121267276.1 for ; Fri, 13 Jun 2025 08:25:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1749828320; x=1750433120; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nLBwQeoFr9oCLfiw/yHROpbDBnE11hBXVxYow8WnAKg=; b=MuUgf2QLw9FxTufN4aoLMSFXNMmh7GdX7YD2cMr2vpu7Mp79Jc/nvoNzAULBqlQ8e2 1P61bnch2/fT/Hq7PbSn1gdPeThIu4SIoykQ5cnixVAWufM9zIN9tCr+A3QdHcr00zsR ZIgKaJYayphGFxEZtRXJhwuHCS1020tqK/Xs7jc1pn5zzEFrGUPY5Tvw6KtgrZQp1A42 oOP+u+hb3DoAJr1U0ONx9k5TpJjEOCckjZW8SWlgRWDO3oQJKfMxGD+Rof6ZGnkKenIQ mUiOHxMa67fcUH1H9RaqVNtZ6HkaRlEpIe22FpI0X9Yt2pu1+v2czNPDJaYRSBYOG1Ro G/ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749828320; x=1750433120; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nLBwQeoFr9oCLfiw/yHROpbDBnE11hBXVxYow8WnAKg=; b=KJrNl4i/ztvOWZp+ds7rvr32ZWhDeDaPmfvtZUXTmAAQ/GcOiKkJh5tke6F/aJzkuD mnwIfJAOUo7ngzrXkrjjNrrcb2BX76KcGixla7moGpTPDRsGOV3PrRjtZqpREa53hE0V Eag65OqAgQsHBhxPP0ZFOxTCKkPeXeXFKKw9DUCXE1J/V0SYJp73nM4OF/vhb9D5C5TP e903oFM80FVl5j81XXTEz8gA4UQhJf3EGWkWrkzrYO/kqmhqYmJvtkN631mR+jjL9suo gVEKAeUrWygs9tuPcQZ1LNchj8oIuv0cc/jFPND7n17FJeZYK3bYEVhFvdAL6N8rbKB4 HYbA== X-Forwarded-Encrypted: i=1; AJvYcCWjiaiMKyiFvIbwAu/+q6hnKk7hKNJjcmjntZ5Nk7ZsK5WaxjjuLlwGsRzggB+izCkoi5UMqvBleg==@kvack.org X-Gm-Message-State: AOJu0YyhD4xaYwPf2FcAnnE8znyuWeOPJxcE7c05Cvrp2j4UmRupK1V7 zxJTYmEOoj3oYKLiz76wowhwGM/9lkNsDDEC3FUR0u8yK9nQ7NzM3+uj X-Gm-Gg: ASbGncsQ8OEcAPkG49sdoU7FUsjQr+a4WuqGUJsmbS3iLhECdbJdFQrb2g3Izw3PVgl 4mVzDti9+vrZZOFI0kNSIZgcWsIU+5BLwi7BqlHNS6uwJNIHtsZgXi/nWQv/Ickv9eJ2VByeiQj H/RYU/9o5+ehyMROB7FghNnwdsA72akxqiGd+0iOJDGrfArwI9dEHl2ZKssy6txfrPl3s822AI+ gOMhfjfMCqIi/Xgy0YSDl9VefRnERboxiq/3S0tIgnYLI8VyUEwPipyPy6LFOz30bdcVPeTWh6A 8k2wuOQFBC7lpHapJE3+OuJqu4GeU2yiwGpM0111gaaPlWNhhyk5JPaKtGV3ig== X-Google-Smtp-Source: AGHT+IHXDiLtYHaZndPHQeLdgKaaAgPSkVlO6f5fnt8QbVgvy/P9bE8XmqcR3tg67nBnU0fusVEqMw== X-Received: by 2002:a05:6902:1ac4:b0:e7b:3d15:10f0 with SMTP id 3f1490d57ef6-e822ad5f1e5mr140623276.31.1749828319830; Fri, 13 Jun 2025 08:25:19 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:4b::]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e820e0a6d0fsm1156117276.25.2025.06.13.08.25.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Jun 2025 08:25:19 -0700 (PDT) From: Joshua Hahn To: Bijan Tabatabai Cc: damon@lists.linux.com, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, sj@kernel.org, akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, joshua.hahnjy@gmail.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Subject: Re: [RFC PATCH 0/4] mm/damon: Add DAMOS action to interleave data across nodes Date: Fri, 13 Jun 2025 08:25:09 -0700 Message-ID: <20250613152517.225529-1-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250612181330.31236-1-bijan311@gmail.com> References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 18B4B40017 X-Stat-Signature: f5ndekbq4bh4a7bc8ofb4zxgyt8ar8bc X-Rspam-User: X-HE-Tag: 1749828320-221220 X-HE-Meta: U2FsdGVkX1/yw5Y64ueic9de6Bj5Ri/CKO9LFyEWfUOnhTp/Uiddfda2DkALj+GSnfNnLSn3pwuLfypET8GhYfxoxNJfbVsJznO+12KfRrQHgDkmuFwFiAkC3FMRmZL/zqT+dLvnJZ+vGL0magSuaKGgijkUpkZE8fvsZJe4mWheTqekXk9UoZhkW83lJ3YdarEzRwTr3vpZ+I07fa9vjC83Na99nMGGZ4YCJ8yxJztQ7oAY8Hqrv0VE10Hn2G2VhksszC3HnfFz0o6s+EqQv9VQ1pMzI+AUNWqTR1ZMUFkZ1ZBykqAZpyOLTDDRoZm6agIp3hllNzLTfenXJTecs5aNTQgN8JvCbmFwG+b9/YqcdCwfQTtPRy9gWwpLaTTYuynmC2qcZuU/G43Zozbdmy5a1F3rmzoN86S5VaQtuT1YSr4+bUd8Iarxsxk6IU+WJFRTvF1lhnb5IhOkovV+KiSWCvZb6f6pZ6Cm78PQPLoxzEl5eJjxz5NFpCR5juBIKbkP0PRv0VPT41AN9qdFgtS3BoZaMCYSR4qMQ9Isaubd3KdQxf88nAUPsc/VWMl+PhMX8liqkNavDzqbTPTtYSwygz/OUjFlUdZnwA4/ovbMfSUREkiMc+MIQL3grGnBQthTHTNGjwvKFWAKSDLbQODLZPIAGyeaFgdJgx8Zhq51yWI2XAla405xpOsRwEbZOaO3bHK9u7QoTeFD2nyIdry7NbLFSmcCUjECVvfvVykigcVW35hAXfgqIJZZNkJlIwDNfcEbJtMfFxurqTY1c0GZlriGidYrvEbrD1B6F2ltU1795r9rMv/vDOmFNR4wuaAQ9px6NO18UE1BFww49Ss47ztBniQk2KUvCsC6/Yh28zvQn0yHhzvYv4SZrglC2PQhJjA0QYyb6Oy/4L/Ah8j1oOcp/aZzoWgDGZMJhOt/nLkAiVHpXpHGRicpFth8Yl/7P/NSGcbpVN3s3Ek N+mvZD3S rJ9RYpV+TcSnA5tFuxSnFHYKV3rUTWYvZn8iD5DXdaAwcXQro+zGkNEUy3kBwi1X+FD/T5S1c3Y+Rq4WQmKEJ4ps9X8yftGhWyetcJM+C4D0PS9HW6+/gPQqlwayKs7glBZmHyr98dVGhYu9TQD49DRKuZlBptD8z3vesGQotI5hQYGzYuyETMwC/2a3me5Zdf0VQR60ceYTPHNVHAAVjxRO9KaDDzjeMOcgWqSGYpyrsCknik1oq3htkPVVvDMlRX4MDxZsVd3t7ykGHee+gzwRPvLuWuVc22K2hX03t2K7GFZpvBrBy68YPixzADRWB49LH9AidbIw8ytTMqMMmpOvoAMhp6Q7Kqd8dXOtoJeVeQqmMGv5WC3ZFimWGOb5nfX5oN6oiIP8Emp8NTpb2+BWHPhT4/Pp1J4AWla4TNVbMMve3ZGXm/k9o/ah1osLtEIHIxx0r/uA+vVCGeQWDAYJl/oPXt8+hBjPXFvWZ0bWwviWN7v3t79pjTSVcfjBF9kft+89qcBmbqsbwX10HncKWtYSnChq2hoDBZbF4buDWzBW4CkAOrg+HLWyxcY+/dAIGlCpbEvGAD4Z8tIsjfY4ODRC1uy7F08jWUpVBo8z/psM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 12 Jun 2025 13:13:26 -0500 Bijan Tabatabai wrote: > From: Bijan Tabatabai > > A recent patch set automatically set the interleave weight for each node > according to the node's maximum bandwidth [1]. In another thread, the patch > set's author, Joshua Hahn, wondered if/how these weights should be changed > if the bandwidth utilization of the system changes [2]. Hi Bijan, Thank you for this patchset, and thank you for finding interest in my question! > This patch set adds the mechanism for dynamically changing how application > data is interleaved across nodes while leaving the policy of what the > interleave weights should be to userspace. It does this by adding a new > DAMOS action: DAMOS_INTERLEAVE. We implement DAMOS_INTERLEAVE with both > paddr and vaddr operations sets. Using the paddr version is useful for > managing page placement globally. Using the vaddr version limits tracking > to one process per kdamond instance, but the va based tracking better > captures spacial locality. > > DAMOS_INTERLEAVE interleaves pages within a region across nodes using the > interleave weights at /sys/kernel/mm/mempolicy/weighted_interleave/node > and the page placement algorithm in weighted_interleave_nid via > policy_nodemask. We chose to reuse the mempolicy weighted interleave > infrastructure to avoid reimplementing code. However, this has the awkward > side effect that only pages that are mapped to processes using > MPOL_WEIGHTED_INTERLEAVE will be migrated according to new interleave > weights. This might be fine because workloads that want their data to be > dynamically interleaved will want their newly allocated data to be > interleaved at the same ratio. I think this is generally true. Maybe until a user says that they have a usecase where they would like to have a non-weighted-interleave policy to allocate pages, but would like to place them according to a set weight, we can leave support for other mempolicies out for now. > If exposing policy_nodemask is undesirable, we have two alternative methods > for having DAMON access the interleave weights it should use. We would > appreciate feedback on which method is preferred. > 1. Use mpol_misplaced instead > pros: mpol_misplaced is already exposed publically > cons: Would require refactoring mpol_misplaced to take a struct vm_area > instead of a struct vm_fault, and require refactoring mpol_misplaced and > get_vma_policy to take in a struct task_struct rather than just using > current. Also requires processes to use MPOL_WEIGHTED_INTERLEAVE. > 2. Add a new field to struct damos, similar to target_nid for the > MIGRATE_HOT/COLD schemes. > pros: Keeps changes contained inside DAMON. Would not require processes > to use MPOL_WEIGHTED_INTERLEAVE. > cons: Duplicates page placement code. Requires discussion on the sysfs > interface to use for users to pass in the interleave weights. Here I agree with SJ's sentiment -- I think mpol_misplaced runs with the context of working with current / fault contexts, like you pointed out. Perhaps it is best to keep the scope of the changes as local as possible : -) As for duplicating page placement code, I think that is something we can refine over iterations of this patchset, and maybe SJ will have some great ideas on how this can best be done as well. > This patchset was tested on an AMD machine with a NUMA node with CPUs > attached to DDR memory and a cpu-less NUMA node attached to CXL memory. > However, this patch set should generalize to other architectures and number > of NUMA nodes. I think moving the test results to the cover letter will help reviewers better understand the intent of the work. Also, I think it will also be very helpful to include some potential use-cases in here as well. That is, what workloads would benefit from placing pages according to a set ratio, rather than using existing migration policies that adjust this based on hotness / coldness? One such use case that I can think of is using this patchset + weighted interleave auto-tuning, which would help alleviate bandwidth limitations by ensuring that past the allocation stage, pages are being accessed in a way that maximizes the bandwidth usage of the system (at the cost of latency, which may or may not even be true based on how bandwidth-bound the workload is). Thank you again for the amazing patchset! Have a great day : -) Joshua Sent using hkml (https://github.com/sjp38/hackermail)