From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D3D4C7115B for ; Mon, 23 Jun 2025 14:57:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8B336B00C2; Mon, 23 Jun 2025 10:57:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3D836B00CD; Mon, 23 Jun 2025 10:57:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2A836B00CE; Mon, 23 Jun 2025 10:57:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BD3056B00C2 for ; Mon, 23 Jun 2025 10:57:49 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 73F9B1D7095 for ; Mon, 23 Jun 2025 14:57:49 +0000 (UTC) X-FDA: 83586969858.21.937400E Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) by imf10.hostedemail.com (Postfix) with ESMTP id 83FC3C000B for ; Mon, 23 Jun 2025 14:57:47 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=E8rfcjzz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of bijan311@gmail.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=bijan311@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1750690667; a=rsa-sha256; cv=none; b=Lo7xEwTDEVh1h34lTCP0y/wDIqpsarMKp8tjJQdYm6+zlQz/NwzrSOu2ae01u4tCj72hj/ jA9Z+GyZHMO/jcYOVUh0WbMqXY6ihoYvxuc2sun32DfPvPx9k7pi8f29l93/MxRJgPm1/l YZ0E6T9PVJVQR4JvQYVn/luKZ0FkOu0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=E8rfcjzz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of bijan311@gmail.com designates 209.85.208.46 as permitted sender) smtp.mailfrom=bijan311@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750690667; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pu9Ih/QcFbUQzus3T5HcRHRjyftZN8WMC5c94j7UZRU=; b=oLYIl3CP7i6rhJO4cQK4tFcs7K09gWuYv56dkEI4IQcgtYoDqxRscpm3roQ3dgzLdkn58V 6z7X0s3IwKKF3eZF6RWUQz7k/aBamxdsDS/nfZMfFvSXqYS8mm96cz8DAnLJQfl8xLiaXU qfQ8kHSkgjfzk7Q5gL2UOI1iYssn8Eo= Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-607cc1a2bd8so6487293a12.2 for ; Mon, 23 Jun 2025 07:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750690666; x=1751295466; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=pu9Ih/QcFbUQzus3T5HcRHRjyftZN8WMC5c94j7UZRU=; b=E8rfcjzzLJ92hW//wNTuhX+rNgDZFuE3TWwEMJSMwTZCX4dsNCsj3q2k0tJfsJdYEP KVXUzjngmG5p5srLK6HmRJcOzINM0LGY51l8FYCvkCNeRVuUOOQwnpPoODF7WD1f8tfc duTOmSioJAT2EGhgPWSZW13z/GAWfnZEdzLhw3eLoc5RVWccgx47cCA3jfwvFqwlij/p d/cFp20wVpz9AMEjY4uhT4xfVuV7sUuRIAW17HQfUiBLl/9CprbSwoL2RFlW7mAH7NuO Z9CiKW4sRMWeg58j9r7q2yoAAd7mX3PQxez8RKz5ZACvwdaSdHcHCSjpVqUaBGhS8hl0 H7cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750690666; x=1751295466; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pu9Ih/QcFbUQzus3T5HcRHRjyftZN8WMC5c94j7UZRU=; b=aLIJsBC5u4FTBIG4sSbyZpxQXFZPwR8Bgj8T1CEUXazWqJkEqS45dwOE3B0NYxXMe2 +NxoAykujqS2d8c0wCW/vi/Lq15Hq3aReguEftgP6/UNn0++M1q8JMjjXtTq/15mASrE Vd8emxzJABH9BWVPKOjH27zOLjfQBbxX6j7VvIZ/mul1P+7WQdS2GYX+fk3yLVc4gZy5 0Bg0L0ZsseDLBJfc463Pu9rRly2VnyGv0li0IPTdj2TLUjooBoCM998YvkhUZsL8kGU7 r93TgzjhavjEXU+SprbnlL3WitVoy3KUELztWpV6XJeL9DF01KURsckjk75IPKOWKZ3B 9O+g== X-Forwarded-Encrypted: i=1; AJvYcCWv4GNJ4FCIUQRur6gRnaRvUS24vz3H0X05IV3A/T+hma559TlVzU+w5qeSqQG/4Ayu/tN3Y+aOww==@kvack.org X-Gm-Message-State: AOJu0YzgmXmYCj2w5D9jFIDJuIwT5rgWSbOPUfvw7/80xW6YXU/I045r WJA1sn5fAK19ypNymd0DqcLfRyVRPTMyIzYLfwjcnZsLsn1gpA92skrqGUXW+ryyNo7LUHTr/Dt TA9BGg08ldd2fsU7I/A88G2iJKtR8lQg= X-Gm-Gg: ASbGncvxOiOX0zGslHgEpPd9sXI2M0kqdjnaInhPB7V5mQXKdk8D+7sprjjWdHKo5CF dvcR0fDqsyzbyK9RLM1HRzfGvXrxfriQYZxJYlWVfX4fYxqDiKZkJjTJnjewcDwmvYJAt1scoE6 ZB1zsmQyyh65xQnBcynKg0kUPyYoktWpYFcwlraCuvKygxdlywBrCzB5Zlr9+8B/xrbZ+QeQa/B fN6tA== X-Google-Smtp-Source: AGHT+IGXKKO40ILnL3YoRR5459rm4GnrhaPtevLb3eJiUTwSri6c4aiU0gObCtK6uhbrpKoJuoEo7jlLWoxV+Vz7iNo= X-Received: by 2002:a17:907:7f29:b0:ae0:54b9:f8a with SMTP id a640c23a62f3a-ae057ef74ffmr1276581566b.39.1750690665601; Mon, 23 Jun 2025 07:57:45 -0700 (PDT) MIME-Version: 1.0 References: <20250620180458.5041-1-bijan311@gmail.com> <20250623134550.2367733-1-joshua.hahnjy@gmail.com> In-Reply-To: <20250623134550.2367733-1-joshua.hahnjy@gmail.com> From: Bijan Tabatabai Date: Mon, 23 Jun 2025 09:57:33 -0500 X-Gm-Features: AX0GCFuDT7kRq5r15XheIRRwt9P0C3EBB7kazhFRcdKb_Nq7cSmIIMffkqiCDis Message-ID: Subject: Re: [RFC PATCH v2 0/2] mm/damon/paddr: Allow interleaving in migrate_{hot,cold} actions To: Joshua Hahn Cc: damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, sj@kernel.org, akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, matthew.brost@intel.com, rakie.kim@sk.com, byungchul@sk.com, gourry@gourry.net, ying.huang@linux.alibaba.com, apopple@nvidia.com, bijantabatab@micron.com, venkataravis@micron.com, emirakhur@micron.com, ajayjoshi@micron.com, vtavarespetr@micron.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: acjr9fo7awd4rqkozegf5cg7g1eyg1qc X-Rspamd-Queue-Id: 83FC3C000B X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1750690667-114765 X-HE-Meta: U2FsdGVkX1/cFKO8Ge7wqYRXS73b4eX781t9hIFMQnV0gEMGs1VfRsZNYZlV4QDLcjNMWHW2g7xObMkqS+g1Pdw7ny1tbE6qH0AtJZEYTSszwQfUE0+AMPLCyz/HTWkEEXaifPQWuwigrAat+SiIudo4DDHAJzLemkTqn7oA+5MHuI/Sliw3Io1bkX5VtJEQ/C69dcgCe+IbYHjRx+EuzVg7iiMrqW3hbEi3pXIJy6FUb5bnesrbYEWxB337z9O8ntH7y9qFjVjdtwEmLU00TYfyBBmlCSs4ald57oMYJ6zmaovQrF8zJ+LP1NUriFsyFjqIm+aulKklKidrsujkp4UPPmpXhBvVPGZTAVcNl7e1ppww6R0FPqE4wvvSRryO9y1ejjRu4pbsFcY5gOVYktxmC2tCgnIzNY2+WuMtUjCJHGFNzpzRfL/mziFWDJGrjTd4pK++vPwOdV0q7OQurwFV8QeWGhsolxrciXdjpd0OCTpvG9lMuc0Okhz2HfiQWazJPRDpEZ/fPk20yKrgAe79dkYij3cr4/XSO+MnoBZP36KGBc8MeH39vzmSasOciuunDpfA25RYN9aRAacQbMK3943biC9Q7zK2l5Nke6u8/zAdZvLh/KsWh4dvXsAFzIWnITP07fesiQ0qeDEG6BnlVPEXEtY4gdvt77ZPEfIQZZaAeFN7CpuV1bxUpRwm55Ky2nugtC/SeDoHNQZE4eYEBvuyvgSXJRrQfomVhfs0vw66U+Zn3tR2/Cjup+Ny3ngDC+l2/V7R4qAoLh9O9CndyOdBVPlfw4Vncqpyik+PdWzC+JzC3gsqOWzjT8LkaTIiv/sA1tOkHd/haBoqxN3byVE5TmelzY6weR2lbNxiDRB7cBtkOPSVbXz/zh0IbaGbiarieWPkZ5dRIwXeqCWhsZc/kxiwBeULojIRWk/yvVBbGXi0Xj+Vx2Kg/271CwbBxJbdUWDKNDss/mc +V8tv9/G 175a6VUbCPyED1nd7pcK363Q5Tw1ZrknV8dhvNZdjIk7au03z3gQOFrHCKKVYGDUZwooYCYcalS5nOAUQv82NwpbSGrbKrpaSsA6kFIwKqN+tl3rQGVCDEPpYRX67YD3EQsXAIqVgIAB8AUOreEvvcQqbJd/XZksEF9dY9pRc9uIIgmqQ02HZqzyaUvt4q5ffniGAjevfIA+Lo42mBF1yUp4ItdEoTRC0o36MJm6ZLYRuOJQMQ++kGkdeoZztmJvocWwG+OySuD/rUOfEL2SQBZfnouEJ7Agkreuzdlx6OPbIexIFdwEsSiPLDvavk/aCMlQInxVxlcIXFf3/a33Ma4276c+hKpB1UyRayOFqM0FtbUZ3lOeIyfxNgZS9pK00+/XiE6W0yzKcrbarXxjQOBWZc2P9JWrHcWlpcpAg+mQiM4y2P0FVouEcrxPHoUa+HqseHVaD6lJTBd/KZopa8+tsBvj4Qhd+57RxTcI8wWW7QGM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Joshua, On Mon, Jun 23, 2025 at 8:45=E2=80=AFAM Joshua Hahn wrote: > > On Fri, 20 Jun 2025 13:04:56 -0500 Bijan Tabatabai w= rote: > > Hi Bijan, > > I hope you are doing well! Sorry for the late response. No need to be sorry. I have no expectation patches sent on a Friday afternoon will be looked at over the weekend. [...] > > Performance Test > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Below is a simple example showing that interleaving application data us= ing > > these patches can improve application performance. > > To do this, we run a bandwidth intensive embedding reduction applicatio= n > > [5]. This workload is useful for this test because it reports the time = it > > takes each iteration to run and reuses its buffers between allocation, > > allowing us to clearly see the benefits of the migration. > > > > We evaluate this a 128 core/256 thread AMD CPU, with 72 GB/s of local D= DR > > bandwidth and 26 GB/s of CXL memory. > > > > Before we start the workload, the system bandwidth utilization is low, = so > > we start with interleave weights biased as much as possible to the loca= l > > node. When the workload begins, it saturates the local bandwidth, makin= g > > the page placement suboptimal. To alleviate this, we modify the interle= ave > > weights, triggering DAMON to migrate the workload's data. > > > > $ cd /sys/kernel/mm/damon/admin/kdamonds/0/ > > $ sudo cat ./contexts/0/schemes/0/action > > migrate_hot > > $ sudo cat ./contexts/0/schemes/0/target_nid > > 0-1 > > $ echo 255 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/no= de0 > > $ echo 1 | sudo tee /sys/kernel/mm/mempolicy/weighted_interleave/node= 1 > > $ /eval_baseline -d amazon_All -c 255 -r 100 > > > > Eval Phase 3: Running Baseline... > > > > REPEAT # 0 Baseline Total time : 9043.24 ms > > REPEAT # 1 Baseline Total time : 7307.71 ms > > REPEAT # 2 Baseline Total time : 7301.4 ms > > REPEAT # 3 Baseline Total time : 7312.44 ms > > REPEAT # 4 Baseline Total time : 7282.43 ms > > # Interleave weights changed to 3:1 > > REPEAT # 5 Baseline Total time : 6754.78 ms > > REPEAT # 6 Baseline Total time : 5322.38 ms > > REPEAT # 7 Baseline Total time : 5359.89 ms > > REPEAT # 8 Baseline Total time : 5346.14 ms > > REPEAT # 9 Baseline Total time : 5321.98 ms > > > > Updating the interleave weights, and having DAMON migrate the workload > > data according to the weights resulted in an approximately 25% speedup. > > Thank you for sharing these very impressive results! So if I can understa= nd > correctly, this workload allocates once (mostly), and each iteration just > re-uses the same allocation, meaning the effects of the weighted interlea= ve > change are isolated mostly to the migration portion. That's correct. > Based on that understanding, I'm wondering if a longer benchmark would he= lp > demonstrate the effects of this patch a bit better. That is, IIRC short-l= ived > workloads should see most of its benefits come from correct allocation, > while longer-lived workloads should see most of its benefits come from > correct migration policies. I don't have a good idea of what the threshol= d > is for characterizing short vs. long workloads, but I think this could be > another prospective test you can use to demonstrate the gains of your pat= ch. You might be right. I'll try to think of something for the next revision, but no promises. [...] > > Questions for Reviewers > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 1. Are you happy with the changes to the DAMON sysfs interface? > > 2. Setting an interleave weight to 0 is currently not allowed. This mak= es > > sense when the weights are only used for allocation. Does it make se= nse > > to allow 0 weights now? > > If the goal of 0 weights is to prevent migration to that node, I think th= at > we should try to re-use existing mechanisms. There was actually quite a b= it > of discussion on whether 0 weights should be allowed (the entire converst= aion > was split across multiple versions, but I think this is the first instanc= e [1]). Thanks, I'll look over this. > How about using nodemasks instead? I think that they serve a more explici= t > purpose of preventing certain nodes from being used. Please let me know i= f > I'm missing something as to why we cannot use nodemasks here : -) I think since we're moving towards DAMON having its own weights, this would only apply to mempolicy. Changing an application's mempolic nodemask would be nice, but based off Gregory's previous comments, having something outside the application change that application's nodemask might be a bit difficult [1]. Also, I think it would be easier to change one weight rather than every affected application's mempolicy. > [...snip...] > > One last thing that I wanted to note -- given that these weights now serv= e > a dual purpose of setting allocation & migration weights, does it make se= nse > to update the weighted interleave documentation with this information as = well? > Or, since it really only affects DAMON users, should we be ok with leavin= g it > out? > > My preference is that we include it in weighted interleave documentation > (Documentation/ABI/testing/sysfs-kernel-mm-mempolikcy-weighted-interleave= ) > so that anyone who edits weighted interleave code in the future will at l= east > be aware that the changes they make will have effects in other subsystems= . I think if we continued to use the mempolicy weights, it would make sense to document that. However, it seems like we are moving towards using DAMON specific weights. Thanks for your feedback, Bijan [1] https://lore.kernel.org/damon/aFBXuTtwhAV7BHeY@gourry-fedora-PF4VCD3F/