From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AFD8E7718B for ; Sat, 28 Dec 2024 03:38:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF546B007B; Fri, 27 Dec 2024 22:38:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67E5B6B0082; Fri, 27 Dec 2024 22:38:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5456C6B0083; Fri, 27 Dec 2024 22:38:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 36FF76B007B for ; Fri, 27 Dec 2024 22:38:52 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6BDAD161D82 for ; Sat, 28 Dec 2024 03:38:51 +0000 (UTC) X-FDA: 82942959498.27.698CA02 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf22.hostedemail.com (Postfix) with ESMTP id 309CFC0004 for ; Sat, 28 Dec 2024 03:38:04 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=ngvg6qVO; dmarc=none; spf=pass (imf22.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.42 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735357088; a=rsa-sha256; cv=none; b=mfclOUPltoLp63Rxfzz8GEkXC26BMB+fiSu/pw+wDGHhZh5e7kZVYTgUOkxtdxL1HCNhA9 wPWve48zgEAHDocLHoKaTLTq4sNotKeeHJg59qmM/QFlD1LxXHc79WDHSod3j0rx7JPtkX vshNXNHBvPilGhuQxxt6NaU9+1CGJvQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=ngvg6qVO; dmarc=none; spf=pass (imf22.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.42 as permitted sender) smtp.mailfrom=gourry@gourry.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735357088; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZHoWyMeD0simD6rzZZ407DZSH4sL8HtAX2h8hHCq/do=; b=7FocBvNwpg+C8mD3NAxzAPPIe/9u/74ipmZfRnkbz14C7M2VB07LxYfAQSK/9cTA/KbBq5 d1mJyuxzeyZYhdv9biXq5qZCfCy2Nj04Xa3qecjGriaNX4Cid/ARC7vAtI2gBbTG2+rV5+ aZxTQYLmQ7ps4jfBoEnCkshRAcx2jzQ= Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-6d884e8341bso55938826d6.0 for ; Fri, 27 Dec 2024 19:38:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1735357128; x=1735961928; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZHoWyMeD0simD6rzZZ407DZSH4sL8HtAX2h8hHCq/do=; b=ngvg6qVOGdR27xNYBRIdcUDAIKZNemxv6h74bk7eyUiRCffYN7JH5cNkt3GextL1lC w/tDW07U0enCH5uyq2VRD+/tA6QWOMn8t3L7j7CVHu4nLABYbeBIAQAVCp5oWwRyE/OZ Ky4DQNJDc5JiZRLXsvBujEZSsoU8WpKo3i5sS1rNWtafNgHUD/xr8ydhOuvLOzxohJ0J 3Py8NWj0jbDlMsRN35OVpW2mfFnbudAR9hCjeeSIqK6P7/CqNKZtCqEa6ozuEie7CfMK 9RpjT7JUVAyTb/tMWPQ5It5IXGD6AuXAAzHJlxPfUGH7C4AqsqlEGYw1DvpybCghG0BZ +nQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735357128; x=1735961928; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZHoWyMeD0simD6rzZZ407DZSH4sL8HtAX2h8hHCq/do=; b=RSLNckj/+/DzSYV8d8ibSrD3K1wPH+ja3VB3IvPwGifcs3rInNrMvtbB5wP1wDWol6 wvEPG/eTQMV5nSVYnEHqRZGzjNnUfbRMf85jMikkwQPiYxx58y7B7P6UI1tVNsQ/zil9 MHTU3rxyFQ6QnWJpqh5UO0ciMZDY3vLIvYoFJbEso9XQwi2YwRo/0Jf47j8oXITlXC5u WdISs0HUY7hzX041O31lMCW4f2do1UMozW5Uy/WCV8ii9KDjhk2AVWJnwSHrBE+0h5WZ aBa2wEHYMBIXxPbbdjrsmc/lHskW9RKs4AwiVryPIVlkrw7JskBo2ElNT37ateuQ+3GI uJnw== X-Gm-Message-State: AOJu0YzJ7z3dlSbxvHUFt/TP8U2j9uHvLo/SShdjrunAoj5HmV23wnRl rXp//F+pO7JxomMTpPVN0WT1vt7v4GGBSWlm8VfBY1o+3JIQk4iwj5yRfook8zA= X-Gm-Gg: ASbGncvMRSo1NJQ5V9o7/2JD2gBuonhzCc032zV/AialkuRc+qMdd7TXvX/I8Nz0jkx LRxLjPEQcr8W8bFmGjNbg3UlsUmKvnrI8eSvED7xkawqVv4Nm20p1ApEiyVdCEecY+JSo4SAXFZ SkuvM3SuO6auouxsVjMSvL56OKfLnsgwF7WM/n/f2gT/klJq6pEniN6IqVbAxo2hKK+bBqCldkX WHiuJco+qGI47QkVFLyBE9l8xqI55KRBbBdEpoYgieExs6Zl1w70wMzxHbTpoVDilOKvNCevfgg U5K8eIkVEtbgHptea4PyRsHx+WPr3GXrvc/Ns74= X-Google-Smtp-Source: AGHT+IGKz6QK+/uf1/3a6peL/xApYCX6a2P3FtsHH4J1utqrJ1T20aaf3i7Y4eSSTMB1ws2RSpp0pQ== X-Received: by 2002:a05:6214:19c2:b0:6d8:d5f6:8c52 with SMTP id 6a1803df08f44-6dd2339d943mr512258066d6.38.1735357128581; Fri, 27 Dec 2024 19:38:48 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd18110112sm84089166d6.33.2024.12.27.19.38.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Dec 2024 19:38:47 -0800 (PST) Date: Fri, 27 Dec 2024 22:38:45 -0500 From: Gregory Price To: "Huang, Ying" Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com Subject: Re: [RFC v2 PATCH 0/5] Promotion of Unmapped Page Cache Folios. Message-ID: References: <20241210213744.2968-1-gourry@gourry.net> <87o715r4vn.fsf@DESKTOP-5N7EMDA> <87wmfsi47b.fsf@DESKTOP-5N7EMDA> <87v7v5g99x.fsf@DESKTOP-5N7EMDA> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: mp3z4nki7bjy8fe57n4hepfkqe531ipy X-Rspam-User: X-Rspamd-Queue-Id: 309CFC0004 X-Rspamd-Server: rspam08 X-HE-Tag: 1735357084-135762 X-HE-Meta: U2FsdGVkX18H3gK4BPF6GMu3ginP2WkGmAzO/ekRRLZZsAzj+rSMFJlJ6QsWDKuE5TKS5bbiKrUhNRqrdPnwEbOVBEIDDfQuIxLimWsZVZgW+dUmE4N2w1zm2M0klN8tY7B1en7NVd+V9q6ffg7bqiWMYJXcgHEp3qboB62G+mp0gDi24l2rL//0c+oyCWDa7hgiP/Wb3IywPhrAHM6aF8vYwsoAqQmbxyNDkrTqt42coft4pKrw37K879dcNRMQIWNOOeWGmJ2SyU27c9GAiefziwzDxlANQnnNK/hOrK/MQdkwxvuEpSZYNqR8PTl2hVLJeewUzjpqdH6mL5o9WBaMh8yW8D4STv5vDkLKqa0wfWpslOHtgJxkl5cps+fFBchHBfIh2K6956+p94/MBaP9Pe2z4zv9ogGsH0pIkEtElBQNoaVnEtWd8rkNRzY4uxEr47zYdRcKMAVxJCQ3STmkgY0SYK3FPhXqEZPfsJ19sQWg0EJTzJ0ezF/c98TmtVlFRGnQ5axSGHiOxKZiZdD09wAYR6EFxKp13IP0FF6w+d5wKpRwU1Wmm6fik2XI867TROnMMB98YTnxUHDCdkpG2Kdorjjd1uUGv2l7ucBjbXUqN4OKAqb0hH9PRekGwArFbg52GrlrOwaQaNsdi6ZzZqcB7GYuFfYAAjyc8fwcK9sBi9xnyJhhrlLh12cVDb9t1XUIB8Kjl6oWkCYlLcKLpwo3ErYtsM+FrqxNuwx0ULHnc5Yor1Blx55HBnVHpx36Z6s7DiOhEHtPn08JsHge0Q9BKlwSV8isbfT84MgiBjAS79aHhqmwSETXZUhaLXeUhrzT/2A4ISC+mci5Ds8xodgGXaeAzSvcRiPX5Gn2NJPeC9skT8pmULUNiGNDYuZF20qVT88xJ+4Te/wvDGxCKs3w1Op58pgE5fXuJF+JjCfdaI1WrR5D6wTkOHq8ntUj2/sV2KUBduRMWvj ZW3Ynddb 8kKluxt/lG8x6ER8AJXIPdVbPWseuf6FT8jH2zirM92SmXIXT3ycA1EBN+aMJ38TN04mjU/ihNXRNzgVsdyDQPpP0TSvKenA+97D0Km41sxCOHT8z7liFHqdfSBX/yYpeGV4bCvmdAjCAZbxIVWCfPt5YTDhxzY0mXsHFqb+fQEu6Oh3BaYzZhFxVRP99tc0PYpwALJjyXO8R2Or0v5rCWC2zx0ZlQ/YypvQ97PLnwks3Ybj84aFH+tSvIVcRYoGYLG3kGy6xk0mfZ1XRNwUICIU6hPCyYvRtPBnpuPZYvPnRX3WeR0aKo9HA3sMZh4r3HuKbzYH+2gjUlyM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.010210, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Dec 27, 2024 at 02:09:50PM -0500, Gregory Price wrote: > On Fri, Dec 27, 2024 at 10:40:36AM -0500, Gregory Price wrote: just adding some follow-up data test is essentially membind(1) - node1 is cxl read() - filecache is initialized on cxl set_mempolicy(MPOL_DEFAULT) - allow migrations while true: start = time() read() print(time()-start) // external events cause migration/drop cache while running baseline: .93-1s/read() from cxl: ~1.15-1.2s/read() So we are seeing anywhere from 20-25% overhead from the filecache living on CXL right out of the box. At least we have good clear signal, right? tests: echo 3 > drop_cache - filecache refills into node 1 result => ~.95-1s/read() we return back to the baseline, which is expected enable promotion - numactl shows promotion occurs result => ~1.15-1.2s/read() No effect?! Even offlining the dax devices does nothing. enable promotion, wait for it to complete, drop cache after promotion => 1.15-1.2s/read after drop cache => .95-1s/read() Back to baseline! This seems to imply that the overhead we're seeing from read() even when filecache is on the remote node isn't actually related to the memory speed, but instead likely related to some kind of stale metadata in the filesystem or filecache layers. This is going to take me a bit to figure out. I need to isolate the filesystem influence (we are using btrfs, i want to make sure this behavior is consistent on other file systems). ~Gregory