From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73364E7718C for ; Sat, 21 Dec 2024 14:49:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB03A6B007B; Sat, 21 Dec 2024 09:49:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B603C6B0082; Sat, 21 Dec 2024 09:49:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A00206B0083; Sat, 21 Dec 2024 09:49:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 824D56B007B for ; Sat, 21 Dec 2024 09:49:03 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 09A05A0FBC for ; Sat, 21 Dec 2024 14:49:03 +0000 (UTC) X-FDA: 82919247432.10.0A3B1A6 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) by imf08.hostedemail.com (Postfix) with ESMTP id 55D95160016 for ; Sat, 21 Dec 2024 14:48:37 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=hHDFpJee; spf=pass (imf08.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.54 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734792524; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5i66wyNJ4GlJ5wDCHDiVB8aDJ2Bcpwt+j2OanevERxg=; b=A+nPQtTr67TPcRPe+VhFLXSgDFEG5LuWB7eKL4uInmYxZilssjNyimXlFO6Dkhfqw8A4+I XsaLAuE4W8O2533h0VQT+xnRlm5jvnO1MwVkUrTkX/+GRR6PlhiV3L9UxnAaQoq7ZtMWLP Dwn/FwVatS+5QoT5yUMBcOzmPr7trwE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734792524; a=rsa-sha256; cv=none; b=goMklqreL5GbBHz+jZOLY74o4yMhErJzPcGTNRlLShzxhEGlpirksB6iuUGqNQDnmd30Uy 42NIHPQheNSl/BpaahWwVdLZXZomPkOHbmKQGHUIRrsThFZhRwmjNXXjZj8JmV0pA12K9M kSmeY0nC11xye04g3sEephr05GWyesc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gourry.net header.s=google header.b=hHDFpJee; spf=pass (imf08.hostedemail.com: domain of gourry@gourry.net designates 209.85.219.54 as permitted sender) smtp.mailfrom=gourry@gourry.net; dmarc=none Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6d89a727a19so35059696d6.0 for ; Sat, 21 Dec 2024 06:49:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1734792540; x=1735397340; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=5i66wyNJ4GlJ5wDCHDiVB8aDJ2Bcpwt+j2OanevERxg=; b=hHDFpJeepj6rgiUz0z8UtMglCCzpKqVDlkumxI28aRorp05vltTs3hTMjogw5YsI7z hBlZy+0ysS4a9M2Uzes12eXPhRkbbUjDMDLhKy6a7QnaGiuoNQs2XduUwXX8nwAWIEwx K+fsCK7rQAI88m2PU+zRhwUBmgKjza1IO/SgUPk21JKn9mBWGCFOdE9S0FTW7TLh+k53 NLy9TY9s6cpgsHXlqQBWiBW2n/CgRLBNF/4IlxnCjAXAUFC+3hLIWNXlQPOSpQ1A+KJD IrmelWtdh3DKHkK/eSU4//rGgPP/+JcX8Yr+y4640QloUACoboWVDYv6khxBAJRxXksQ +LJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734792540; x=1735397340; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=5i66wyNJ4GlJ5wDCHDiVB8aDJ2Bcpwt+j2OanevERxg=; b=DLQJ3F7HBm9HvOUOiJxYg2wZSNkyHQEHTQIjhXuUEzQIcKdEQO7OTk4MDHRT9GaEsx KhrLyQdb1/i+fXKWNijvqubTX1Yf6h7MvmGgCmgOqUQhSrjHiAXn/LxFThHjkrrdXbGm JNowztxmIfNt961MLrmW1qh3zaRLOPLMbk6PCusoXycLLNY85hrP2usdar/O7y86TOP0 lQcORAYqpEPKBun+j9OZ9OGepQ73GzSJNklkpapxbN87TjweDm1Ch/81BJfWsb0xrjji k8RwfI1hIwIQPje/yBu+9cjvOYt3qEk5HJhr/8hy8YeDf8sglFB5hMavcwqSTBxwR08r Lf4w== X-Forwarded-Encrypted: i=1; AJvYcCVTyZwQs4TrwOxuratojqGmkHPdZw0Ea9YTc6EAwYLjq+cy6n5IUfrVHz/HskeSCqeQVOaWaQ6nGQ==@kvack.org X-Gm-Message-State: AOJu0YyJmng+xR3AX2nTwsAe82V6E1db9Qppj52JkOJ82kDExHC+ZQA2 6xQIRQbLBG6LPXYFjW/sn2ByqOm/IA8l5k7DbxN14pT96K0ywSxnRuem5DM+mdA= X-Gm-Gg: ASbGncsdRfwRB5BRottOADgMA5xMqlinFzGv5WbLumwq5KKkXV+vrZjkCFEgI9cU84H jUoAebz9T/2vI1WEirQ9wO/Tysv9lzTYzabCzcTyDmIVSmYdMTzHHGN62N+CtwGitHmVSPIfu+J GZ/Y8vUH6ZFefyLfS+t1B2ElIeru1jXZY3nhA6ZwilUwasXtg397pWkYqj5g0jVHyUGNS6EACXR AizqELrxfm9K550bj34teusE31lELvmtMDf1X9DW4k9G5SL2ymGJxQ5JX8VmTLRxA0N7ibqgddP zkMNP3naejhYsUblge75juQQUkt+AilG7kdnOTBvMePye7AfOCvlsvM= X-Google-Smtp-Source: AGHT+IE08WY46z05aeVGGZfbqV6PpIY33PC1rYA5PmRSYOOUtDhRdu7FjN8tcxXoII4nrAfaU8n7TA== X-Received: by 2002:a05:6214:428c:b0:6d8:7a85:59ca with SMTP id 6a1803df08f44-6dd2360e187mr121468676d6.16.1734792540122; Sat, 21 Dec 2024 06:49:00 -0800 (PST) Received: from gourry-fedora-PF4VCD3F (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6dd1810d5casm26920556d6.39.2024.12.21.06.48.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Dec 2024 06:48:59 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price Date: Sat, 21 Dec 2024 09:48:57 -0500 To: "Huang, Ying" Cc: Gregory Price , linux-mm@kvack.org, linux-kernel@vger.kernel.org, nehagholkar@meta.com, abhishekd@meta.com, kernel-team@meta.com, david@redhat.com, nphamcs@gmail.com, akpm@linux-foundation.org, hannes@cmpxchg.org, kbusch@meta.com Subject: Re: [RFC v2 PATCH 0/5] Promotion of Unmapped Page Cache Folios. Message-ID: References: <20241210213744.2968-1-gourry@gourry.net> <87o715r4vn.fsf@DESKTOP-5N7EMDA> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87o715r4vn.fsf@DESKTOP-5N7EMDA> X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 55D95160016 X-Stat-Signature: roomwgx3yhey8bjrgntq575kwjpk6gta X-Rspam-User: X-HE-Tag: 1734792517-821469 X-HE-Meta: U2FsdGVkX1/yoOxPbm5pRQDDN+oC3wjb4ShgFzB16h+QMTn71Tj7DGdo+9pHnBnRyeiiMtFgxUKGAy6J1ljda1PJlJJ8z+ddpHqSdlvDQA6rzXa/idz9QL6h6G5zzK9EB2h/gPpsHVhGwJbH30zgMVEbCNxfjt3sVKe0Go2D2bav6e+lgPDHw6ogaWFcWGSAS7qjGjrKEuZ08y8OlIyBiK+hrxXI3jOjnvtFwG1FQkxOwghIOoAbSRCnbuct4HY4gxl8rk0GjHaDwl71i5/uQkiIFazyD7pgw3DXI8fBa6ycx9WXyckGr97e5DGRjBYw7+syHX5vlWN8YB8yCbD9Vnmqq2yX/onYLQVGROWtqdgaW10qSr9vMnYZdvIKUM4gtyfc4vRZuyFOWkhKG/VpsqTIYOmo+lisAEI+9v67BY1tPeCSrP/5omd9AuuCVKAZSn9o2dy4F1JpQ/AUOyv/Nmi30EVFy3qZC1VIoc4DnJR0xjDppqCyUJMU8AwMDYSLtvoHppQ+qoeQj363ll2cDMgT00BpqSr2Ff2yM3+McTvGjr+fmQ0/908QczGcS+TsyNWWUqs8Qgxl7qakZ8fqupF7/1Y7h32ZvlGWZeu03+dclgCo5fQ9UCWFIDdk58fIuw7DUYx72Dn/VSdXYEfyRBecyWnvM4PvA9vWdm7Is0NiI5xiUk3u/MV6fVcVT/mawsgDJLcNL7BXF2Iahz96yn+nvWBx0r2enXnh4wxEKJfy/Y/fap5l9nElK8KYN30I8IOPSzqfIppXlBEyPgSQGj9HVJbqymoxQKYOgRjDyDpK7gAN6rr1PrFuCnoImGKUHMOk31pHT3F4RnRsQXO05UWcHncc37x2KNR07bevkpqnQySkcW6p6mfCSo2M5phnJYEso6NNgZOQgdQqYaeW33lFpn1qc7uOOV6hA50rkTPFWNUIU8iO438iO43MhNTTvwXMOTBBoHLXhpDvFla JBo/dBwu phXZxGRXZJ+PgMsHbSZPWLM9sVFexhKUPV2ouJDdFd2roTzjo0VoyweacSEu7yTxV3MAkET7dSz9EkmQbHy+K6efq4Ub7kmQXOC6eKeZP+fadwDLG/f7fwSl1/ZX/wxouDsnRFZglYcXvn+EunuFGsHTt/NeVXNUjz1E1zV4jdq24TB/Sit+JJkC2mbzyfHJjKJWPMi5vOHZS+7ecMQJ8WkGvEIMTVX/MwWy+kN4cdHMl9q+Hgf3UIqI9f3L/FYbx4vCh2fLXbs+Vageea618mTgdLc0kqjkCVUX+QFYmHMY7VgFYAf6G5g4uFuwpmu0uNJJZaJK8yFbL0JyreRnqrsuYR8N+zgvej/t/6U/0+aXopmF8RRmvUx45XZKKvpwBRO6SWGHWkbQ0YTY= X-Bogosity: Unsure, tests=bogofilter, spamicity=0.482941, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Dec 21, 2024 at 01:18:04PM +0800, Huang, Ying wrote: > Gregory Price writes: > > > > > Single-reader DRAM: ~16.0-16.4s > > Single-reader CXL (after demotion): ~16.8-17s > > The difference is trivial. This makes me thought that why we need this > patchset? > That's 3-6% performance in this contrived case. We're working to testing a real workload we know suffers from this problem as it is long-running. Should be early in the new year hopefully. > > Next we turned promotion on with only a single reader running. > > > > Before promotions: > > Node 0 MemFree: 636478112 kB > > Node 0 FilePages: 59009156 kB > > Node 1 MemFree: 250336004 kB > > Node 1 FilePages: 14979628 kB > > Why are there some many file pages on node 1 even if there're a lot of > free pages on node 0? You moved some file pages from node 0 to node 1? > This was explicit and explained in the test notes: First we ran with promotion disabled to show consistent overhead as a result of forcing a file out to CXL memory. We first ran a single reader to see uncontended performance, launched many readers to force demotions, then dropped back to a single reader to observe. The goal here was to simply demonstrate functionality and stability. > > After promotions: > > Node 0 MemFree: 632267268 kB > > Node 0 FilePages: 72204968 kB > > Node 1 MemFree: 262567056 kB > > Node 1 FilePages: 2918768 kB > > > > Single-reader (after_promotion): ~16.5s This represents a 2.5-6% speedup depending on the spread. > > > > numa_migrate_prep: 93 - time(3969867917) count(42576860) > > migrate_misplaced_folio_prepare: 491 - time(3433174319) count(6985523) > > migrate_misplaced_folio: 1635 - time(11426529980) count(6985523) > > > > Thoughts on a good throttling heuristic would be appreciated here. > > We do have a throttle mechanism already, for example, you can used > > $ echo 100 > /proc/sys/kernel/numa_balancing_promote_rate_limit_MBps > > to rate limit the promotion throughput under 100 MB/s for each DRAM > node. > Can easily piggyback on that, just wasn't sure if overloading it was an acceptable idea. Although since that promotion rate limit is also per-task (as far as I know, will need to read into it a bit more) this is probably fine. ~Gregory