From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42585C0219B for ; Mon, 10 Feb 2025 02:34:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 74B046B007B; Sun, 9 Feb 2025 21:34:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FAA96B0083; Sun, 9 Feb 2025 21:34:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C2256B0085; Sun, 9 Feb 2025 21:34:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3F6DB6B007B for ; Sun, 9 Feb 2025 21:34:00 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A767E81DD9 for ; Mon, 10 Feb 2025 02:33:59 +0000 (UTC) X-FDA: 83102464998.03.3B70B04 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf23.hostedemail.com (Postfix) with ESMTP id C4E92140004 for ; Mon, 10 Feb 2025 02:33:57 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e1LE0t5H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739154837; a=rsa-sha256; cv=none; b=2+CH2ig+VzHx6J0VKXBxyUQ1mfEnx2mOxvE+shfSWzeL6XH9EPgKnT9DmPdxhvX6HF1iEB hzcQhZkg1bTIGZAp+V3I92S5TOfBXu2UPBNaSQkfzVthhzmcMi6p09dWIPUSXrd38exlwR 7i6jUrOLFttZvC9r7c4qWPRGL3hsCqE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e1LE0t5H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739154837; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2LHy9O0Q+cIdK9P5Oe27RcBt/wwySHmUVPiYjBKRzVU=; b=zkUPLEVKGrkK2p5Yd8327BvR6SBjiw/zCxmV1ocy1pKldFKpEAizzXHbTgdGdLrFg8+oN4 I59P7ydBAHaqjpMQYDOgkVr0iwYdqv4CpleUzyiRQfwmNpZyCJ8HETTjKkm4/19/VoDjcw YmP6jV9Em1YtZNE38IKx9ff7C8V871U= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21f710c17baso18619405ad.1 for ; Sun, 09 Feb 2025 18:33:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739154837; x=1739759637; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=2LHy9O0Q+cIdK9P5Oe27RcBt/wwySHmUVPiYjBKRzVU=; b=e1LE0t5HaN4PtmfDjzlY+TaxVi3OEfsvdwklCSaHY9nRHDuyePthK/2GmqdEWImoOw IJokOtCCZDizcWnO5H3VimWgxHfHWmh1Wvsmhks0zMdJ6oRuJQLgJ0emtjowsHuVh+Sr rcf0UUVX3sYKoGXGD7KoyeV/Nhqnd/KovLsOU2puMfb1xoMf3Dcc0LdPxjUBh4iBHzdD /DfM8OegdyanD7X5yrluvz8BeMMr5E1Jdkz8J7woKGAAnxzC9d7375EcBgcGOlTFiRRW OoGbC4J6hew7GvtYrSOqiKzjBbuRzQLNzswUMDiC3ieLsIJ7kOtwtagYsbV8hWtyYsLp U64Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739154837; x=1739759637; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2LHy9O0Q+cIdK9P5Oe27RcBt/wwySHmUVPiYjBKRzVU=; b=U6da4uKT8XtxijIs0FAhWB8EZwUHE957lkxUyQwADhQ8GlPBJjWhUIEHf01JRiXZBz 7JcSq2PPulf+Qg88ChVgBFD2hIGguaGb8dUzLfRIkwi0MbLfdsG2vp+DiAFfIfJ9jGW+ egmU1/EDas8462la/SvpIsZKSKUM+mbXXShQKo0s68syFXf/H2tZ28vAcHRDLJ1aKbOg NtweF7LbGymu4gZOCzangjFEwdorxiOkqgynsrB/WPodxBoWDaSZcGIHYuh6mZhN2U0A +zi7Sv/G/nNdlMW/nr+dbY53kFuBz2OF7Xuta1jAfq4SljSFr8A9sMpFFZR6cCAM3+Ej z8tA== X-Forwarded-Encrypted: i=1; AJvYcCWi46hwBEu3XKpCjTzL2sNHNG/T96aAjTeJxTY0GbTrUOCJqvNdDV09ni5UOCgl4V7Xdt/6XgH1Ug==@kvack.org X-Gm-Message-State: AOJu0YzgNgtMTbdIWhdA2UtdeRe7bfw7m+sxU7LcCgYBIBBAfkTRDR9t mujb/6/m5qZ2u8MaqyN8WJTmauBFCFD9XQWdK0CFjtO4b/KtC6FQ X-Gm-Gg: ASbGncu/tUErrhVHHg9xzWvSOmtJSN+h8BFP5S6gbI7Zjo6izJeoBb7u1C5l4vkNl+O iZnuXNLws+xM7Chq/dIqOboWMJuxDSBjjMz+hnqtbiKSZnTPQpGwESkOzcX/5N+tE7EyjRDFcgd tFDI9FQLTkLCv7Zsm+4FP+XAV2sfCG+6NWZcz0cJlaBulOwf4monUNZPtJV/EFHuMLYHPzJ9zgU yzEX9Z8LTvex1ePnEO9aBbyGrlF+tikk+cW3pm2clTNd/HXCDAofcV7SSU1TAPzjCbNHhZghj0+ xuYbnGOm6Ygtv4aa8M9dLqAHTSG764jb25mrgA== X-Google-Smtp-Source: AGHT+IGR450gXz9KJwUJnT7T725UtrcbaCiY5e5ht7Rj5/IYKCs0AAKsEkN5x5JHedKYw93ADjnHlA== X-Received: by 2002:a05:6300:630e:b0:1ee:48a:c2fe with SMTP id adf61e73a8af0-1ee048ac301mr16364304637.2.1739154836431; Sun, 09 Feb 2025 18:33:56 -0800 (PST) Received: from MacBook-Air-5.local ([210.123.245.130]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73048ad2688sm6472529b3a.47.2025.02.09.18.33.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 09 Feb 2025 18:33:55 -0800 (PST) Date: Mon, 10 Feb 2025 11:33:47 +0900 From: "Harry (Hyeonggon) Yoo" <42.hyeyoo@gmail.com> To: Gregory Price Cc: Honggyu Kim , Byungchul Park , kernel_team@skhynix.com, Matthew Wilcox , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-cxl@vger.kernel.org Subject: Re: [LSF/MM/BPF TOPIC] Restricting or migrating unmovable kernel allocations from slow tier Message-ID: References: <20250207072024.GA48419@system.software.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: C4E92140004 X-Rspamd-Server: rspam12 X-Stat-Signature: hmoyo9winp3sdaroq1mb8rukwhtpjdyw X-HE-Tag: 1739154837-749279 X-HE-Meta: U2FsdGVkX1+SKKZ6NRYqcxGUX4Mz4z7NATtylYjsJKTImjYBJJypPEL6bZdFrhMG9JnQhCk/n18e9Zpzs20Q2Gv9NuKabKxZrOSMYuIbxo0ykmbYl0rJcflyvq+E97yjFg+eOzYawYQZZEGT5c/gp8A9aCixK3mNY/i/4fESW4XbsVbgXeT7DL91F8s7VnWEBQnDY2URswnCAnl0EcX8MEqBu7hoSNwck01sV87OElGuvUIBF7tsLx3iMuTCxgzzdLB8+NpmLHAwHq/YJB8pwGYhFeoYc3FiziqAm4nyjXfyTtEtK8zspFeTtIPu30cu97c7Gbtr/9++nZmATLJV/ayHhKNua9pttUkFunwee/gnmdRUhhkOS+1klJ4tQBGaA6N5uVyIwPAhC9m91A4JxgKsT4KQ/qdh2oArfsqltqSjz+eRcUSqKxSOcOh5UKJklpDnvwxJJ5f9FEZijHF4Lcw6h1kEwiEvwdxlCtbPDTFQio1g63PULuJvxnqS2m3cmoeHPoshzKzmE9aqM167XH4ozSfrTWx2CarUFBewqWDqJYYGaQx5EGIJK8jLS4OvrROaokKRZ/dOb640ZaE9UUrd9Q6+vgaIXyt6ykepoe4l8T61GI2nSk94NE/Z5agl8/Fmpxuw3dhw6D7IQOuouk8+yiZHVEPUPi3io4NGrMzfs/2voJLgQhQTGfLQ3Vvah9sqHDMg80ghgC+SpT3bfmTrwVuDAP4cmJFrzIlQ+N4g5JdjjU7Uw+QAgi1L2o2RMW/HQFuHQf5At+iWUH7u2t1RnR00rt9izSfzO4CPtNYCIWFWusTO1f4VwKp9b5yYxrj5hfbTubA8zeyz41Ryh3AXDN07z0T9bIRcJ623CWwMSfaQhyFfDX/r1Nzf6iGvmb9WTkW6IHa7GZDCgvCZ7fVQf90/F7MYoub+iUXD1fTtyiPQpW3w8Fpb7rUxDE5NJ1rL6HWeA0GaUELtnVp UeFZZs2S S0CPN+vE6LZZVVR+nLYrnVIMUGF5wblGNEuB8BOvPGpEpiDXQSvpipTCduEMAPrhQK/2QvzUpyWQS+K4r2XMo1g0Gn94q8dYlTOR+S8fvP+D4bT2RAsCZLJ/5gARexh8TFnM9ur0HYxt3CQEaZaGMCq1caHPLRHFp+ppnZhwdSTuKDJ7VevR20Z2d3eyltSA4lumEYOyDkaobPvqJ16KrJxEcouofuuyuq+CuDs0+Mg3zEWoLV+lUdWvfCOAPw5I5WByzOSqe/1ExXHFQVZPvhSjiuKYuWmZkmkG48oBcxcNCU3sSdEjVM6DKovcVGzjMDPmXzrJQUbOrAJUqx3QnjQlgzfBmehkKQNAa4Gwhw9VHO5Bo5Pw6n9x2HnSI+eIG6fwL4Nv34q/FErkb+J3WVd0pwnVgdX7Lf2uNpWSQK6orIBk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 07, 2025 at 04:54:10AM -0500, Gregory Price wrote: > On Fri, Feb 07, 2025 at 06:34:43PM +0900, Honggyu Kim wrote: > > On 2/7/2025 5:57 PM, Gregory Price wrote: > > > > > The default kernel stack size is like 16kb. You'd need like 100,000 > > > threads to eat up 1.5GB, and 2048 threads only eats like 32MB. > > > > > > It's not an interesting amount of memory if you have a 20TB system. > > > > The amount might be small, but having those data in slow tier can > > make performance degradation if it is heavily accessed. > > > > The number of accesses isn't linearly corelated to the size of the > > memory region. > > > > Right, I started by saying: > > [CXL is] "generally not fit for kernel use" > > I have the opinion that CXL memory should be defaulted to ZONE_MOVABLE, Agreed, when the ratio of slow to fast capacity makes it feasible. > but I understand the pressure on ZONE_NORMAL means this may not be > possible for large capacities. Yes, I this is when we start consider some ZONE_NORMAL capacity on CXL memory. > I don't think the solution is to make kernel memory migratable and allow > kernel allocations on CXL. IMHO the relevant questions here are: Premise: Some ZONE_NORMAL capacity exists on CXL memory due to its large capacity. Q1. How aggressively should the kernel avoid allocating kernel allocations from ZONE_NORMAL in slow tier (and instead reclaim pages in fast tier)? e.g.: - Only when there's no easily reclaimable memory? - Or as a last resort before OOM? - Or should certain types of kernel allocations simply not be allowed from slow tier? Q2. If kernel allocations are made from slow tier anyway, would it be worthwhile to migrate _certain types_ of kernel memory back to fast tier later when free space becomes available? (sounds like a promotion policy) > There's a reason most kernel allocations are not swappable. Because most kernel allocations cannot be swapped, with a few exceptions. However, there's non-LRU page migration functionality where kernel allocations can be migrated. I don't understand why we shouldn't introduce more kernel movable memory if that turns out to be beneficial? -- Harry