From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75F18C4332F for ; Wed, 23 Nov 2022 22:30:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC0556B0071; Wed, 23 Nov 2022 17:30:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C708B6B0072; Wed, 23 Nov 2022 17:30:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B11C66B0074; Wed, 23 Nov 2022 17:30:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A34606B0071 for ; Wed, 23 Nov 2022 17:30:33 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 73A18ABB8C for ; Wed, 23 Nov 2022 22:30:33 +0000 (UTC) X-FDA: 80166152346.15.23788C1 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf25.hostedemail.com (Postfix) with ESMTP id D8086A0010 for ; Wed, 23 Nov 2022 22:30:31 +0000 (UTC) Received: by mail-qk1-f170.google.com with SMTP id p18so13475748qkg.2 for ; Wed, 23 Nov 2022 14:30:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=2PsyXXA9CudTt1GF7QXo+ry29gXPUEu+QzCWmZI6Zig=; b=XMfsHKwK6gxd9tt2hChYTe5pQ25W+G022dNNp25j8jGYTdJNaF40WrrHzq8fOVjvNQ 6oCZ7b3jr+K2sYCzBFxmbnk24Ep4ujNlf4doTSuQkHNDrqeL5ghS/4l4Xl148Unxbptt iHkxYQ0vlJAVtsjKS2tOCk6WFNQGGe/k97tPlHGzXftZ9/eWz1QepcoTWTyk0oklY6S3 h+08Iq4aqap3gfcMhOfLp5DkiOX/fswAoTFAJQKh2cvH3o9OTCQAUIuAGNpMkSmNnicN S2SqUR1boFkW3FREVyDD/Sx6b827THmq1YMvB30nC/aeRC118qw2xmUFWjVSv1lxqE1J J+AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=2PsyXXA9CudTt1GF7QXo+ry29gXPUEu+QzCWmZI6Zig=; b=eV7RUq2lwjij5wBz1wYsGjmKsWRhCjQHCEQ3wdcvVYqDM0NdejIYSbcCMJhCzn0D3J bs1kLwKQtBQVafSykWlJBxXcZzmEW99ht63/CN7jurvMTsA2WqMRkYpFdAeZrbyEgH4l y5h4Px0RHS/RNUeQZPBEk5bGor7aUrycT5iokqrygQ/ZXAW7VG+quwXtoHQH3YEKroXf tm3+J54vsPRjo544rwMRSBTfVxJPPoL+9Z6uUJUJjTo77qnUUqigNl71oMZO5qG7Yd5J JmiXaR6SunF+D1Q4c0F7y7ovy6eBTWKHBL2KiZSI2IjPFliTOAgwYYOfYPSx55+riU9K B/kg== X-Gm-Message-State: ANoB5pn07D84sc1pQmBY8fK+GfdKtMZIWe71dIYdmqkgPRgzFO3rJ1VN c8Y1fAarcWQWaxDacmUP22I6I+19NR6MBw== X-Google-Smtp-Source: AA0mqf7i13+gr+nV4F+Pz1qThWBzQ5ZvPLsujUe0vIKUyAVKNbj+hpveo1MQ3Y6HRoXfeC9fe2BM7w== X-Received: by 2002:a05:620a:1310:b0:6f9:ffc6:45d1 with SMTP id o16-20020a05620a131000b006f9ffc645d1mr18525732qkj.663.1669242631003; Wed, 23 Nov 2022 14:30:31 -0800 (PST) Received: from localhost ([2620:10d:c091:480::1:bc4]) by smtp.gmail.com with ESMTPSA id fg22-20020a05622a581600b003a580cd979asm10695213qtb.58.2022.11.23.14.30.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Nov 2022 14:30:30 -0800 (PST) Date: Wed, 23 Nov 2022 17:30:56 -0500 From: Johannes Weiner To: Yosry Ahmed Cc: Mina Almasry , Huang Ying , Yang Shi , Tim Chen , weixugc@google.com, shakeelb@google.com, gthelen@google.com, fvdl@google.com, Michal Hocko , Roman Gushchin , Muchun Song , Andrew Morton , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH V1] mm: Disable demotion from proactive reclaim Message-ID: References: <20221122203850.2765015-1-almasrymina@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669242632; a=rsa-sha256; cv=none; b=B6avwa2IhgCd2KaRUBtmYf8x/93GgHLxxrrQJNTl0tP/C731GMwX4+GPLa52v6yP6LEQJJ AKHSiBFpIQHRr1x9o6XfIV4b1XmzQfBZTnieFK8cbrEijcaMnOEj/zvbRwylM5hpUIqXFr IifWNMHE7RQhB2jqKn0Gb6cCpSuizKA= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=XMfsHKwK; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669242632; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2PsyXXA9CudTt1GF7QXo+ry29gXPUEu+QzCWmZI6Zig=; b=a58BlwyyIMwkBCAJuv+KvQedBjau2Jgsz6txl8+n5lB3P8P9fDwiyPhwdguowJD+5dhTX2 APVoyeTDFgBj6tUuBCNonAU1NyAChg/XPe9htowyzwjlkLVzMp0A4F3UitFIV+cNDGd9JO 1L/WO3RLeCMJyPWpLUr4OUdttlToEnk= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D8086A0010 Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=XMfsHKwK; spf=pass (imf25.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.170 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Rspam-User: X-Stat-Signature: t8r9o488t8ips1d8w1mygn1rd453aw7q X-HE-Tag: 1669242631-865926 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Nov 23, 2022 at 01:35:13PM -0800, Yosry Ahmed wrote: > On Wed, Nov 23, 2022 at 1:21 PM Mina Almasry wrote: > > > > On Wed, Nov 23, 2022 at 10:00 AM Johannes Weiner wrote: > > > > > > Hello Mina, > > > > > > On Tue, Nov 22, 2022 at 12:38:45PM -0800, Mina Almasry wrote: > > > > Since commit 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg > > > > reclaim""), the proactive reclaim interface memory.reclaim does both > > > > reclaim and demotion. This is likely fine for us for latency critical > > > > jobs where we would want to disable proactive reclaim entirely, and is > > > > also fine for latency tolerant jobs where we would like to both > > > > proactively reclaim and demote. > > > > > > > > However, for some latency tiers in the middle we would like to demote but > > > > not reclaim. This is because reclaim and demotion incur different latency > > > > costs to the jobs in the cgroup. Demoted memory would still be addressable > > > > by the userspace at a higher latency, but reclaimed memory would need to > > > > incur a pagefault. > > > > > > > > To address this, I propose having reclaim-only and demotion-only > > > > mechanisms in the kernel. There are a couple possible > > > > interfaces to carry this out I considered: > > > > > > > > 1. Disable demotion in the memory.reclaim interface and add a new > > > > demotion interface (memory.demote). > > > > 2. Extend memory.reclaim with a "demote=" flag to configure the demotion > > > > behavior in the kernel like so: > > > > - demote=0 would disable demotion from this call. > > > > - demote=1 would allow the kernel to demote if it desires. > > > > - demote=2 would only demote if possible but not attempt any > > > > other form of reclaim. > > > > > > Unfortunately, our proactive reclaim stack currently relies on > > > memory.reclaim doing both. It may not stay like that, but I'm a bit > > > wary of changing user-visible semantics post-facto. > > > > > > In patch 2, you're adding a node interface to memory.demote. Can you > > > add this to memory.reclaim instead? This would allow you to control > > > demotion and reclaim independently as you please: if you call it on a > > > node with demotion targets, it will demote; if you call it on a node > > > without one, it'll reclaim. And current users will remain unaffected. > > > > Hello Johannes, thanks for taking a look! > > > > I can certainly add the "nodes=" arg to memory.reclaim and you're > > right, that would help in bridging the gap. However, if I understand > > the underlying code correctly, with only the nodes= arg the kernel > > will indeed attempt demotion first, but the kernel will also merrily > > fall back to reclaiming if it can't demote the full amount. I had > > hoped to have the flexibility to protect latency sensitive jobs from > > reclaim entirely while attempting to do demotion. > > > > There are probably ways to get around that in the userspace. I presume > > the userspace can check if there is available memory on the node's > > demotion targets, and if so, the kernel should demote-only. But I feel > > that wouldn't be reliable as the demotion logic may change across > > kernel versions. The userspace may think the kernel would demote but > > instead demotion failed due to whatever heuristic introduced into the > > new kernel version. > > > > The above is just one angle of the issue. Another angle (which Yosry > > would care most about I think) is that at Google we call > > memory.reclaim mainly when memory.current is too close to memory.max > > and we expect the memory usage of the cgroup to drop as a result of a > > success memory.reclaim call. I suspect once we take in commit > > 3f1509c57b1b ("Revert "mm/vmscan: never demote for memcg reclaim""), > > we would run into that regression, but I defer to Yosry here, he may > > have a solution for that in mind already. > > We don't exactly rely on memory.current, but we do have a separate > proactive reclaim policy today from demotion, and we do expect > memory.reclaim to reclaim memory and not demote it. So it is important > that we can control reclaim vs. demotion separately. Having > memory.reclaim do demotions by default is not ideal for our current > setup, so at least having a demote= argument to control it (no > demotions, may demote, only demote) is needed. With a nodemask you should be able to only reclaim by specifying terminal memory tiers that do that, and leave out higher tiers that demote. That said, it would actually be nice if reclaim policy wouldn't have to differ from demotion policy longer term. Ultimately it comes down to mapping age to memory tier, right? Such that hot pages are in RAM, warm pages are in CXL, cold pages are in storage. If you apply equal presure on all tiers, it's access frequency that should determine which RAM pages to demote, and which CXL pages to reclaim. If RAM pages are hot and refuse demotion, and CXL pages are cold in comparison, CXL should clear out. If RAM pages are warm, they should get demoted to CXL but not reclaimed further from there (and rotate instead). Do we know what's preventing this from happening today? What's the reason you want to control them independently?