From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B5E0C3A5A7 for ; Thu, 8 Dec 2022 08:09:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 147348E0003; Thu, 8 Dec 2022 03:09:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CF3B8E0001; Thu, 8 Dec 2022 03:09:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB1B58E0003; Thu, 8 Dec 2022 03:09:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D52F88E0001 for ; Thu, 8 Dec 2022 03:09:43 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A4A9D1A04DD for ; Thu, 8 Dec 2022 08:09:43 +0000 (UTC) X-FDA: 80218415046.03.2FB7E71 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf16.hostedemail.com (Postfix) with ESMTP id D3B6D180017 for ; Thu, 8 Dec 2022 08:09:41 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=iNWSrfnt; spf=pass (imf16.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670486982; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r6HuzxxK1R5LWZiX2Ry1QA4a/oTdpKiXcJsN1wfL8/4=; b=5EmLvYF/OmHmBLCNsfB34XXbOxEQhdIj+0QDT7pw0yK08G6tR08hUlS9MnpUtsJZYJnvQV vHV6UU4rCts7NIp6oNU93C5IQxZ3frHLHKubGFKDv3Ao55magLuC412vl+1NSntTzEFwIL hCQlGUu/6KilUo/zwYyWe/cNIKkHiJQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=iNWSrfnt; spf=pass (imf16.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670486982; a=rsa-sha256; cv=none; b=QAnq3EQpj6Iku7FU7GjPHLbQJbqESFbx16PzY7vWppH+0/HonQHyPQfEKO2+VZOsMFoF7q pI92OOz6pRS+596A5/kuLLHkukW0AZUAF2AWUzhwaXjXKeQ0I90xG88wbzZQNIg/iA/TlO TnIMPJkbf7GTKUNW0aW63L7C3tqy8AA= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 97FBC2075E; Thu, 8 Dec 2022 08:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1670486980; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=r6HuzxxK1R5LWZiX2Ry1QA4a/oTdpKiXcJsN1wfL8/4=; b=iNWSrfntGjAMT0fkfWVxQ3zKRHO5wRDAm28xHnmTDcW+rll/wk7LJLDYvhG2jUEehAE1ah ozgpzedy9wg/LeC+STFaVFTVlLkBJoTBFni5WkfYOnB26NpuFc59i0w1jW5DI/2hlW7xzT 0Jx0W42wQDYBg3BwKryIr0+KKJvq/F8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 79FC5138E0; Thu, 8 Dec 2022 08:09:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id soQnG8SbkWPbGwAAMHmgww (envelope-from ); Thu, 08 Dec 2022 08:09:40 +0000 Date: Thu, 8 Dec 2022 09:09:39 +0100 From: Michal Hocko To: Mina Almasry Cc: Andrew Morton , Johannes Weiner , Roman Gushchin , Shakeel Butt , Muchun Song , Huang Ying , Yang Shi , Yosry Ahmed , weixugc@google.com, fvdl@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v3] [mm-unstable] mm: Fix memcg reclaim on memory tiered systems Message-ID: References: <20221206023406.3182800-1-almasrymina@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: D3B6D180017 X-Stat-Signature: ryzkjb3bwxc4qa735hcnfg481ba1izbm X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1670486981-787297 X-HE-Meta: U2FsdGVkX199IIMh9mdknmHU5NyVgsgPZoTPX+XorV9jCYJgBhPgeqzPFsG0L3hEsUsAzRUJFoj0Z4b4zV1Uhr4TzpUIm//yj1kVTBKIDk/1fjjyLkgZbwXHhvElGG8WUCVR8yVQkigp3c4FEhaBBfBPlg0RViFEPoKI+3wj8gE0Sj3waUgcwLrBRJAoYXUCy4mqb9FFeJ4daU7qGyRWU3l3PuW5Wc5TMlLdGaLdaQ8V8rPYaiY9fQCU57y1i5NcnWTLQw2DsurdSA+QC9lJRRpRpXRoQTIAxqnIdwp8SLyCiavMfcE1ZDFjBRwstlEtRsi5z/C/N/z7tea1A/KotpjWqznp2jXzk0PmyAgyS/cNz8y2WGZW/msYiPM2w6xDluUYpiQa12535pg+3CuaOqyCDVgpg18pwqZT8UFOqdG7i1lgkBe/Dq94ufP2qz8Bl/AcDCSXybAU6jZ8MTBdz37AA4/rrsie1UjZW8oditiW+IlWiHKKyKF31Wi5OJfImdfJoMhLiGqKShzDwnluHPRMQ6EwcOmn5sM+tpMMCFokZcgy3dLzMktWe87FiOxwtZXSbmphV8YTyeeJo0thq/JpwxcyKwcKIib8zJ7nuIrs+jMuhbhh8FV+BJ5oPs7hY2oWNAgQv+cME0c+D7znXqfBNDiwiPd9TpIJuYmKET5lyfJlydZKKOPrHP82DDQO5ygaY+E6brfKovL0nTO9PGnt0nV25DbFqkbBowZUehn2TZnpKgWeo0NqwGHoMng+pPZKAuRQ1Zpn+RRImDCWBuATUgUad5bUIu3prDi+ayG3EkoSsM3stpRlHw1h10n7RPSk0YMUwRRK7qjGqUWUZjJTyTpqNmtlSFr3blwtRp03NxMB7x8/+w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 07-12-22 13:43:55, Mina Almasry wrote: > On Wed, Dec 7, 2022 at 3:12 AM Michal Hocko wrote: [...] > > Anyway a proper nr_reclaimed tracking should be rather straightforward > > but I do not expect to make a big difference in practice > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 026199c047e0..1b7f2d8cb128 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -1633,7 +1633,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, > > LIST_HEAD(ret_folios); > > LIST_HEAD(free_folios); > > LIST_HEAD(demote_folios); > > - unsigned int nr_reclaimed = 0; > > + unsigned int nr_reclaimed = 0, nr_demoted = 0; > > unsigned int pgactivate = 0; > > bool do_demote_pass; > > struct swap_iocb *plug = NULL; > > @@ -2065,8 +2065,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, > > } > > /* 'folio_list' is always empty here */ > > > > - /* Migrate folios selected for demotion */ > > - nr_reclaimed += demote_folio_list(&demote_folios, pgdat); > > + /* > > + * Migrate folios selected for demotion. > > + * Do not consider demoted pages to be reclaimed for the memcg reclaim > > + * because no charges are really freed during the migration. Global > > + * reclaim aims at releasing memory from nodes/zones so consider > > + * demotion to reclaim memory. > > + */ > > + nr_demoted += demote_folio_list(&demote_folios, pgdat); > > + if (!cgroup_reclaim(sc)) > > + nr_reclaimed += nr_demoted; > > + > > /* Folios that could not be demoted are still in @demote_folios */ > > if (!list_empty(&demote_folios)) { > > /* Folios which weren't demoted go back on @folio_list for retry: */ > > > > [...] > > Thank you again, but this patch breaks the memory.reclaim nodes arg > for me. This is my test case. I run it on a machine with 2 memory > tiers. > > Memory tier 1= nodes 0-2 > Memory tier 2= node 3 > > mkdir -p /sys/fs/cgroup/unified/test > cd /sys/fs/cgroup/unified/test > echo $$ > cgroup.procs > head -c 500m /dev/random > /tmp/testfile > echo $$ > /sys/fs/cgroup/unified/cgroup.procs > echo "1m nodes=0-2" > memory.reclaim > > In my opinion the expected behavior is for the kernel to demote 1mb of > memory from nodes 0-2 to node 3. > > Actual behavior on the tip of mm-unstable is as expected. > > Actual behavior with your patch cherry-picked to mm-unstable is that > the kernel demotes all 500mb of memory from nodes 0-2 to node 3, and > returns -EAGAIN to the user. This may be the correct behavior you're > intending, but it completely breaks the use case I implemented the > nodes= arg for and listed on the commit message of that change. Yes, strictly speaking the behavior is correct albeit unexpected. You have told the kernel to _reclaim_ that much memory but demotion are simply aging handling rather than a reclaim if the demotion target has a lot of memory free. This would be the case without any nodemask as well btw. I am worried this will popping up again and again. I thought your nodes subset approach could deal with this but I have overlooked one important thing in your patch. The user provided nodemask controls where to reclaim from but it doesn't constrain demotion targets. Is this intentional? Would it actually make more sense to control demotion by addint demotion nodes into the nodemask? -- Michal Hocko SUSE Labs