From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E915AC5321D for ; Thu, 22 Aug 2024 08:44:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 782D06B015B; Thu, 22 Aug 2024 04:44:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 731F580017; Thu, 22 Aug 2024 04:44:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F90A6B015F; Thu, 22 Aug 2024 04:44:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3DD7C6B015B for ; Thu, 22 Aug 2024 04:44:48 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E17CDA5621 for ; Thu, 22 Aug 2024 08:44:47 +0000 (UTC) X-FDA: 82479245814.17.B41D3AA Received: from mail-vk1-f172.google.com (mail-vk1-f172.google.com [209.85.221.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 2BE0718000F for ; Thu, 22 Aug 2024 08:44:45 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A3bEv6zO; spf=pass (imf06.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.172 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724316205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KIO21vDdB9JW0rAwIaxs62jHknF62iZEFTwbURftOkk=; b=GRZ+/zMPe2If2gTPzhYnzSsY5P6VPFM/6vuXlinzWaw2IzBMRWy818wG5mpGUmEdZlLGIs T2kBpBGgFfHF0V5iqBJQt+SKZ2MN1P/WGv2PnwyI6cBz501FhT+DuNQEdpJrdb5z262Xvg aWJ0ltEFbQOEFZ0AeCcY4N/zPB99o0k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724316205; a=rsa-sha256; cv=none; b=Xn75oisYiXXu4N6YU7lMpQo0/hzu54Bv1ol4/jGf+3tUduiEa8B1WC3K7YIC6empqHIGxA Y9AD4Gyxyv8qvkzQ15pja7Q03P3GxjNYYoy60Isdc+n4T9KNKGJLKnGxe1KdgHZtfsWicd WQkXYDtSpPsF4+UiBDc04hCTdOEWWcU= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=A3bEv6zO; spf=pass (imf06.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.172 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-vk1-f172.google.com with SMTP id 71dfb90a1353d-4fceb60e169so256120e0c.3 for ; Thu, 22 Aug 2024 01:44:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724316285; x=1724921085; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KIO21vDdB9JW0rAwIaxs62jHknF62iZEFTwbURftOkk=; b=A3bEv6zO88/dWqwiL6pXp+JLV0oltOfFUrzrdzn34RGB/kcx0yDTM2dkK3yomavJ1Y eCzwOS63zCoZYvOBRoe9ZXZViRMsovHhFnm8EGtyq62RwgKrdVD5KGOYdE3dO6ch8MWS G0uhJcWBZEwBXG2q2Chx5asWE5B87O+4xB6utNehH83NFT3DBBaTHTcxkqVuwmWBXyOP HGYC72Rz5ZzxeKrZTR2EWvf9QBEAAmznprQyVP0RN8xmQgsTxuwO4y/+9OPJY2irN57x 40ns2wWdbEGai+i9zVPIY6DWFWbAgFedY4LVHCs5Cf94OpAG+wVDkPONHyM4XMZJzrOH iU7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724316285; x=1724921085; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KIO21vDdB9JW0rAwIaxs62jHknF62iZEFTwbURftOkk=; b=Uz1uEHlyhYzoeNdBGPkfEjZ4N650jcKMyWQ5HepB/+Gj/6MzZOMptAOYAZ1tjhBJVG 0n8PJzULtFqwF2fuoBqUfHu6fTmQIlqMzodSjL/p6+twCphgzD1b1/3meWY14Zl8PCcL AJ3TwTY+DaGwji2/P3O0xPkHUBOeZbSUJXiUmGay8tZZTwslWIr9pNWte1dyEuJLC8Zf re3AkSSJcyt2ZHUrD/kR46b6dMfqtagPDlvlohSs8U28PpphmGQQILI3cFg5kvNQ41EV b1hytU34dTOSytzg90Tm3SR4SfjX2d24ESoW/SmkT+8UEPiAqiXZW12KbV2c2WDYKebY TkzA== X-Forwarded-Encrypted: i=1; AJvYcCU2G8OjHaENIGjWhtEs40hJoRKHRC+lVgr4jbQ4JqEuA6EkgyjfFj/4FFHnyXENx1Vba++2CSYuzw==@kvack.org X-Gm-Message-State: AOJu0YzgSj/EJi/T4KEu5rxZEMA4+HKbXiZmLrd6ehrM4MwmmxghDmwB ZV5OoLX7UubKCgSKmAznH09HvccfGskytYLiiNcY/4xPC70xvV9gSFx0FXfq/LgJxQu3IoESHOh 21lDyO/KzHrxfnB4V5VA5X24SqEs= X-Google-Smtp-Source: AGHT+IGMSTK8oyJMHzMp0SaYBAXM8UyTM8XVkRv+OC7S9RdFR/EBMu8FLmIS2TiZf3+irMJk9R6s5QNkrqPnLnEjQfY= X-Received: by 2002:a05:6122:3199:b0:4ed:52b:dd29 with SMTP id 71dfb90a1353d-4fd098c38c9mr1503865e0c.3.1724316285074; Thu, 22 Aug 2024 01:44:45 -0700 (PDT) MIME-Version: 1.0 References: <20240811224940.39876-1-21cnbao@gmail.com> <20240811224940.39876-2-21cnbao@gmail.com> <3572ae2e-2141-4a70-99da-850b2e7ade41@redhat.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Thu, 22 Aug 2024 20:44:33 +1200 Message-ID: Subject: Re: [PATCH v2 1/2] mm: collect the number of anon large folios To: David Hildenbrand Cc: akpm@linux-foundation.org, linux-mm@kvack.org, baolin.wang@linux.alibaba.com, chrisl@kernel.org, hanchuanhua@oppo.com, ioworker0@gmail.com, kaleshsingh@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, ziy@nvidia.com, yuanshuai@oppo.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: im1o7m1ccn4pcra55ipp971m9maroyzh X-Rspamd-Queue-Id: 2BE0718000F X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724316285-115650 X-HE-Meta: U2FsdGVkX1/WIDEn8AbUt1UvssbvJ6USQubswe2YJArpWKb8+zU5u1ZQk3LDaW+/1PseucsbgMnnlVlFdsR/Hf8n17KmtcMQFLhBul0HrxCQcY3HisiYM7N/XkJtOozPxqxqodt8j8ci34zP9LVZ1ZQ7fRBLDE+Lsvw8yp0hNLg+oasJkO+doqRLjIcMXDtVKCYuQXnkDWhH01rPLkG/AqL8cy/vCd5C6BH5aVvNz8Wg7t9meT1j7IJrpSZ/3m/4c0GCXBunYXeO7AA5RG9Dc0s3xlwzux6kk7VpHh8ltnjAm/RK3v9eA74kCZvS5jAtOlkTZ22REPmCpm+P0+XW4Y5Y5vOwtpGGbCLaLv1SfGDh5YBz5b7MCRO1NCKpnQ6JD0v7xGBYpiijA2PinH0eHEJvsiBwdRRdTC9zNCUIO76bGplFC537IlUKN5+xYQ1iwdm7C/WgmGvn4ldyQW+w9NngMl65jV+HYLwnj3nIbupWDptgmc3+fFZAA28bI5Hb0cwh7/kSFK5pFjbzSsD+bYNXM+tdy1/xk+fyMa8v3nwr5Fb+p1FnXnA4qNwn3CoKWruyZYFOqoVoxb6UteUXCd6er7/vNQVhqp/SBskkGhD3tci3baRMH6aHAN1U13yIsewyFWybENJufs+rP6BXj2pgnSlNUHFeGQMb+SgVq6R76RfVBj0XM/J4aUk39tF1TX8d4/QIf+Rh8+KUvm8VbSaXAzlCfs4+mMvDtlWv9PkfW7GFaIUc0yUkMQqkhYvHnAl15WOLIulbrFKT9RkQX8fp6nJUzGdTXdnv6pAXNjpkagkU88lhqxT5MJfUwyOisl6wmZOZKGP1cixgCP1xe4RWGaF/RBg4tI5XNQzkwlYUX7PsgYLPGFpOdw49uemd+qJA4GDcVszfS9pmf/kkic8MJAE/NlbeMULlWgAToq86lrIrDaqK6ZAUlBcC0ACstSmRq8iThp7R4D26o2l 9k4R5iuI 4N/F5Aw0y1PzF8mZXsJfUqBQuEPRCLThm3RDKwnqL92BrbZtGYlBUioMG/0O1pXWb+OVrszqh2FglRp3isvrTXitynJfc+P1WqoSkxfhNEQ2nCfQzEIsnB09RT0yDT6T04y+NYQgZweKTUtVmlYX3PZpjyfGp9TTn8YzCQGsA6dhD2S060xwqAR+TyhsjnOksJ8t66RSljRSD1IiHU4SVgXoEEvWoWCBcPqgfdK0ddoMVQQHMX8ElBMPgjf1Wsm4gTaXf7yNl0rhimKTODU4x+2YNO7twU4u31xr5VHFInH3fyOfKokO6M255Z9qeZPed9p0Ps7ruWzYC/oZb78cQup8O1sLqG63uqZjKEWmZpM4WzVxdxDAQcgmihdGmDb75FNOhfXChyAfWp6N6vp4Xd28+BA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Aug 22, 2024 at 12:52=E2=80=AFPM Barry Song <21cnbao@gmail.com> wro= te: > > On Thu, Aug 22, 2024 at 5:34=E2=80=AFAM David Hildenbrand wrote: > > > > On 12.08.24 00:49, Barry Song wrote: > > > From: Barry Song > > > > > > Anon large folios come from three places: > > > 1. new allocated large folios in PF, they will call folio_add_new_ano= n_rmap() > > > for rmap; > > > 2. a large folio is split into multiple lower-order large folios; > > > 3. a large folio is migrated to a new large folio. > > > > > > In all above three counts, we increase nr_anon by 1; > > > > > > Anon large folios might go either because of be split or be put > > > to free, in these cases, we reduce the count by 1. > > > > > > Folios that have been added to the swap cache but have not yet receiv= ed > > > an anon mapping won't be counted. This is consistent with the AnonPag= es > > > statistics in /proc/meminfo. > > > > Thinking out loud, I wonder if we want to have something like that for > > any anon folios (including small ones). > > > > Assume we longterm-pinned an anon folio and unmapped/zapped it. It woul= d > > be quite interesting to see that these are actually anon pages still > > consuming memory. Same with memory leaks, when an anon folio doesn't ge= t > > freed for some reason. > > > > The whole "AnonPages" counter thingy is just confusing, it only counts > > what's currently mapped ... so we'd want something different. > > > > But it's okay to start with large folios only, there we have a new > > interface without that legacy stuff :) > > We have two options to do this: > 1. add a new separate nr_anon_unmapped interface which > counts unmapped anon memory only > 2. let the nr_anon count both mapped and unmapped anon > folios. > > I would assume 1 is clearer as right now AnonPages have been > there for years. and counting all mapped and unmapped together, > we are still lacking an approach to find out anon memory leak > problem you mentioned. > > We are right now comparing nr_anon(including mapped folios only) > with AnonPages to get the distribution of different folio sizes in > performance profiling. > > unmapped_nr_anon should be normally always quite small. otherwise, > something must be wrong. > > > > > > > > > Signed-off-by: Barry Song > > > --- > > > Documentation/admin-guide/mm/transhuge.rst | 5 +++++ > > > include/linux/huge_mm.h | 15 +++++++++++++-- > > > mm/huge_memory.c | 13 ++++++++++--- > > > mm/migrate.c | 4 ++++ > > > mm/page_alloc.c | 5 ++++- > > > mm/rmap.c | 1 + > > > 6 files changed, 37 insertions(+), 6 deletions(-) > > > > > > diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentati= on/admin-guide/mm/transhuge.rst > > > index 058485daf186..9fdfb46e4560 100644 > > > --- a/Documentation/admin-guide/mm/transhuge.rst > > > +++ b/Documentation/admin-guide/mm/transhuge.rst > > > @@ -527,6 +527,11 @@ split_deferred > > > it would free up some memory. Pages on split queue are goin= g to > > > be split under memory pressure, if splitting is possible. > > > > > > +nr_anon > > > + the number of anon huge pages we have in the whole system. > > > > "transparent ..." otherwise people might confuse it with anon hugetlb > > "huge pages" ... :) > > > > I briefly tried coming up with a better name than "nr_anon" but failed. > > > > > > if we might have unmapped_anon counter later, maybe rename it to > nr_anon_mapped? and the new interface we will have in the future > might be nr_anon_unmapped? On second thought, this might be incorrect as well. Concepts like 'anon', 'shmem', and 'file' refer to states after mapping. If an 'anon' has been unmapped but is still pinned and not yet freed, it isn't technically an 'anon' anymore? On the other hand, implementing nr_anon_unmapped could be extremely tricky. I have no idea how to implement it as we are losing those mapping flags. However, a page that is read-ahead but not yet mapped can still become an anon, which seems slightly less tricky to count though seems still difficult - except anon pages, shmem can be also swapped-backed? > > > [...] > > > > > @@ -447,6 +449,8 @@ static int __folio_migrate_mapping(struct address= _space *mapping, > > > */ > > > newfolio->index =3D folio->index; > > > newfolio->mapping =3D folio->mapping; > > > + if (folio_test_anon(folio) && folio_test_large(folio)) > > > + mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1)= ; > > > folio_ref_add(newfolio, nr); /* add cache reference */ > > > if (folio_test_swapbacked(folio)) { > > > __folio_set_swapbacked(newfolio); > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > > index 84a7154fde93..382c364d3efa 100644 > > > --- a/mm/page_alloc.c > > > +++ b/mm/page_alloc.c > > > @@ -1084,8 +1084,11 @@ __always_inline bool free_pages_prepare(struct= page *page, > > > (page + i)->flags &=3D ~PAGE_FLAGS_CHECK_AT_PRE= P; > > > } > > > } > > > - if (PageMappingFlags(page)) > > > + if (PageMappingFlags(page)) { > > > + if (PageAnon(page) && compound) > > > + mod_mthp_stat(order, MTHP_STAT_NR_ANON, -1); > > > > I wonder if you could even drop the "compound" check. mod_mthp_stat > > would handle order =3D=3D 0 just fine. Not that I think it makes much > > difference. > > i think either is fine as mod_mthp_stat will filter out order=3D=3D0 > right now. > > > > > > > Nothing else jumped at me. > > > > Acked-by: David Hildenbrand > > > > Thanks! > > > -- > > Cheers, > > > > David / dhildenb > > > > Barry Thanks Barry