From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA392C4706C for ; Fri, 12 Jan 2024 06:04:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DCC36B008A; Fri, 12 Jan 2024 01:04:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48C6F6B008C; Fri, 12 Jan 2024 01:04:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32C336B0092; Fri, 12 Jan 2024 01:04:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 21BD76B008A for ; Fri, 12 Jan 2024 01:04:16 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E830A140D67 for ; Fri, 12 Jan 2024 06:04:15 +0000 (UTC) X-FDA: 81669618870.17.94E86B2 Received: from mail-ua1-f48.google.com (mail-ua1-f48.google.com [209.85.222.48]) by imf06.hostedemail.com (Postfix) with ESMTP id D4070180007 for ; Fri, 12 Jan 2024 06:04:06 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=adqK3vEu; spf=pass (imf06.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.222.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705039446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p8OENQ8Svy25bRX9eWY34yAdiICgNerJ6rQcIdexfDY=; b=zfR/DQ015G7D5EXOfxR98wmuOLvPhz8Srh4QSxZfrj2It6O7/RVFMRurNMY9eOSQy0Kbt1 l3hdcUWGubLST4GwhmlGn1/b5q1AoEAAsg1VMif6fxpj0Ckl9RLdnf3ZLkIahe9wyU5Q5K IeQXJQif8rXrE7eDZXFGwMyLuVQtSfc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705039446; a=rsa-sha256; cv=none; b=YPLQuG6AYEIHlfV6T0zDDDqeOfq8gjJkUpeLiFF+8/wHWZ1LGfFRIzcoG3sAAJXbE+5zfj by5MZgiWD1qja4jTWCYx3zsNXCq8LXKmCPuqeIubjBEktGBAyAfNzV585ZpkqNQjwRaD8D 6q4rN1Hx1NJ9PV6//A/CZelHbo5h9e8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=adqK3vEu; spf=pass (imf06.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.222.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ua1-f48.google.com with SMTP id a1e0cc1a2514c-7ce6bffb9easo981601241.1 for ; Thu, 11 Jan 2024 22:04:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705039445; x=1705644245; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=p8OENQ8Svy25bRX9eWY34yAdiICgNerJ6rQcIdexfDY=; b=adqK3vEuCiJ7VLPtMSI3Y9fffd2HAywjhHXYkCmrTVuu9GGMOuXfDh/4lNCEQqUFSB I1pkOr/YSfn3gy4jZlPv7CV1351dmS4XBFywcLxLkm3ekIKFvFLHsx//nALKqN5jKUlK Om/cm10/LddWckBUMvt8/n8OCrWP4gfw8NfvkcZSj4Xrj8AHA0eo3mtV3T+Mkpr24sj9 7VxjWc6xPK/sc5xfzTaj0ODj2EFO78GvbXTQBAKotp6lYl/OtviYaL5cYQr/rb7ga4JT oxoeiM9bkVqnys5lGGZjXMhQHHA9duXpr5vJXtKJPGaLt1xAKJCY2l8zq21skCrYf84e 4HGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705039445; x=1705644245; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p8OENQ8Svy25bRX9eWY34yAdiICgNerJ6rQcIdexfDY=; b=LR1++HJIOA1amJe5yMBs2G692t2rJH5Sgn57+/e5UwktDEKoV0Ms+A0eWGx3Xa2o1l TuI7efO9cCj0K1r6ahhGEHkNHr9BZSIKjQ0vcpG6z2tD9ar5XHFDxXyGd1lxPNiIYUNZ CEubMGh5DXzDR5hXEsxjS9grOHIMWVqO5kxR5G3lI6EozC/Ia/5iaRftmLSLRKD8QolP LX6P1xiOvOB1p1f/whhdcnWXB/FNkY6Yn0TrTjxx8yRSjayrXRp2APz4kL7ppRrHKOhP GmNu541DsC6uMfc0SPHj0iNxUIQNz2I++6M26PHXk2Pg+nawIl67TX6xVen7XVaH0sxW TXTA== X-Gm-Message-State: AOJu0YwzwttJzpI4u4Sns5bw1wvDLcgcBWpU6cwVGY8qnHCicEgfQGwP jOxLMvsLb7pB8gbrst45opBdfwl9EFIf/FZ24rg= X-Google-Smtp-Source: AGHT+IEfOoDzHnZZabvVqL2lAYfONtI9/NapcZACz7qDQJjMpfhHLMMGO7k3O/dRla/eX3/Y1ExA+EY8brTVokuZ/P8= X-Received: by 2002:a05:6122:3108:b0:4b6:c5ae:26cc with SMTP id cg8-20020a056122310800b004b6c5ae26ccmr619578vkb.14.1705039444724; Thu, 11 Jan 2024 22:04:04 -0800 (PST) MIME-Version: 1.0 References: <20240102153828.1002295-1-ryan.roberts@arm.com> <3bd5e4a3-9f67-4483-9a0e-9abb5eb783cd@arm.com> <94ebe62b-5f55-4be9-b464-4105b4692496@arm.com> <68d5ce7e-6587-47c6-bd0f-988adf5d92a4@arm.com> <974a2670-7fa9-425e-921e-8d54a596e6cf@arm.com> <6c77f143-9c2c-4d17-9a2a-d69d9adf2eea@arm.com> <40945967-3f16-4574-8f5d-80e0fcb0bcb4@redhat.com> <915d1397-2ac4-497d-9f94-c4b405f99f9a@redhat.com> In-Reply-To: <915d1397-2ac4-497d-9f94-c4b405f99f9a@redhat.com> From: Barry Song <21cnbao@gmail.com> Date: Fri, 12 Jan 2024 19:03:52 +1300 Message-ID: Subject: Re: [RFC PATCH v1] tools/mm: Add thpmaps script to dump THP usage info To: David Hildenbrand Cc: Ryan Roberts , John Hubbard , Andrew Morton , Zenghui Yu , Matthew Wilcox , Kefeng Wang , Zi Yan , Alistair Popple , linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: D4070180007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: wpbqddbup4buqsyexapec7xtufrwdqc5 X-HE-Tag: 1705039446-698698 X-HE-Meta: U2FsdGVkX19iQpfHObq3m3/WC2TyK94DkALQmqLTOyhzSZIQZWJTbKZsUnG//4luJUZ6flzKInlj7emzrWw3SrpSvfEnc7KXesOSROOkhX1L0CXT3XSMG/FP/sTxSYLpZ7xEsWFogdyGVo6gEtgfqlzYIG0Tb/0HKodJbJLfLkwTPnJGVvmxrv++MbPf3JgBmeNiK2h+0wi8IRlRHLduIm07NIs7RKePWRRS6nqc8ckb3iS+DNdaVMHajNbfCCLe6n8YQoCmFJMAiEyqqmnqTbF5GCW/yTiN95YqgFVoNeWyiRut/RPBc+4t2Fq3ghZX+NkyAfCj4LfuJPfF1bL4AkVveSCoCepO7I9y+4Wl5O4EYl7BRGrGnObR7n7lKQUSg2lhFZJYVI1k1pCViyPC+qqt+JMsa4hKfxvnZ6SNHmda/zrE8Cb19UR/q2oz8LYWrIzRCTVquT61QNRCKwyXbGxq63umYNSlJ/c7Un8coUFzqGs1Rpb/xUv2sB/tWGf1TSlIHN6ma2t7MdCATMElMLfSr7pN5YPrZ8wjz0WjaiuKBUsMvSPwZ095nfJDxvpPDk1GgVhEYfbFr1wRM2ouocN7G6+mJm7oMab5J6Hb7yQCOoO4OeXeYJ+q8i9/VES6LXnEtarnZ0tuw6jsDGZ6KjqYa6eWFKRKy6eYk60VqEvO3JeWB8dFzsi1J7wPJGt5gn9UKCFgjzLb/aOiwfgjoBpJkmj7XaGFb4rcRNRyzXKnfoQ4rqJCS2lQHX47X++Cs4Ww5iXbztcqTnAFlDpZpglYjrrrbWbvC92Msk4w2zlCQKUeFq/cvMZQzg74o/PWZEUXN0/Fnm63SEER80HNmz2U/rj5JXHst4lZuUH3qhleDB7Na2KSbvxHaZ9uccMBM1tUSvIaxZyJFLA9ZROW2QWOVSVITqsAE/Rf0yjEfdiiipfgGojqZSLAGgYQ4MgJFBugWnzgTiD9N3GDFqx lc1vchpr q3VAZ3F5PfjkMkrwZb03PLHq8c0Y6a5tJfk/eNfcdPM5gcINHdbORvSBnWiwSEnXqAG3Uln534tI6D3oJCJjMyVirTAhNc/gSMcQ2Kh9WMctQcybdwBUGcgPp8T1j+N6ziZ1NZzuzlWS9tmeg6FG2zETMtZBLoo8/zSoVDWfRsu3XBCtyZm24k1ORrOQjQmmfFmgLYQIL58ZKYSFVILh2k85q32RQ4tWBI1c4ueFQd1/kkK2vN1AVwRTd//3T2IFPsrimPZQsRBdCqeteSjdWSn8Vwxxfg3HKUYrJF+MrpF1SNvnqb9jH2uFIQ0FBR6X7hGm3u5FeVlgTS3Xlht4XU0WnTTxQNZmkhcnnsBoXFaPNEuq4wSoYQlg63Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 12, 2024 at 9:28=E2=80=AFAM David Hildenbrand wrote: > > On 11.01.24 21:21, Barry Song wrote: > > On Fri, Jan 12, 2024 at 2:18=E2=80=AFAM David Hildenbrand wrote: > >> > >> On 11.01.24 13:25, Ryan Roberts wrote: > >>> On 10/01/2024 22:14, Barry Song wrote: > >>>> On Wed, Jan 10, 2024 at 7:59=E2=80=AFPM Ryan Roberts wrote: > >>>>> > >>>>> On 10/01/2024 11:38, Barry Song wrote: > >>>>>> On Wed, Jan 10, 2024 at 7:21=E2=80=AFPM Ryan Roberts wrote: > >>>>>>> > >>>>>>> On 10/01/2024 11:00, David Hildenbrand wrote: > >>>>>>>> On 10.01.24 11:55, Ryan Roberts wrote: > >>>>>>>>> On 10/01/2024 10:42, David Hildenbrand wrote: > >>>>>>>>>> On 10.01.24 11:38, Ryan Roberts wrote: > >>>>>>>>>>> On 10/01/2024 10:30, Barry Song wrote: > >>>>>>>>>>>> On Wed, Jan 10, 2024 at 6:23=E2=80=AFPM Ryan Roberts wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 10/01/2024 09:09, Barry Song wrote: > >>>>>>>>>>>>>> On Wed, Jan 10, 2024 at 4:58=E2=80=AFPM Ryan Roberts wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 10/01/2024 08:02, Barry Song wrote: > >>>>>>>>>>>>>>>> On Wed, Jan 10, 2024 at 12:16=E2=80=AFPM John Hubbard wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On 1/9/24 19:51, Barry Song wrote: > >>>>>>>>>>>>>>>>>> On Wed, Jan 10, 2024 at 11:35=E2=80=AFAM John Hubbard = > >>>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>> Hi Ryan, > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> One thing that immediately came up during some recent= testing of mTHP > >>>>>>>>>>>>>>>>>>> on arm64: the pid requirement is sometimes a little a= wkward. I'm > >>>>>>>>>>>>>>>>>>> running > >>>>>>>>>>>>>>>>>>> tests on a machine at a time for now, inside various = containers and > >>>>>>>>>>>>>>>>>>> such, and it would be nice if there were an easy way = to get some > >>>>>>>>>>>>>>>>>>> numbers > >>>>>>>>>>>>>>>>>>> for the mTHPs across the whole machine. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Just to confirm, you're expecting these "global" stats be= truely global > >>>>>>>>>>>>>>> and not > >>>>>>>>>>>>>>> per-container? (asking because you exploicitly mentioned = being in a > >>>>>>>>>>>>>>> container). > >>>>>>>>>>>>>>> If you want per-container, then you can probably just cre= ate the container > >>>>>>>>>>>>>>> in a > >>>>>>>>>>>>>>> cgroup? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I'm not sure if that changes anything about thpmaps h= ere. Probably > >>>>>>>>>>>>>>>>>>> this is fine as-is. But I wanted to give some initial= reactions from > >>>>>>>>>>>>>>>>>>> just some quick runs: the global state would be conve= nient. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks for taking this for a spin! Appreciate the feedbac= k. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> +1. but this seems to be impossible by scanning pagema= p? > >>>>>>>>>>>>>>>>>> so may we add this statistics information in kernel ju= st like > >>>>>>>>>>>>>>>>>> /proc/meminfo or a separate /proc/mthp_info? > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Yes. From my perspective, it looks like the global stat= s are more useful > >>>>>>>>>>>>>>>>> initially, and the more detailed per-pid or per-cgroup = stats are the > >>>>>>>>>>>>>>>>> next level of investigation. So feels odd to start with= the more > >>>>>>>>>>>>>>>>> detailed stats. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> probably because this can be done without the modificati= on of the kernel. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Yes indeed, as John said in an earlier thread, my previou= s attempts to add > >>>>>>>>>>>>>>> stats > >>>>>>>>>>>>>>> directly in the kernel got pushback; DavidH was concerned= that we don't > >>>>>>>>>>>>>>> really > >>>>>>>>>>>>>>> know exectly how to account mTHPs yet > >>>>>>>>>>>>>>> (whole/partial/aligned/unaligned/per-size/etc) so didn't = want to end up > >>>>>>>>>>>>>>> adding > >>>>>>>>>>>>>>> the wrong ABI and having to maintain it forever. There ha= s also been some > >>>>>>>>>>>>>>> pushback regarding adding more values to multi-value file= s in sysfs, so > >>>>>>>>>>>>>>> David > >>>>>>>>>>>>>>> was suggesting coming up with a whole new scheme at some = point (I know > >>>>>>>>>>>>>>> /proc/meminfo isn't sysfs, but the equivalent files for N= UMA nodes and > >>>>>>>>>>>>>>> cgroups > >>>>>>>>>>>>>>> do live in sysfs). > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Anyway, this script was my attempt to 1) provide a short = term solution > >>>>>>>>>>>>>>> to the > >>>>>>>>>>>>>>> "we need some stats" request and 2) provide a context in = which to explore > >>>>>>>>>>>>>>> what > >>>>>>>>>>>>>>> the right stats are - this script can evolve without the = ABI problem. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The detailed per-pid or per-cgroup is still quite useful= to my case in > >>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>> we set mTHP enabled/disabled and allowed sizes according= to vma types, > >>>>>>>>>>>>>>>> eg. libc_malloc, java heaps etc. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Different vma types can have different anon_name. So I c= an use the > >>>>>>>>>>>>>>>> detailed > >>>>>>>>>>>>>>>> info to find out if specific VMAs have gotten mTHP prope= rly and how many > >>>>>>>>>>>>>>>> they have gotten. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> However, Ryan did clearly say, above, "In future we may= wish to > >>>>>>>>>>>>>>>>> introduce stats directly into the kernel (e.g. smaps or= similar)". And > >>>>>>>>>>>>>>>>> earlier he ran into some pushback on trying to set up /= proc or /sys > >>>>>>>>>>>>>>>>> values because this is still such an early feature. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I wonder if we could put the global stats in debugfs fo= r now? That's > >>>>>>>>>>>>>>>>> specifically supposed to be a "we promise *not* to keep= this ABI stable" > >>>>>>>>>>>>>>>>> location. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Now that I think about it, I wonder if we can add a --glo= bal mode to the > >>>>>>>>>>>>>>> script > >>>>>>>>>>>>>>> (or just infer global when neither --pid nor --cgroup are= provided). I > >>>>>>>>>>>>>>> think I > >>>>>>>>>>>>>>> should be able to determine all the physical memory range= s from > >>>>>>>>>>>>>>> /proc/iomem, > >>>>>>>>>>>>>>> then grab all the info we need from /proc/kpageflags. We = should then be > >>>>>>>>>>>>>>> able to > >>>>>>>>>>>>>>> process it all in much the same way as for --pid/--cgroup= and provide the > >>>>>>>>>>>>>>> same > >>>>>>>>>>>>>>> stats, but it will apply globally. What do you think? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Having now thought about this for a few mins (in the shower= , if anyone wants > >>>>>>>>>>>>> the > >>>>>>>>>>>>> complete picture :) ), this won't quite work. This approach= doesn't have the > >>>>>>>>>>>>> virtual mapping information so the best it can do is tell u= s "how many of > >>>>>>>>>>>>> each > >>>>>>>>>>>>> size of THP are allocated?" - it doesn't tell us anything a= bout whether they > >>>>>>>>>>>>> are > >>>>>>>>>>>>> fully or partially mapped or what their alignment is (all n= ecessary if we > >>>>>>>>>>>>> want > >>>>>>>>>>>>> to know if they are contpte-mapped). So I don't think this = approach is > >>>>>>>>>>>>> going to > >>>>>>>>>>>>> be particularly useful. > >>>>>>>>>>>>> > >>>>>>>>>>>>> And this is also the big problem if we want to gather stats= inside the > >>>>>>>>>>>>> kernel; > >>>>>>>>>>>>> if we want something equivalant to /proc/meminfo's > >>>>>>>>>>>>> AnonHugePages/ShmemPmdMapped/FilePmdMapped, we need to cons= ider not just the > >>>>>>>>>>>>> allocation of the THP but also whether it is mapped. That's= easy for > >>>>>>>>>>>>> PMD-mappings, because there is only one entry to consider -= when you set it, > >>>>>>>>>>>>> you > >>>>>>>>>>>>> increment the number of PMD-mapped THPs, when you clear it,= you decrement. > >>>>>>>>>>>>> But > >>>>>>>>>>>>> for PTE-mappings it's harder; you know the size when you ar= e mapping so its > >>>>>>>>>>>>> easy > >>>>>>>>>>>>> to increment, but you can do a partial unmap, so you would = need to scan the > >>>>>>>>>>>>> PTEs > >>>>>>>>>>>>> to figure out if we are unmapping the first page of a previ= ously > >>>>>>>>>>>>> fully-PTE-mapped THP, which is expensive. We would need a c= heap mechanism to > >>>>>>>>>>>>> determine "is this folio fully and contiguously mapped in a= t least one > >>>>>>>>>>>>> process?". > >>>>>>>>>>>> > >>>>>>>>>>>> as OPPO's approach I shared to you before is maintaining two= mapcount > >>>>>>>>>>>> 1. entire map > >>>>>>>>>>>> 2. subpage's map > >>>>>>>>>>>> 3. if 1 and 2 both exist, it is DoubleMapped. > >>>>>>>>>>>> > >>>>>>>>>>>> This isn't a problem for us. and everytime if we do a partia= l unmap, > >>>>>>>>>>>> we have an explicit > >>>>>>>>>>>> cont_pte split which will decrease the entire map and increa= se the > >>>>>>>>>>>> subpage's mapcount. > >>>>>>>>>>>> > >>>>>>>>>>>> but its downside is that we expose this info to mm-core. > >>>>>>>>>>> > >>>>>>>>>>> OK, but I think we have a slightly more generic situation goi= ng on with the > >>>>>>>>>>> upstream; If I've understood correctly, you are using the PTE= _CONT bit in the > >>>>>>>>>>> PTE to determne if its fully mapped? That works for your case= where you only > >>>>>>>>>>> have 1 size of THP that you care about (contpte-size). But fo= r the upstream, we > >>>>>>>>>>> have multi-size THP so we can't use the PTE_CONT bit to deter= mine if its fully > >>>>>>>>>>> mapped because we can only use that bit if the THP is at leas= t 64K and aligned, > >>>>>>>>>>> and only on arm64. We would need a SW bit for this purpose, a= nd the mm would > >>>>>>>>>>> need to update that SW bit for every PTE one the full -> part= ial map > >>>>>>>>>>> transition. > >>>>>>>>>> > >>>>>>>>>> Oh no. Let's not make everything more complicated for the purp= ose of some stats. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> Indeed, I was intending to argue *against* doing it this way. F= undamentally, if > >>>>>>>>> we want to know what's fully mapped and what's not, then I don'= t see any way > >>>>>>>>> other than by scanning the page tables and we might as well do = that in user > >>>>>>>>> space with this script. > >>>>>>>>> > >>>>>>>>> Although, I expect you will shortly make a proposal that is sim= ple to implement > >>>>>>>>> and prove me wrong ;-) > >>>>>>>> > >>>>>>>> Unlikely :) As you said, once you have multiple folio sizes, it = stops really > >>>>>>>> making sense. > >>>>>>>> > >>>>>>>> Assume you have a 128 kiB pageache folio, and half of that is ma= pped. You can > >>>>>>>> set cont-pte bits on that half and all is fine. Or AMD can benef= it from it's > >>>>>>>> optimizations without the cont-pte bit and everything is fine. > >>>>>>> > >>>>>>> Yes, but for debug and optimization, its useful to know when THPs= are > >>>>>>> fully/partially mapped, when they are unaligned etc. Anyway, the = script does > >>>>>>> that for us, and I think we are tending towards agreement that th= ere are > >>>>>>> unlikely to be any cost benefits by moving it into the kernel. > >>>>>> > >>>>>> frequent partial unmap can defeat all purpose for us to use large = folios. > >>>>>> just imagine a large folio can soon be splitted after it is formed= . we lose > >>>>>> the performance gain and might get regression instead. > >>>>> > >>>>> nit: just because a THP gets partially unmapped in a process doesn'= t mean it > >>>>> gets split into order-0 pages. If the folio still has all its pages= mapped at > >>>>> least once then no further action is taken. If the page being unmap= ped was the > >>>>> last mapping of that page, then the THP is put on the deferred spli= t queue, so > >>>>> that it can be split in future if needed. > >>>>>> > >>>>>> and this can be very frequent, for example, one userspace heap man= agement > >>>>>> is releasing memory page by page. > >>>>>> > >>>>>> In our real product deployment, we might not care about the second= partial > >>>>>> unmapped, we do care about the first partial unmapped as we can u= se this > >>>>>> to know if split has ever happened on this large folios. an partia= l unmapped > >>>>>> subpage can be unlikely re-mapped back. > >>>>>> > >>>>>> so i guess 1st unmap is probably enough, at least for my product. = I mean we > >>>>>> care about if partial unmap has ever happened on a large folio mor= e than how > >>>>>> they are exactly partially unmapped :-) > >>>>> > >>>>> I'm not sure what you are suggesting here? A global boolean that te= lls you if > >>>>> any folio in the system has ever been partially unmapped? That will= almost > >>>>> certainly always be true, even for a very well tuned system. > >>>> > >>>> not a global boolean but a per-folio boolean. in case userspace maps= a region > >>>> and has no userspace management, then we are fine as it is unlikely = to have > >>>> partial unmap/map things; in case userspace maps a region, but manag= es it > >>>> by itself, such as heap things, we might result in lots of partial m= ap/unmap, > >>>> which can lead to 3 problems: > >>>> 1. potential memory footprint increase, for example, while userspace= releases > >>>> some pages in a folio, we might still keep it as frequent splitting = folio into > >>>> basepages and releasing the unmapped subpage might be too expensive. > >>>> 2. if cont-pte is involved, frequent dropping cont-pte/tlb shootdown > >>>> might happen. > >>>> 3. other maintenance overhead such as splitting large folios etc. > >>>> > >>>> We'd like to know how serious partial map things are happening. so e= ither > >>>> we will disable mTHP in this kind of VMAs, or optimize userspace to = do > >>>> some alignment according to the size of large folios. > >>>> > >>>> in android phones, we detect lots of apps, and also found some apps = might > >>>> do things like > >>>> 1. mprotect on some pages within a large folio > >>>> 2. mlock on some pages within a large folio > >>>> 3. madv_free on some pages within a large folio > >>>> 4. madv_pageout on some pages within a large folio. > >>>> > >>>> it would be good if we have a per-folio boolean to know how serious = userspace > >>>> is breaking the large folios. for example, if more than 50% folios i= n a vma has > >>>> this problem, we can find it out and take some action. > >>> > >>> The high level value of these stats seems clear - I agree we need to = be able to > >>> get these insights. I think the issues are more around the implementa= tion > >>> though. I'm struggling to understand exactly how we could implement a= lot of > >>> these things cheaply (either in the kernel or in user space). > >>> > >>> Let me try to work though what I think you are suggesting: > >>> > >>> - every THP is initially fully mapped > >> > >> Not for pagecache folios. > >> > >>> - when an operation causes a partial unmap, mark the folio as havi= ng at least > >>> one partial mapping > >>> - on transition from "no partial mappings" to "at least one partia= l mapping" > >>> increment a "anon-partial-kB" (one for each supported foli= o size) > >>> counter by the folio size > >>> - on transition from "at least one partial mapping" to "fully unam= pped > >>> everywhere" decrement the counter by the folio size > >>> > >>> I think the issue with this is that a folio that is fully mapped in a= process > >>> that gets forked, then is partially unmapped in 1 process, will be ac= counted as > >>> partially mapped even after the process that partially unmapped it ex= its, even > >>> though that folio is now fully mapped in all processes that map it. I= s that a > >>> problem, perhaps not? I'm not sure. > >> > >> What I can offer with my total mapcount I am working on (+ entire/pmd > >> mapcount, but let's put that aside): > >> > >> 1) total_mapcount not multiples of folio_nr_page -> at least one proce= ss > >> currently maps the folio partially > >> > >> 2) total_mapcount is less than folio_nr_page -> surely partially mappe= d > >> > >> I think for most of anon memory (note that most folios are always > >> exclusive in our system, not cow-shared) 2) would already be sufficien= t. > > > > if we can improve Ryan's "mm: Batch-copy PTE ranges during fork()" to > > add nr_pages in copy_pte_range for rmap. > > copy_pte_range() > > { > > folio_try_dup_anon_rmap_ptes(...nr_pages....) > > } > > and at the same time, in zap_pte_range(), we remove the whole anon_rmap > > if the zapped-range covers the whole folio. > > > > Replace the for-loop > > for (i =3D 0; i < nr; i++, page++) { > > add_rmap(1); > > } > > for (i =3D 0; i < nr; i++, page++) { > > remove_rmap(1); > > } > > by always using add_rmap(nr_pages) and remove_rmap(nr_pages) if we > > are doing the entire mapping/unmapping > > That's precisely what I have already running as protoypes :) And I > promised Ryan to get to this soon, clean it up and sent it out. Cool. Glad we'll have it soon. > > . > > > > then we might be able to TestAndSetPartialMapped flag for this folio an= ywhile > > 1. someone is adding rmap with a number not equal nr_pages > > 2. someone is removing rmap with a number not equal nr_pages > > That means we are doing partial mapping or unmapping. > > and we increment partialmap_count by 1, let debugfs or somewhere presen= t > > this count. > > Yes. The only "ugly" corner case if you have a split VMA. We're not > batching rmap exceeding that. I am sorry I don't quite get what the problem is. Do you mean splitting vma is crossing a PTE-mapped mTHP or a PMD-mapped THP? for the latter, I see __split_huge_pmd_locked() does have some mapcount operation but it is batched by folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, vma, haddr, rmap_flags); for the former, I don't find any special mapcount thing is needed. Do I miss something? > > -- > Cheers, > > David / dhildenb Thanks Barry