From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4155EC38142 for ; Wed, 1 Feb 2023 00:24:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98EEC6B0071; Tue, 31 Jan 2023 19:24:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 917FF6B0072; Tue, 31 Jan 2023 19:24:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 791276B0074; Tue, 31 Jan 2023 19:24:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 637B46B0071 for ; Tue, 31 Jan 2023 19:24:54 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 29C56A0CE4 for ; Wed, 1 Feb 2023 00:24:54 +0000 (UTC) X-FDA: 80416827708.12.8C756E0 Received: from mail-vs1-f49.google.com (mail-vs1-f49.google.com [209.85.217.49]) by imf23.hostedemail.com (Postfix) with ESMTP id 5FB47140019 for ; Wed, 1 Feb 2023 00:24:52 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=UV3OKEii; spf=pass (imf23.hostedemail.com: domain of jthoughton@google.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675211092; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n3z8uh1TyZYnel8a52K+i7m9SXQH4sbMWkRc16OkIxE=; b=I7yFouslGRGG9+SScIcIckdBtdLqopttaJm5+goxDMqxuTMYyxdsy0oKHpH62DaSHT8dka IsdYibhkGhFRbmzd7Bjm5QsZqZK+tUgKdt2WwFvxjb6le9UWWs8HJKrINMx3Jw1+Fg2sKn ppXKtNW+5tqXyn4XiiM4vjkcvKOE2fY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=UV3OKEii; spf=pass (imf23.hostedemail.com: domain of jthoughton@google.com designates 209.85.217.49 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675211092; a=rsa-sha256; cv=none; b=pOEYoitkZq2Z3XPkfwsGHu2DCZLjWWpMn1e/XuvEl5K8AXGOFOq8bH4602i2WrMZWmxd2U DvBySH9rnaelFCucWywVElYwFD9ClGeI8OSd/kk9hIMtS1cK/kbmfhvcZBJQJICXCwyT8h D/Ve1VhdGnAHZqAsBsasd3CFBzXvK5s= Received: by mail-vs1-f49.google.com with SMTP id 3so17914485vsq.7 for ; Tue, 31 Jan 2023 16:24:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=n3z8uh1TyZYnel8a52K+i7m9SXQH4sbMWkRc16OkIxE=; b=UV3OKEiiY7SRdeddEPzqdQHfDAMjRdASUmDwaatREgTLb8Hbei0soIrezvlSZIMzLk N72VOuqmC2TFw4EE9d2Z1u8eIwGdQQDpBi4t5xI23NMtjF6UkmlN9XCvnOiKqtmrYnw+ GQdY9aK9v7L5GWeY3+ImC/BLq/oX+teynz7Lr4IgjIpCsJm6aecq6YOpAHfs2T9lcuM5 RUXDm7dtgCJ8rAkgfLPOdDb2jt7tDfkHm17Hk4eQa2Vc7u0rnOfSq1UTKQTkUXpGAMH5 LIgzuGxeezNtFvlVZIiiPJzFyOv2owzHOt36mlVDzSsbXvkcZdu8X89R4GKhk3YDj+h8 gDIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=n3z8uh1TyZYnel8a52K+i7m9SXQH4sbMWkRc16OkIxE=; b=0UF8oF/WnyTFVQJKN8rJYi/aRDemEXzc0jP1Zb9gEtGYeyVJtlb/mGT0Pw+CkVxYlh yusa3b2vfwlYKGRdoS6WLY5xXmg4+yFSgpmMhOjYHr56lVXJsOB9wf/wrfPRz9RtEFYM CGWLZ5sCAEWx+A1uztIQHs/3Mghqyl9uqP7Iqff/4NeNAAcWe3pI1Lvh8jfe4nuQb3nq 9uVAUHjMf5abhiGWVRHUOmtxjyEI25Pn4iswRz6D3/idofmrXj7fD+OHhtsjoDplAbuC 1OvpmQ43+WZYwGNBaMkQEnGPRB5uDnJqM/Y8WmLU2CWHJVwuoqMj2BS5LpmHsfgOdpUL wM2A== X-Gm-Message-State: AO0yUKXtOJM1h+8FT4ndCqGMSbztY54aS6E7FLictXMYoiseij1DVGES KD39ylh/sZcV1OcbeX8AhjMtcBOCMwg/8jI/sdfeAg== X-Google-Smtp-Source: AK7set85z+ikcKfjWSpAKAbk/uDZcQXz8m+4cesN/rq6G5l7hEVVrNzs47CBhqCH1cqGHrtnBB9ER39dDHudM9IRdmo= X-Received: by 2002:a67:e3c1:0:b0:3fa:3c7f:59b4 with SMTP id k1-20020a67e3c1000000b003fa3c7f59b4mr145046vsm.15.1675211091356; Tue, 31 Jan 2023 16:24:51 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: James Houghton Date: Tue, 31 Jan 2023 16:24:15 -0800 Message-ID: Subject: Re: [PATCH 21/46] hugetlb: use struct hugetlb_pte for walk_hugetlb_range To: Peter Xu Cc: Mike Kravetz , David Hildenbrand , Muchun Song , David Rientjes , Axel Rasmussen , Mina Almasry , "Zach O'Keefe" , Manish Mishra , Naoya Horiguchi , "Dr . David Alan Gilbert" , "Matthew Wilcox (Oracle)" , Vlastimil Babka , Baolin Wang , Miaohe Lin , Yang Shi , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: dzwwyft8ee9oj1b35xfe8qkdz6xmsxop X-Rspamd-Queue-Id: 5FB47140019 X-HE-Tag: 1675211092-517572 X-HE-Meta: U2FsdGVkX1/QDZPBupTqGURH3ZsHl9jbb9Xm3EReyGVU8SNyD0mqfS8jJ8bMA+P9OmVCIz5/QWP5q9yGnkkF2Ui71Gp6MLgAXSWTL8Jr1lrAG8HKqbze1bn6fqKqfa3PxAxIcxeh7MW2n8VvR/fwi3OyJa9xEvXbBWtALzJhiunBO9HVdgUtrS2tlF26wGRovxDY9lmY1+92EJwfwiRkBEOojGGwjBj4ArAgf+sLp62wRJ0/uKfLpXFHvOjDn/t1B8O8W3GMkvF8Taqtu3Wk3qoobGJuE9Ri2Rg6+j9W69wJQq7rYXeXvi5uTWejs7pyloVmf6ysSwV1jFF2Y4gqh5oTcQHlz7HwBxCLAvFOF1I3f5vCfdEVBstJPu2jDmtzKo8dhXhIEA6GmcnLrZRi48z83IqAISEldv4mSB2cFncFyjVdvnNfeiL2FMQBnKDf7k65VElIISb2es4yFnIj6VBu2j+RPzbnkOUVoDbpwmnIxETBe75oQqbe0QymH2qUC/Lrjq9F1ejxflM4/It1CiZWidwT7QQgg83y4j4QiCjtBsPEB3YtsS3Hw2QG6y0Vl+sYSLonhdPplKotz9GPmT9d2YuDEpPyN9fdcMmxLM+OFJT636iy22+4BAu+YR9V+6U/E/tgN91w7cCeUWUTk8gfEHpBYwLJO35BcNZZALfv6tM9Fz2bZvgxZ0+puW3Kd3/Wa9WlWLOQb1V1UbJfliC4kcayQIIgLiXo4M8091tEsNLGJrvpGcwuxoWq5598Yp/VbhFJp4m+xBVpl7PdU22hgKxo1/8cTgQ4fxB5plBlkIvjLBNrIBXMljRg6HkW0j9YXZMS7RPGzsPRXI5KhI61d7R+Z9tFPpp5huSv6f4PNqcV/N44nAG1KqgA4CvETO5Wo8ZGO+23RrepFuNYVAzVorf4ida1fR8EA0LNa+HSPhRKCuGszC6alwgE5NqDRx3LCYYxtl6CUVoIXCt 1S0yPZof AL/svwuRxZ3RDEM1gaetbyKQuBWWCQXFELaE/Rc6kSBQbyCkyk7yqHeGvLA+Y3HJ5S9oM8cmvgEBJ0t3FWa8nGGXXHMhAUsBu2/eeu80jtQkx3uqPWEA3TxUm1babiz+f6cfvN4v5+hmt6apKkr0tAnIWUqz5mmNlnhVC2isqMUOo9b40dTua37tpYNBsiedA0poia9uxNYFfvn3xrwBLDXLuV+/SJKbs4Pu5vzu2Y8/fdWlGbpNu9liQCAQu9+ONDZrnkLlRXs1AxNvlXLleO4UxPfkbiLPXmO/nLZYrxWxTQqsD47Lp3pfm4HYM4+nQ2YufGNGcRAo1wa5yiXRYz56dckhIqHkia2J6SfdFIp6664tLOPDMbLV4PV+/FJeWDs1Rt1FgOoRcm/RUIg+msIcWL9nOVjqclfulHGtd54lUx2c77XD0VLTPkeExoAUzNGAepmx2rCOWSLs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jan 30, 2023 at 1:14 PM Peter Xu wrote: > > On Mon, Jan 30, 2023 at 10:38:41AM -0800, James Houghton wrote: > > On Mon, Jan 30, 2023 at 9:29 AM Peter Xu wrote: > > > > > > On Fri, Jan 27, 2023 at 01:02:02PM -0800, James Houghton wrote: > > > > On Thu, Jan 26, 2023 at 12:31 PM Peter Xu wrote: > > > > > > > > > > James, > > > > > > > > > > On Thu, Jan 26, 2023 at 08:58:51AM -0800, James Houghton wrote: > > > > > > It turns out that the THP-like scheme significantly slows down > > > > > > MADV_COLLAPSE: decrementing the mapcounts for the 4K subpages becomes > > > > > > the vast majority of the time spent in MADV_COLLAPSE when collapsing > > > > > > 1G mappings. It is doing 262k atomic decrements, so this makes sense. > > > > > > > > > > > > This is only really a problem because this is done between > > > > > > mmu_notifier_invalidate_range_start() and > > > > > > mmu_notifier_invalidate_range_end(), so KVM won't allow vCPUs to > > > > > > access any of the 1G page while we're doing this (and it can take like > > > > > > ~1 second for each 1G, at least on the x86 server I was testing on). > > > > > > > > > > Did you try to measure the time, or it's a quick observation from perf? > > > > > > > > I put some ktime_get()s in. > > > > > > > > > > > > > > IIRC I used to measure some atomic ops, it is not as drastic as I thought. > > > > > But maybe it depends on many things. > > > > > > > > > > I'm curious how the 1sec is provisioned between the procedures. E.g., I > > > > > would expect mmu_notifier_invalidate_range_start() to also take some time > > > > > too as it should walk the smally mapped EPT pgtables. > > > > > > > > Somehow this doesn't take all that long (only like 10-30ms when > > > > collapsing from 4K -> 1G) compared to hugetlb_collapse(). > > > > > > Did you populate as much the EPT pgtable when measuring this? > > > > > > IIUC this number should be pretty much relevant to how many pages are > > > shadowed to the kvm pgtables. If the EPT table is mostly empty it should > > > be super fast, but OTOH it can be much slower if when it's populated, > > > because tdp mmu should need to handle the pgtable leaves one by one. > > > > > > E.g. it should be fully populated if you have a program busy dirtying most > > > of the guest pages during test migration. > > > > That's what I was doing. I was running a workload in the guest that > > just writes 8 bytes to a page and jumps ahead a few pages on all > > vCPUs, touching most of its memory. > > > > But there is more to understand; I'll collect more results. I'm not > > sure why the EPT can be unmapped/collapsed so quickly. > > Maybe something smart done by the hypervisor? Doing a little bit more digging, it looks like the invalidate_range_start notifier clears the sptes, and then later on (on the next EPT violation), the page tables are freed. I still need to look at how they end up being so much faster still, but I thought that was interesting. > > > > > > > > > Write op should be the worst here case since it'll require the atomic op > > > being applied; see kvm_tdp_mmu_write_spte(). > > > > > > > > > > > > > > > > > Since we'll still keep the intermediate levels around - from application > > > > > POV, one other thing to remedy this is further shrink the size of COLLAPSE > > > > > so potentially for a very large page we can start with building 2M layers. > > > > > But then collapse will need to be run at least two rounds. > > > > > > > > That's exactly what I thought to do. :) I realized, too, that this is > > > > actually how userspace *should* collapse things to avoid holding up > > > > vCPUs too long. I think this is a good reason to keep intermediate > > > > page sizes. > > > > > > > > When collapsing 4K -> 1G, the mapcount scheme doesn't actually make a > > > > huge difference: the THP-like scheme is about 30% slower overall. > > > > > > > > When collapsing 4K -> 2M -> 1G, the mapcount scheme makes a HUGE > > > > difference. For the THP-like scheme, collapsing 4K -> 2M requires > > > > decrementing and then re-incrementing subpage->_mapcount, and then > > > > from 2M -> 1G, we have to decrement all 262k subpages->_mapcount. For > > > > the head-only scheme, for each 2M in the 4K -> 2M collapse, we > > > > decrement the compound_mapcount 512 times (once per PTE), then > > > > increment it once. And then for 2M -> 1G, for each 1G, we decrement > > > > mapcount again by 512 (once per PMD), incrementing it once. > > > > > > Did you have quantified numbers (with your ktime treak) to compare these? > > > If we want to go the other route, I think these will be materials to > > > justify any other approach on mapcount handling. > > > > Ok, I can do that. GIve me a couple days to collect more results and > > organize them in a helpful way. > > > > (If it's helpful at all, here are some results I collected last week: > > [2]. Please ignore it if it's not helpful.) > > It's helpful already at least to me, thanks. Yes the change is drastic. That data only contains THP-like mapcount performance, no performance for the head-only way. But the head-only scheme makes the 2M -> 1G very good ("56" comes down to about the same everything else, instead of being ~100-500x bigger). > > > > > > > > > > > > > > The mapcount decrements are about on par with how long it takes to do > > > > other things, like updating page tables. The main problem is, with the > > > > THP-like scheme (implemented like this [1]), there isn't a way to > > > > avoid the 262k decrements when collapsing 1G. So if we want > > > > MADV_COLLAPSE to be fast and we want a THP-like page_mapcount() API, > > > > then I think something more clever needs to be implemented. > > > > > > > > [1]: https://github.com/48ca/linux/blob/hgmv2-jan24/mm/hugetlb.c#L127-L178 > > > > > > I believe the whole goal of HGM is trying to face the same challenge if > > > we'll allow 1G THP exist and being able to split for anon. > > > > > > I don't remember whether we discussed below, maybe we did? Anyway... > > > > > > Another way to not use thp mapcount, nor break smaps and similar calls to > > > page_mapcount() on small page, is to only increase the hpage mapcount only > > > when hstate pXd (in case of 1G it's PUD) entry being populated (no matter > > > as leaf or a non-leaf), and the mapcount can be decreased when the pXd > > > entry is removed (for leaf, it's the same as for now; for HGM, it's when > > > freeing pgtable of the PUD entry). > > > > Right, and this is doable. Also it seems like this is pretty close to > > the direction Matthew Wilcox wants to go with THPs. > > I may not be familiar with it, do you mean this one? > > https://lore.kernel.org/all/Y9Afwds%2FJl39UjEp@casper.infradead.org/ Yep that's it. > > For hugetlb I think it should be easier to maintain rather than any-sized > folios, because there's the pgtable non-leaf entry to track rmap > information and the folio size being static to hpage size. > > It'll be different to folios where it can be random sized pages chunk, so > it needs to be managed by batching the ptes when install/zap. Agreed. It's probably easier for HugeTLB because they're always "naturally aligned" and yeah they can't change sizes. > > > > > Something I noticed though, from the implementation of > > folio_referenced()/folio_referenced_one(), is that folio_mapcount() > > ought to report the total number of PTEs that are pointing on the page > > (or the number of times page_vma_mapped_walk returns true). FWIW, > > folio_referenced() is never called for hugetlb folios. > > FWIU folio_mapcount is the thing it needs for now to do the rmap walks - > it'll walk every leaf page being mapped, big or small, so IIUC that number > should match with what it expects to see later, more or less. I don't fully understand what you mean here. > > But I agree the mapcount/referenced value itself is debatable to me, just > like what you raised in the other thread on page migration. Meanwhile, I > am not certain whether the mapcount is accurate either because AFAICT the > mapcount can be modified if e.g. new page mapping established as long as > before taking the page lock later in folio_referenced(). > > It's just that I don't see any severe issue either due to any of above, as > long as that information is only used as a hint for next steps, e.g., to > swap which page out. I also don't see a big problem with folio_referenced() (and you're right that folio_mapcount() can be stale by the time it takes the folio lock). It still seems like folio_mapcount() should return the total number of PTEs that map the page though. Are you saying that breaking this would be ok?