From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5792C433EF for ; Wed, 29 Jun 2022 16:11:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D3058E0012; Wed, 29 Jun 2022 12:11:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 483838E0009; Wed, 29 Jun 2022 12:11:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34A4C8E0012; Wed, 29 Jun 2022 12:11:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 222138E0009 for ; Wed, 29 Jun 2022 12:11:05 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E057B350CE for ; Wed, 29 Jun 2022 16:11:04 +0000 (UTC) X-FDA: 79631762448.16.87399A4 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf19.hostedemail.com (Postfix) with ESMTP id 3346B1A0031 for ; Wed, 29 Jun 2022 16:11:04 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id i64so15552465pfc.8 for ; Wed, 29 Jun 2022 09:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=QNGWyKk7b7OH2uoXfGLlgbosCtDAoP4ImNMPROj2AzI=; b=qM2+WP3NRFaf4WYrXWOZpW2B8d6z3I/tgsa0zNU7KHL9wJl1oKSVa67x0UqOiq6stC r+7VkOVeUA9IGogWN8nnlKrETvu9nf1uow079WedbfFc4IooPDT+VN1w11KsgeDnpt14 4hjLaqzI1A5L+raNkDcAMDDDmrhlByzWW2vs/AlqzYmgAuiREwGlavsRTZYbbQTB0dVO +MgdG22YNOBxdwwmWbSy8cvn0blRdvs8LMs9Cye+7yHW/8mCULOo6DuL/Ajk2vImKzNq cAxvRxZSHXWABIS9tbF+p1kuNK2cAAw0+UNBEKFVqbkNFN3tg4jpAWCAW5cvMP1MgF2J kFWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QNGWyKk7b7OH2uoXfGLlgbosCtDAoP4ImNMPROj2AzI=; b=1HSp/jsJ29gfSOmoH3Bd993EiwoYQJ+OBwUu04twtbZEsa82MUHQoL3K4WMWkPt7FD 36fVUUPERSwBW01q+sMUZpafrVYT4eAG3QcNdZdxfUwUCStUSzfJkMBH1TcGKdKNemx/ l9dY2IBPVDoWxdr1Q2pUCKtjpKvRbjCgx0T1ZdnOmEJKiKOyjJTWHMAHnZLMl+nCvikL lCMP7+2QchZXjvNVW0dZ7YfIOigxYPrBi/PFfJE4+oNmd9so/XiAQtj3HhLiFJRWuQH4 ymaC/SGq5TJQ89gWh1frcXUn9XNmiQr3xUUCc1EC7/s+PJGyDJyt4ApVcrW3gyLSgjjV TV9Q== X-Gm-Message-State: AJIora9aKdUck12KnAegluzXu6ara3vVSsagsscYSGrrl8qBeuq9c9df yywAaHXeGXTLzA9Nyzap/rNSDXEKAQNK3WGD7inutw== X-Google-Smtp-Source: AGRyM1vWHTZK4sOOG39AoVUnM0BpQXEFUcvFDkwKfvwQMAUgV4SZLSj0NfZmGZYl0bbj9R9oFqyWfk9rMl5oTBkIclg= X-Received: by 2002:a63:5449:0:b0:40d:c8d4:ed6a with SMTP id e9-20020a635449000000b0040dc8d4ed6amr3556661pgm.9.1656519063000; Wed, 29 Jun 2022 09:11:03 -0700 (PDT) MIME-Version: 1.0 References: <20220624173656.2033256-1-jthoughton@google.com> <20220624173656.2033256-13-jthoughton@google.com> <142229ce-fb50-992b-3b11-a1fb5f9175f9@nutanix.com> In-Reply-To: <142229ce-fb50-992b-3b11-a1fb5f9175f9@nutanix.com> From: James Houghton Date: Wed, 29 Jun 2022 09:10:52 -0700 Message-ID: Subject: Re: [RFC PATCH 12/26] hugetlb: add HugeTLB splitting functionality To: "manish.mishra" Cc: Mike Kravetz , Muchun Song , Peter Xu , David Hildenbrand , David Rientjes , Axel Rasmussen , Mina Almasry , Jue Wang , "Dr . David Alan Gilbert" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656519064; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QNGWyKk7b7OH2uoXfGLlgbosCtDAoP4ImNMPROj2AzI=; b=27wR/p42nygFBk1IiiMmAjlst+bUKfAWlZTrKieGMg7o0vIKuRzQHV5omfSudCYmhSqqjH FHvZlP/8UzsHgIuOyZ14N7q7CRCFY3zIfW1pAzZUbdzkwGzkhcVrdWLpMJNl7EvMXEfdnW F1KOdORFyst9DRDseWppOPzXZeCTIpA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qM2+WP3N; spf=pass (imf19.hostedemail.com: domain of jthoughton@google.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656519064; a=rsa-sha256; cv=none; b=T6/UYWu/kDL9+23S3Em2dW7tOUWfVCDZYh1YLCcbMslM5GW1A444yPTI+MfUxM4afm8VW5 JIBiXVAtG84ECmAAf5HCzXYk7VhCqyn6xG7pRipgJ7uhNkRAG71vyigPrISyWd0qia7nAv dkbivdUnh4karFFUgG8Gao3wTnp9ud0= Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=qM2+WP3N; spf=pass (imf19.hostedemail.com: domain of jthoughton@google.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Stat-Signature: sqfa75j8d7ar6t6otpn3msff8fg4ij6e X-Rspamd-Queue-Id: 3346B1A0031 X-HE-Tag: 1656519064-368679 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 27, 2022 at 6:51 AM manish.mishra wrote: > > > On 24/06/22 11:06 pm, James Houghton wrote: > > The new function, hugetlb_split_to_shift, will optimally split the page > > table to map a particular address at a particular granularity. > > > > This is useful for punching a hole in the mapping and for mapping small > > sections of a HugeTLB page (via UFFDIO_CONTINUE, for example). > > > > Signed-off-by: James Houghton > > --- > > mm/hugetlb.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 122 insertions(+) > > > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > > index 3ec2a921ee6f..eaffe7b4f67c 100644 > > --- a/mm/hugetlb.c > > +++ b/mm/hugetlb.c > > @@ -102,6 +102,18 @@ struct mutex *hugetlb_fault_mutex_table ____cacheline_aligned_in_smp; > > /* Forward declaration */ > > static int hugetlb_acct_memory(struct hstate *h, long delta); > > > > +/* > > + * Find the subpage that corresponds to `addr` in `hpage`. > > + */ > > +static struct page *hugetlb_find_subpage(struct hstate *h, struct page *hpage, > > + unsigned long addr) > > +{ > > + size_t idx = (addr & ~huge_page_mask(h))/PAGE_SIZE; > > + > > + BUG_ON(idx >= pages_per_huge_page(h)); > > + return &hpage[idx]; > > +} > > + > > static inline bool subpool_is_free(struct hugepage_subpool *spool) > > { > > if (spool->count) > > @@ -7044,6 +7056,116 @@ static unsigned int __shift_for_hstate(struct hstate *h) > > for ((tmp_h) = hstate; (shift) = __shift_for_hstate(tmp_h), \ > > (tmp_h) <= &hstates[hugetlb_max_hstate]; \ > > (tmp_h)++) > > + > > +/* > > + * Given a particular address, split the HugeTLB PTE that currently maps it > > + * so that, for the given address, the PTE that maps it is `desired_shift`. > > + * This function will always split the HugeTLB PTE optimally. > > + * > > + * For example, given a HugeTLB 1G page that is mapped from VA 0 to 1G. If we > > + * call this function with addr=0 and desired_shift=PAGE_SHIFT, will result in > > + * these changes to the page table: > > + * 1. The PUD will be split into 2M PMDs. > > + * 2. The first PMD will be split again into 4K PTEs. > > + */ > > +static int hugetlb_split_to_shift(struct mm_struct *mm, struct vm_area_struct *vma, > > + const struct hugetlb_pte *hpte, > > + unsigned long addr, unsigned long desired_shift) > > +{ > > + unsigned long start, end, curr; > > + unsigned long desired_sz = 1UL << desired_shift; > > + struct hstate *h = hstate_vma(vma); > > + int ret; > > + struct hugetlb_pte new_hpte; > > + struct mmu_notifier_range range; > > + struct page *hpage = NULL; > > + struct page *subpage; > > + pte_t old_entry; > > + struct mmu_gather tlb; > > + > > + BUG_ON(!hpte->ptep); > > + BUG_ON(hugetlb_pte_size(hpte) == desired_sz); > can it be BUG_ON(hugetlb_pte_size(hpte) <= desired_sz) Sure -- I think that's better. > > + > > + start = addr & hugetlb_pte_mask(hpte); > > + end = start + hugetlb_pte_size(hpte); > > + > > + i_mmap_assert_write_locked(vma->vm_file->f_mapping); > > As it is just changing mappings is holding f_mapping required? I mean in future > > is it any paln or way to use some per process level sub-lock? We don't need to hold a per-mapping lock here, you're right; a per-VMA lock will do just fine. I'll replace this with a per-VMA lock for the next version. > > > + > > + BUG_ON(!hpte->ptep); > > + /* This function only works if we are looking at a leaf-level PTE. */ > > + BUG_ON(!hugetlb_pte_none(hpte) && !hugetlb_pte_present_leaf(hpte)); > > + > > + /* > > + * Clear the PTE so that we will allocate the PT structures when > > + * walking the page table. > > + */ > > + old_entry = huge_ptep_get_and_clear(mm, start, hpte->ptep); > > + > > + if (!huge_pte_none(old_entry)) > > + hpage = pte_page(old_entry); > > + > > + BUG_ON(!IS_ALIGNED(start, desired_sz)); > > + BUG_ON(!IS_ALIGNED(end, desired_sz)); > > + > > + for (curr = start; curr < end;) { > > + struct hstate *tmp_h; > > + unsigned int shift; > > + > > + for_each_hgm_shift(h, tmp_h, shift) { > > + unsigned long sz = 1UL << shift; > > + > > + if (!IS_ALIGNED(curr, sz) || curr + sz > end) > > + continue; > > + /* > > + * If we are including `addr`, we need to make sure > > + * splitting down to the correct size. Go to a smaller > > + * size if we are not. > > + */ > > + if (curr <= addr && curr + sz > addr && > > + shift > desired_shift) > > + continue; > > + > > + /* > > + * Continue the page table walk to the level we want, > > + * allocate PT structures as we go. > > + */ > > As i understand this for_each_hgm_shift loop is just to find right size of shift, > > then code below this line can be put out of loop, no strong feeling but it looks > > more proper may make code easier to understand. Agreed. I'll clean this up. > > > + hugetlb_pte_copy(&new_hpte, hpte); > > + ret = hugetlb_walk_to(mm, &new_hpte, curr, sz, > > + /*stop_at_none=*/false); > > + if (ret) > > + goto err; > > + BUG_ON(hugetlb_pte_size(&new_hpte) != sz); > > + if (hpage) { > > + pte_t new_entry; > > + > > + subpage = hugetlb_find_subpage(h, hpage, curr); > > + new_entry = make_huge_pte_with_shift(vma, subpage, > > + huge_pte_write(old_entry), > > + shift); > > + set_huge_pte_at(mm, curr, new_hpte.ptep, new_entry); > > + } > > + curr += sz; > > + goto next; > > + } > > + /* We couldn't find a size that worked. */ > > + BUG(); > > +next: > > + continue; > > + } > > + > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, vma->vm_mm, > > + start, end); > > + mmu_notifier_invalidate_range_start(&range); > > sorry did not understand where tlb flush will be taken care in case of success? > > I see set_huge_pte_at does not do it internally by self. A TLB flush isn't necessary in the success case -- pages that were mapped before will continue to be mapped the same way, so the TLB entries will still be valid. If we're splitting a none P*D, then there's nothing to flush. If we're splitting a present P*D, then the flush will come if/when we clear any of the page table entries below the P*D. > > > + return 0; > > +err: > > + tlb_gather_mmu(&tlb, mm); > > + /* Free any newly allocated page table entries. */ > > + hugetlb_free_range(&tlb, hpte, start, curr); > > + /* Restore the old entry. */ > > + set_huge_pte_at(mm, start, hpte->ptep, old_entry); > > + tlb_finish_mmu(&tlb); > > + return ret; > > +} > > #endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ > > > > /*