From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C47AC433B4 for ; Tue, 27 Apr 2021 16:13:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B880661151 for ; Tue, 27 Apr 2021 16:13:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B880661151 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 15B286B0073; Tue, 27 Apr 2021 12:13:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 135BF6B0074; Tue, 27 Apr 2021 12:13:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDACE6B0075; Tue, 27 Apr 2021 12:13:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0025.hostedemail.com [216.40.44.25]) by kanga.kvack.org (Postfix) with ESMTP id C364C6B0074 for ; Tue, 27 Apr 2021 12:13:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8475E181AF5D7 for ; Tue, 27 Apr 2021 16:13:33 +0000 (UTC) X-FDA: 78078642306.04.3847119 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 9968F2000247 for ; Tue, 27 Apr 2021 16:13:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619540012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P2JkGq2vO8wj3u4HClWKqDzaqtOjDoD0JZSILmTLsRU=; b=Fgc5RyLivRQOMZAwbssuBg158Eriy4YswBwuWAs5aQv+7i6R9+yXSIcEjibuUdGORMv+Jw 8aKSfitiW+kCohAe7A7CR96BSzBevI0ThxpaCmR6CA9LsFnd6p2uZp7lGO331xXPfw5D2r 4KBz32I0vAOnaK0ynd6+EcZ9PWHTjkI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619540012; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P2JkGq2vO8wj3u4HClWKqDzaqtOjDoD0JZSILmTLsRU=; b=Fgc5RyLivRQOMZAwbssuBg158Eriy4YswBwuWAs5aQv+7i6R9+yXSIcEjibuUdGORMv+Jw 8aKSfitiW+kCohAe7A7CR96BSzBevI0ThxpaCmR6CA9LsFnd6p2uZp7lGO331xXPfw5D2r 4KBz32I0vAOnaK0ynd6+EcZ9PWHTjkI= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-393-B9MnVn2nN0K2wl6exVSBug-1; Tue, 27 Apr 2021 12:13:30 -0400 X-MC-Unique: B9MnVn2nN0K2wl6exVSBug-1 Received: by mail-qv1-f72.google.com with SMTP id y14-20020a0cf14e0000b029019ff951fd16so25248471qvl.12 for ; Tue, 27 Apr 2021 09:13:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P2JkGq2vO8wj3u4HClWKqDzaqtOjDoD0JZSILmTLsRU=; b=UzwHTqmuVdE4R5BRssJucaNclnq6Q4sBbvtadseGMQNHSoPGg6cjVEg+d+EN5oWBb3 Bs7Nk6YcoG8XVRG4gnCcQZ2xDRXXXBiu2dFchN3K41gHKJmQ59BAowgQZEi8qH7IGvZb 6Ek1VFa+NT3wgkFaxGa4JDqLjgvV6eR30wiRfjG694KXysPkofA7v1TZCQM63O9Kh9dj LvZhqb5PuHSaTeXbRwvPLFI/fjr7J80wMuRe4nrztV/qHxo33jy9kfr3qLtWlNABvjfO zS0ii/BFVkd35AFQn5D7EhEPNbXe+zL1rFDn0Qa1kmGi6fwQU8uvKs1RvOA1t8en0gbf 2+Sw== X-Gm-Message-State: AOAM532VjKMzTtG69vNT7ZWBG3akkYmx2Yr1Hk5cKg9RB7KFUPqPo0+w GBNEDWrCS8E01767/UtuNOMbrbqKharGYAG2ugVxJrm97JKMGuUM024J4UJ2Qscd2z6jgsyxSB5 Qi/0JnlXk78BLmi33q2XO/km5rSpPLkGVGW18vTAqptWmmYsPElsBcyolD/Ue X-Received: by 2002:a37:b685:: with SMTP id g127mr6501930qkf.42.1619540009659; Tue, 27 Apr 2021 09:13:29 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwx5mLIHNqVmAY+l6qskoKTYPSrLswCa0ymj3l43bBwGXFObHOSp0TR6zVhhrJneunIDW/6Kg== X-Received: by 2002:a37:b685:: with SMTP id g127mr6501882qkf.42.1619540009216; Tue, 27 Apr 2021 09:13:29 -0700 (PDT) Received: from xz-x1.redhat.com (bras-base-toroon474qw-grc-77-184-145-104-227.dsl.bell.ca. [184.145.104.227]) by smtp.gmail.com with ESMTPSA id v66sm3103621qkd.113.2021.04.27.09.13.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Apr 2021 09:13:28 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Nadav Amit , Miaohe Lin , Mike Rapoport , Andrea Arcangeli , Hugh Dickins , peterx@redhat.com, Jerome Glisse , Mike Kravetz , Jason Gunthorpe , Matthew Wilcox , Andrew Morton , Axel Rasmussen , "Kirill A . Shutemov" Subject: [PATCH v2 05/24] shmem/userfaultfd: Handle uffd-wp special pte in page fault handler Date: Tue, 27 Apr 2021 12:12:58 -0400 Message-Id: <20210427161317.50682-6-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210427161317.50682-1-peterx@redhat.com> References: <20210427161317.50682-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="US-ASCII" X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9968F2000247 X-Stat-Signature: n3axyktbhfjkwnzhu5yxnbr7gar5jmb4 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619540014-388535 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: File-backed memories are prone to unmap/swap so the ptes are always unsta= ble. This could lead to userfaultfd-wp information got lost when unmapped or s= wapped out on such types of memory, for example, shmem. To keep such an informa= tion persistent, we will start to use the newly introduced swap-like special p= tes to replace a null pte when those ptes were removed. Prepare this by handling such a special pte first before it is applied. = Here a new fault flag FAULT_FLAG_UFFD_WP is introduced. When this flag is set= , it means the current fault is to resolve a page access (either read or write= ) to the uffd-wp special pte. The handling of this special pte page fault is similar to missing fault, = but it should happen after the pte missing logic since the special pte is design= ed to be a swap-like pte. Meanwhile it should be handled before do_swap_page()= so that the swap core logic won't be confused to see such an illegal swap pt= e. This is a slow path of uffd-wp handling, because unmap of wr-protected sh= mem ptes should be rare. So far it should only trigger in two conditions: (1) When trying to punch holes in shmem_fallocate(), there will be a pre-unmap optimization before evicting the page. That will create unmapped shmem ptes with wr-protected pages covered. (2) Swapping out of shmem pages Because of this, the page fault handling is simplifed too by not sending = the wr-protect message in the 1st page fault, instead the page will be instal= led read-only, so the message will be generated until the next do_wp_page() c= all. Disable fault-around for such a special page fault, because the introduce= d new flag (FAULT_FLAG_UFFD_WP) only applies to current pte rather than all the= pages around it. Doing fault-around with the new flag could confuse all the re= st of pages when installing ptes from page cache when there's a cache hit. Signed-off-by: Peter Xu --- include/linux/userfaultfd_k.h | 11 +++++ mm/memory.c | 80 ++++++++++++++++++++++++++++++++--- 2 files changed, 86 insertions(+), 5 deletions(-) diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.= h index bc733512c6905..fefebe6e96560 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -89,6 +89,17 @@ static inline bool uffd_disable_huge_pmd_share(struct = vm_area_struct *vma) return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); } =20 +/* + * Don't do fault around for FAULT_FLAG_UFFD_WP because it means we want= to + * recover a previously wr-protected pte. This flag is a per-pte inform= ation, + * so it could confuse all the pages around the current page when faulte= d in. + * Similar reason for MINOR mode faults. + */ +static inline bool uffd_disable_fault_around(struct vm_area_struct *vma) +{ + return vma->vm_flags & (VM_UFFD_WP | VM_UFFD_MINOR); +} + static inline bool userfaultfd_missing(struct vm_area_struct *vma) { return vma->vm_flags & VM_UFFD_MISSING; diff --git a/mm/memory.c b/mm/memory.c index 235857ccfaa11..02db41bad3340 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3786,6 +3786,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct = page *page) void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long a= ddr) { struct vm_area_struct *vma =3D vmf->vma; + bool uffd_wp =3D pte_swp_uffd_wp_special(vmf->orig_pte); bool write =3D vmf->flags & FAULT_FLAG_WRITE; bool prefault =3D vmf->address !=3D addr; pte_t entry; @@ -3798,6 +3799,8 @@ void do_set_pte(struct vm_fault *vmf, struct page *= page, unsigned long addr) =20 if (write) entry =3D maybe_mkwrite(pte_mkdirty(entry), vma); + if (unlikely(uffd_wp)) + entry =3D pte_mkuffd_wp(pte_wrprotect(entry)); /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES); @@ -3865,8 +3868,12 @@ vm_fault_t finish_fault(struct vm_fault *vmf) vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); ret =3D 0; - /* Re-check under ptl */ - if (likely(pte_none(*vmf->pte))) + + /* + * Re-check under ptl. Note: this will cover both none pte and + * uffd-wp-special swap pte + */ + if (likely(pte_same(*vmf->pte, vmf->orig_pte))) do_set_pte(vmf, page, vmf->address); else ret =3D VM_FAULT_NOPAGE; @@ -3970,9 +3977,21 @@ static vm_fault_t do_fault_around(struct vm_fault = *vmf) return vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff); } =20 +/* Return true if we should do read fault-around, false otherwise */ +static inline bool should_fault_around(struct vm_fault *vmf) +{ + /* No ->map_pages? No way to fault around... */ + if (!vmf->vma->vm_ops->map_pages) + return false; + + if (uffd_disable_fault_around(vmf->vma)) + return false; + + return fault_around_bytes >> PAGE_SHIFT > 1; +} + static vm_fault_t do_read_fault(struct vm_fault *vmf) { - struct vm_area_struct *vma =3D vmf->vma; vm_fault_t ret =3D 0; =20 /* @@ -3980,7 +3999,7 @@ static vm_fault_t do_read_fault(struct vm_fault *vm= f) * if page by the offset is not ready to be mapped (cold cache or * something). */ - if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) { + if (should_fault_around(vmf)) { ret =3D do_fault_around(vmf); if (ret) return ret; @@ -4293,6 +4312,57 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf= , pud_t orig_pud) return VM_FAULT_FALLBACK; } =20 +static vm_fault_t uffd_wp_clear_special(struct vm_fault *vmf) +{ + vmf->pte =3D pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + /* + * Be careful so that we will only recover a special uffd-wp pte into a + * none pte. Otherwise it means the pte could have changed, so retry. + */ + if (pte_swp_uffd_wp_special(*vmf->pte)) + pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; +} + +/* + * This is actually a page-missing access, but with uffd-wp special pte + * installed. It means this pte was wr-protected before being unmapped. + */ +static vm_fault_t uffd_wp_handle_special(struct vm_fault *vmf) +{ + /* Careful! vmf->pte unmapped after return */ + if (!pte_unmap_same(vmf)) + return 0; + + /* + * Just in case there're leftover special ptes even after the region + * got unregistered - we can simply clear them. + */ + if (unlikely(!userfaultfd_wp(vmf->vma) || vma_is_anonymous(vmf->vma))) + return uffd_wp_clear_special(vmf); + + /* + * Here we share most code with do_fault(), in which we can identify + * whether this is "none pte fault" or "uffd-wp-special fault" by + * checking the vmf->orig_pte. + */ + return do_fault(vmf); +} + +static vm_fault_t do_swap_pte(struct vm_fault *vmf) +{ + /* + * We need to handle special swap ptes before handling ptes that + * contain swap entries, always. + */ + if (unlikely(pte_swp_uffd_wp_special(vmf->orig_pte))) + return uffd_wp_handle_special(vmf); + + return do_swap_page(vmf); +} + /* * These routines also need to handle stuff like marking pages dirty * and/or accessed for architectures that don't do it in hardware (most @@ -4367,7 +4437,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) } =20 if (!pte_present(vmf->orig_pte)) - return do_swap_page(vmf); + return do_swap_pte(vmf); =20 if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); --=20 2.26.2