From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 739DEC77B7F for ; Fri, 27 Jun 2025 15:47:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8964E6B00C4; Fri, 27 Jun 2025 11:47:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 847CF6B00C5; Fri, 27 Jun 2025 11:47:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E73F6B00C6; Fri, 27 Jun 2025 11:47:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5A0906B00C4 for ; Fri, 27 Jun 2025 11:47:14 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 0BDE51A0124 for ; Fri, 27 Jun 2025 15:47:14 +0000 (UTC) X-FDA: 83601609588.23.78075B2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id CEB7A4000A for ; Fri, 27 Jun 2025 15:47:11 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="V+AFN/T2"; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751039231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dLsEBLaMb/b3Vgp2d6YxUIW6WEnZzDk0PEHjsGDj8XY=; b=noVHQomdIoozVRJNVdsdEy6FfUOwaY/KFAAbIWWcCujORb8G/h3t5ilwGOA5eTbneMvx/e fyOTwsTRKoeHmrx/glSHRvhbKM+RRmSfcpZzT0ygMCOa2fBq+IV1ovUZWrv5Fu4t1WGA+w 3sPd/oyeJLglSpgFun4Cx+q0JQqztSg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="V+AFN/T2"; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=quarantine) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751039231; a=rsa-sha256; cv=none; b=NaFRDQCBUpNNje+TNENNshgiEFcOQ1+obMHgzy+Wede37HZeViR1ae63zsrX8i7yQSmP2r 4ZElJkFATcx9REicZuGPUL4bvtOS53tNMmx6eceKlCUdQXMLjHtAzHXiJhWTf19XFJpIYF KuYGpTvxVLXm/Ipt7AP/udz4uxvOpQ8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1751039231; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dLsEBLaMb/b3Vgp2d6YxUIW6WEnZzDk0PEHjsGDj8XY=; b=V+AFN/T2KacgzOki9lenpp84pb6Gj2MUou5f9HFWGX3Pf6fKcMfNxnMFO2y89JSUJWdat8 uY3MonKhUGb7jzoNAxPsnjRs7xkZilZZjxiqoHmxMVMr6Vm6wUfrrz/jJQdiiMDvswW+UX +wkwHXO3gl5jqj8cSmyjnvGp8fLDjyk= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-558-rco7dL8uNWalEU0NI4INKw-1; Fri, 27 Jun 2025 11:47:10 -0400 X-MC-Unique: rco7dL8uNWalEU0NI4INKw-1 X-Mimecast-MFC-AGG-ID: rco7dL8uNWalEU0NI4INKw_1751039230 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7ceb5b5140eso315391385a.2 for ; Fri, 27 Jun 2025 08:47:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751039229; x=1751644029; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dLsEBLaMb/b3Vgp2d6YxUIW6WEnZzDk0PEHjsGDj8XY=; b=XPAC8r3ti4g/gR/44+iCNqcLqF1bRQpXiScgGgadU8Mx61L1lvrtNPh7pr4p/jqAjL Igaxnhtew43RPrcmJ+AFjyZZKhONrGOERacjgwGMo/0DeCM1aQgDZxzWMADkqSCphQgV 9D8C0kG8GhgdLF3IpokyS74BkcWanjHF2ZuGzymzar5OO90pMIZDsq5oC4gCnZro+CpR 8o0KFud/KhP41n1c1PrisTe48AyW5PTh1ijrmr2hobrnr/lLWslMk3OBTZ//2LyQ5JQZ 2fsDuq5yLATq/009qAHksBsCGh6GMkFYWxat15NZwgYZwC4Uc7MdgrW2+X3i/5bb5OZv XvOA== X-Gm-Message-State: AOJu0YzlkfLp913FBC/88KLmUN0CekIvtG+eZEjBSB2Vw1RrfrG9seJz A/5smoNpJbj2/9KGb+g2qn+OY7GM5vaLjfrJO1FX1fqjyD9izShFbv63aooJXS/OgMHTleN5n1D V8TtNYKfz5Ye654S/JKWfios308jILr1FbsAv2J3ENIsmekBup/Et6OBwPc+lhPvXHzI7MBfWVh UYjTWR+vFxOuaiut3Wa2emufRHP5xXtvgcYw== X-Gm-Gg: ASbGncuv2M04ZiDBN6AsW7akoR1MdUWL9wlQ66fZEpo56eQvixEOZwVqVID8XYXdoUA En+wiK1vYApP7jNLJ7R8gEqM90JzP2XRGMoSmiP2gy+zYFlnjyNS7y2uKFzWPqJGIT++bYKwPAL 5aLMNtKphbvrRrAv7zwyjwKXHLNt/NqtDd0BKyE9QO6ovdNFnVjyPJ9bNC2b5nJ4seVof/3jQ7T Bn0MFNJ/tkxi1GdNMHA1I59yBotTyjAEVHeRdofV49AlFip+LgzLL2Yio6ZrFMxUJBvReE2jFV8 RSPbQkSYl6w= X-Received: by 2002:a05:6214:4587:b0:6d8:99cf:d2db with SMTP id 6a1803df08f44-70003c8d7fdmr69296936d6.38.1751039228877; Fri, 27 Jun 2025 08:47:08 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFkTQ3X+dYDWCeaFKDWyYkyNZOuxb4Hf3+//g3kzB/Ownzy/PsHVMHjJ/Tpv6Rh1L1/WGJ+JQ== X-Received: by 2002:a05:6214:4587:b0:6d8:99cf:d2db with SMTP id 6a1803df08f44-70003c8d7fdmr69296386d6.38.1751039228368; Fri, 27 Jun 2025 08:47:08 -0700 (PDT) Received: from x1.com ([85.131.185.92]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6fd772e4fddsm22296066d6.65.2025.06.27.08.47.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 27 Jun 2025 08:47:07 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Vlastimil Babka , Suren Baghdasaryan , Muchun Song , Mike Rapoport , Lorenzo Stoakes , Hugh Dickins , Andrew Morton , James Houghton , peterx@redhat.com, "Liam R . Howlett" , Nikita Kalyazin , Michal Hocko , David Hildenbrand , Andrea Arcangeli , Oscar Salvador , Axel Rasmussen , Ujwal Kundur Subject: [PATCH v2 4/4] mm: Apply vm_uffd_ops API to core mm Date: Fri, 27 Jun 2025 11:46:55 -0400 Message-ID: <20250627154655.2085903-5-peterx@redhat.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250627154655.2085903-1-peterx@redhat.com> References: <20250627154655.2085903-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 2SBTjuIYUKs_CpNMdRjQeZlw3d0_aReakghgWkXYbNc_1751039230 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit content-type: text/plain; charset="US-ASCII"; x-default=true X-Rspamd-Queue-Id: CEB7A4000A X-Stat-Signature: uybfw1ei64xm9hozb4dn7py31an5iq97 X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1751039231-885884 X-HE-Meta: U2FsdGVkX197LpIJNd4p3o0NndHETvlbhsOJIBKm4R39A0K1XpPa4FfX+iDVJP8in1zCsPuQs2c1ozJa+6E4yxkVc/l4jwPJgw8dCNIlt01Dgb0iKuct0CwZMobdESJSjRkklJGVKw99JMDx9d/tVOakZtjnVtGiaWwK4aIr885QYXAX8G7yPx3eu0yRRV4Gbhaf60AkPh2Zz+/pGCpvFc7miT4Qr6BOnZrbJQNOJOP0WL6kTRuQKsZkH351EyGfQ9WDeDbPvFMCZT/D2JYf3O1Iz910zjKTtitOL6rDuwQh4Vwc071s5eDjHDrvEH1CfO6VWinr1e+YoGRLKNbWBWQbKlbSofGzRQJk+LEIdLoU+liPDeoOEmGtD3x3lTcw+JKwD4jDAgd3cbDSRcRVl7zsRv3P7y/adCxiYfQcatrJwt20S21Wjo3auZVQODs4+Cm/NwXGV5wUyYrLQKntxHSSw/dfAYvU36RewIghKB3p3s+XyHUDw2AzOPECR3b1MfDWnD7Qs44u5n0EbrzhNWRt/bTFiBRXpsDVWPA4mNtr/ewZOK93ywqPD1KyRi3vI15vlV0aN1zX5sH3ztSvjnd+ip9vkaYGTPE8U4mPkIz9ftg24xWEk3rs8lMqR/GATZAH3oBGa8ef60jxDSKyOlb6AGj+J9epzrQGYBlFhdlkuS7NN2wLHsxnHPOijGUvo2fVX+QVmJCiRQVbXMAGlnOb2SXN9f0cAswPcVEPnSgy+gPPtV4zCYPTEKDptqmEJheIwOSPVPOSmb/ZrC2wEF5nXM1E2Xo6bLiANX73KHMdmIDD855pU9BpiZrwJelMxTFujoJCiyq0i14AizRZa0D5uZ/EqI6q83fsGWoBc6SOPXPgyaxXCI6+8Rm2JpMcS/jReqWEXOhCOl7Rr/DslX3eaJUOefLb80Y4uNUMMK/jbiBpthoFCxCmF8L4HnmaYidc9i1dXQk/xOHjICF kmrUXc6V 2Jqg+IrHkL7+QnzdFe6BRQPoZsvEU77E6BxqGlc60BNvBEY6l8BY3TX3kxz7TbKjtqA/BPAGjhJI9O7I+cKzkS1W1id1PDud+r5/sSzvfOywOnT69s5mOz6zvLCznK8ITAaQLaH2dJvvY05aYyigxUgyYr4tIm375jAkOdQ6W/cUjYerHdWIVTkmkUysbcnm3xmr10P5LuM9TL5Kfi6tRPMzKuN05mgvWN53wf1+Kqkj6AzlDSVb14qp+xUbLsuswXH8H3UNI6qOoGgkJLlfrMUjJtcCN81iNU6EYyVdgneNW0lj1R6hSvbiQ3r3atJs/QD+jwJtu6pOzTX71kYmnCj5Yh7MYIclRdGvaxzEDZxYrzgvByGGEjIndTh+CpDzNOtl6qZHIy5ihe0PbfRxvcjSxawevcOCVhC1LwhHEK8UeV6+14TU8jI50MeX6bu6FmPc2cMc/7lVg0W9QgA0U6rOdGrhSO5YfFXj/Grqq0EpcslftWMpUwe/Q1Y06BPRwV/1V284yndAuJAU9o6Yr9BXsRHY3trzoAp9xJPFohUR6bLU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch completely moves the old userfaultfd core to use the new vm_uffd_ops API. After this change, existing file systems will start to use the new API for userfault operations. When at it, moving vma_can_userfault() into mm/userfaultfd.c instead, because it's getting too big. It's only used in slow paths so it shouldn't be an issue. Move the pte marker check before wp_async, which might be more intuitive because wp_async depends on pte markers. That shouldn't cause any functional change though because only one check would take effect depending on whether pte marker was selected in config. This will also remove quite some hard-coded checks for either shmem or hugetlbfs. Now all the old checks should still work but with vm_uffd_ops. Note that anonymous memory will still need to be processed separately because it doesn't have vm_ops at all. Reviewed-by: James Houghton Signed-off-by: Peter Xu --- include/linux/shmem_fs.h | 14 ----- include/linux/userfaultfd_k.h | 46 ++++---------- mm/shmem.c | 2 +- mm/userfaultfd.c | 115 +++++++++++++++++++++++++--------- 4 files changed, 101 insertions(+), 76 deletions(-) diff --git a/include/linux/shmem_fs.h b/include/linux/shmem_fs.h index 6d0f9c599ff7..2f5b7b295cf6 100644 --- a/include/linux/shmem_fs.h +++ b/include/linux/shmem_fs.h @@ -195,20 +195,6 @@ static inline pgoff_t shmem_fallocend(struct inode *inode, pgoff_t eof) extern bool shmem_charge(struct inode *inode, long pages); extern void shmem_uncharge(struct inode *inode, long pages); -#ifdef CONFIG_USERFAULTFD -#ifdef CONFIG_SHMEM -extern int shmem_mfill_atomic_pte(pmd_t *dst_pmd, - struct vm_area_struct *dst_vma, - unsigned long dst_addr, - unsigned long src_addr, - uffd_flags_t flags, - struct folio **foliop); -#else /* !CONFIG_SHMEM */ -#define shmem_mfill_atomic_pte(dst_pmd, dst_vma, dst_addr, \ - src_addr, flags, foliop) ({ BUG(); 0; }) -#endif /* CONFIG_SHMEM */ -#endif /* CONFIG_USERFAULTFD */ - /* * Used space is stored as unsigned 64-bit value in bytes but * quota core supports only signed 64-bit values so use that diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index c9a093c4502b..1aa9b246fb84 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -149,9 +149,14 @@ typedef struct vm_uffd_ops vm_uffd_ops; #define MFILL_ATOMIC_FLAG(nr) ((__force uffd_flags_t) MFILL_ATOMIC_BIT(nr)) #define MFILL_ATOMIC_MODE_MASK ((__force uffd_flags_t) (MFILL_ATOMIC_BIT(0) - 1)) +static inline enum mfill_atomic_mode uffd_flags_get_mode(uffd_flags_t flags) +{ + return (__force enum mfill_atomic_mode)(flags & MFILL_ATOMIC_MODE_MASK); +} + static inline bool uffd_flags_mode_is(uffd_flags_t flags, enum mfill_atomic_mode expected) { - return (flags & MFILL_ATOMIC_MODE_MASK) == ((__force uffd_flags_t) expected); + return uffd_flags_get_mode(flags) == expected; } static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_atomic_mode mode) @@ -260,41 +265,16 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) return vma->vm_flags & __VM_UFFD_FLAGS; } -static inline bool vma_can_userfault(struct vm_area_struct *vma, - vm_flags_t vm_flags, - bool wp_async) +static inline const vm_uffd_ops *vma_get_uffd_ops(struct vm_area_struct *vma) { - vm_flags &= __VM_UFFD_FLAGS; - - if (vma->vm_flags & VM_DROPPABLE) - return false; - - if ((vm_flags & VM_UFFD_MINOR) && - (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma))) - return false; - - /* - * If wp async enabled, and WP is the only mode enabled, allow any - * memory type. - */ - if (wp_async && (vm_flags == VM_UFFD_WP)) - return true; - -#ifndef CONFIG_PTE_MARKER_UFFD_WP - /* - * If user requested uffd-wp but not enabled pte markers for - * uffd-wp, then shmem & hugetlbfs are not supported but only - * anonymous. - */ - if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) - return false; -#endif - - /* By default, allow any of anon|shmem|hugetlb */ - return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || - vma_is_shmem(vma); + if (vma->vm_ops && vma->vm_ops->userfaultfd_ops) + return vma->vm_ops->userfaultfd_ops; + return NULL; } +bool vma_can_userfault(struct vm_area_struct *vma, + unsigned long vm_flags, bool wp_async); + static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma) { struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx; diff --git a/mm/shmem.c b/mm/shmem.c index 9a8b8dd4709b..fc85002dd8af 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3158,7 +3158,7 @@ static int shmem_uffd_get_folio(struct inode *inode, pgoff_t pgoff, return shmem_get_folio(inode, pgoff, 0, folio, SGP_NOALLOC); } -int shmem_mfill_atomic_pte(pmd_t *dst_pmd, +static int shmem_mfill_atomic_pte(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index cbed91b09640..52d7d5f144b8 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -14,12 +14,48 @@ #include #include #include -#include #include #include #include "internal.h" #include "swap.h" +bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags, + bool wp_async) +{ + unsigned long supported; + + if (vma->vm_flags & VM_DROPPABLE) + return false; + + vm_flags &= __VM_UFFD_FLAGS; + +#ifndef CONFIG_PTE_MARKER_UFFD_WP + /* + * If user requested uffd-wp but not enabled pte markers for + * uffd-wp, then any file system (like shmem or hugetlbfs) are not + * supported but only anonymous. + */ + if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) + return false; +#endif + /* + * If wp async enabled, and WP is the only mode enabled, allow any + * memory type. + */ + if (wp_async && (vm_flags == VM_UFFD_WP)) + return true; + + if (vma_is_anonymous(vma)) + /* Anonymous has no page cache, MINOR not supported */ + supported = VM_UFFD_MISSING | VM_UFFD_WP; + else if (vma_get_uffd_ops(vma)) + supported = vma_get_uffd_ops(vma)->uffd_features; + else + return false; + + return !(vm_flags & (~supported)); +} + static __always_inline bool validate_dst_vma(struct vm_area_struct *dst_vma, unsigned long dst_end) { @@ -384,11 +420,15 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, { struct inode *inode = file_inode(dst_vma->vm_file); pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); + const vm_uffd_ops *uffd_ops = vma_get_uffd_ops(dst_vma); struct folio *folio; struct page *page; int ret; - ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); + if (WARN_ON_ONCE(!uffd_ops || !uffd_ops->uffd_get_folio)) + return -EINVAL; + + ret = uffd_ops->uffd_get_folio(inode, pgoff, &folio); /* Our caller expects us to return -EFAULT if we failed to find folio */ if (ret == -ENOENT) ret = -EFAULT; @@ -504,18 +544,6 @@ static __always_inline ssize_t mfill_atomic_hugetlb( u32 hash; struct address_space *mapping; - /* - * There is no default zero huge page for all huge page sizes as - * supported by hugetlb. A PMD_SIZE huge pages may exist as used - * by THP. Since we can not reliably insert a zero page, this - * feature is not supported. - */ - if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) { - up_read(&ctx->map_changing_lock); - uffd_mfill_unlock(dst_vma); - return -EINVAL; - } - src_addr = src_start; dst_addr = dst_start; copied = 0; @@ -686,14 +714,55 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t *dst_pmd, err = mfill_atomic_pte_zeropage(dst_pmd, dst_vma, dst_addr); } else { - err = shmem_mfill_atomic_pte(dst_pmd, dst_vma, - dst_addr, src_addr, - flags, foliop); + const vm_uffd_ops *uffd_ops = vma_get_uffd_ops(dst_vma); + + if (WARN_ON_ONCE(!uffd_ops || !uffd_ops->uffd_copy)) { + err = -EINVAL; + } else { + err = uffd_ops->uffd_copy(dst_pmd, dst_vma, + dst_addr, src_addr, + flags, foliop); + } } return err; } +static inline bool +vma_uffd_ops_supported(struct vm_area_struct *vma, uffd_flags_t flags) +{ + enum mfill_atomic_mode mode = uffd_flags_get_mode(flags); + const vm_uffd_ops *uffd_ops; + unsigned long uffd_ioctls; + + if ((flags & MFILL_ATOMIC_WP) && !(vma->vm_flags & VM_UFFD_WP)) + return false; + + /* Anonymous supports everything except CONTINUE */ + if (vma_is_anonymous(vma)) + return mode != MFILL_ATOMIC_CONTINUE; + + uffd_ops = vma_get_uffd_ops(vma); + if (!uffd_ops) + return false; + + uffd_ioctls = uffd_ops->uffd_ioctls; + switch (mode) { + case MFILL_ATOMIC_COPY: + return uffd_ioctls & BIT(_UFFDIO_COPY); + case MFILL_ATOMIC_ZEROPAGE: + return uffd_ioctls & BIT(_UFFDIO_ZEROPAGE); + case MFILL_ATOMIC_CONTINUE: + if (!(vma->vm_flags & VM_SHARED)) + return false; + return uffd_ioctls & BIT(_UFFDIO_CONTINUE); + case MFILL_ATOMIC_POISON: + return uffd_ioctls & BIT(_UFFDIO_POISON); + default: + return false; + } +} + static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, unsigned long dst_start, unsigned long src_start, @@ -752,11 +821,7 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, dst_vma->vm_flags & VM_SHARED)) goto out_unlock; - /* - * validate 'mode' now that we know the dst_vma: don't allow - * a wrprotect copy if the userfaultfd didn't register as WP. - */ - if ((flags & MFILL_ATOMIC_WP) && !(dst_vma->vm_flags & VM_UFFD_WP)) + if (!vma_uffd_ops_supported(dst_vma, flags)) goto out_unlock; /* @@ -766,12 +831,6 @@ static __always_inline ssize_t mfill_atomic(struct userfaultfd_ctx *ctx, return mfill_atomic_hugetlb(ctx, dst_vma, dst_start, src_start, len, flags); - if (!vma_is_anonymous(dst_vma) && !vma_is_shmem(dst_vma)) - goto out_unlock; - if (!vma_is_shmem(dst_vma) && - uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) - goto out_unlock; - while (src_addr < src_start + len) { pmd_t dst_pmdval; -- 2.49.0