From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE272EB64DA for ; Wed, 5 Jul 2023 15:59:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 333D38D0002; Wed, 5 Jul 2023 11:59:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E2E38D0001; Wed, 5 Jul 2023 11:59:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 184338D0002; Wed, 5 Jul 2023 11:59:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 071B88D0001 for ; Wed, 5 Jul 2023 11:59:20 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D1EA11A0322 for ; Wed, 5 Jul 2023 15:59:19 +0000 (UTC) X-FDA: 80978017638.17.8FBF652 Received: from mail-qt1-f173.google.com (mail-qt1-f173.google.com [209.85.160.173]) by imf24.hostedemail.com (Postfix) with ESMTP id 6039A180025 for ; Wed, 5 Jul 2023 15:59:16 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pLUgYATe; spf=pass (imf24.hostedemail.com: domain of jthoughton@google.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688572756; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h6JhVHABPh7lO/2EMETmTryaZgiMx7eDsKAe5JV5KYU=; b=v4ZVttxvhoSicn0ft7UVCvZO5Wop9SiHTqK5ncAXOSOISU1rFjUyZRmy3bx6aWnnJpmA+k c29yBY8YLD/Z5jBlKlL0Y3qVL/cHFV7rO+F2dvn4qaNOHWE8cPi3Vopc1G+BpXjELSJRZ/ OOtlqoP6tzzUCBSgyw9twhwIb7iXx0g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688572756; a=rsa-sha256; cv=none; b=gBF05g5+WPXV9iA8G1DORgYCxCGVSVcPtEszd4FbAMxDmpYa7yFnGwxEPjeyU+/BX2rMkS Z2VPUedXUMtpDolklm8fS+5IAk0pDgECmYvmypWI949COBWXslNVqjNCXxINv0HbAVBifi jp0TdQ73YDnmVFg32ItatD57R06IB0k= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pLUgYATe; spf=pass (imf24.hostedemail.com: domain of jthoughton@google.com designates 209.85.160.173 as permitted sender) smtp.mailfrom=jthoughton@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qt1-f173.google.com with SMTP id d75a77b69052e-4036bd4fff1so549201cf.0 for ; Wed, 05 Jul 2023 08:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688572755; x=1691164755; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=h6JhVHABPh7lO/2EMETmTryaZgiMx7eDsKAe5JV5KYU=; b=pLUgYATeQLAhBUTbgrhhlZYAwXib3nS35jsB/W96kuDJLoBVSr+86HTjnFhCeCF3j3 DXP3QxMLgUYWRJBrYtaE0/YVZHH5UGXbneqvixA3RrOSh+RkUDpmkJNzfD2aIOlJVfdy uXUFrZZlhnf/flfmDvko+Ozpf3sZBR5ZhY089WPPRS7LRRzur0K23f1RIDBwqB1BGfmw FLKJpJKawgxCA9YVlLKzGGa9fsIiOS03C2N5HZUDYbSkVPCPJ/TxgdDT3QGClGJ+S4bM OfRWNqxmJr5dlUoOeRs+ktGOB9q1ehUYDBeX1HPBVIYx0MAu7Z3yw9p3+BEpXJj8O9Mf lGDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688572755; x=1691164755; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h6JhVHABPh7lO/2EMETmTryaZgiMx7eDsKAe5JV5KYU=; b=ln0teh7pjJ7hFnnFoiMPGoCMJ6HDvYpytexDZFbqBOMzMDeOExp8Cr+xKC8gA2JEYK +wdbFVDZjHX1bGIxFchXBI/aVh6OgzeW1ioFxuDJ0dv8PsNEEachnG4YUC9+tjq9CM7q dWtokvGA9z3vxgpBJwm/nVIHwGGCS4zfe7Rb7ORI4E6SGaoDe5hsg4bLTroqw84EVuH+ UK/Ifa/nlJ6JCGw2XxmClgXeq3eiGn1vF+J7YbtKqpJYOil5VYIWlre+ycLlA2hui7dg YW+sdcOcayyuH6xNbIrA20yx94E3aL16tuz0aTkwN4JDj+C5TpSdWiZ/pvaHIs/tNgAc YpWA== X-Gm-Message-State: ABy/qLaeZdk/S//stx7cAnw8Xx34aVOYy/w0uHoVprXLPDZLHdf3X+kF olONfPo6jKTHzryAPrP2NEjYl9qWlubLmnB9SFAX/A== X-Google-Smtp-Source: APBJJlGJNT/E8ZZhIEEgE2ILiniZAcPLBGTAwq2Ws9047VexCF8eCP/Z1jcDbIviBT3vnm7pLiVMCyuhJMv/E9WeiSE= X-Received: by 2002:ac8:5c0f:0:b0:3f6:97b4:1a4a with SMTP id i15-20020ac85c0f000000b003f697b41a4amr111006qti.20.1688572755236; Wed, 05 Jul 2023 08:59:15 -0700 (PDT) MIME-Version: 1.0 References: <20230629205040.665834-1-axelrasmussen@google.com> In-Reply-To: <20230629205040.665834-1-axelrasmussen@google.com> From: James Houghton Date: Wed, 5 Jul 2023 08:58:39 -0700 Message-ID: Subject: Re: [PATCH v2 1/6] mm: userfaultfd: add new UFFDIO_POISON ioctl To: Axel Rasmussen Cc: Alexander Viro , Andrew Morton , Christian Brauner , David Hildenbrand , Huang Ying , Hugh Dickins , Jiaqi Yan , Jonathan Corbet , "Liam R. Howlett" , Miaohe Lin , Mike Kravetz , "Mike Rapoport (IBM)" , Muchun Song , Nadav Amit , Naoya Horiguchi , Peter Xu , Shuah Khan , ZhangPeng , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: fsg3p3qio8c4yq75rfz16s564rnj1gei X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6039A180025 X-Rspam-User: X-HE-Tag: 1688572756-465215 X-HE-Meta: U2FsdGVkX18nO7UagobuK7bOkHtJwRC2yHoASBZnoTxIgro7RAhIQQBC1ig+IxlHGVjJZXAppCzWmHzLQhpn4BEt1zb7pDLG1lCrA7T13QchYDV0/tPBmsrcab7XyIK5EU0oruF98C9feiznGITBg4qrhrRkhtJhUuxHS4R4bDzUfOxcDCvEFpjt+vhkLasRyjyr99XJSTEcl90DwQJR0ebEvHLBr1XKsCbIepoghh/6k27A6sPR2c7/n0hobfYV1b+y3oDr3ZTJW9T1w8rOZjB8WqLw8h6sgg4HePt3bWKA5l7XH0MreamFGdMUnQVzCYrll+3znKmGkLx/Gqy6dha4foSxv317XO1oZ+6TntXaKB2iUkF0Uq8u5pmTA5/vkof5nscbb2+kjqkzBJ7oXaX+a0+P1U+qtKM876DQaMnmvJNTtdFO1+GqVHaN6YPic/8gp2pUz1LmSY4mu8XSLhQ1GFzgvvx+uQrey4NGSDBy84Eblvh4K3nOi0BZsk67NKy2+Pjic8sfM8UMSOp5siDS9bVyy1RSUS+rx0mWkh4o5b10DRKTRc78c9r5AZ7Y6qbzRF7ZqVmuoxg0cElGsK8bAqjc6hJUg2rv5Fi1RWsVNRBDpQGiGite2Npnh3y3DGlkMjELGgANpQGDIj0yjjZUE8WmOWVWziwR9ktew++EYXI22kMohr4IpzyGIguZhErAe3XM1wYMPwKpRJY5qp4QZ0UEtfX/AuZcNosUW2kB7tY30IojOV7XZWs8W0b5I+YdOBeVSUJtYkxExfbk5WeY54d9Y14SPoDZter//OImXGiEnuPN4Ott8iWX+3sAckVxytrWFevoTNnHi/j163pyPJIYXyP8xGxDP7hxDAwsZhX7oBBEKJDs3tLDfureWhP6FvssTh9lggP0Hxv8IHfIrpRw2yEsnkW/5jy38L6eLuu7mdkbCgP2M3Ss+cFs9eZfDI9o8L7bvxzJyjg Q5JgVsDC FidH26fKd9vAZ9C2erqrOCTngP7nYfNLLqhbezb44/TDFsewxU0loda+votydWBJfJp7N9YmDGou3iNmKH+8VQws+qoUY/qOqEiyiQPWQfhtnckWY/84gucyAmt54hZi52G5I2vGc2G9w3ZhqJBK6S9ZQjTVTwLcujgzZ/ta1aE4252A65HjJvSnxvie2k6l3+eK0KuNA+UgTctI5ZRVe86fNiVAMnaLyc7W8HKiE1MQROtfVFy3s4DqN26cNWgN2vgt3n8bThc7IlpSZN7Ywp9yko4L2Qd4vuBmlzRpnLeeCwS4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jun 29, 2023 at 1:50=E2=80=AFPM Axel Rasmussen wrote: > > The basic idea here is to "simulate" memory poisoning for VMs. A VM > running on some host might encounter a memory error, after which some > page(s) are poisoned (i.e., future accesses SIGBUS). They expect that > once poisoned, pages can never become "un-poisoned". So, when we live > migrate the VM, we need to preserve the poisoned status of these pages. > > When live migrating, we try to get the guest running on its new host as > quickly as possible. So, we start it running before all memory has been > copied, and before we're certain which pages should be poisoned or not. > > So the basic way to use this new feature is: > > - On the new host, the guest's memory is registered with userfaultfd, in > either MISSING or MINOR mode (doesn't really matter for this purpose). > - On any first access, we get a userfaultfd event. At this point we can > communicate with the old host to find out if the page was poisoned. > - If so, we can respond with a UFFDIO_POISON - this places a swap marker > so any future accesses will SIGBUS. Because the pte is now "present", > future accesses won't generate more userfaultfd events, they'll just > SIGBUS directly. > > UFFDIO_POISON does not handle unmapping previously-present PTEs. This > isn't needed, because during live migration we want to intercept > all accesses with userfaultfd (not just writes, so WP mode isn't useful > for this). So whether minor or missing mode is being used (or both), the > PTE won't be present in any case, so handling that case isn't needed. > > Why return VM_FAULT_HWPOISON instead of VM_FAULT_SIGBUS when one of > these markers is encountered? For "normal" userspace programs there > isn't a big difference, both yield a SIGBUS. The difference for KVM is > key though: VM_FAULT_HWPOISON will result in an MCE being injected into > the guest (which is the behavior we want). With VM_FAULT_SIGBUS, the > hypervisor would need to catch the SIGBUS and deal with the MCE > injection itself. For a normal userspace program, the si_code will be different (BUS_MCEERR_AR vs. BUS_ADRERR), maybe that's worth mentioning? Also KVM doesn't inject an MCE automatically. When KVM_RUN gets -EHWPOISON back from GUP (for x86 and arm64 anyway), KVM will queue up a BUS_MCERR_AR signal for userspace and return 0 to userspace. Then userspace may choose to inject an MCE. If we gave back VM_FAULT_SIGBUS instead of VM_FAULT_HWPOISON, KVM_RUN would instead return -EFAULT to userspace without queueing a signal. Today, without the signal, it becomes impossible (I think) to determine what the appropriate MCE to inject is. > > Signed-off-by: Axel Rasmussen > --- > fs/userfaultfd.c | 63 ++++++++++++++++++++++++++++++++ > include/linux/swapops.h | 3 +- > include/linux/userfaultfd_k.h | 4 ++ > include/uapi/linux/userfaultfd.h | 25 +++++++++++-- > mm/memory.c | 4 ++ > mm/userfaultfd.c | 62 ++++++++++++++++++++++++++++++- > 6 files changed, 156 insertions(+), 5 deletions(-) > > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > index 7cecd49e078b..c26a883399c9 100644 > --- a/fs/userfaultfd.c > +++ b/fs/userfaultfd.c > @@ -1965,6 +1965,66 @@ static int userfaultfd_continue(struct userfaultfd= _ctx *ctx, unsigned long arg) > return ret; > } > > +static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsign= ed long arg) > +{ > + __s64 ret; > + struct uffdio_poison uffdio_poison; > + struct uffdio_poison __user *user_uffdio_poison; > + struct userfaultfd_wake_range range; > + > + user_uffdio_poison =3D (struct uffdio_poison __user *)arg; > + > + ret =3D -EAGAIN; > + if (atomic_read(&ctx->mmap_changing)) > + goto out; > + > + ret =3D -EFAULT; > + if (copy_from_user(&uffdio_poison, user_uffdio_poison, > + /* don't copy the output fields */ > + sizeof(uffdio_poison) - (sizeof(__s64)))) > + goto out; > + > + ret =3D validate_range(ctx->mm, uffdio_poison.range.start, > + uffdio_poison.range.len); > + if (ret) > + goto out; > + > + ret =3D -EINVAL; > + /* double check for wraparound just in case. */ > + if (uffdio_poison.range.start + uffdio_poison.range.len <=3D > + uffdio_poison.range.start) { > + goto out; > + } > + if (uffdio_poison.mode & ~UFFDIO_POISON_MODE_DONTWAKE) > + goto out; > + > + if (mmget_not_zero(ctx->mm)) { > + ret =3D mfill_atomic_poison(ctx->mm, uffdio_poison.range.= start, > + uffdio_poison.range.len, > + &ctx->mmap_changing, 0); > + mmput(ctx->mm); > + } else { > + return -ESRCH; > + } > + > + if (unlikely(put_user(ret, &user_uffdio_poison->updated))) > + return -EFAULT; > + if (ret < 0) > + goto out; > + > + /* len =3D=3D 0 would wake all */ > + BUG_ON(!ret); > + range.len =3D ret; > + if (!(uffdio_poison.mode & UFFDIO_POISON_MODE_DONTWAKE)) { > + range.start =3D uffdio_poison.range.start; > + wake_userfault(ctx, &range); > + } > + ret =3D range.len =3D=3D uffdio_poison.range.len ? 0 : -EAGAIN; > + > +out: > + return ret; > +} > + > static inline unsigned int uffd_ctx_features(__u64 user_features) > { > /* > @@ -2066,6 +2126,9 @@ static long userfaultfd_ioctl(struct file *file, un= signed cmd, > case UFFDIO_CONTINUE: > ret =3D userfaultfd_continue(ctx, arg); > break; > + case UFFDIO_POISON: > + ret =3D userfaultfd_poison(ctx, arg); > + break; > } > return ret; > } > diff --git a/include/linux/swapops.h b/include/linux/swapops.h > index 4c932cb45e0b..8259fee32421 100644 > --- a/include/linux/swapops.h > +++ b/include/linux/swapops.h > @@ -394,7 +394,8 @@ typedef unsigned long pte_marker; > > #define PTE_MARKER_UFFD_WP BIT(0) > #define PTE_MARKER_SWAPIN_ERROR BIT(1) > -#define PTE_MARKER_MASK (BIT(2) - 1) > +#define PTE_MARKER_UFFD_POISON BIT(2) > +#define PTE_MARKER_MASK (BIT(3) - 1) > > static inline swp_entry_t make_pte_marker_entry(pte_marker marker) > { > diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.= h > index ac7b0c96d351..ac8c6854097c 100644 > --- a/include/linux/userfaultfd_k.h > +++ b/include/linux/userfaultfd_k.h > @@ -46,6 +46,7 @@ enum mfill_atomic_mode { > MFILL_ATOMIC_COPY, > MFILL_ATOMIC_ZEROPAGE, > MFILL_ATOMIC_CONTINUE, > + MFILL_ATOMIC_POISON, > NR_MFILL_ATOMIC_MODES, > }; > > @@ -83,6 +84,9 @@ extern ssize_t mfill_atomic_zeropage(struct mm_struct *= dst_mm, > extern ssize_t mfill_atomic_continue(struct mm_struct *dst_mm, unsigned = long dst_start, > unsigned long len, atomic_t *mmap_ch= anging, > uffd_flags_t flags); > +extern ssize_t mfill_atomic_poison(struct mm_struct *dst_mm, unsigned lo= ng start, > + unsigned long len, atomic_t *mmap_chan= ging, > + uffd_flags_t flags); > extern int mwriteprotect_range(struct mm_struct *dst_mm, > unsigned long start, unsigned long len, > bool enable_wp, atomic_t *mmap_changing); > diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfa= ultfd.h > index 66dd4cd277bd..62151706c5a3 100644 > --- a/include/uapi/linux/userfaultfd.h > +++ b/include/uapi/linux/userfaultfd.h > @@ -39,7 +39,8 @@ > UFFD_FEATURE_MINOR_SHMEM | \ > UFFD_FEATURE_EXACT_ADDRESS | \ > UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ > - UFFD_FEATURE_WP_UNPOPULATED) > + UFFD_FEATURE_WP_UNPOPULATED | \ > + UFFD_FEATURE_POISON) > #define UFFD_API_IOCTLS \ > ((__u64)1 << _UFFDIO_REGISTER | \ > (__u64)1 << _UFFDIO_UNREGISTER | \ > @@ -49,12 +50,14 @@ > (__u64)1 << _UFFDIO_COPY | \ > (__u64)1 << _UFFDIO_ZEROPAGE | \ > (__u64)1 << _UFFDIO_WRITEPROTECT | \ > - (__u64)1 << _UFFDIO_CONTINUE) > + (__u64)1 << _UFFDIO_CONTINUE | \ > + (__u64)1 << _UFFDIO_POISON) > #define UFFD_API_RANGE_IOCTLS_BASIC \ > ((__u64)1 << _UFFDIO_WAKE | \ > (__u64)1 << _UFFDIO_COPY | \ > + (__u64)1 << _UFFDIO_WRITEPROTECT | \ > (__u64)1 << _UFFDIO_CONTINUE | \ > - (__u64)1 << _UFFDIO_WRITEPROTECT) > + (__u64)1 << _UFFDIO_POISON) > > /* > * Valid ioctl command number range with this API is from 0x00 to > @@ -71,6 +74,7 @@ > #define _UFFDIO_ZEROPAGE (0x04) > #define _UFFDIO_WRITEPROTECT (0x06) > #define _UFFDIO_CONTINUE (0x07) > +#define _UFFDIO_POISON (0x08) > #define _UFFDIO_API (0x3F) > > /* userfaultfd ioctl ids */ > @@ -91,6 +95,8 @@ > struct uffdio_writeprotect) > #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ > struct uffdio_continue) > +#define UFFDIO_POISON _IOWR(UFFDIO, _UFFDIO_POISON, \ > + struct uffdio_poison) > > /* read() structure */ > struct uffd_msg { > @@ -225,6 +231,7 @@ struct uffdio_api { > #define UFFD_FEATURE_EXACT_ADDRESS (1<<11) > #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) > #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) > +#define UFFD_FEATURE_POISON (1<<14) > __u64 features; > > __u64 ioctls; > @@ -321,6 +328,18 @@ struct uffdio_continue { > __s64 mapped; > }; > > +struct uffdio_poison { > + struct uffdio_range range; > +#define UFFDIO_POISON_MODE_DONTWAKE ((__u64)1<<0) > + __u64 mode; > + > + /* > + * Fields below here are written by the ioctl and must be at the = end: > + * the copy_from_user will not read past here. > + */ > + __s64 updated; > +}; > + > /* > * Flags for the userfaultfd(2) system call itself. > */ > diff --git a/mm/memory.c b/mm/memory.c > index d8a9a770b1f1..7fbda39e060d 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -3692,6 +3692,10 @@ static vm_fault_t handle_pte_marker(struct vm_faul= t *vmf) > if (WARN_ON_ONCE(!marker)) > return VM_FAULT_SIGBUS; > > + /* Poison emulation explicitly requested for this PTE. */ > + if (marker & PTE_MARKER_UFFD_POISON) > + return VM_FAULT_HWPOISON; > + > /* Higher priority than uffd-wp when data corrupted */ > if (marker & PTE_MARKER_SWAPIN_ERROR) > return VM_FAULT_SIGBUS; > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > index a2bf37ee276d..87b62ca1e09e 100644 > --- a/mm/userfaultfd.c > +++ b/mm/userfaultfd.c > @@ -286,6 +286,51 @@ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, > goto out; > } > > +/* Handles UFFDIO_POISON for all non-hugetlb VMAs. */ > +static int mfill_atomic_pte_poison(pmd_t *dst_pmd, > + struct vm_area_struct *dst_vma, > + unsigned long dst_addr, > + uffd_flags_t flags) > +{ > + int ret; > + struct mm_struct *dst_mm =3D dst_vma->vm_mm; > + pte_t _dst_pte, *dst_pte; > + spinlock_t *ptl; > + > + _dst_pte =3D make_pte_marker(PTE_MARKER_UFFD_POISON); > + dst_pte =3D pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); > + > + if (vma_is_shmem(dst_vma)) { > + struct inode *inode; > + pgoff_t offset, max_off; > + > + /* serialize against truncate with the page table lock */ > + inode =3D dst_vma->vm_file->f_inode; > + offset =3D linear_page_index(dst_vma, dst_addr); > + max_off =3D DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); > + ret =3D -EFAULT; > + if (unlikely(offset >=3D max_off)) > + goto out_unlock; > + } > + > + ret =3D -EEXIST; > + /* > + * For now, we don't handle unmapping pages, so only support fill= ing in > + * none PTEs, or replacing PTE markers. > + */ > + if (!pte_none_mostly(*dst_pte)) > + goto out_unlock; > + > + set_pte_at(dst_mm, dst_addr, dst_pte, _dst_pte); > + > + /* No need to invalidate - it was non-present before */ > + update_mmu_cache(dst_vma, dst_addr, dst_pte); > + ret =3D 0; > +out_unlock: > + pte_unmap_unlock(dst_pte, ptl); > + return ret; > +} > + > static pmd_t *mm_alloc_pmd(struct mm_struct *mm, unsigned long address) > { > pgd_t *pgd; > @@ -336,8 +381,12 @@ static __always_inline ssize_t mfill_atomic_hugetlb( > * supported by hugetlb. A PMD_SIZE huge pages may exist as used > * by THP. Since we can not reliably insert a zero page, this > * feature is not supported. > + * > + * PTE marker handling for hugetlb is a bit special, so for now > + * UFFDIO_POISON is not supported. > */ > - if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE)) { > + if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE) || > + uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) { > mmap_read_unlock(dst_mm); > return -EINVAL; > } > @@ -481,6 +530,9 @@ static __always_inline ssize_t mfill_atomic_pte(pmd_t= *dst_pmd, > if (uffd_flags_mode_is(flags, MFILL_ATOMIC_CONTINUE)) { > return mfill_atomic_pte_continue(dst_pmd, dst_vma, > dst_addr, flags); > + } else if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON)) { > + return mfill_atomic_pte_poison(dst_pmd, dst_vma, > + dst_addr, flags); > } > > /* > @@ -702,6 +754,14 @@ ssize_t mfill_atomic_continue(struct mm_struct *dst_= mm, unsigned long start, > uffd_flags_set_mode(flags, MFILL_ATOMIC_CONTI= NUE)); > } > > +ssize_t mfill_atomic_poison(struct mm_struct *dst_mm, unsigned long star= t, > + unsigned long len, atomic_t *mmap_changing, > + uffd_flags_t flags) > +{ > + return mfill_atomic(dst_mm, start, 0, len, mmap_changing, > + uffd_flags_set_mode(flags, MFILL_ATOMIC_POISO= N)); > +} > + > long uffd_wp_range(struct vm_area_struct *dst_vma, > unsigned long start, unsigned long len, bool enable_wp= ) > { > -- > 2.41.0.255.g8b1d071c50-goog >