From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB7ADC636CC for ; Fri, 17 Feb 2023 02:00:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D21E6B0072; Thu, 16 Feb 2023 21:00:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 080246B0073; Thu, 16 Feb 2023 21:00:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB2A66B0074; Thu, 16 Feb 2023 21:00:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D96DC6B0072 for ; Thu, 16 Feb 2023 21:00:39 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 82C3781321 for ; Fri, 17 Feb 2023 02:00:39 +0000 (UTC) X-FDA: 80475129798.07.E843126 Received: from mail-lj1-f178.google.com (mail-lj1-f178.google.com [209.85.208.178]) by imf11.hostedemail.com (Postfix) with ESMTP id 571C340004 for ; Fri, 17 Feb 2023 02:00:35 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=fNk5zNOC; spf=pass (imf11.hostedemail.com: domain of stevensd@chromium.org designates 209.85.208.178 as permitted sender) smtp.mailfrom=stevensd@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676599235; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y2ocsOZ5sFWY4NltltauFVvgFvdTgNqHwTi9Mu49FDg=; b=HTxPUj96du3hyiu4bCccUipV8ADaSBmlC+Ibs3wLpTgFiZpIL4NRjNUjGbBkEoMCVKA36n 5VJabocXCcbgpffwH7SXlj3l5pMu9EFnPhxk5L7g1BNbLRD2uZtnqg1UhuXJ/pyzlAOmH+ GASwIhvAI658BQNCY71prR4GuEl1P+U= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=fNk5zNOC; spf=pass (imf11.hostedemail.com: domain of stevensd@chromium.org designates 209.85.208.178 as permitted sender) smtp.mailfrom=stevensd@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676599235; a=rsa-sha256; cv=none; b=6r41zD+kGQzbE1dN2hSsMoP0QnJVHOFd4fW4WU3qcU7S4AqXqols4MQmubcs2MfwR9ZCQI vVMEYo7Lj2wcOt1QH4OgvKB5BuYxkRa9g9ZteVAO0kMyEkkzGuL0zAiH4WCPSau6QTJQ5u cfJsTBLBopZVa+pqRB6JOZM9f9GwVRQ= Received: by mail-lj1-f178.google.com with SMTP id a9so3868138ljr.13 for ; Thu, 16 Feb 2023 18:00:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Y2ocsOZ5sFWY4NltltauFVvgFvdTgNqHwTi9Mu49FDg=; b=fNk5zNOCGdQ4EnoLYyGTzws0gKUiffwFQeo68HV71/jh2lAx+2XqheIrmXxZLuqkxT I1cvaXkFewBfLU+JW9ew2g2//MSA1Nl3obqTH6p3UQ+k8Jq49cWB7iZOjJybj29KNIIc 76Gk9ZdJNrvIxvh0sFZsd6mhmA3tcSvh2Xs8s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Y2ocsOZ5sFWY4NltltauFVvgFvdTgNqHwTi9Mu49FDg=; b=1jsIf8WjYKJWTy94+3arffVJanKxVgZJdpD7tzSmMksOYoD+ncafAIEycUmaPaUIxM yVNVUL4PUp3dIMWjSu0TDF+JyfHkyXSXK+os3Pxrd1X7yDiBtDZ+e+tXDE5U0XMLU8yP g9BzOaNPfubtdA2PDGk9iNnW21SDk4EvnPzSMUXCb44PmKqm3joIKjSROMURjLbdtr7I UWNQ00rHSg0hSacWcXZKeV/f27gtXYOb7HWB9ahIqSRjAg/7aOGPGKzlEIxRXNIosCTT yNWDdJA7W4yNAQH+KP/EazwLxWJvrPsS4qb/BGWWQWWxDB8Jl1ly9ea48ETCeWZHErUs trpQ== X-Gm-Message-State: AO0yUKWmQTNRcKJILC0pvU8afu7zqArAGuNTubkPmYtCv405E0E311gy MJyb/QDjNs1xBWTzNrasCwtXwqddk+LGj9YCSYVJDA== X-Google-Smtp-Source: AK7set9FLLImfo/3mu20BiERlsz/quZ1Wba+WRmkZxOxFJfoiexyz5N6nl5jlT9uhbJcsbWBrRgrhT+XB1zoPJWJAsM= X-Received: by 2002:a2e:8e2a:0:b0:293:4e16:89b3 with SMTP id r10-20020a2e8e2a000000b002934e1689b3mr2231758ljk.3.1676599233372; Thu, 16 Feb 2023 18:00:33 -0800 (PST) MIME-Version: 1.0 References: <20230214075710.2401855-1-stevensd@google.com> <20230214075710.2401855-2-stevensd@google.com> In-Reply-To: From: David Stevens Date: Fri, 17 Feb 2023 11:00:22 +0900 Message-ID: Subject: Re: [PATCH 2/2] mm/khugepaged: skip shmem with userfaultfd To: Peter Xu Cc: linux-mm@kvack.org, Matthew Wilcox , Andrew Morton , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 571C340004 X-Stat-Signature: ow1cam6zu6t7gmnznjf4fyiucosqqj5d X-HE-Tag: 1676599235-80391 X-HE-Meta: U2FsdGVkX18E2ObjYVUHv6nb0U4conDWWi8+QAmLyRqWYlraU/zWPlBMbkcvSMBSXMEAkjN+IhVgmKlXguBBZsu/MXCZPtJbx5GQmBfUsbRJON7zZwSgkPCbr35xK28+GgBSxb0S3LUtXU1C6rqWkbDV/IryJlhqOOmVy95FQVZ9a8auiOE/HI33KKfpz9GgVBFgiul+BZsYGyhlePcFU064G5Bya8d9FvTuIjwgsFKGkZB4LEyvWJV5n6pxEUCmDuLHVcturJGEB66QsCHAQcnbPoTBIeDoxiYD6GH3epMMvnns4x2b9xda8QNEzTZ8aThnVCYyfLIVnZlP1TQpYz6ujUjws0BwrHA66JJGdYaq4+kMB/VGjun3bX6HachkuzrxUVhQFJS5ROdtMCHZ09D/me/vZFvQGtcmpw81H6Vx5iARrYiIYCKBSLEKenEjhXBx2qvEL60RrTRwK2rzYjpRDu/v6yqPo5ook2xawOK45JvFwpsow47REBBC6unk5Qr5tRdZkHtGJ+S0Al7rTtF6Lel5fTgW4ZICXY7jKbs6z02BvG3SHolZmO6m/L/bIhYJhySfyUDZlYDZGtWVAIaN54mKlUt3Kvx2Ygs14KwbkCW548UQ30VZ6ebQRlxyQgqDJS/hXXeR0mL6T5IrHQTaPnDYZ0aIBOwwZSCFcHIbVvJCvAfw1KqsTflC3RJxJbFKptbLne0JgKiPjT+m27O6Fv5Y4JgnA/S67B9Pt7UZH4KQ3X6pK6ttA/Lb9l5At1p3nglcKHYeylV7l4RPm2peQvoWZO7gX6hqgg/tA/8/tBbvWvYm+AxPjfFtoasCP31vzYrbx+0dEhCgoBCNHynPauf/LtnjxeCtYqqSXiMN2wNpgcafa5JzJ5PiH3a42Q4ieu8J6o3BWLJdgYF8xm0KoZNdYN4T4dRS+6F6nEfY56asm0tS+mR5aGHoQy5QX5IOD5m4Msd2ok7R+2N NONvngnp 0srMslpyhvmFoSIXX3n0nw7lE7enHyEYFv/pAMaz5GWzl9lD9lRaZHKemQGzSlkeImUNrSXBNo8oxfBlwNVEzj2U/vdrgcr+6X5mSDnzG1pdTRxaRM7ShFyLWt6+DEjCHkt41g1h7LgSFSBYYk6Rkth2W1yHpHWxUQecvyjoO/507htxyfNT9Z+L2VpALOiCAslujLYdFRo2dnkGnEXBjATbcyFT8hloC/enAYUaVsyYSGkc8HKZd4RD9ZgCkuk+8aLfz2IW1E1Se36MDSCK4KAwo3sbiaLktOkME98TgU9+XtaOfbSceksB/cfYlUTZn0BTtNCYJJBSA+y8eJKfcxC2QY3AttoVNbkU00G3u0K8bpujyo8nXqvkhXxQjo0VG51x9Q959aRcaMHbfsShxkiGK+RwdDAkStgOV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 16, 2023 at 11:41 PM Peter Xu wrote: > > On Thu, Feb 16, 2023 at 10:37:47AM +0900, David Stevens wrote: > > On Thu, Feb 16, 2023 at 7:48 AM Peter Xu wrote: > > > > > > On Tue, Feb 14, 2023 at 04:57:10PM +0900, David Stevens wrote: > > > > From: David Stevens > > > > > > > > Make sure that collapse_file respects any userfaultfds registered with > > > > MODE_MISSING. If userspace has any such userfaultfds registered, then > > > > for any page which it knows to be missing, it may expect a > > > > UFFD_EVENT_PAGEFAULT. This means collapse_file needs to take care when > > > > collapsing a shmem range would result in replacing an empty page with a > > > > THP, so that it doesn't break userfaultfd. > > > > > > > > Synchronization when checking for userfaultfds in collapse_file is > > > > tricky because the mmap locks can't be used to prevent races with the > > > > registration of new userfaultfds. Instead, we provide synchronization by > > > > ensuring that userspace cannot observe the fact that pages are missing > > > > before we check for userfaultfds. Although this allows registration of a > > > > userfaultfd to race with collapse_file, it ensures that userspace cannot > > > > observe any pages transition from missing to present after such a race. > > > > This makes such a race indistinguishable to the collapse occurring > > > > immediately before the userfaultfd registration. > > > > > > > > The first step to provide this synchronization is to stop filling gaps > > > > during the loop iterating over the target range, since the page cache > > > > lock can be dropped during that loop. The second step is to fill the > > > > gaps with XA_RETRY_ENTRY after the page cache lock is acquired the final > > > > time, to avoid races with accesses to the page cache that only take the > > > > RCU read lock. > > > > > > > > This fix is targeted at khugepaged, but the change also applies to > > > > MADV_COLLAPSE. MADV_COLLAPSE on a range with a userfaultfd will now > > > > return EBUSY if there are any missing pages (instead of succeeding on > > > > shmem and returning EINVAL on anonymous memory). There is also now a > > > > window during MADV_COLLAPSE where a fault on a missing page will cause > > > > the syscall to fail with EAGAIN. > > > > > > > > The fact that intermediate page cache state can no longer be observed > > > > before the rollback of a failed collapse is also technically a > > > > userspace-visible change (via at least SEEK_DATA and SEEK_END), but it > > > > is exceedingly unlikely that anything relies on being able to observe > > > > that transient state. > > > > > > > > Signed-off-by: David Stevens > > > > --- > > > > mm/khugepaged.c | 66 +++++++++++++++++++++++++++++++++++++++++++------ > > > > 1 file changed, 58 insertions(+), 8 deletions(-) > > > > > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > > index b648f1053d95..8c2e2349e883 100644 > > > > --- a/mm/khugepaged.c > > > > +++ b/mm/khugepaged.c > > > > @@ -55,6 +55,7 @@ enum scan_result { > > > > SCAN_CGROUP_CHARGE_FAIL, > > > > SCAN_TRUNCATED, > > > > SCAN_PAGE_HAS_PRIVATE, > > > > + SCAN_PAGE_FILLED, > > > > > > PS: You may want to also touch SCAN_STATUS in huge_memory.h next time. > > > > > > > }; > > > > > > > > #define CREATE_TRACE_POINTS > > > > @@ -1725,8 +1726,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, > > > > * - allocate and lock a new huge page; > > > > * - scan page cache replacing old pages with the new one > > > > * + swap/gup in pages if necessary; > > > > - * + fill in gaps; > > > > > > IIUC it's not a complete removal, but just moved downwards: > > > > > > > * + keep old pages around in case rollback is required; > > > > + * - finalize updates to the page cache; > > > > > > + fill in gaps with RETRY entries > > > + detect race conditions with userfaultfds > > > > > > > * - if replacing succeeds: > > > > * + copy data over; > > > > * + free old pages; > > > > @@ -1805,13 +1806,12 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, > > > > result = SCAN_TRUNCATED; > > > > goto xa_locked; > > > > } > > > > - xas_set(&xas, index); > > > > + xas_set(&xas, index + 1); > > > > } > > > > if (!shmem_charge(mapping->host, 1)) { > > > > result = SCAN_FAIL; > > > > goto xa_locked; > > > > } > > > > - xas_store(&xas, hpage); > > > > nr_none++; > > > > continue; > > > > } > > > > @@ -1970,6 +1970,56 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, > > > > put_page(page); > > > > goto xa_unlocked; > > > > } > > > > + > > > > + if (nr_none) { > > > > + struct vm_area_struct *vma; > > > > + int nr_none_check = 0; > > > > + > > > > + xas_unlock_irq(&xas); > > > > + i_mmap_lock_read(mapping); > > > > + xas_lock_irq(&xas); > > > > + > > > > + xas_set(&xas, start); > > > > + for (index = start; index < end; index++) { > > > > + if (!xas_next(&xas)) { > > > > + xas_store(&xas, XA_RETRY_ENTRY); > > > > + nr_none_check++; > > > > + } > > > > + } > > > > + > > > > + if (nr_none != nr_none_check) { > > > > + result = SCAN_PAGE_FILLED; > > > > + goto immap_locked; > > > > + } > > > > + > > > > + /* > > > > + * If userspace observed a missing page in a VMA with an armed > > > > + * userfaultfd, then it might expect a UFFD_EVENT_PAGEFAULT for > > > > + * that page, so we need to roll back to avoid suppressing such > > > > + * an event. Any userfaultfds armed after this point will not be > > > > + * able to observe any missing pages due to the previously > > > > + * inserted retry entries. > > > > + */ > > > > + vma_interval_tree_foreach(vma, &mapping->i_mmap, start, start) { > > > > + if (userfaultfd_missing(vma)) { > > > > + result = SCAN_EXCEED_NONE_PTE; > > > > + goto immap_locked; > > > > + } > > > > + } > > > > + > > > > +immap_locked: > > > > + i_mmap_unlock_read(mapping); > > > > + if (result != SCAN_SUCCEED) { > > > > + xas_set(&xas, start); > > > > + for (index = start; index < end; index++) { > > > > + if (xas_next(&xas) == XA_RETRY_ENTRY) > > > > + xas_store(&xas, NULL); > > > > + } > > > > + > > > > + goto xa_locked; > > > > + } > > > > + } > > > > + > > > > > > Until here, all look fine to me (ignoring patch 1 for now; assuming the > > > hpage is always uptodate). > > > > > > My question is after here we'll release page cache lock again before > > > try_to_unmap_flush(), but is it safe to keep RETRY entries after releasing > > > page cache lock? It means other threads can be spinning. I assume page > > > lock is always safe and sleepable, but not sure about the page cache lock > > > here. > > > > We insert the multi-index entry for hpage before releasing the page > > cache lock, which should replace all of the XA_RETRY_ENTRYs. So the > > page cache will be fully up to date when we release the lock, at least > > in terms of which pages it contains. > > IIUC we released it before copying the pages: > > xa_locked: > xas_unlock_irq(&xas); <-------------------------------- here > xa_unlocked: > > /* > * If collapse is successful, flush must be done now before copying. > * If collapse is unsuccessful, does flush actually need to be done? > * Do it anyway, to clear the state. > */ > try_to_unmap_flush(); > > Before insertion of the multi-index: > > /* Join all the small entries into a single multi-index entry. */ > xas_set_order(&xas, start, HPAGE_PMD_ORDER); > xas_store(&xas, hpage); Okay, I realize what's going on. There is a change in mm-everything [1] that significantly rewrites collapse_file, and my patch is going to conflict with that patch. I'll see if I can rework my patches on top of that change. [1] https://lore.kernel.org/all/20221205234059.42971-3-jiaqiyan@google.com/T/#u -David