From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E797C433EF for ; Mon, 27 Sep 2021 17:36:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 156AA60E09 for ; Mon, 27 Sep 2021 17:36:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 156AA60E09 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8967E6B006C; Mon, 27 Sep 2021 13:36:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 845736B0071; Mon, 27 Sep 2021 13:36:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70D09900002; Mon, 27 Sep 2021 13:36:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0109.hostedemail.com [216.40.44.109]) by kanga.kvack.org (Postfix) with ESMTP id 61B366B006C for ; Mon, 27 Sep 2021 13:36:31 -0400 (EDT) Received: from smtpin32.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 201AB8249980 for ; Mon, 27 Sep 2021 17:36:31 +0000 (UTC) X-FDA: 78634057782.32.2F74211 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf17.hostedemail.com (Postfix) with ESMTP id C040BF00038D for ; Mon, 27 Sep 2021 17:36:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632764178; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Kx1DzZsvr2Xg6OJE0dxcX64G8Fhnxv7OEFrQ0oSnlMA=; b=G4ykE/2T/+cFHcIayRsjVxi4izBey91KT5/jA+SZjbA0t9Qb6jr3hh8lIhImduQELw/6pL 33LHJuBbFowCYVMmXL/8nqb0Oqhg66UntIwAE7QxKCK2q1tXYw4ThlSklWE3lBwdaF0aWq Mmf+4bHS3WboP6nHvGlkD6a8NNXjYN8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632764190; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Kx1DzZsvr2Xg6OJE0dxcX64G8Fhnxv7OEFrQ0oSnlMA=; b=ZUR4yZkZw3JPOJIgSr7Qmvcs2dqCE+bEIi6/9t4lqSJ3ZAqs7IzAp+Ify0aCE9ICUiiOLp kAmLIfCe7G7hr2cNtctdjPWP8qCqQgcUe+pn1gPT4VK7HMl5MVwPVy1c3nake9GZnxJRHy Noj5bBMFJx5VOEWtweDUrwQQf0whFRo= Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-446-y9cIzRrmNOCbQ9GMuVG9dw-1; Mon, 27 Sep 2021 13:36:14 -0400 X-MC-Unique: y9cIzRrmNOCbQ9GMuVG9dw-1 Received: by mail-io1-f70.google.com with SMTP id i11-20020a056602134b00b005be82e3028bso22283988iov.18 for ; Mon, 27 Sep 2021 10:36:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Kx1DzZsvr2Xg6OJE0dxcX64G8Fhnxv7OEFrQ0oSnlMA=; b=TEE5ciBFmXacDG5jxvXGfyqrjFpJNRufEyxLk7Qz2K+sGb62YFw7bKhr+qoBexgTiC T70Sz87h+ujIoLmLqANCgAIJCcO74RPI/gQcDxcUkYpDyTkfRCX2r3GKcTgkO0OT2PTE 2RLa2gy2bMrK4Sr5oVG10A5q1Ks6znEtHhHFeNULeWhlnCnhsCiMtj8Q24xu0zh5Avo/ eBRrASaBsDZi8VSwEZY2bkqSW5zIZy0rojx7yV4wuajX8Au1G9alINQuBIsLNDEawb0r QBTJHbs023lXJT+iwkx+9Y0ROLA3/593pgurfstfiLh3Y6MCQrF4k5Pta13D+Z1PkrQE aXdQ== X-Gm-Message-State: AOAM533XMeTu1DZ4cW19Xv4gRtMv0f7DP6VR5fpjUDN/VnJpp6HjGyxD XhYH5YFwbXYimEcVrvexIto4FOycxX3dR7CY9GYPoJp+FHSirsudXWlmSKsCIAgyZ0oekhLf0At 71JqMVD8K3+c= X-Received: by 2002:a6b:f805:: with SMTP id o5mr638399ioh.131.1632764173455; Mon, 27 Sep 2021 10:36:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQvaPSgeWlbzwDiqYEYVMbq1Xj6c+GiHHrarcR7/YoxjzTe8wNONJynFvm0oNyk1dAFEsafw== X-Received: by 2002:a6b:f805:: with SMTP id o5mr638377ioh.131.1632764173188; Mon, 27 Sep 2021 10:36:13 -0700 (PDT) Received: from t490s ([2607:fea8:56a2:9100::d3ec]) by smtp.gmail.com with ESMTPSA id h5sm9318414ioz.31.2021.09.27.10.36.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Sep 2021 10:36:12 -0700 (PDT) Date: Mon, 27 Sep 2021 13:36:10 -0400 From: Peter Xu To: Axel Rasmussen Cc: Linux MM , LKML , Andrew Morton , Andrea Arcangeli , Nadav Amit , Li Wang Subject: Re: [PATCH] mm/userfaultfd: selftests: Fix memory corruption with thp enabled Message-ID: References: <20210923232512.210092-1-peterx@redhat.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: C040BF00038D X-Stat-Signature: pccr3mozuhma8f8n99nsmaebeswbjk4y Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="G4ykE/2T"; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZUR4yZkZ; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf17.hostedemail.com: domain of peterx@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=peterx@redhat.com X-HE-Tag: 1632764190-706629 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Sep 24, 2021 at 03:59:32PM -0400, Peter Xu wrote: > On Fri, Sep 24, 2021 at 10:21:30AM -0700, Axel Rasmussen wrote: > > On Thu, Sep 23, 2021 at 4:25 PM Peter Xu wrote: > > > > > > In RHEL's gating selftests we've encountered memory corruption in the uffd > > > event test even with upstream kernel: > > > > > > # ./userfaultfd anon 128 4 > > > nr_pages: 32768, nr_pages_per_cpu: 32768 > > > bounces: 3, mode: rnd racing read, userfaults: 6240 missing (6240) 14729 wp (14729) > > > bounces: 2, mode: racing read, userfaults: 1444 missing (1444) 28877 wp (28877) > > > bounces: 1, mode: rnd read, userfaults: 6055 missing (6055) 14699 wp (14699) > > > bounces: 0, mode: read, userfaults: 82 missing (82) 25196 wp (25196) > > > testing uffd-wp with pagemap (pgsize=4096): done > > > testing uffd-wp with pagemap (pgsize=2097152): done > > > testing events (fork, remap, remove): ERROR: nr 32427 memory corruption 0 1 (errno=0, line=963) > > > ERROR: faulting process failed (errno=0, line=1117) > > > > > > It can be easily reproduced when global thp enabled, which is the default for > > > RHEL. > > > > > > It's also known as a side effect of commit 0db282ba2c12 ("selftest: use mmap > > > instead of posix_memalign to allocate memory", 2021-07-23), which is imho right > > > itself on using mmap() to make sure the addresses will be untagged even on arm. > > > > > > The problem is, for each test we allocate buffers using two allocate_area() > > > calls. We assumed these two buffers won't affect each other, however they > > > could, because mmap() could have found that the two buffers are near each other > > > and having the same VMA flags, so they got merged into one VMA. > > > > > > It won't be a big problem if thp is not enabled, but when thp is agressively > > > enabled it means when initializing the src buffer it could accidentally setup > > > part of the dest buffer too when there's a shared THP that overlaps the two > > > regions. Then some of the dest buffer won't be able to be trapped by > > > userfaultfd missing mode, then it'll cause memory corruption as described. > > > > > > To fix it, do release_pages() after initializing the src buffer. > > > > But, if I understand correctly, release_pages() will just free the > > physical pages, but not touch the VMA(s). So, with the right > > max_ptes_none setting, why couldn't khugepaged just decide to > > re-collapse (with zero pages) immediately after we release the pages, > > causing the same problem? It seems to me this change just > > significantly narrows the race window (which explains why we see less > > of the issue), but doesn't fix it fundamentally. > > Did you mean you can reproduce the issue even with this patch? > > It is a good point anyway, indeed I don't see anything stops it from happening. > > I wanted to prepare a v2 by releasing the pages after uffdio registration where > we'll do the vma split, but it won't simply work because release_pages() will > cause the process to hang death since that test registers with EVENT_REMOVE, > and release_pages() upon the thp will trigger synchronous EVENT_REMOVE which > cannot be handled by anyone. > > Another solution is to map some PROT_NONE regions between the buffers, to make > sure they won't share a VMA. I'll need to think more about which is better.. Axel, let me know if you can reproduce an issue with this patch, or otherwise would you mind we keep this patch in -mm and fix the issue first? I can never reproduce any issue with current patch even if I agree you're probably right, however before the patch is mostly 100% reproducable to fail. It's just that after the weekend when I look back I still don't see a 100% clean way to fix it yet. Mapping 4K PROT_NONE before/after each allocation is the most ideal but still looks tricky to me. Would you have time on looking for a better solution, so as to (see it a way to) complete what commit 8ba6e8640844 whats to do afterwards? Thanks, -- Peter Xu