From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE806EB64DA for ; Wed, 5 Jul 2023 17:14:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C04B8D0003; Wed, 5 Jul 2023 13:14:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 670858D0001; Wed, 5 Jul 2023 13:14:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 560848D0003; Wed, 5 Jul 2023 13:14:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 487388D0001 for ; Wed, 5 Jul 2023 13:14:53 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id D4A6A40352 for ; Wed, 5 Jul 2023 17:14:52 +0000 (UTC) X-FDA: 80978208024.07.C14B9E3 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by imf08.hostedemail.com (Postfix) with ESMTP id B78E7160020 for ; Wed, 5 Jul 2023 17:14:50 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OIdlq2Gg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of surenb@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688577290; a=rsa-sha256; cv=none; b=o1Bs3LCB5/vZQJ4/pKWpc/Je0v+/HIveh7zyFUdACs9K5pc12Lk+AIBotgjrot6P6BMUUn nA/LbMDSDFvD7SAYInuXBXWy8ylWSizHJwwmFXdg86mWFKP1dcmPtscGMaWiDzWyy9XXm4 7KvN7PDZrYet1Nhu9eVn0C4361VdSqA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OIdlq2Gg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf08.hostedemail.com: domain of surenb@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688577290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7af1C7YlyT/T6R9rGIVx9ssRIh5Vdv+9xO7E9CdybRo=; b=7dxLtiOf1ive1eo9f8qDWXl8yWtZIv9nz9Qxuohcw1RagKU6H+yOKlFuagZuO17AOMRnmI wv09HSuc+2orFCAAyZ8ESSgyrG0KAdRNJVGjkNmTKAY4lV/91C8TnwW5WcDYxIkjRezDQu f6Lkr2BMiAxqW7p6sEK4lHHgx1Y4kLw= Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-c4dd86f5d78so1478682276.0 for ; Wed, 05 Jul 2023 10:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688577289; x=1691169289; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=7af1C7YlyT/T6R9rGIVx9ssRIh5Vdv+9xO7E9CdybRo=; b=OIdlq2Gg0GIxH/8ccdJ5pXUa+Fq2yqk2DSw7FGlIvRZBFAKrerdZJU/11ibBT+6ecn VcPHex1XI7Mmme3RLUbdILWhN5u1LovIbsJzPEfeNV+SWdRZOkx9+2vXys4p1qDnpGi8 BT7Rq0CnNgJqBu736F725DQbIxqJ/yuM6+n+a9oum3bvcc4+sfRxWcmnPIAOac5FgHcT +B+bNBKfZ/M1onZt/0G3aoGfh/aCSpfTlmIbBiTkgKta/M2pdCuZs3gsNocn09JnVWWj Sbjor9kwA85MYH6pIbRTi1SVeYcoSLDlVwSszhLhbm/+0UOe03PswjIB7Dy91aLB2rd+ zxPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688577289; x=1691169289; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7af1C7YlyT/T6R9rGIVx9ssRIh5Vdv+9xO7E9CdybRo=; b=JiRvdHK3HZ0bNz8AFsi0a8joxpvdeDHe0EgYKuynSo1Y/h4S+4tYkyHvdsuUrMOY+C vsycq8J7p/4ydTxxgN/kpQAXyKgTGKuMQOr3SbZ2hFVC6vH9jNMKrHfvzrUmBslWfQGZ ZFnjVH/7IXWtYN9v8rCwq+/PTMIixey3K/yx+Rji9RUs+qWUsgpSUdajqz84jPRBuqXi yjxw6Ow0wMrUZHKE4qQO0pjIE5i3c/bkAIqWjpF63YeY5pBG4qtnqGcY1F/AOnHuEwuG ccrIoYPNGZNWqcxf5d+LwfhmM5mwEB/ZaYTs0PuGZzB7/eww6hnnUQFHlKlo9vIZRJP2 Loug== X-Gm-Message-State: ABy/qLaUETH19MRjVjfAACdmgYxPtLhiECsiLMQxFsScKI2De8rd9Fwt Du4CKABfGHm+7/C73TsYk4UKcJg6btPjEqnqerUnqw== X-Google-Smtp-Source: APBJJlEX3rgKhr8mlEaVqEbAP0IHoIbU3dQBjYg7kFhHKFndrDa9V5NAcLVaYvpx08xjlz8fYPwgK76mxaEeSW0UxTk= X-Received: by 2002:a25:add6:0:b0:b9e:c516:6e32 with SMTP id d22-20020a25add6000000b00b9ec5166e32mr2028417ybe.24.1688577289460; Wed, 05 Jul 2023 10:14:49 -0700 (PDT) MIME-Version: 1.0 References: <20230705063711.2670599-1-surenb@google.com> <20230705063711.2670599-2-surenb@google.com> <970295ab-e85d-7af3-76e6-df53a5c52f8b@redhat.com> In-Reply-To: From: Suren Baghdasaryan Date: Wed, 5 Jul 2023 10:14:38 -0700 Message-ID: Subject: Re: [PATCH v2 1/2] fork: lock VMAs of the parent process when forking To: David Hildenbrand Cc: akpm@linux-foundation.org, jirislaby@kernel.org, jacobly.alt@gmail.com, holger@applied-asynchrony.com, hdegoede@redhat.com, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B78E7160020 X-Stat-Signature: g1wd19cnnwf68xqosfx3iinrqikrxi3n X-HE-Tag: 1688577290-603187 X-HE-Meta: U2FsdGVkX1/Sn8/HXSQfzXxhHS/iaW3R2qy4clucuHH83g2izpxEQqNGiUixUPAAD2eVTjiGZXEOaF6j/4haNH5TJteQkHD6PbNygbgQBtTMnwV/8XuZ9IrO6/c+nwqxcwmbKjKKqzzAhUdwtDWNR4ehqVm8P3HOzQzX4VDpv3CfrR+GqCV0kMddb4juILJwIKvfdBNZV3Q08n5yUjGCfRlUxnOG2yLYWuM1j2Gukth+WkQNyasYL20D5JfLiSeJJimBPI7AEOdTM91cuDx5qXNE2Wz4YqycETxEyhgpsaGPTjKcD2qDTGx7yvrs+Z/sN147ry8vDZHxtz9qfsQwi1/x5ONovXNnQmnUVJcuy029kngoYZLhr6HEey8N8YiM/8RnMLxm7mWgUofJYF9YhsCBMN/X2x9rzHJpaR2d4Dp9mjtzh09tlynacKQzmE4I8A0kR/wX1DTWyoEEBABFoXIpAMkZCs0/h2TkjJBPzZGTGqbx0bbzdq2LVZChfz8k0HF5d7KZCBjha6DoU551ri6SdyXys3Rzp8+2I+7fK+oeH5WqvCnk8D7lhO02RKjvjppFNldT7R3ARUsmhGbLYrEvZMtP8gpZ/118uRylkJ1/RhNcxNJSKwPz/Uy4kZDjBs2QMQIHmJDmgi4+X18wRz/H9r1v21Cch+nVWPxj73lrsw81I/ktnA5snHPve12Xv7T/CKhi8EaNCIei+EPdMQloINuU6qEUIH6qVpd0MNHtH6JOuKrbH1bK8Zl4TmOS9ekD0h7aWWIGdQJKfTYZxLk12ZtBJExnjr83nhO15qzmAKmIPi8ukpZYF51lCoPcVaFirsIiAaLJhFIAUNntSDc3amkmmIhxqEIDj88gql8U/XmKdz+nj3wyTtHZRRrJ9SOulH6xYLx3WGHrXTXyGoQgQebKTGXr1LPSSjl1+MqdI7uFEUQmmru07I7F9VY+7uYjHN9ppasd6jJIL/L SRGn+oka Z/GqTUmhfQSbQx96+rzsxjy9+hZpRVwmsyDBx3c2OCfUY0F2IDL8+LSW1RijmMDFOvU/TUR8z9UzwlUx03W+7BT38u3+5S1f/s2uBpw5MVKoFQ2VNlmTIYQYDxe7s4PCBWIiM+8JTbt/P7Kofed5pMk+Cwj1xBam2j6hHuQ7ys1xpgYnZbj8Xxi6xuoiU9AN8VIySaaWB4wL0oU3V7SFCXlCh/RZVoovYil5hUynsLbyBXI3MbhUHlvDX6a48A6kHNbv/mEk4EUc4blwcZvSan8VeWcuSl/wS2zfHGcrNaMZooPr2wX050cHVUltzvs4nyhXHdrhmn7O4b8UJmiKOTpjyNDc+5ysh8GSnVaK8+IvI/VbPuyMa1lzgYCjlrCUKS0wDZfkZPwG7ZpCqT60vDlb54V0jHs77D8WVsfWmGVUdLsLpLw/G4AGujXCMiKl2fHktYzPoyh//soZV1dfgBzFpFTIKGmVKh25vzLlfu3+vmUU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 5, 2023 at 9:10=E2=80=AFAM Suren Baghdasaryan wrote: > > On Wed, Jul 5, 2023 at 1:08=E2=80=AFAM David Hildenbrand wrote: > > > > On 05.07.23 08:37, Suren Baghdasaryan wrote: > > > When forking a child process, parent write-protects an anonymous page > > > and COW-shares it with the child being forked using copy_present_pte(= ). > > > Parent's TLB is flushed right before we drop the parent's mmap_lock i= n > > > dup_mmap(). If we get a write-fault before that TLB flush in the pare= nt, > > > and we end up replacing that anonymous page in the parent process in > > > do_wp_page() (because, COW-shared with the child), this might lead to > > > some stale writable TLB entries targeting the wrong (old) page. > > > Similar issue happened in the past with userfaultfd (see flush_tlb_pa= ge() > > > call inside do_wp_page()). > > > Lock VMAs of the parent process when forking a child, which prevents > > > concurrent page faults during fork operation and avoids this issue. > > > This fix can potentially regress some fork-heavy workloads. Kernel bu= ild > > > time did not show noticeable regression on a 56-core machine while a > > > stress test mapping 10000 VMAs and forking 5000 times in a tight loop > > > shows ~5% regression. If such fork time regression is unacceptable, > > > disabling CONFIG_PER_VMA_LOCK should restore its performance. Further > > > optimizations are possible if this regression proves to be problemati= c. > > > > Out of interest, did you also populate page tables / pages for some of = these > > VMAs, or is this primarily looping over 10000 VMAs that don't actually = copy any > > page tables? > > I did not populate the page tables, therefore this represents the > worst case scenario (the share of time used to lock the VMAs is > maximized). > > > > > > > > > Suggested-by: David Hildenbrand > > > Reported-by: Jiri Slaby > > > Closes: https://lore.kernel.org/all/dbdef34c-3a07-5951-e1ae-e9c6e3cdf= 51b@kernel.org/ > > > Reported-by: Holger Hoffst=C3=A4tte > > > Closes: https://lore.kernel.org/all/b198d649-f4bf-b971-31d0-e8433ec2a= 34c@applied-asynchrony.com/ > > > Reported-by: Jacob Young > > > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=3D217624 > > > Fixes: 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling = first") > > > Cc: stable@vger.kernel.org > > > Signed-off-by: Suren Baghdasaryan > > > --- > > > kernel/fork.c | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/kernel/fork.c b/kernel/fork.c > > > index b85814e614a5..d2e12b6d2b18 100644 > > > --- a/kernel/fork.c > > > +++ b/kernel/fork.c > > > @@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_st= ruct *mm, > > > for_each_vma(old_vmi, mpnt) { > > > struct file *file; > > > > > > + vma_start_write(mpnt); > > > if (mpnt->vm_flags & VM_DONTCOPY) { > > > vm_stat_account(mm, mpnt->vm_flags, -vma_pages(= mpnt)); > > > continue; > > > > After the mmap_write_lock_killable(), there will still be a period wher= e page > > faults can happen. Essentially, page faults can happen for a VMA until = we lock that VMA. > > > > I cannot immediately name something that is broken allowing for that, a= nd this change > > should fix the issue at hand, but exotic things like > > > > flush_cache_dup_mm(oldmm); > > > > make me wonder if we really want to allow for that or if there is some = other corner case > > in fork() handling that really doesn't expect concurrent page faults (a= nd, thereby, page > > table modifications) with fork. > > > > For example, documentation/core-api/cachetlb.rst says > > > > 2) ``void flush_cache_dup_mm(struct mm_struct *mm)`` > > > > This interface flushes an entire user address space from > > the caches. That is, after running, there will be no cache > > lines associated with 'mm'. > > > > This interface is used to handle whole address space > > page table operations such as what happens during fork. > > > > This option is separate from flush_cache_mm to allow some > > optimizations for VIPT caches. > > > > I see. So, we really need to lock all VMAs before > flush_cache_dup_mm(). Makes sense. I'll post an update to this patch > shortly. v3 of the patchset with this fix is posted at https://lore.kernel.org/all/20230705171213.2843068-1-surenb@google.com/ > Thanks, > Suren. > > > > > An alternative that requires another VMA walk would be > > > > diff --git a/kernel/fork.c b/kernel/fork.c > > index 41c964104b58..0f182d3f049b 100644 > > --- a/kernel/fork.c > > +++ b/kernel/fork.c > > @@ -662,6 +662,13 @@ static __latent_entropy int dup_mmap(struct mm_str= uct *mm, > > retval =3D -EINTR; > > goto fail_uprobe_end; > > } > > + > > + /* Disallow any page faults early by locking all VMAs. */ > > + if (IS_ENABLED(CONFIG_PER_VMA_LOCK)) { > > + for_each_vma(old_vmi, mpnt) > > + vma_start_write(mpnt); > > + vma_iter_init(old_vmi, old_mm, 0); > > + } > > flush_cache_dup_mm(oldmm); > > uprobe_dup_mmap(oldmm, mm); > > /* > > -- > > 2.41.0 > > > > > > Unless there are other thoughts, I guess you change is fine regarding t= he problem > > at hand. Not so sure regarding any other corner cases, that's why I'm s= pelling it out. > > > > > > Acked-by: David Hildenbrand > > > > -- > > Cheers, > > > > David / dhildenb > >