From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A69BEB64DA for ; Wed, 5 Jul 2023 16:11:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10CE68D0002; Wed, 5 Jul 2023 12:11:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 095178D0001; Wed, 5 Jul 2023 12:11:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E781C8D0002; Wed, 5 Jul 2023 12:11:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D1B6C8D0001 for ; Wed, 5 Jul 2023 12:11:16 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9C97A403C5 for ; Wed, 5 Jul 2023 16:11:16 +0000 (UTC) X-FDA: 80978047752.01.27B8487 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) by imf04.hostedemail.com (Postfix) with ESMTP id 759E640009 for ; Wed, 5 Jul 2023 16:11:13 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p3sGXarQ; spf=pass (imf04.hostedemail.com: domain of surenb@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688573473; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U4CizjzMyVryUzo9zkH8yPfHNB6eZIgaisjI66Xz0ps=; b=XYPUsFymTyjitFnn+ZVQWxNy3vI8coJCnbcxAJZ9irh5erWKblh7j8aF8UVAoaxsIfisn9 sPHbljIIb5P9l0VnftWjoHUXI2/L1ZPt3oAIt3b6xngiPpuUaiE1+nYTBs0/VZg/KFxtLn SGCYfRngama2cozCBgVw2zPLWP8+ALQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p3sGXarQ; spf=pass (imf04.hostedemail.com: domain of surenb@google.com designates 209.85.219.171 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688573473; a=rsa-sha256; cv=none; b=UkboT8k6cwf8qTAlBjFvsrN64uhAV1DP3csdjCDcl+uECbCqTXxVMqWBaccMcs/HjD1fF6 ROlt/QOMf2lWk87dAhBbgcKmqgVzH3KnGtlalzdj1BN3IMQi9vCqTRBOVHaqHrKIlGpZ7h aytN8som+2IfqCoTJmjawgrTGPq6Ues= Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-bacf685150cso7888881276.3 for ; Wed, 05 Jul 2023 09:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688573472; x=1691165472; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=U4CizjzMyVryUzo9zkH8yPfHNB6eZIgaisjI66Xz0ps=; b=p3sGXarQz4x4/EmK2Xq5/rWClhl7cjpV3yLtHQHjTD/GnYOz7/mfQyb18mnyca94W0 sroj10MWFWDJM8ZkO/U8ygDvPd9iBDHjw3VuReL4hsYcQ205TAAn4Tp2GJjLs2//L/VQ rVq8XGqc6+Qbdn9CgKabb9EgMoFGPWEnP3AyKvPFLmI5YsLWmA7/n8N3j/iXFjuiSlct bRg7ZCqB9z/ASsrjPlzxypGqGuvFrayFmgyyapxuPag/8Syfiyz57BG1M6LSygq3A8WS s8dKKhJJvCF8MgCVfxOH5FMLGwn313ijAVrU7BMPNN3Ky3JR2Tf+j0REf1U5rgUXDsxV bHIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688573472; x=1691165472; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=U4CizjzMyVryUzo9zkH8yPfHNB6eZIgaisjI66Xz0ps=; b=cx0K2yt/bpL5jFg1/F9jsCgTh1YsD4AgD5yylGH3D4yz5ZppmWTlXx5ms1dNp6SZg2 kfefReCYpixjgvp8j3w1O+Fmj2RJmi6r74fXwYM1MCK/GkjcybyBo5Zse+bv/barrCPL IsRqIIDGxzV1iekIxxWBktqcMememWH5lE2YDX5Y60vWo6ouIYiBvCcsqEVJbxXTEM8q RPIff8yEWB9dPbyJOJFqaTSPOhglq9Rw6ly+//cQolbDG4vIa7532eLLM3IlgMbdG/t4 FxdrrGSG0GipgoqzRjD1mFEvb5tb5trNxLObbfQebFyxy37WVPbacfRxOqiJzbHGPv+o DiqA== X-Gm-Message-State: ABy/qLaC08KA1F+mOd67WO4DLtGOnjQFAI0Vfgw71Aw5Gps02SdZqnvb SAUyFzKicYy6HLNuvleVhizevR8BOfWSbJjbD24xwQ== X-Google-Smtp-Source: APBJJlHulVtZG5SP3B5FT5kXTl3vzTRyAFjlJZNFEC+4RVdD1bhcdNpFSDHB6IB+VKa5q1dQjsSw3rX4BYZHYErisWs= X-Received: by 2002:a25:b315:0:b0:c5d:6992:6fa7 with SMTP id l21-20020a25b315000000b00c5d69926fa7mr4528975ybj.9.1688573471234; Wed, 05 Jul 2023 09:11:11 -0700 (PDT) MIME-Version: 1.0 References: <20230705063711.2670599-1-surenb@google.com> <20230705063711.2670599-2-surenb@google.com> <970295ab-e85d-7af3-76e6-df53a5c52f8b@redhat.com> In-Reply-To: <970295ab-e85d-7af3-76e6-df53a5c52f8b@redhat.com> From: Suren Baghdasaryan Date: Wed, 5 Jul 2023 09:10:58 -0700 Message-ID: Subject: Re: [PATCH v2 1/2] fork: lock VMAs of the parent process when forking To: David Hildenbrand Cc: akpm@linux-foundation.org, jirislaby@kernel.org, jacobly.alt@gmail.com, holger@applied-asynchrony.com, hdegoede@redhat.com, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 759E640009 X-Rspam-User: X-Stat-Signature: gd9c13wg31x6ubmd1711kkonxpwu5gia X-Rspamd-Server: rspam01 X-HE-Tag: 1688573473-681248 X-HE-Meta: U2FsdGVkX1+Ni4zva4O8dai+scoRTN1ykjx1xiohq02gXE6KkzT9SP2yqxMRkAV94dSj2lotWyWzpY+JZfMYTFfNCmofXENKrUYPcZA1031z6dYyQBLxMgepgRx1QfYUeeaXP3hn81esp42ZRg6h2e+L/VDhyVe0KqSD3GZoVb6s9gGbW2t22vL6KtpRJyGGeenG7ifUPoDaG3uehiaWCxzk6xMAsPnDxYvPN715QIWVtYh9RRbvMHEfqq0Rs6w9atTl+dQ+MS85YsN9/4kfYwfdXdw6a/SNdXwFF2dh9BDdCZpyftUEFuFqqM+ceryeLu/CG7TqbsKgEanb+modDqpi0mZN3VejbOAyKRKhwdIDbyFeFC7V1DALx0sHLqDCqBIcJrQp4XZ7iZxDNAebVnD06OsyuDi4l05U7jvjm61Ev/lJCHtxoZCw+WPICaxs3qd2EU0hrTlkL/D89Bm17yuFzMmAtOuBihMC1LPwjKbP/v3Bu36gTeMqiS4uyJki+ArhcV99TvUXCBMOBm9rC1O4eshzagI8XK75MXxCHC5B9FJ6p5JItbPYYgNy/1F8Cpp9ygQrX4GQdIS5zH2k/p+R9a/aqIVltPybzSSfBl1iow/HIbbP5v8uvHtNvFLBebj1uQ2bu3UzRoYqKVZ4mopbeX1bVJ6lzdSzeTdCf0fQtb0+XUnfAmqGGDvV4m+kaaZgdnphQ7bnSIrdvylNcgcyQy6ikpHpxxFEugOYaTAnCUwj1Lp/qYB009s0NEVdWeS5DNz9DdELREaKhNJ+LwYlzH8xeBaWpHUYAJPLNLaFTe2iUJd3uYQScr/MDYDZTzkRI27OZxC2MCssrShfqTp6ZKBNt2x/5Rap+hFmkgImvjr/CNYOO8VjT3lMhUoGJH7cRVzx36A6xUX+oQPlRGkO0uF0HIr3PacHNNQRw3qIil/GzCMDs5Ff1G0iWhkwaVUwsfHSqYzgfsIX0uA 01oqgthE y76AUuxVYk2Q9IgoioOvpq+J7wmgcj/jNpE8pUwsD4JXRupq8E3JL/aBt4WM+LX1O7OugdXk+iviRfGe6hlpfb/aXd+urswCVqHUroILwiOWLoZAtdNc8nXavgCymOxdcRV2J3EFw/UoDB9NmZY1psjGPFV+0HmoUCa25lAKAjw/xUHK5GxOQKTxVO9KiZ4RO2brxGe+KRY+L0a/h57uTfeZ3Tzzmg92how1YhZLzmnucR1feLuPf04hypcS2jn9qsrNzCAfCsJEqM1iQMntggL6m1QuE9i0pDmPGnq2rKvMUeuvftX3PseumE5s4+x1HWmOk+gcKoAkzVklGkXofUJDZ6qcEmmQnxtKlxbK6HjOc+rgKS6oLfICxxEB38sdTE4zMFFMJ3caJtsdmIPPGBeDjsJ4iPfrr8QK15tWtkmf0VZAKRH6FnaFwbeqq9WlWjQwDCN6IzNwfF2vRYmkQCIfKAWrsqsc2tUGIKeAXU+1pxjM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 5, 2023 at 1:08=E2=80=AFAM David Hildenbrand = wrote: > > On 05.07.23 08:37, Suren Baghdasaryan wrote: > > When forking a child process, parent write-protects an anonymous page > > and COW-shares it with the child being forked using copy_present_pte(). > > Parent's TLB is flushed right before we drop the parent's mmap_lock in > > dup_mmap(). If we get a write-fault before that TLB flush in the parent= , > > and we end up replacing that anonymous page in the parent process in > > do_wp_page() (because, COW-shared with the child), this might lead to > > some stale writable TLB entries targeting the wrong (old) page. > > Similar issue happened in the past with userfaultfd (see flush_tlb_page= () > > call inside do_wp_page()). > > Lock VMAs of the parent process when forking a child, which prevents > > concurrent page faults during fork operation and avoids this issue. > > This fix can potentially regress some fork-heavy workloads. Kernel buil= d > > time did not show noticeable regression on a 56-core machine while a > > stress test mapping 10000 VMAs and forking 5000 times in a tight loop > > shows ~5% regression. If such fork time regression is unacceptable, > > disabling CONFIG_PER_VMA_LOCK should restore its performance. Further > > optimizations are possible if this regression proves to be problematic. > > Out of interest, did you also populate page tables / pages for some of th= ese > VMAs, or is this primarily looping over 10000 VMAs that don't actually co= py any > page tables? I did not populate the page tables, therefore this represents the worst case scenario (the share of time used to lock the VMAs is maximized). > > > > > Suggested-by: David Hildenbrand > > Reported-by: Jiri Slaby > > Closes: https://lore.kernel.org/all/dbdef34c-3a07-5951-e1ae-e9c6e3cdf51= b@kernel.org/ > > Reported-by: Holger Hoffst=C3=A4tte > > Closes: https://lore.kernel.org/all/b198d649-f4bf-b971-31d0-e8433ec2a34= c@applied-asynchrony.com/ > > Reported-by: Jacob Young > > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=3D217624 > > Fixes: 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handling fi= rst") > > Cc: stable@vger.kernel.org > > Signed-off-by: Suren Baghdasaryan > > --- > > kernel/fork.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/kernel/fork.c b/kernel/fork.c > > index b85814e614a5..d2e12b6d2b18 100644 > > --- a/kernel/fork.c > > +++ b/kernel/fork.c > > @@ -686,6 +686,7 @@ static __latent_entropy int dup_mmap(struct mm_stru= ct *mm, > > for_each_vma(old_vmi, mpnt) { > > struct file *file; > > > > + vma_start_write(mpnt); > > if (mpnt->vm_flags & VM_DONTCOPY) { > > vm_stat_account(mm, mpnt->vm_flags, -vma_pages(mp= nt)); > > continue; > > After the mmap_write_lock_killable(), there will still be a period where = page > faults can happen. Essentially, page faults can happen for a VMA until we= lock that VMA. > > I cannot immediately name something that is broken allowing for that, and= this change > should fix the issue at hand, but exotic things like > > flush_cache_dup_mm(oldmm); > > make me wonder if we really want to allow for that or if there is some ot= her corner case > in fork() handling that really doesn't expect concurrent page faults (and= , thereby, page > table modifications) with fork. > > For example, documentation/core-api/cachetlb.rst says > > 2) ``void flush_cache_dup_mm(struct mm_struct *mm)`` > > This interface flushes an entire user address space from > the caches. That is, after running, there will be no cache > lines associated with 'mm'. > > This interface is used to handle whole address space > page table operations such as what happens during fork. > > This option is separate from flush_cache_mm to allow some > optimizations for VIPT caches. > I see. So, we really need to lock all VMAs before flush_cache_dup_mm(). Makes sense. I'll post an update to this patch shortly. Thanks, Suren. > > An alternative that requires another VMA walk would be > > diff --git a/kernel/fork.c b/kernel/fork.c > index 41c964104b58..0f182d3f049b 100644 > --- a/kernel/fork.c > +++ b/kernel/fork.c > @@ -662,6 +662,13 @@ static __latent_entropy int dup_mmap(struct mm_struc= t *mm, > retval =3D -EINTR; > goto fail_uprobe_end; > } > + > + /* Disallow any page faults early by locking all VMAs. */ > + if (IS_ENABLED(CONFIG_PER_VMA_LOCK)) { > + for_each_vma(old_vmi, mpnt) > + vma_start_write(mpnt); > + vma_iter_init(old_vmi, old_mm, 0); > + } > flush_cache_dup_mm(oldmm); > uprobe_dup_mmap(oldmm, mm); > /* > -- > 2.41.0 > > > Unless there are other thoughts, I guess you change is fine regarding the= problem > at hand. Not so sure regarding any other corner cases, that's why I'm spe= lling it out. > > > Acked-by: David Hildenbrand > > -- > Cheers, > > David / dhildenb >