From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 646FEEB64DA for ; Thu, 6 Jul 2023 00:20:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E36EB8D0002; Wed, 5 Jul 2023 20:20:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBEF58D0001; Wed, 5 Jul 2023 20:20:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C877C8D0002; Wed, 5 Jul 2023 20:20:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B79688D0001 for ; Wed, 5 Jul 2023 20:20:20 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 87EC51C85DD for ; Thu, 6 Jul 2023 00:20:20 +0000 (UTC) X-FDA: 80979280200.04.28775B8 Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) by imf01.hostedemail.com (Postfix) with ESMTP id CD8A140013 for ; Thu, 6 Jul 2023 00:20:18 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p032VP+I; spf=pass (imf01.hostedemail.com: domain of surenb@google.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688602818; a=rsa-sha256; cv=none; b=pqsCBSDeqJD6nL/+9duJmYkw8cy4nyO5kJgf81kU0zKvqPy9VSkjSg95bp+IaEu9/PQFpx CCybcUJ/+d7P+7jX2BygqRxNI/hddvj7jXC8uy/4qcPER4GHZp+I3aTOGDZssQcMXvMnm7 2bp3Z+AKUZW27qgwaOvLVcbRNh3PeJs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=p032VP+I; spf=pass (imf01.hostedemail.com: domain of surenb@google.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688602818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=knzXabcAOJG0/E2AKfeBSXuBhoKGya+QyzfiqVLDY0c=; b=bKg8bE5wFtbvVkB3eHs0m0aAIRUvGeczUwPRTjzpVYxbI3+ErQqvSHa8i5SL8SxyXeaOJP GsEAZGUrvRmaEvadSraepWDAeZj3G6y8n/C6hF4azM6si6BJjJw3WNq3VOMaXUOqgjsre0 jluNSVkG+Va+GApfhd+qfYK9nICKKxU= Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-57688a146ecso1989857b3.2 for ; Wed, 05 Jul 2023 17:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688602817; x=1691194817; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=knzXabcAOJG0/E2AKfeBSXuBhoKGya+QyzfiqVLDY0c=; b=p032VP+I2cT4M0GRmBEuHaLjt1qMukmrGcg8nfFGTMhfFwdBKiYmHMidsUKlN72pQJ 89p0zkgDGkg05ayMLp2Zy15EuH4npY69hHErx+Ej+kA/kC/qzOm3CUzu+xFCHRpnfDnm p+KpKUqfjUwdApExidI7GYGs6y1jPKFnFsxomzO7q8al/MgFZF01MxK56YWvcJU0wn5Z CpQN8XPo+70kUOPhKaQyYvA604uKpOxsoFZ36HqckrVSP5UAPYW49hCmhn0+PcWwmCgk WAJXaJRJc7irtS3p+QPkcUaxDzgRLG+WvpJWgiBHERoneGF7rrpmU/2NPfF8/nnutgVm cYWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688602817; x=1691194817; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=knzXabcAOJG0/E2AKfeBSXuBhoKGya+QyzfiqVLDY0c=; b=Y9dSgNzx2YiTs96U3PIuHjzxcvpko641GZ7O3JNjuV9Ow6B0ZioverToi1c0Nv8GQz zIQM5F/gvuHldImtucuZVojgtPgIkj2ZZSAMDdzPvg/wOh37umvE6+8eXtcPJYiG88bH 26Z3bd7R+Sh5u8DUdkFkKth65MnmddVWoB+mpOqyUeUaJ9/EOPlz7sHIa09tM+KpJKBm nAI/X5JV8JeICzUA9S+gvjbPUYzwZiglHN0b0lIB5Wu0GFei8haHPZHzthun1dzJESNS 3LzrkXvVlBA8w4gk3LilSzM2laUD7toLIXOIdhq3FuSj1c0VaKxLF/L0QSEJZc3IJqqr TcHg== X-Gm-Message-State: ABy/qLZC82MtTncpVZYeRmUr8PMdpaaFmDwxfyIV04uBHrfqClDDQ79f wupz0vIMGjqfcm7VEmE4201Vyy3GjVHhzwbnNhanpQ== X-Google-Smtp-Source: APBJJlG9B/AgV8ZlpmDyoFeYHyoyaXzxDtkaKfnNIU8/AaWY2+nG/ix41X59c785FqRwTiYk/QEqPu6RBiViX4Se3rM= X-Received: by 2002:a25:1657:0:b0:bad:125f:9156 with SMTP id 84-20020a251657000000b00bad125f9156mr353741ybw.35.1688602817224; Wed, 05 Jul 2023 17:20:17 -0700 (PDT) MIME-Version: 1.0 References: <20230705171213.2843068-1-surenb@google.com> <20230705171213.2843068-2-surenb@google.com> <10c8fe17-fa9b-bf34-cb88-c758e07c9d72@redhat.com> <20230705230647.twq3n5nb2iabr7uk@revolver> In-Reply-To: <20230705230647.twq3n5nb2iabr7uk@revolver> From: Suren Baghdasaryan Date: Wed, 5 Jul 2023 17:20:06 -0700 Message-ID: Subject: Re: [PATCH v3 1/2] fork: lock VMAs of the parent process when forking To: "Liam R. Howlett" , Suren Baghdasaryan , David Hildenbrand , akpm@linux-foundation.org, jirislaby@kernel.org, jacobly.alt@gmail.com, holger@applied-asynchrony.com, hdegoede@redhat.com, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: CD8A140013 X-Stat-Signature: mcexofmckiuk4yx4dcwhz49rsdsk8gk3 X-Rspam-User: X-HE-Tag: 1688602818-940157 X-HE-Meta: U2FsdGVkX18cvtxTDWlScXER0z7Izosxv76l3OAGbRMvNzskpsH8wMXkeq+YTJ2ZALWRCqRURblfr7rxwcBLQCMXgTiBdwcA3bRsBZeQwjOSAVhsdc5e9Txl+595R/ffukHy+CYfvrT72tByvqFrQV+QkJA0+y569FoTrB+DupmVjlQFpAaL0U9zWyUcmMsEJoBbV1Pbkk0yOwRG7DHElOuMDlCB57b8Rq1XIrKdPTp22POzRDa+p9+QPzJHMFuXFX2ZWxECxNsLUt1uw5jRgbJGEp2M03z9pNg6+0DxrUhSMkK/vE4KAp1fEEioSGsMnAYOS8DzvXjMwhlgtApKfYhLBILJLMkspxEtn8JIrrOQbtw8NLguSARb/DUwhf63zMdLlbHtF7RSiJ1IolG/sEY6n4oZwfHYmuOQBDkJukeqZkkKwpQDEI4ETXtPArSCO8Q0Ll4hGT72R76/iBmi5PqXYixsJ243BT5nDy8VcMdXHTo80WtaObX77TpMlXgw53ZgVuLvdNfGjQ4whtdfd6fzoU2sY5ze11uHyzaNDE7kB3EaA7MlLeWgMZ+G/KAi6lW1/TNeG3U+IKNjltRm/Vw/xHPjj0FSHnoSWmXhzu5kQzIz24W+FUNdO7GH58P8A6LiLuJhu9b6aRC9s916WLBqAuIESvYwFL38ifwS2ppGaHG9UTLsAGphQVEwYsh0pr0dkJNqalBZWo/iOrZ2CEXsAmbI5Xkhsc3clqGVUTwk2bU+lc6WauPjaJzWvDRAtiSGiqpIHlW6U3kmVk8lEw7uZAlIicdK+IwbsauzwlX9PSholsx8FrHMVRoFIYbz+qfsSsmzOQqVvE8kF0e0lB1x9DCNFyb+SdoBcxErbZBKTgD+sKspr4z13uj+OFdUeIH8dNgB3EHTtBjYWf/sb5Zj6T5D1csrxKDeQyF5OxuIhLJlOpJ/0dc+ecKPfSDK4VR0jvG63WQUxu90LHT KsoiRBsx QOILI0th/kzcueZfoDN3HcTpD3QZQDCi/R2Nl9D6kYcJuFWMmnOD6Q0/XjD6uWl1uecrzgWT0P/VGBVLDbRd2haNwjqr32MpA9UXnzwTsTTPbSuj3iGn/LZfNUkseNda+85s+adMelrnMJRBiM3jODPWv9JH4LU9D5elkw0rzfCzXvGDxlNox2yr9gNaklSTpusz3aVqX+bA6jE2y+TTxiJCxMEDt1u90Ps/cavEO3IX33dJlx/1Qdp1dVuTnYY4sztY2FTAhkCvXiYlKYK5WrUuSCtfal7aLBlh7RdvuVrh7mbEdOIRq8TXJDYQpvvX8ikxVqPJZwCfIQ2GwcTa6StPuU/X8nkmBqQkIzslbno3DxdrZd/MaJhIp11LCnebOqDMn46lXXpANyzCnk5O0RbAyio45ZuNWrNwl+PAG5OFA2Pu+N/P4pJdkv/DA3L+Eq1XgZdtzJSWC+stVxfrC6RCybRTUI197AOU0nkzoMprE7kalcYKBfrdftjRIkKrM2v2G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 5, 2023 at 4:07=E2=80=AFPM Liam R. Howlett wrote: > > * Suren Baghdasaryan [230705 13:24]: > > On Wed, Jul 5, 2023 at 10:14=E2=80=AFAM David Hildenbrand wrote: > > > > > > On 05.07.23 19:12, Suren Baghdasaryan wrote: > > > > When forking a child process, parent write-protects an anonymous pa= ge > > > > and COW-shares it with the child being forked using copy_present_pt= e(). > > > > Parent's TLB is flushed right before we drop the parent's mmap_lock= in > > > > dup_mmap(). If we get a write-fault before that TLB flush in the pa= rent, > > > > and we end up replacing that anonymous page in the parent process i= n > > > > do_wp_page() (because, COW-shared with the child), this might lead = to > > > > some stale writable TLB entries targeting the wrong (old) page. > > > > Similar issue happened in the past with userfaultfd (see flush_tlb_= page() > > > > call inside do_wp_page()). > > > > Lock VMAs of the parent process when forking a child, which prevent= s > > > > concurrent page faults during fork operation and avoids this issue. > > > > This fix can potentially regress some fork-heavy workloads. Kernel = build > > > > time did not show noticeable regression on a 56-core machine while = a > > > > stress test mapping 10000 VMAs and forking 5000 times in a tight lo= op > > > > shows ~5% regression. If such fork time regression is unacceptable, > > > > disabling CONFIG_PER_VMA_LOCK should restore its performance. Furth= er > > > > optimizations are possible if this regression proves to be problema= tic. > > > > > > > > Suggested-by: David Hildenbrand > > > > Reported-by: Jiri Slaby > > > > Closes: https://lore.kernel.org/all/dbdef34c-3a07-5951-e1ae-e9c6e3c= df51b@kernel.org/ > > > > Reported-by: Holger Hoffst=C3=A4tte > > > > Closes: https://lore.kernel.org/all/b198d649-f4bf-b971-31d0-e8433ec= 2a34c@applied-asynchrony.com/ > > > > Reported-by: Jacob Young > > > > Closes: https://bugzilla.kernel.org/show_bug.cgi?id=3D217624 > > > > Fixes: 0bff0aaea03e ("x86/mm: try VMA lock-based page fault handlin= g first") > > > > Cc: stable@vger.kernel.org > > > > Signed-off-by: Suren Baghdasaryan > > > > --- > > > > kernel/fork.c | 6 ++++++ > > > > 1 file changed, 6 insertions(+) > > > > > > > > diff --git a/kernel/fork.c b/kernel/fork.c > > > > index b85814e614a5..403bc2b72301 100644 > > > > --- a/kernel/fork.c > > > > +++ b/kernel/fork.c > > > > @@ -658,6 +658,12 @@ static __latent_entropy int dup_mmap(struct mm= _struct *mm, > > > > retval =3D -EINTR; > > > > goto fail_uprobe_end; > > > > } > > > > +#ifdef CONFIG_PER_VMA_LOCK > > > > + /* Disallow any page faults before calling flush_cache_dup_mm= */ > > > > + for_each_vma(old_vmi, mpnt) > > > > + vma_start_write(mpnt); > > > > + vma_iter_init(&old_vmi, oldmm, 0); > > vma_iter_set(&old_vmi, 0) is probably what you want here. Ok, I send another version with that. > > > > > +#endif > > > > flush_cache_dup_mm(oldmm); > > > > uprobe_dup_mmap(oldmm, mm); > > > > /* > > > > > > The old version was most probably fine as well, but this certainly lo= oks > > > even safer. > > > > > > Acked-by: David Hildenbrand > > I think this is overkill and believe setting the vma_start_write() will > synchronize with any readers since it's using the per-vma rw semaphore > in write mode. Anything faulting will need to finish before the fork > continues and faults during the fork will fall back to a read lock of > the mmap_lock. Is there a possibility of populate happening outside the > mmap_write lock/vma_lock? Yes, I think we understand the loss of concurrency in the parent's ability to fault pages while forking. Is that a real problem though? > > Was your benchmarking done with this loop at the start? No, it was done with the initial version where the lock was inside the existing loop. I just reran the benchmark and while kernel compilation times did not change, the stress test shows ~7% regression now, probably due to that additional tree walk. I'll update that number in the new patch. Thanks! > > Thanks, > Liam