From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA593C3DA4A for ; Thu, 8 Aug 2024 21:02:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 766026B008C; Thu, 8 Aug 2024 17:02:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EF3A6B0092; Thu, 8 Aug 2024 17:02:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 569186B0095; Thu, 8 Aug 2024 17:02:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 33C8B6B008C for ; Thu, 8 Aug 2024 17:02:49 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D6D7612053A for ; Thu, 8 Aug 2024 21:02:48 +0000 (UTC) X-FDA: 82430302416.07.42975AE Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) by imf10.hostedemail.com (Postfix) with ESMTP id 15923C0026 for ; Thu, 8 Aug 2024 21:02:46 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QS1qpJYR; spf=pass (imf10.hostedemail.com: domain of surenb@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723150934; a=rsa-sha256; cv=none; b=Isd/3r4aP299CQcQ+Q7a/9UQhsJyogIP7Hl9ZWyItmG0EyLQGpVryRRW8Y8MX0d36s91Ok 36Laac/zlU2738ulw3riTdN9lnEEamLP/OleeDzM+tZWpLL1H7f0uKSq+f8vxPFBPnwbUo u9TyDI16T6QbeCLYCu2RO/U8HMm6Az8= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=QS1qpJYR; spf=pass (imf10.hostedemail.com: domain of surenb@google.com designates 209.85.128.172 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723150934; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VK0xxjAchJmTqLyqblW6a1BSUgp4bmsDLqA1sO88AlI=; b=mi/LuYKCRStwYnB34fyhI0hsnPICnPpy+xmqhyD0WaHjOGNiSUnnwjxiQBX3hJuNxKLWDz yyXV9YefRbntc1EBJthjFVzypvfog68zmVWjKBx8UA5nYHizGvG/ivvjVD60tdTRYnoCjX D/aHYBT1o9tZMAdDXA/eLsnXKSIIqdw= Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-690404fd27eso13325007b3.2 for ; Thu, 08 Aug 2024 14:02:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723150966; x=1723755766; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=VK0xxjAchJmTqLyqblW6a1BSUgp4bmsDLqA1sO88AlI=; b=QS1qpJYRy+1260mc2SKp5Lo4Ej91/hzGC8wpcT1a0EQttTg13B6DBCv8qv0Eq6IYcj RlsCv8uKHBsdxCJ5JsLdkn6TrHSZtdeDov48aBWkaI6OmB+TKvklvAwgy0AEL48/qyKC s5ySMuBYcJMxCts5P0fe4pA3HOyTI80sqqeYKQp/LgegIXQssPaW0iwZUGrlMqULHyKU qGNn906Y33bCseInuNGGxL9BivSvuWUctQ5vqwk119vHX4KzJMHpsiTp/oIbl0q+cvdI PZrVO4MHMCSWl5ZN7whwC9LCr34J1tbJgSu6d8BWC5pXo+/9TG/IqyQnzAuB/MJyN9Ah jbFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723150966; x=1723755766; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VK0xxjAchJmTqLyqblW6a1BSUgp4bmsDLqA1sO88AlI=; b=pIf7bdwm9nQ00I0aRae9tb/w+DVTBkZGkUl/HaAfvUiORkOmD79ENXpJz8+bhStE8x F17rLKx6jkbz8fxQHUjWsN1WsnLtVYJWGxz5OQ7PbKjOxQI5BKPLojc5sZWA78Rm1pni WHKLP26JBbNwHAh4IWtM43JufOjUbTixwN+UL6jjZEqgkWedN1GhAkOqhDtekkVp2Taa 97nJFG5lnW19f8/ptgphbPlTRKpImmiJnqdrTxHa790hPiJ0vJm2mAGPyBDK6HvWg6zm GreQ0BHRBsuqRRbvJoxfsZJKq+zuqepifAYmu4NV6jfrAMdqPzZNn37HSK5m28Vj+Jy+ 4EHw== X-Forwarded-Encrypted: i=1; AJvYcCWttifo9F1pvHh/KgZ6EJvPuPfSh7e++c4jpPauaZTLG/ph3PR2jRImQIS9u+97O97KzGVj+WrHLK7jVycCXl0FvfM= X-Gm-Message-State: AOJu0Yx7FLNi4VgN5QAW3udaHmsX+VZtnRlmgKZT50bPXLsmcNHVQImE SP9pUu5CJME5D8iyY2d4LPGx3g9Oj0WhLEvePcyQ99w8RKD88hENLoQOKTJganWqRzUehMaEbse VXBZM7PpzW0XsDMe1xEH8jJ8gBryXXb+uHfb1 X-Google-Smtp-Source: AGHT+IEtGpSOUY+7BEqAMsrA6RDC+KM1NYPNuRKKblL87JlMO05gMjbAol7QNVp6PBC4L80ueZKGVTxHTa6ugxo2rps= X-Received: by 2002:a05:690c:618a:b0:64b:2cf2:391c with SMTP id 00721157ae682-69bf81ddd67mr29616357b3.18.1723150965620; Thu, 08 Aug 2024 14:02:45 -0700 (PDT) MIME-Version: 1.0 References: <20240807182325.2585582-1-surenb@google.com> In-Reply-To: From: Suren Baghdasaryan Date: Thu, 8 Aug 2024 21:02:32 +0000 Message-ID: Subject: Re: [RFC 1/1] mm: introduce mmap_lock_speculation_{start|end} To: Andrii Nakryiko Cc: akpm@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jann Horn , Matthew Wilcox , Vlastimil Babka , Michal Hocko Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: uodnkw5wmsatc8pww3r3hykzyuqntrnf X-Rspamd-Queue-Id: 15923C0026 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723150966-791370 X-HE-Meta: U2FsdGVkX19GeG/e9xhnqJ7RL/ALkdYp4yojC3PLXceniUVlwULuIZavC/IiTnyMiT/0QkEHwq0rlKvqLdCGLd7snaKrQXtiL9pfUvAIMHLON6mV6DpTQLGy1hDXuDQREOE4F+IPOE8g71GDs5AoaN1YqIf/jIfJa8PMpi2/0aRQ21saYf7ck3fq6wzorWsnjdGEiH9DbI4kXAtQ6nKWsEeuxbDBznNhi/BcCDqaHZCNDaC0ckVrzAUiyAFXe6Zyo5/4DeX6FVHfj62FZ2y6volnetWG+Gf2CqoQGqFpKu2C/coNCJu35QKwjGA+JiusuBBgIxPW+ptYrk4kM974gZBnrph2vVIysEhybl/TQtqBPcQmQJ9HhXxH24lKDg8sFqrCeR41WLl3cjjryLl2hG7u/5zIP2q++7ePPbbZS8yqOgPIR7Cls+Cyx4DJ+Ia6FZzBj3rcyr9h35rUjeuzfpl2HAl0pwNbtSiU9Iy/uB0J4qM4C66imNPEnqVqqF/ec2b4Nw3/Lj3EIgReAqZ6KfQp/YITQ2TKE+3D7V6UpIO4jk9CkLX2NJ9usTAQ5fQlD3iLq6pG7DmqTKw5AQPm48uE5BiU0ivBNbovo1AXQIPBtTg/OQDTFko7AiSda1NjFY4pwe7PyHikaZLgV1S8+Az9pTZu3A0oI5k+c7Yc/ee09j7lDQT3XtCxDHSZOUn55uEtofP+XDsBNrXxpPHBygceSFMKv1bIRsZXSGROYfvhtpkgFm8dV6e1DvBBR5e5Wmz8DPGd8qaQ0RiIMv8giepx/jnXFPkiwWqTiTjJ5IrVe3aTm0IE6Rge6JJP6dZOuar94hCRb9ytAdhLj4/zPbVjZ+sRCh4WWI+3+tqald3rbmicjfzaWr+PtTL+mZ1V9V0S+3OzJlovni86oiKL7+GQ2KjLUScgYEogHNk39Hub4bkd8peBm1rB7IWx4svC6fepz20iKhVMt/Ux40s cZZOZGgW 5DWWejFAVKgyJyoZFz8Pt6uoMJUpitTNOONh7O5cnmf0lrEXUYXd9PioVUL6aAcI5h24RCc5laCp15NmGgcS3LvgBgIJBaC4pGcSMyLeQMb3G+isGZKmeZEBK1j34MnTmW7KxDXOG3YCOiTtg7zbeOyn0uXSy8q/ecFO2L8Yh/7yZQHpFX79NTAk8cCZ5Y8WbNRxdtPHqKNuYB8bGSco8ryfvnaREMRKFU9ckELIImfv/e5qsd6zc8BDmxdaQiVS3TeXfTko/x0J+tWhSVOVdwIr7ImJ3+yOGVMPhs52MQpAeBC7b7S2Vx8lBElSww0z+Hh+qKNhx8h+NQ0A0ecuWlLyszbdNiDJ+Dt2JnKYkNnhY/lz3E1qtuoN/IQCOD7U8HbdtuNKkunGQOch/wjS/cVZaoI5VQdWzmVbBLoM2AhvWWffIPqaKzhDIhLGahe8qlzFdtcNxs0+CCwUDB9ZAaJZUi+phK6VRDiiksfwXz2uBBh7yjxZTa9abWB0ONTy2bKvvNRTzQuVv5sxXxPtd9AbQyq5d/3nJNMuSugg/+9pEj+NYiMCmDJvqTDObtMUxdfdu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Aug 8, 2024 at 8:19=E2=80=AFPM Andrii Nakryiko wrote: > > On Wed, Aug 7, 2024 at 11:23=E2=80=AFAM Suren Baghdasaryan wrote: > > > > Add helper functions to speculatively perform operations without > > read-locking mmap_lock, expecting that mmap_lock will not be > > write-locked and mm is not modified from under us. > > > > Signed-off-by: Suren Baghdasaryan > > Suggested-by: Peter Zijlstra > > Cc: Andrii Nakryiko > > --- > > This change makes sense and makes mm's seq a bit more useful and > meaningful. I've also tested it locally with uprobe stress-test, and > it seems to work great, I haven't run into any problems with a > multi-hour stress test run so far. Thanks! Thanks for testing and feel free to include this patch into your set. I've been thinking about this some more and there is a very unlikely corner case if between mmap_lock_speculation_start() and mmap_lock_speculation_end() mmap_lock is write-locked/unlocked so many times that mm->mm_lock_seq (int) overflows and just happen to reach the same value as we recorded in mmap_lock_speculation_start(). This would generate a false positive, which would show up as if the mmap_lock was never touched. Such overflows are possible for vm_lock as well (see: https://elixir.bootlin.com/linux/v6.10.3/source/include/linux= /mm_types.h#L688) but they are not critical because a false result would simply lead to a retry under mmap_lock. However for your case this would be a critical issue. This is an extremely low probability scenario but should we still try to handle it? I'm CC'ing several mm folks and Jann Horn to chime in. > > Acked-by: Andrii Nakryiko > > > Discussion [1] follow-up. If proves to be useful can be included in tha= t > > patchset. Based on mm-unstable. > > > > [1] https://lore.kernel.org/all/20240730134605.GO33588@noisy.programmin= g.kicks-ass.net/ > > > > include/linux/mm_types.h | 3 +++ > > include/linux/mmap_lock.h | 53 +++++++++++++++++++++++++++++++-------- > > kernel/fork.c | 3 --- > > 3 files changed, 46 insertions(+), 13 deletions(-) > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > index 003619fab20e..a426e6ced604 100644 > > --- a/include/linux/mm_types.h > > +++ b/include/linux/mm_types.h > > @@ -887,6 +887,9 @@ struct mm_struct { > > * Roughly speaking, incrementing the sequence number i= s > > * equivalent to releasing locks on VMAs; reading the s= equence > > * number can be part of taking a read lock on a VMA. > > + * Incremented every time mmap_lock is write-locked/unl= ocked. > > + * Initialized to 0, therefore odd values indicate mmap= _lock > > + * is write-locked and even values that it's released. > > * > > * Can be modified under write mmap_lock using RELEASE > > * semantics. > > diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h > > index de9dc20b01ba..5410ce741d75 100644 > > --- a/include/linux/mmap_lock.h > > +++ b/include/linux/mmap_lock.h > > @@ -71,15 +71,12 @@ static inline void mmap_assert_write_locked(const s= truct mm_struct *mm) > > } > > > > #ifdef CONFIG_PER_VMA_LOCK > > -/* > > - * Drop all currently-held per-VMA locks. > > - * This is called from the mmap_lock implementation directly before re= leasing > > - * a write-locked mmap_lock (or downgrading it to read-locked). > > - * This should normally NOT be called manually from other places. > > - * If you want to call this manually anyway, keep in mind that this wi= ll release > > - * *all* VMA write locks, including ones from further up the stack. > > - */ > > -static inline void vma_end_write_all(struct mm_struct *mm) > > +static inline void init_mm_lock_seq(struct mm_struct *mm) > > +{ > > + mm->mm_lock_seq =3D 0; > > +} > > + > > +static inline void inc_mm_lock_seq(struct mm_struct *mm) > > { > > mmap_assert_write_locked(mm); > > /* > > @@ -91,19 +88,52 @@ static inline void vma_end_write_all(struct mm_stru= ct *mm) > > */ > > smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); > > } > > + > > +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, i= nt *seq) > > +{ > > + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ > > + *seq =3D smp_load_acquire(&mm->mm_lock_seq); > > + /* Allow speculation if mmap_lock is not write-locked */ > > + return (*seq & 1) =3D=3D 0; > > +} > > + > > +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int= seq) > > +{ > > + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ > > + return seq =3D=3D smp_load_acquire(&mm->mm_lock_seq); > > +} > > + > > #else > > -static inline void vma_end_write_all(struct mm_struct *mm) {} > > +static inline void init_mm_lock_seq(struct mm_struct *mm) {} > > +static inline void inc_mm_lock_seq(struct mm_struct *mm) {} > > +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, i= nt *seq) { return false; } > > +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int= seq) { return false; } > > #endif > > > > +/* > > + * Drop all currently-held per-VMA locks. > > + * This is called from the mmap_lock implementation directly before re= leasing > > + * a write-locked mmap_lock (or downgrading it to read-locked). > > + * This should normally NOT be called manually from other places. > > + * If you want to call this manually anyway, keep in mind that this wi= ll release > > + * *all* VMA write locks, including ones from further up the stack. > > + */ > > +static inline void vma_end_write_all(struct mm_struct *mm) > > +{ > > + inc_mm_lock_seq(mm); > > +} > > + > > static inline void mmap_init_lock(struct mm_struct *mm) > > { > > init_rwsem(&mm->mmap_lock); > > + init_mm_lock_seq(mm); > > } > > > > static inline void mmap_write_lock(struct mm_struct *mm) > > { > > __mmap_lock_trace_start_locking(mm, true); > > down_write(&mm->mmap_lock); > > + inc_mm_lock_seq(mm); > > __mmap_lock_trace_acquire_returned(mm, true, true); > > } > > > > @@ -111,6 +141,7 @@ static inline void mmap_write_lock_nested(struct mm= _struct *mm, int subclass) > > { > > __mmap_lock_trace_start_locking(mm, true); > > down_write_nested(&mm->mmap_lock, subclass); > > + inc_mm_lock_seq(mm); > > __mmap_lock_trace_acquire_returned(mm, true, true); > > } > > > > @@ -120,6 +151,8 @@ static inline int mmap_write_lock_killable(struct m= m_struct *mm) > > > > __mmap_lock_trace_start_locking(mm, true); > > ret =3D down_write_killable(&mm->mmap_lock); > > + if (!ret) > > + inc_mm_lock_seq(mm); > > __mmap_lock_trace_acquire_returned(mm, true, ret =3D=3D 0); > > return ret; > > } > > diff --git a/kernel/fork.c b/kernel/fork.c > > index 3d590e51ce84..73e37af8a24d 100644 > > --- a/kernel/fork.c > > +++ b/kernel/fork.c > > @@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_struct= *mm, struct task_struct *p, > > seqcount_init(&mm->write_protect_seq); > > mmap_init_lock(mm); > > INIT_LIST_HEAD(&mm->mmlist); > > -#ifdef CONFIG_PER_VMA_LOCK > > - mm->mm_lock_seq =3D 0; > > -#endif > > mm_pgtables_bytes_init(mm); > > mm->map_count =3D 0; > > mm->locked_vm =3D 0; > > > > base-commit: 98808d08fc0f78ee638e0c0a88020fbbaf581ec6 > > -- > > 2.46.0.rc2.264.g509ed76dc8-goog > >