From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57CC6C3DA4A for ; Thu, 8 Aug 2024 21:12:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C29D36B0089; Thu, 8 Aug 2024 17:11:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD8046B008A; Thu, 8 Aug 2024 17:11:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A787D6B008C; Thu, 8 Aug 2024 17:11:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 883FE6B0089 for ; Thu, 8 Aug 2024 17:11:59 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3F68E40105 for ; Thu, 8 Aug 2024 21:11:59 +0000 (UTC) X-FDA: 82430325558.27.DDAFD9B Received: from mail-lf1-f50.google.com (mail-lf1-f50.google.com [209.85.167.50]) by imf10.hostedemail.com (Postfix) with ESMTP id 5170EC0024 for ; Thu, 8 Aug 2024 21:11:57 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BmVVRkjT; spf=pass (imf10.hostedemail.com: domain of andrii.nakryiko@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=andrii.nakryiko@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723151484; a=rsa-sha256; cv=none; b=RBsXQYszrvtnKoPGFJ7NtuyRVyjZlQdcFsYhushVq2wOLIUHVcnV2JAJcKe5nFt8AV/l7Y G4d8R1kJADU2e230YWZOr1qN5gQ3XF/wnVNlHPBIxqbgm/31GHT5KZ1LY4eImBkRo2pnVH exam+FLx3cDXVsMmvS3/zV4M4kL+fzc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=BmVVRkjT; spf=pass (imf10.hostedemail.com: domain of andrii.nakryiko@gmail.com designates 209.85.167.50 as permitted sender) smtp.mailfrom=andrii.nakryiko@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723151484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8evJkR69sPV/KRCSOPqEBKgOd9wCQjS5v2YcO5rlL1g=; b=a4nIMEG1u/untJ9w6f5EkEaDxjk515oPTEvK7xHeAf0n/1A1yOoSqoLRg9HWGefGcm9fWR C9ABQZSBK7VfduEU1icalKb8CEjwuR6JeTNs4FkBf5R/s8DMaINtuYNgpsjcnFhsMdRUeS 4Uk1CXeOCifAGv+ley9RRhvCXn+p+ns= Received: by mail-lf1-f50.google.com with SMTP id 2adb3069b0e04-52efdf02d13so2720760e87.2 for ; Thu, 08 Aug 2024 14:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723151515; x=1723756315; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8evJkR69sPV/KRCSOPqEBKgOd9wCQjS5v2YcO5rlL1g=; b=BmVVRkjT3Snd+pPBZJRgZjdMtFOi4LZlvvG3I1cyOA1bewi2+7aPfLw76vh8e8l2DS 74zGZsR7GJD9WUA6J9b3Id5QdN5TLXdXFXDwOxxfh7eao9kZ7m1IbN7UMYroG99kFWMI jWRTdRV07VGmjhXw3FClSfBS1ZW2dwmipGQGu0Oav4MLGJWi1ITuOimaZL1qduJHPrFC vo6XS4rddNZegg9b+WfGlyBKElKVGQad1J41r0QnJjuWiEp5ycjoCFIv770J+Xo0NG8D gdzR2LqZLeoABdWv30FDTK+EQr6tjOOGC3fWmuyHhAg+JohWnEIkBsy9jsG/lMy3LFUN TK3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723151515; x=1723756315; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8evJkR69sPV/KRCSOPqEBKgOd9wCQjS5v2YcO5rlL1g=; b=NvIDifTMwPP2cnQQsuu0PuMKgXQ2uHjyiQ+UaLZUYc6OT45IsmEiMmE/LfglISRVmS 5ShTvL1D9uKw4EYzYZbynIyGb5zipR92e+TDhPqi1LUMVgKR+gcIVQ4qMuaHgYVghb7m gBz6S5323q/wZCX6t3zaMAUd/+yeZjdxZyqjbZ8hPKyrnPvA+jISFPrflETfoXJeP/Lp aZbtwsYzW3QnqBT7rXvo46sNGegxDKPr8V5YklTjWu86ZqOMrwkOsNUhRyzSmujKnikj hzFdhngPQSEL6IQq6gafPkYzxA8CJY+p5iZnVsicHWjh3immhL/B0Zc1XGHU1ZXF0fcz CS4w== X-Forwarded-Encrypted: i=1; AJvYcCVbpaTWdNoYUYm3HC7QSH25Q0zBEA5s1hBM8APVZ0PeX9CbWIGC5CqIypDlWEmifMcywLDxSAXLIg==@kvack.org X-Gm-Message-State: AOJu0YzzRKwUNNNrmVL3LbDW7CCQZPoIyJH8P7c/67PQ3RIFnppP81Wu lztSqM6HJOfQzZKIQmNyjmEfGR77gcFchoL2Q9lOoZwkbwWww8Y0Ywlw945I9DH8GHumLQ4iD4V IfRZmeKt91LTe3eiiRRUrekqJqDs= X-Google-Smtp-Source: AGHT+IGC7XvQ9WqG1TKyA8QJMF9wu8iQyRy1rsF4xmNcCm6iwESG4HbLGhx0u1eJxtkBZFWajniUuy9uvkXNPzKeNSI= X-Received: by 2002:a05:6512:b9b:b0:52c:e05f:9052 with SMTP id 2adb3069b0e04-530e5876975mr2886465e87.47.1723151515133; Thu, 08 Aug 2024 14:11:55 -0700 (PDT) MIME-Version: 1.0 References: <20240807182325.2585582-1-surenb@google.com> In-Reply-To: From: Andrii Nakryiko Date: Thu, 8 Aug 2024 14:11:37 -0700 Message-ID: Subject: Re: [RFC 1/1] mm: introduce mmap_lock_speculation_{start|end} To: Suren Baghdasaryan Cc: akpm@linux-foundation.org, peterz@infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Jann Horn , Matthew Wilcox , Vlastimil Babka , Michal Hocko Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 4zdbi98uq7ih17ufc8o4i4krtg1e5wqi X-Rspamd-Queue-Id: 5170EC0024 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723151517-553735 X-HE-Meta: U2FsdGVkX1/uJt5Pn7ViJdhrsaguJ5N89/hHUN/nNP0ZOVsDwvfJNs4u0pEO4acJpsbJa2ua7tCLI9r0hxooGMFQbM6VU4jU2lSitJW3r5W+LCD/rtw45uFI19zkJvMzLRwHo7WdBHpN9lmW0ly/9pY6m52XNosNPcfJ29CFpLzWCDIQoeYucbynK6Nddqi8XVkO++p0M5YGR6zufAxcI9/OqNEip70KWJf3C7T99ihizMvUL5rozOmbLz7zKnSgWjazyhf5nqMnhEzGAvlEb42yr7ix5Vdf1QTz5lPhQq2W1lpPNGx9hKxbfP4AaPDGNmsp/+0eXYutEUMGRxRnzOGYnXWzJiXD+SE51L48JAlH7VslUMm8L9Wx2lyIuPwlcuxorRP/gyfr6YMgTvGUlVzMQJGXho8tKLcQpyBE/ehJHHKThAK+iwziZByadmF3j/nnl7hBwOMxwj8aI0/+ds0nSdDy1t6Swb4zBSxV1jvBGCUSY4QtvnZILYGBkSL1QM2Frd+CRsjSViwZQ0HLsDHINrU98s7KPkMqiavcOUHfzEKdi7ByEfXzx8N9i972wD9HBoDxP2We3N09TPOUXC8FDmg7N0UaCJOFZJ1dQhr6DCXLPp1UJVj2Uh/LtiMWmtOUdAsROgdbMi5Rjvd5UkB/IvGrR9HL+HUj70lh6yT48+BrkIK+24Da595vfciWW0Fic3VrEDdx9eyCNeye52+wz7qnJfOiWIstRP8KhpZpj6yNWgoXC/XBueYaHGZ4dE4f60kqCIW4Ryn1Rsmlyb6z7jp0HwYQ5tF8tNNING5KlhkCAWdSo8yqNaIuJ2viiIXsOr3Z9gInQPd2/fM35W0Pcb4wDgTtiY3W0S4MBMQHAx9GdQeU29nYyG/98CRbnoZp+UjcmAYyVoXoM07K7fVxjcQHBB6VL/uknrj5mNcGTvPxtpWWAfm0A2inEvWBfFo8iQtMQZd2PYdfnFn nWK9Xs/F eShkGpWZ4K2TZJsTj0G+uOjZp87qFtPXehQlT6TvQGQEDBrMTllq8qrE/eMmMFAO+zFpUq5DUsFoJrXxO1os0A2xekLgW03ZD870P5dHnMQMV5M3Cn7FKw1QwXcEnEevmezwsEfK4t5F+QjG8UqCwh/0UEtRYIJywaxmgaqfTn+kfUBvD27GltT0+zRQiSarM4akW7cYyKPHTKHgHcKwGOfIAICaUAgR/2RN3tdLCFJcajDwmtmOFamDEQ/Kp54Y72xCeFIdW6YKGKOBoMU3VKEx0xbmQc+KCEB065uiM+ogPIYPV40Ga7VH5BoakN6tBxE8cZtuRnhyNRVgSllv0lUjLFbrFD4xklkzfTxdxnfDAYMavqNLxgmtKtX7YBHvosIoVV5+ZGsdvvBJg6BYje3vCJM1kd4WCUJbVgTpqATdQqZsscRCPcnydv7lse0i57OXhiJlssY+QaWaWsxUXcnxeq/LSxJhMYp2s1zFBcdlWLYDu1SZ9e1XlN83OK/s0+QPA7xCWUv3W0AQzmBcdEbo0wAxC6Y1k/DQiGRDTQ1JuQGc74zJMjGf6b/pQqJOJp1eQaLj7ZElcQ52J5P5J37D66UorQvcDnZgm8Kf1hgPZwVtmYHHNLbIdEQy5eM548qpwuQgR9Xj4VcczL/ZWmEmZcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Aug 8, 2024 at 2:02=E2=80=AFPM Suren Baghdasaryan wrote: > > On Thu, Aug 8, 2024 at 8:19=E2=80=AFPM Andrii Nakryiko > wrote: > > > > On Wed, Aug 7, 2024 at 11:23=E2=80=AFAM Suren Baghdasaryan wrote: > > > > > > Add helper functions to speculatively perform operations without > > > read-locking mmap_lock, expecting that mmap_lock will not be > > > write-locked and mm is not modified from under us. > > > > > > Signed-off-by: Suren Baghdasaryan > > > Suggested-by: Peter Zijlstra > > > Cc: Andrii Nakryiko > > > --- > > > > This change makes sense and makes mm's seq a bit more useful and > > meaningful. I've also tested it locally with uprobe stress-test, and > > it seems to work great, I haven't run into any problems with a > > multi-hour stress test run so far. Thanks! > > Thanks for testing and feel free to include this patch into your set. Will do! > > I've been thinking about this some more and there is a very unlikely > corner case if between mmap_lock_speculation_start() and > mmap_lock_speculation_end() mmap_lock is write-locked/unlocked so many > times that mm->mm_lock_seq (int) overflows and just happen to reach > the same value as we recorded in mmap_lock_speculation_start(). This > would generate a false positive, which would show up as if the > mmap_lock was never touched. Such overflows are possible for vm_lock > as well (see: https://elixir.bootlin.com/linux/v6.10.3/source/include/lin= ux/mm_types.h#L688) > but they are not critical because a false result would simply lead to > a retry under mmap_lock. However for your case this would be a > critical issue. This is an extremely low probability scenario but > should we still try to handle it? > No, I think it's fine. Similar problems could happen with refcount_t, for instance (it has a logic to have a sticky "has overflown" state, which I believe relies on the fact that we'll never be able to increment refcount 2bln+ times in between some resetting logic). Anyways, I think it's utterly unrealistic and should be considered impossible. > I'm CC'ing several mm folks and Jann Horn to chime in. > > > > > Acked-by: Andrii Nakryiko > > > > > Discussion [1] follow-up. If proves to be useful can be included in t= hat > > > patchset. Based on mm-unstable. > > > > > > [1] https://lore.kernel.org/all/20240730134605.GO33588@noisy.programm= ing.kicks-ass.net/ > > > > > > include/linux/mm_types.h | 3 +++ > > > include/linux/mmap_lock.h | 53 +++++++++++++++++++++++++++++++------= -- > > > kernel/fork.c | 3 --- > > > 3 files changed, 46 insertions(+), 13 deletions(-) > > > > > > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > > > index 003619fab20e..a426e6ced604 100644 > > > --- a/include/linux/mm_types.h > > > +++ b/include/linux/mm_types.h > > > @@ -887,6 +887,9 @@ struct mm_struct { > > > * Roughly speaking, incrementing the sequence number= is > > > * equivalent to releasing locks on VMAs; reading the= sequence > > > * number can be part of taking a read lock on a VMA. > > > + * Incremented every time mmap_lock is write-locked/u= nlocked. > > > + * Initialized to 0, therefore odd values indicate mm= ap_lock > > > + * is write-locked and even values that it's released= . > > > * > > > * Can be modified under write mmap_lock using RELEAS= E > > > * semantics. > > > diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h > > > index de9dc20b01ba..5410ce741d75 100644 > > > --- a/include/linux/mmap_lock.h > > > +++ b/include/linux/mmap_lock.h > > > @@ -71,15 +71,12 @@ static inline void mmap_assert_write_locked(const= struct mm_struct *mm) > > > } > > > > > > #ifdef CONFIG_PER_VMA_LOCK > > > -/* > > > - * Drop all currently-held per-VMA locks. > > > - * This is called from the mmap_lock implementation directly before = releasing > > > - * a write-locked mmap_lock (or downgrading it to read-locked). > > > - * This should normally NOT be called manually from other places. > > > - * If you want to call this manually anyway, keep in mind that this = will release > > > - * *all* VMA write locks, including ones from further up the stack. > > > - */ > > > -static inline void vma_end_write_all(struct mm_struct *mm) > > > +static inline void init_mm_lock_seq(struct mm_struct *mm) > > > +{ > > > + mm->mm_lock_seq =3D 0; > > > +} > > > + > > > +static inline void inc_mm_lock_seq(struct mm_struct *mm) > > > { > > > mmap_assert_write_locked(mm); > > > /* > > > @@ -91,19 +88,52 @@ static inline void vma_end_write_all(struct mm_st= ruct *mm) > > > */ > > > smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1); > > > } > > > + > > > +static inline bool mmap_lock_speculation_start(struct mm_struct *mm,= int *seq) > > > +{ > > > + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ > > > + *seq =3D smp_load_acquire(&mm->mm_lock_seq); > > > + /* Allow speculation if mmap_lock is not write-locked */ > > > + return (*seq & 1) =3D=3D 0; > > > +} > > > + > > > +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, i= nt seq) > > > +{ > > > + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ > > > + return seq =3D=3D smp_load_acquire(&mm->mm_lock_seq); > > > +} > > > + > > > #else > > > -static inline void vma_end_write_all(struct mm_struct *mm) {} > > > +static inline void init_mm_lock_seq(struct mm_struct *mm) {} > > > +static inline void inc_mm_lock_seq(struct mm_struct *mm) {} > > > +static inline bool mmap_lock_speculation_start(struct mm_struct *mm,= int *seq) { return false; } > > > +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, i= nt seq) { return false; } > > > #endif > > > > > > +/* > > > + * Drop all currently-held per-VMA locks. > > > + * This is called from the mmap_lock implementation directly before = releasing > > > + * a write-locked mmap_lock (or downgrading it to read-locked). > > > + * This should normally NOT be called manually from other places. > > > + * If you want to call this manually anyway, keep in mind that this = will release > > > + * *all* VMA write locks, including ones from further up the stack. > > > + */ > > > +static inline void vma_end_write_all(struct mm_struct *mm) > > > +{ > > > + inc_mm_lock_seq(mm); > > > +} > > > + > > > static inline void mmap_init_lock(struct mm_struct *mm) > > > { > > > init_rwsem(&mm->mmap_lock); > > > + init_mm_lock_seq(mm); > > > } > > > > > > static inline void mmap_write_lock(struct mm_struct *mm) > > > { > > > __mmap_lock_trace_start_locking(mm, true); > > > down_write(&mm->mmap_lock); > > > + inc_mm_lock_seq(mm); > > > __mmap_lock_trace_acquire_returned(mm, true, true); > > > } > > > > > > @@ -111,6 +141,7 @@ static inline void mmap_write_lock_nested(struct = mm_struct *mm, int subclass) > > > { > > > __mmap_lock_trace_start_locking(mm, true); > > > down_write_nested(&mm->mmap_lock, subclass); > > > + inc_mm_lock_seq(mm); > > > __mmap_lock_trace_acquire_returned(mm, true, true); > > > } > > > > > > @@ -120,6 +151,8 @@ static inline int mmap_write_lock_killable(struct= mm_struct *mm) > > > > > > __mmap_lock_trace_start_locking(mm, true); > > > ret =3D down_write_killable(&mm->mmap_lock); > > > + if (!ret) > > > + inc_mm_lock_seq(mm); > > > __mmap_lock_trace_acquire_returned(mm, true, ret =3D=3D 0); > > > return ret; > > > } > > > diff --git a/kernel/fork.c b/kernel/fork.c > > > index 3d590e51ce84..73e37af8a24d 100644 > > > --- a/kernel/fork.c > > > +++ b/kernel/fork.c > > > @@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_stru= ct *mm, struct task_struct *p, > > > seqcount_init(&mm->write_protect_seq); > > > mmap_init_lock(mm); > > > INIT_LIST_HEAD(&mm->mmlist); > > > -#ifdef CONFIG_PER_VMA_LOCK > > > - mm->mm_lock_seq =3D 0; > > > -#endif > > > mm_pgtables_bytes_init(mm); > > > mm->map_count =3D 0; > > > mm->locked_vm =3D 0; > > > > > > base-commit: 98808d08fc0f78ee638e0c0a88020fbbaf581ec6 > > > -- > > > 2.46.0.rc2.264.g509ed76dc8-goog > > >