From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F884EB64DA for ; Fri, 30 Jun 2023 17:40:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 83F6F8E003D; Fri, 30 Jun 2023 13:40:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7EF538E000F; Fri, 30 Jun 2023 13:40:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 690908E003D; Fri, 30 Jun 2023 13:40:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 57D448E000F for ; Fri, 30 Jun 2023 13:40:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 2BB4640D57 for ; Fri, 30 Jun 2023 17:40:44 +0000 (UTC) X-FDA: 80960129208.07.59ECEC7 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf06.hostedemail.com (Postfix) with ESMTP id 4AC4818000F for ; Fri, 30 Jun 2023 17:40:42 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sVIUuw+C; spf=pass (imf06.hostedemail.com: domain of surenb@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688146842; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6+En68YkXQdDYAx0rEbBwu3QYNU0fe/9lrmYA+JJw4=; b=jQVDcZRQgMTsp6cG3OP7nO2mDPllZRBejDQOqX0sUHVzZDng4FWFAvMKqak/AtpEmR3nBC u2+VddwHp9HzS+tKz3ZEmU2iYim3H8XVDhgw7/KAZxJ8QHaaArg+u+zjo4LcLwDGMAhQ6p u4irxOT1JDIw16ZGYY1cL1UHfKlXGkU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688146842; a=rsa-sha256; cv=none; b=4vqVI5pE0j/cj4Fr2sobbDAmIwDYzF+PA+m9pCCbAsvaxDq0mP90IU5rRdeQSK3bWwEx95 29ce0yzpW4hu1r8vAnfvbX+SmThNofAmkT1HgqUBGRWd/CbqlilYFPedHgkXINIYSHdKHA h46pJMedWFhkadKqH09KmcPRO9w0/QI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=sVIUuw+C; spf=pass (imf06.hostedemail.com: domain of surenb@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-577412111f0so11856067b3.0 for ; Fri, 30 Jun 2023 10:40:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688146841; x=1690738841; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=z6+En68YkXQdDYAx0rEbBwu3QYNU0fe/9lrmYA+JJw4=; b=sVIUuw+Ct1U/Bf5d3gHquE2/2YmHeGw4eAOYMZurkPnz7Y24DlbEXN3fMLp3sYcN7w tgfucc0iVLFc34Ms3jMPxnvlW6Eb8x0Lf1MXUlGxByUUNx4WXYipQ9KgO/xl31B0JI4M IIP0u6Xs56Bs3e9lex0861S68WDmmB7+JcKO0UeBjpy21YwOaoSxBmNv6oUU/s1G8oYP 6JLLYQk3SD5QEk4tvBwsEFcdcNFlJ01oX2frs9bkRH3gAdSUAVTk3kbjQfpKTAqXirns C+eZ0I5q0ioihoN8MMcXuY6tPy58xUEn55wZSRbsndeVUUbxVUJZrjADvROmQxbWTf9R HjJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688146841; x=1690738841; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z6+En68YkXQdDYAx0rEbBwu3QYNU0fe/9lrmYA+JJw4=; b=LMhyqOcYm4uYUbuA7XfQHfR+YsX9DgcIMipJSXEoeAXKyW+048Z0rg4tJ9TzQm0rEe c2mvJowwNGUlxlOrJDXVGmtbcWghPhWGClMb0jziyycdIsdAYjbsMPoaR0pQhfG+6qSx VquvAMOLayF+MBfJzXjGYhpoOBTNupavFfchT0qkORUscCQtm3ozfW8yrcKSGbHKFvcK Kz/5mv4WP4mTpE1OuK4KSUQjAmGEn7krRaUHU0xNGZS15nz6PzXpDZ6MaTws4LNiK364 r4fe6qmLjSbIVdIamwR4i2fGQcQePtobJPi1Aa9PLdtCpAvTGl6LtvvAdXoNTV9zUMEI m3bA== X-Gm-Message-State: ABy/qLY4rztn1b9gpAqCN4w5c5TL1Cm+2ZyB++iKbDf6goY/25p9rK0z h+y5iB9buO6PXp4fqzwmrXKxtVSeuSZMOsjO5JhgyA== X-Google-Smtp-Source: APBJJlFXdH9cwo9RRxx4x9iMDrvd6rqmTF9tbASyk8UnyE1yPQ9tRQvTIFcSV/qhYHv9yQckIwxngQeKPyprmMPCXOc= X-Received: by 2002:a0d:d54d:0:b0:577:2f3f:dd20 with SMTP id x74-20020a0dd54d000000b005772f3fdd20mr3237043ywd.34.1688146841016; Fri, 30 Jun 2023 10:40:41 -0700 (PDT) MIME-Version: 1.0 References: <20230227173632.3292573-1-surenb@google.com> <20230227173632.3292573-30-surenb@google.com> <9a8d788c-b8ba-1b8a-fd79-0e25b1b60bed@kernel.org> In-Reply-To: From: Suren Baghdasaryan Date: Fri, 30 Jun 2023 10:40:29 -0700 Message-ID: Subject: Re: [PATCH v4 29/33] x86/mm: try VMA lock-based page fault handling first To: Jiri Slaby Cc: akpm@linux-foundation.org, michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, paulmck@kernel.org, mingo@redhat.com, will@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, chriscli@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, rppt@kernel.org, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, leewalsh@google.com, posk@google.com, michalechner92@googlemail.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4AC4818000F X-Rspam-User: X-Stat-Signature: fyy397f1aibd5wbrmmxhc78yau4hqhry X-Rspamd-Server: rspam03 X-HE-Tag: 1688146842-769183 X-HE-Meta: U2FsdGVkX18+TpWI1MGC0XQ0gp5spYGC+8cAtVcS5hhl1rz1TN3HcfIvW3miD0lyRUUBnbBaleCJHX2qIDupV8A0NLnnjRhSIrxbytpAwngG1zE44MdXkjdwNddGJiFT8uRH+pYMokqPeaiLvDmzoL+uTA5fTWC93i6Slm6G/cfmKcZ0Jec7fSNiqnQoJJmZ13J6uua+pm9S+VdZIfCGNo14iFvbESdMtYrWxEv2YfKx9FxRINd6YkC/r6nMVkp9m1SjiJijeDw023GsPD1W18LBQ36q62Gy/egbf63xPajcV+gBATiu6HDNZYcovDxVJ0cdBwq+AASf5dTwopSqtlFOwAUFS70CtfjR9NpF969RfA/OedTJ43Ed5TyMc6dFYK+z9X0OpBPby8P5SGUz1uzztDlP0EDbCWcTMPCk8POt8fTX1TFPevNuLVdj5idj8Zx9rEJWJ0v6EppZteWn81txtp5GSeD/HxY2+Bp9RPfc6TCAHBJmN/vf7BmX7MzpwABo3uKDCfHXM00NHLyqzSJyrvY8SxiuOeuVTiQk+rzu9kDsM0JG1pAeAli6CKptNK62y3meOdCfMCSwMA7XiaGTR7ks0YcDjII/dbcmIBC1szY8i89xcPBN3pLMefFyEu1JvytAfYPuTg8lgcSf3fFbbGJ93eJr//QlUGQPD2sLr++dOIX/SI8MLr+Ub+Jo66/0eeVDYPskKk2pQYGzWia/OydaPzvQkmgzWaCSOmu5t8/FgOPnz9aZHpkTimq5CCvRAoBoChDE//aF3eUSc8m9g3Y2lKm62W71ulkrr4rEZvzlUF86xLpfdDTJ4B5G3v287bcO6Xzzb1klWYSmRy6uO9VbJwcZoPtv657iMJD7U0j+2p5ycM4D0itXSzuhTsSQ/X5ELoHU4ae0pdPuUehrX4C7Of31fmL1JRY+y4588QPdrdYC5ED6YjwpbWflp3rOGmctibezzqvs/lL ctyz/XE4 H501++6PGJR9vFoq5nFqC9+QsXVu4STqLsPkiOut9ZwTaqtlnTHKg6DBavKhYemosX2K9khaiwfUwoJgmjEjwO8bksowXQdGFcI2sR+RXN8dQBDndbEjsgfUUs52ZaUAfMa2lw27ztKFBjyhGyDJiM4776h5jsc935JtQNAzxj/UwLtBObF09Z5TQW8ww2SIk4qY+XxOkenE/vz5O7nYGNA26sbj6UO0Xbdg02VIOb6OuOPQTDokZ32iVXKu+wV1eN6f3GYkJNtZQuv3264tLLlLa2rj6ilw6HoSkXOWGb5/EuXzmVEz6OgTiRA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jun 30, 2023 at 1:43=E2=80=AFAM Jiri Slaby w= rote: > > On 30. 06. 23, 10:28, Jiri Slaby wrote: > > > 2348 > > clone3({flags=3DCLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREA= D|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, chil= d_tid=3D0x7fcaa5882990, parent_tid=3D0x7fcaa5882990, exit_signal=3D0, stack= =3D0x7fcaa5082000, stack_size=3D0x7ffe00, tls=3D0x7fcaa58826c0} =3D> {paren= t_tid=3D[2351]}, 88) =3D 2351 > > > 2350 <... clone3 resumed> =3D> {parent_tid=3D[2372]}, 88) =3D 2372 > > > 2351 <... clone3 resumed> =3D> {parent_tid=3D[2354]}, 88) =3D 2354 > > > 2351 <... clone3 resumed> =3D> {parent_tid=3D[2357]}, 88) =3D 2357 > > > 2354 <... clone3 resumed> =3D> {parent_tid=3D[2355]}, 88) =3D 2355 > > > 2355 <... clone3 resumed> =3D> {parent_tid=3D[2370]}, 88) =3D 2370 > > > 2370 mmap(NULL, 262144, PROT_READ|PROT_WRITE, > > MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 > > > 2370 <... mmap resumed>) =3D 0x7fca68249000 > > > 2372 <... clone3 resumed> =3D> {parent_tid=3D[2384]}, 88) =3D 2384 > > > 2384 <... clone3 resumed> =3D> {parent_tid=3D[2388]}, 88) =3D 2388 > > > 2388 <... clone3 resumed> =3D> {parent_tid=3D[2392]}, 88) =3D 2392 > > > 2392 <... clone3 resumed> =3D> {parent_tid=3D[2395]}, 88) =3D 2395 > > > 2395 write(2, "runtime: marked free object in s"..., 36 > ...> > > > > I.e. IIUC, all are threads (CLONE_VM) and thread 2370 mapped ANON > > 0x7fca68249000 - 0x7fca6827ffff and go in thread 2395 thinks for some > > reason 0x7fca6824bec8 in that region is "bad". Thanks for the analysis Jiri. Is it possible from these logs to identify whether 2370 finished the mmap operation before 2395 tried to access 0x7fca6824bec8? That access has to happen only after mmap finishes mapping the region. > > As I was noticed, this might be as well be a fail of the go's > inter-thread communication (or alike) too. It might now be only more > exposed with vma-based locks as we can do more parallelism now. Yes, with multithreaded processes like these where threads are mapping and accessing memory areas, per-VMA locks should allow for greater parallelism. So, if there is a race like the one I asked above, it might become more pronounced with per-VMA locks. I'll double check the code, but from Kernel's POV mmap would take the mmap_lock for write then will lock the VMA lock for write. That should prevent any page fault handlers from accessing this VMA in parallel until writers release the locks. Page fault path would try to find the VMA without any lock and then will try to read-lock that VMA. If it fails it will fall back to mmap_lock. So, if the writer started first and obtained the VMA lock, the reader will fall back to mmap_lock and will block until the writer releases the mmap_lock. If the reader got VMA read lock first then the writer will block while obtaining the VMA's write lock. However for your scenario, the reader (page fault) might be getting here before the writer (mmap) and upon not finding the VMA it is looking for, it will fail. Please let me know if you can verify this scenario. Thanks, Suren. > > There are older hard to reproduce bugs in go with similar symptoms (we > see this error sometimes now too): > https://github.com/golang/go/issues/15246 > > Or this 2016 bug is a red herring. Hard to tell... > > >> thanks, > -- > js > suse labs >