From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22481EB64D9 for ; Sun, 2 Jul 2023 12:40:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 980726B0075; Sun, 2 Jul 2023 08:40:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 930E26B0078; Sun, 2 Jul 2023 08:40:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D1228E0059; Sun, 2 Jul 2023 08:40:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 68EF96B0075 for ; Sun, 2 Jul 2023 08:40:45 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 213ADC0468 for ; Sun, 2 Jul 2023 12:40:45 +0000 (UTC) X-FDA: 80966630850.13.0A31D69 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 4CFDE100013 for ; Sun, 2 Jul 2023 12:40:43 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jbybsz1L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of jacobly.alt@gmail.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=jacobly.alt@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688301643; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y1dWZkJloBET8LXDsDpYxpSua9Iq0SwGTujx63V57eI=; b=x8rETNP7Os1Mo4KRKyu2TwtkUqtyMWbB2glBANhjQRdSVuj/su0z57hFiFr1FP/cm3C1UH YoJSAxznE4UHWuAyBASiioWpRg9nRIiwgddmOWLedbS0npya6ejnjpuiJMPbOPVQPbWX8c CkM3FFQQWUOs5CWgseVPQiZis+7DKqQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jbybsz1L; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf14.hostedemail.com: domain of jacobly.alt@gmail.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=jacobly.alt@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688301643; a=rsa-sha256; cv=none; b=Pn3RwtruFyyoPlSCFm/NAcfe4IswlQcY257bwJdINwR+Zx0i9rSewJd6pHlxaIDBrGqz5x 1BkxzhsDZITEbbQUms0hkEXDuyD5KljxXh98rkL/nbOcnGLW6MVKgWby5GcFf3XR3nRI4D hakEOiW4BbMB5RTo5ILOxLBd6cMRHIw= Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-bc379e4c1cbso3987350276.2 for ; Sun, 02 Jul 2023 05:40:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1688301642; x=1690893642; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Y1dWZkJloBET8LXDsDpYxpSua9Iq0SwGTujx63V57eI=; b=jbybsz1L9s0WuAS3sp1dwZ+9EAx4GF26NZ4M1EBM7JQXiqk0tYUwE1MNMIzP1ldxQV JoC9yxPyKW1QAkApUVGBPqwOG2abbcU0W6tr3Sl6KBnZnaEams/m2DCnmP0DLqhkdBkU G2cWXSV8aCw3DCXoFUxgL3y4ql3S/G31xmBAL/N9HziyuXet+syhfjWzyCycnQ7+aN9j 47hqrQbcEzNpm3IvR+WNOkO5XyyvWCUwFBs1rtM863dvpNGKIqexLGnqPRtxrhbKtquF FZTwANlChpjAcfAVotSDxlsvr+E6Os7YOu0Zna+qNQnU1UGWOm5z2OYKc2CoJt5oh5BM kt2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688301642; x=1690893642; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Y1dWZkJloBET8LXDsDpYxpSua9Iq0SwGTujx63V57eI=; b=aAjAHUnh6dlAzIJnYFPF8754+eT8D09/haB3TeMtLKg0kMqpBEBReOidkVh8i15wsZ tkTIGu4qGbD0ZX6LKb1H/761gvRujo3Z67tS6nfnvcW/BsbUp4I2c0B6prw6lLYEEZIl lG61n3JbjGjhB2ORJMdOPYB43TBF+8/XRmCP6PCDp58iQdFKI0gRX6fe2l+FOfLvT7EW WC0r7qVnhHi4KBVlwapXOG9t86rf0ySymW/rhwqsVb6la7Iyp0K/neG+ZbPSSadYe6Bh fyWCdth+34+fxt1pN+3alCB9jAC9aiKrDO5Xb91IhkCNXrokzn7vhyac53TVl4Qfql4K SrSQ== X-Gm-Message-State: ABy/qLaOe/RG6o/W38UIAkVYXiv3BSegx5I6gXQU7N4CUAkQ1RVCxGnZ 21DN4AdHY9LiQF7W40PMjD7yqCOMyjGW6owI8Yk= X-Google-Smtp-Source: APBJJlENfqbSeMy1r9gy6ESToUyNzb/A94w9qUwtS2MvFWh9zE+wFgoXJMCLmvI5mG/CHO6rDV/sAX4ArEiHcf2COjM= X-Received: by 2002:a25:c091:0:b0:c4e:82e1:602 with SMTP id c139-20020a25c091000000b00c4e82e10602mr1786702ybf.38.1688301642236; Sun, 02 Jul 2023 05:40:42 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jacob Young Date: Sun, 2 Jul 2023 08:40:31 -0400 Message-ID: Subject: Re: Memory corruption in multithreaded user space program while calling fork To: Bagas Sanjaya Cc: Suren Baghdasaryan , Andrew Morton , Laurent Dufour , Linux Kernel Mailing List , Linux Regressions , Linux Memory Management , Linux PowerPC , Linux ARM Content-Type: multipart/alternative; boundary="000000000000a8c6cf05ff805b80" X-Rspamd-Queue-Id: 4CFDE100013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 9fkn36mftdz83wmj9skf1dgnfzkim3ge X-HE-Tag: 1688301643-626808 X-HE-Meta: U2FsdGVkX18XxnhSJ3hE7HBLWy7NcHy3isgaEx2h2H14bwy4hpiC9nSmcP4QSiij5tNu3UOmJVMqDyuoNTUdDFIwkxFPPPQZ0udykdVnpkYZ5zoExXgoEJvbDap0KgBKUDzL3QQe7Y1bG13Qjq3y2WjXv+2GTM5DBoTELfLr1LcZg0IDPJXO6FHDmBWdhrsrCS60c3M5KX3w3NBTS8cPpJWKuBR5JtHk1wzJsr7Kkq9HAVno5mySmnmyxjTZv+4MDHHe8nKNgSSI2G19H0JFg1SX/GyT11WtdlIV0Jc/WK8CTiCpJT7YRU8elAzewQGKf6W5ONW4C+yZt0mU4u5kFlgzjcn1RmTzaaFIXgzTKhQ/PcnPWsBVwpv0AiJPn2ynb0uV9y9tHeKJIDp+jOmQJymSfHO6XfjwCI1Ceqjvy5wgt0OHuZF6CUvF5nJ7sv10ID+0opMCY7NRfyZ4sW3xMnmecFNKHSGAoRGi6Dr5NTEJswHMfrPbsp1H0rxLapPvRxoxH0qpEAKuuw7TNCSacktKL/GKcU1bUIgmNh3TJgeYb44BwXXAb+AjO89DayLHaH1r7m9+t6wqrJVBjD8SP7Am+HImDO4ZvzVX7vDlYo8HLjKdBcxUxvDplpWuWKqPqAGkrNeu7STPhjwCjly5jJhuFAG+xdzo+n3kJ6fKeQUlfGnvn1SGsJXoJiNIUGIkV2K6wGsa7a0M7KNtlLnWLq1/AULjWuJa8icik4pdlauHf4gwtS6zUK2Bz6qzZ0Hb9qe9fMOXedOHFRdNe4Gv+9zOdO6yqpQCnI4ncRfNUAb2qxl8S0w4ZcB88BpiUJtwtaDF4/uH8jZHdDpz1qLOe/5ACckLaGEpye9Cy0wQ8czmFyfijJCILSR6mZtIYRjaLljruYzXWgg65kiY+hYlraellrwqszUe1wl3aFUJ64Bseg1s7YDJ/UlJJCi4OF32ggXyouIHZNY/mY7SzSN MlH9acp7 mXXZI7Qk1BB4fg+G0KrFWy/lT3hyeNvvtxaXVzo3wAuAe/TOkQbOgLgZFI/xk7jCWoZnXYIfrL+cT3zsjL2jGta07P6JeILibSMl4RIaME5TcHFbLJAVrdBBuLKOjOga3H9RS/7KF07nGyx6aY078RcCAf2ELtohav0TjbnCaPpTAN+PmQ2WiDSEG83FYq96uAF5YtT1ptwU0WEx/MN+SPpsaTAhRvDkS+he5rAhGfBMQoC4gDq5p73ScYyXZYTVqnrCGKXQ3ooOdjZAqhaNa1z6lUVYV/0g7SSxawUWz0Ly5pzQxYfXyErwZCm8V2f+1e436aDJoCZ3dsbHfhdqUqOdUU6wl+Z+qAFJpZspEq8by0+FJQ1wy4cGQTNPuqLdizL0iqd5CkCkboWEnVZAB9KRHNxuLz4bts6cwiy6lGraBXI2f54R/Cx5ssuIsBTrno8x4qFqXp/t1tAZJthnoTqlNEoEY9XmIYdG8cBHtS+0+8Ae4w/tJzgdMlvG8lN0rabh6BU+JswCCMmWXiT0KEDrgKA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --000000000000a8c6cf05ff805b80 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > Jacob: Can you repeat bisection please? Why did you skip VMA lock-based page fault commits in your bisection? All skips were due to compile errors of the form: make[3]: 'install_headers' is up to date. In file included from ./include/linux/memcontrol.h:20, from ./include/linux/swap.h:9, from ./include/linux/suspend.h:5, from arch/x86/kernel/asm-offsets.c:14: ./include/linux/mm.h: In function =E2=80=98vma_try_start_write=E2=80=99: ./include/linux/mm.h:702:37: error: =E2=80=98struct vm_area_struct=E2=80=99= has no member named =E2=80=98vm_lock=E2=80=99 702 | if (!down_write_trylock(&vma->vm_lock->lock)) | ^~ ./include/linux/mm.h:706:22: error: =E2=80=98struct vm_area_struct=E2=80=99= has no member named =E2=80=98vm_lock=E2=80=99 706 | up_write(&vma->vm_lock->lock); | ^~ make[1]: *** [scripts/Makefile.build:114: arch/x86/kernel/asm-offsets.s] Error 1 make: *** [Makefile:1286: prepare0] Error 2 On Sun, Jul 2, 2023, 08:27 Bagas Sanjaya wrote: > Hi, > > I notice a regression report on Bugzilla [1]. Quoting from it: > > > After upgrading to kernel version 6.4.0 from 6.3.9, I noticed frequent > but random crashes in a user space program. After a lot of reduction, I > have come up with the following reproducer program: > > > > $ uname -a > > Linux jacob 6.4.1-gentoo #1 SMP PREEMPT_DYNAMIC Sat Jul 1 19:02:42 EDT > 2023 x86_64 AMD Ryzen 9 7950X3D 16-Core Processor AuthenticAMD GNU/Linux > > $ cat repro.c > > #define _GNU_SOURCE > > #include > > #include > > #include > > > > void *threadSafeAlloc(size_t n) { > > static size_t end_index =3D 0; > > static char buffer[1 << 25]; > > size_t start_index =3D __atomic_load_n(&end_index, __ATOMIC_SEQ_CST= ); > > while (1) { > > if (start_index + n > sizeof(buffer)) _exit(1); > > if (__atomic_compare_exchange_n(&end_index, &start_index, > start_index + n, 1, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST)) return buffer + > start_index; > > } > > } > > > > int thread(void *arg) { > > size_t i; > > size_t n =3D 1 << 7; > > char *items; > > (void)arg; > > while (1) { > > items =3D threadSafeAlloc(n); > > for (i =3D 0; i !=3D n; i +=3D 1) items[i] =3D '@'; > > for (i =3D 0; i !=3D n; i +=3D 1) if (items[i] !=3D '@') _exit(= 2); > > } > > } > > > > int main(void) { > > static size_t stacks[2][1 << 9]; > > size_t i; > > for (i =3D 0; i !=3D 2; i +=3D 1) clone(&thread, &stacks[i] + 1, > CLONE_THREAD | CLONE_VM | CLONE_SIGHAND, NULL); > > while (1) { > > if (fork() =3D=3D 0) _exit(0); > > (void)wait(NULL); > > } > > } > > $ cc repro.c > > $ ./a.out > > $ echo $? > > 2 > > > > After tuning the various parameters for my computer, exit code 2, which > indicates that memory corruption was detected, occurs approximately 99% o= f > the time. Exit code 1, which occurs approximately 1% of the time, means = it > ran out of statically-allocated memory before reproducing the issue, and > increasing the memory usage any more only leads to diminishing returns. > There is also something like a 0.1% chance that it segfaults due to memor= y > corruption elsewhere than in the statically-allocated buffer. > > > > With this reproducer in hand, I was able to perform the following > bisection: > > > > git bisect start > > # status: waiting for both good and bad commits > > # bad: [6995e2de6891c724bfeb2db33d7b87775f913ad1] Linux 6.4 > > git bisect bad 6995e2de6891c724bfeb2db33d7b87775f913ad1 > > # status: waiting for good commit(s), bad commit known > > # good: [457391b0380335d5e9a5babdec90ac53928b23b4] Linux 6.3 > > git bisect good 457391b0380335d5e9a5babdec90ac53928b23b4 > > # good: [d42b1c47570eb2ed818dc3fe94b2678124af109d] Merge tag > 'devicetree-for-6.4-1' of git:// > git.kernel.org/pub/scm/linux/kernel/git/robh/linux > > git bisect good d42b1c47570eb2ed818dc3fe94b2678124af109d > > # bad: [58390c8ce1bddb6c623f62e7ed36383e7fa5c02f] Merge tag > 'iommu-updates-v6.4' of git:// > git.kernel.org/pub/scm/linux/kernel/git/joro/iommu > > git bisect bad 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f > > # good: [888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0] Merge tag > 'sysctl-6.4-rc1' of git:// > git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux > > git bisect good 888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0 > > # bad: [86e98ed15b3e34460d1b3095bd119b6fac11841c] Merge tag > 'cgroup-for-6.4' of git:// > git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup > > git bisect bad 86e98ed15b3e34460d1b3095bd119b6fac11841c > > # bad: [7fa8a8ee9400fe8ec188426e40e481717bc5e924] Merge tag > 'mm-stable-2023-04-27-15-30' of git:// > git.kernel.org/pub/scm/linux/kernel/git/akpm/mm > > git bisect bad 7fa8a8ee9400fe8ec188426e40e481717bc5e924 > > # bad: [0120dd6e4e202e19a0e011e486fb2da40a5ea279] zram: make > zram_bio_discard more self-contained > > git bisect bad 0120dd6e4e202e19a0e011e486fb2da40a5ea279 > > # good: [fce0b4213edb960859dcc65ea414c8efb11948e1] mm/page_alloc: add > helper for checking if check_pages_enabled > > git bisect good fce0b4213edb960859dcc65ea414c8efb11948e1 > > # bad: [59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8] mm: avoid passing 0 t= o > __ffs() > > git bisect bad 59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8 > > # good: [0050d7f5ee532f92e8ab1efcec6547bfac527973] afs: split > afs_pagecache_valid() out of afs_validate() > > git bisect good 0050d7f5ee532f92e8ab1efcec6547bfac527973 > > # good: [2ac0af1b66e3b66307f53b1cc446514308ec466d] mm: fall back to > mmap_lock if vma->anon_vma is not yet set > > git bisect good 2ac0af1b66e3b66307f53b1cc446514308ec466d > > # skip: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a] mm/mmap: free > vm_area_struct without call_rcu in exit_mmap > > git bisect skip 0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a > > # skip: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a] powerc/mm: try VMA > lock-based page fault handling first > > git bisect skip 70d4cbc80c88251de0a5b3e8df3275901f1fa99a > > # good: [444eeb17437a0ef526c606e9141a415d3b7dfddd] mm: prevent > userfaults to be handled under per-vma lock > > git bisect good 444eeb17437a0ef526c606e9141a415d3b7dfddd > > # bad: [e06f47a16573decc57498f2d02f9af3bb3e84cf2] s390/mm: try VMA > lock-based page fault handling first > > git bisect bad e06f47a16573decc57498f2d02f9af3bb3e84cf2 > > # skip: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829] x86/mm: try VMA > lock-based page fault handling first > > git bisect skip 0bff0aaea03e2a3ed6bfa302155cca8a432a1829 > > # skip: [cd7f176aea5f5929a09a91c661a26912cc995d1b] arm64/mm: try VMA > lock-based page fault handling first > > git bisect skip cd7f176aea5f5929a09a91c661a26912cc995d1b > > # good: [52f238653e452e0fda61e880f263a173d219acd1] mm: introduce per-VM= A > lock statistics > > git bisect good 52f238653e452e0fda61e880f263a173d219acd1 > > # bad: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f] mm: separate vma->loc= k > from vm_area_struct > > git bisect bad c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f > > # only skipped commits left to test > > # possible first bad commit: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f] > mm: separate vma->lock from vm_area_struct > > # possible first bad commit: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a] > mm/mmap: free vm_area_struct without call_rcu in exit_mmap > > # possible first bad commit: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a] > powerc/mm: try VMA lock-based page fault handling first > > # possible first bad commit: [cd7f176aea5f5929a09a91c661a26912cc995d1b] > arm64/mm: try VMA lock-based page fault handling first > > # possible first bad commit: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829] > x86/mm: try VMA lock-based page fault handling first > > > > I do not usually see any kernel log output while running the program, > just occasional logs about user space segfaults. > > See Bugzilla for the full thread. > > Jacob: Can you repeat bisection please? Why did you skip VMA lock-based > page fault commits in your bisection? > > Anyway, I'm adding it to regzbot: > > #regzbot introduced: 0bff0aaea03e2a..c7f8f31c00d187 > https://bugzilla.kernel.org/show_bug.cgi?id=3D217624 > #regzbot > title: Memory corruption in multithreaded user space program while callin= g > fork (possibly caused by trying VMA lock-based page fault) > > Thanks. > > [1]: https://bugzilla.kernel.org/show_bug.cgi?id=3D217624 > > -- > An old man doll... just what I always wanted! - Clara > --000000000000a8c6cf05ff805b80 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
>=20 Jacob: Can you repeat bisection please? Why did you skip VMA lock-based
page fault commits in your bisection?

All skips were due to compile errors of the form:
make[3]:= 'install_headers' is up to date.
In file included from ./includ= e/linux/memcontrol.h:20,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0from ./include/linux/swap.h:9,
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0from ./include/linux/suspend.h:5,
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0from arch/x86= /kernel/asm-offsets.c:14:
./include/linux/mm.h: In function =E2=80=98vma= _try_start_write=E2=80=99:
./include/linux/mm.h:702:37: error: =E2=80=98= struct vm_area_struct=E2=80=99 has no member named =E2=80=98vm_lock=E2=80= =99
=C2=A0 702 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (!down_write_trylock(&am= p;vma->vm_lock->lock))
=C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^~
./include/linux/mm.h:706:22: error: = =E2=80=98struct vm_area_struct=E2=80=99 has no member named =E2=80=98vm_loc= k=E2=80=99
=C2=A0 706 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 up_write(&vma-&g= t;vm_lock->lock);
=C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0^~
make[1]: *** [scripts= /Makefile.build:114: arch/x86/kernel/asm-offsets.s] Error 1
make: *** [M= akefile:1286: prepare0] Error 2

<= div dir=3D"ltr" class=3D"gmail_attr">On Sun, Jul 2, 2023, 08:27 Bagas Sanja= ya <bagasdotme= @gmail.com> wrote:
Hi,

I notice a regression report on Bugzilla [1]. Quoting from it:

> After upgrading to kernel version 6.4.0 from 6.3.9, I noticed frequent= but random crashes in a user space program.=C2=A0 After a lot of reduction= , I have come up with the following reproducer program:
>
> $ uname -a
> Linux jacob 6.4.1-gentoo #1 SMP PREEMPT_DYNAMIC Sat Jul=C2=A0 1 19:02:= 42 EDT 2023 x86_64 AMD Ryzen 9 7950X3D 16-Core Processor AuthenticAMD GNU/L= inux
> $ cat repro.c
> #define _GNU_SOURCE
> #include <sched.h>
> #include <sys/wait.h>
> #include <unistd.h>
>
> void *threadSafeAlloc(size_t n) {
>=C2=A0 =C2=A0 =C2=A0static size_t end_index =3D 0;
>=C2=A0 =C2=A0 =C2=A0static char buffer[1 << 25];
>=C2=A0 =C2=A0 =C2=A0size_t start_index =3D __atomic_load_n(&end_ind= ex, __ATOMIC_SEQ_CST);
>=C2=A0 =C2=A0 =C2=A0while (1) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (start_index + n > sizeof(buffe= r)) _exit(1);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (__atomic_compare_exchange_n(&= end_index, &start_index, start_index + n, 1, __ATOMIC_SEQ_CST, __ATOMIC= _SEQ_CST)) return buffer + start_index;
>=C2=A0 =C2=A0 =C2=A0}
> }
>
> int thread(void *arg) {
>=C2=A0 =C2=A0 =C2=A0size_t i;
>=C2=A0 =C2=A0 =C2=A0size_t n =3D 1 << 7;
>=C2=A0 =C2=A0 =C2=A0char *items;
>=C2=A0 =C2=A0 =C2=A0(void)arg;
>=C2=A0 =C2=A0 =C2=A0while (1) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0items =3D threadSafeAlloc(n);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0for (i =3D 0; i !=3D n; i +=3D 1) ite= ms[i] =3D '@';
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0for (i =3D 0; i !=3D n; i +=3D 1) if = (items[i] !=3D '@') _exit(2);
>=C2=A0 =C2=A0 =C2=A0}
> }
>
> int main(void) {
>=C2=A0 =C2=A0 =C2=A0static size_t stacks[2][1 << 9];
>=C2=A0 =C2=A0 =C2=A0size_t i;
>=C2=A0 =C2=A0 =C2=A0for (i =3D 0; i !=3D 2; i +=3D 1) clone(&thread= , &stacks[i] + 1, CLONE_THREAD | CLONE_VM | CLONE_SIGHAND, NULL);
>=C2=A0 =C2=A0 =C2=A0while (1) {
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (fork() =3D=3D 0) _exit(0);
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(void)wait(NULL);
>=C2=A0 =C2=A0 =C2=A0}
> }
> $ cc repro.c
> $ ./a.out
> $ echo $?
> 2
>
> After tuning the various parameters for my computer, exit code 2, whic= h indicates that memory corruption was detected, occurs approximately 99% o= f the time.=C2=A0 Exit code 1, which occurs approximately 1% of the time, m= eans it ran out of statically-allocated memory before reproducing the issue= , and increasing the memory usage any more only leads to diminishing return= s.=C2=A0 There is also something like a 0.1% chance that it segfaults due t= o memory corruption elsewhere than in the statically-allocated buffer.
>
> With this reproducer in hand, I was able to perform the following bise= ction:
>
> git bisect start
> # status: waiting for both good and bad commits
> # bad: [6995e2de6891c724bfeb2db33d7b87775f913ad1] Linux 6.4
> git bisect bad 6995e2de6891c724bfeb2db33d7b87775f913ad1
> # status: waiting for good commit(s), bad commit known
> # good: [457391b0380335d5e9a5babdec90ac53928b23b4] Linux 6.3
> git bisect good 457391b0380335d5e9a5babdec90ac53928b23b4
> # good: [d42b1c47570eb2ed818dc3fe94b2678124af109d] Merge tag 'devi= cetree-for-6.4-1' of git://g= it.kernel.org/pub/scm/linux/kernel/git/robh/linux
> git bisect good d42b1c47570eb2ed818dc3fe94b2678124af109d
> # bad: [58390c8ce1bddb6c623f62e7ed36383e7fa5c02f] Merge tag 'iommu= -updates-v6.4' of git://git.= kernel.org/pub/scm/linux/kernel/git/joro/iommu
> git bisect bad 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f
> # good: [888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0] Merge tag 'sysc= tl-6.4-rc1' of git://git.k= ernel.org/pub/scm/linux/kernel/git/mcgrof/linux
> git bisect good 888d3c9f7f3ae44101a3fd76528d3dd6f96e9fd0
> # bad: [86e98ed15b3e34460d1b3095bd119b6fac11841c] Merge tag 'cgrou= p-for-6.4' of git://git.kerne= l.org/pub/scm/linux/kernel/git/tj/cgroup
> git bisect bad 86e98ed15b3e34460d1b3095bd119b6fac11841c
> # bad: [7fa8a8ee9400fe8ec188426e40e481717bc5e924] Merge tag 'mm-st= able-2023-04-27-15-30' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
> git bisect bad 7fa8a8ee9400fe8ec188426e40e481717bc5e924
> # bad: [0120dd6e4e202e19a0e011e486fb2da40a5ea279] zram: make zram_bio_= discard more self-contained
> git bisect bad 0120dd6e4e202e19a0e011e486fb2da40a5ea279
> # good: [fce0b4213edb960859dcc65ea414c8efb11948e1] mm/page_alloc: add = helper for checking if check_pages_enabled
> git bisect good fce0b4213edb960859dcc65ea414c8efb11948e1
> # bad: [59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8] mm: avoid passing 0 = to __ffs()
> git bisect bad 59f876fb9d68a4d8c20305d7a7a0daf4ee9478a8
> # good: [0050d7f5ee532f92e8ab1efcec6547bfac527973] afs: split afs_page= cache_valid() out of afs_validate()
> git bisect good 0050d7f5ee532f92e8ab1efcec6547bfac527973
> # good: [2ac0af1b66e3b66307f53b1cc446514308ec466d] mm: fall back to mm= ap_lock if vma->anon_vma is not yet set
> git bisect good 2ac0af1b66e3b66307f53b1cc446514308ec466d
> # skip: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a] mm/mmap: free vm_ar= ea_struct without call_rcu in exit_mmap
> git bisect skip 0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a
> # skip: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a] powerc/mm: try VMA = lock-based page fault handling first
> git bisect skip 70d4cbc80c88251de0a5b3e8df3275901f1fa99a
> # good: [444eeb17437a0ef526c606e9141a415d3b7dfddd] mm: prevent userfau= lts to be handled under per-vma lock
> git bisect good 444eeb17437a0ef526c606e9141a415d3b7dfddd
> # bad: [e06f47a16573decc57498f2d02f9af3bb3e84cf2] s390/mm: try VMA loc= k-based page fault handling first
> git bisect bad e06f47a16573decc57498f2d02f9af3bb3e84cf2
> # skip: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829] x86/mm: try VMA loc= k-based page fault handling first
> git bisect skip 0bff0aaea03e2a3ed6bfa302155cca8a432a1829
> # skip: [cd7f176aea5f5929a09a91c661a26912cc995d1b] arm64/mm: try VMA l= ock-based page fault handling first
> git bisect skip cd7f176aea5f5929a09a91c661a26912cc995d1b
> # good: [52f238653e452e0fda61e880f263a173d219acd1] mm: introduce per-V= MA lock statistics
> git bisect good 52f238653e452e0fda61e880f263a173d219acd1
> # bad: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f] mm: separate vma->= ;lock from vm_area_struct
> git bisect bad c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f
> # only skipped commits left to test
> # possible first bad commit: [c7f8f31c00d187a2c71a241c7f2bd6aa102a4e6f= ] mm: separate vma->lock from vm_area_struct
> # possible first bad commit: [0d2ebf9c3f7822e7ba3e4792ea3b6b19aa2da34a= ] mm/mmap: free vm_area_struct without call_rcu in exit_mmap
> # possible first bad commit: [70d4cbc80c88251de0a5b3e8df3275901f1fa99a= ] powerc/mm: try VMA lock-based page fault handling first
> # possible first bad commit: [cd7f176aea5f5929a09a91c661a26912cc995d1b= ] arm64/mm: try VMA lock-based page fault handling first
> # possible first bad commit: [0bff0aaea03e2a3ed6bfa302155cca8a432a1829= ] x86/mm: try VMA lock-based page fault handling first
>
> I do not usually see any kernel log output while running the program, = just occasional logs about user space segfaults.

See Bugzilla for the full thread.

Jacob: Can you repeat bisection please? Why did you skip VMA lock-based
page fault commits in your bisection?

Anyway, I'm adding it to regzbot:

#regzbot introduced: 0bff0aaea03e2a..c7f8f31c00d187 https://bugzilla.kernel.org/show_bug.cgi?id=3D217624=
#regzbot title: Memory corruption in multithreaded user space program w= hile calling fork (possibly caused by trying VMA lock-based page fault)

Thanks.

[1]: https://bugzilla.kernel.org/sh= ow_bug.cgi?id=3D217624

--
An old man doll... just what I always wanted! - Clara
--000000000000a8c6cf05ff805b80--