From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 612FFC43334 for ; Wed, 8 Jun 2022 00:21:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB66B6B0071; Tue, 7 Jun 2022 20:21:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A65F86B0072; Tue, 7 Jun 2022 20:21:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92CB26B0073; Tue, 7 Jun 2022 20:21:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 80BC66B0071 for ; Tue, 7 Jun 2022 20:21:00 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 57B7B137D for ; Wed, 8 Jun 2022 00:21:00 +0000 (UTC) X-FDA: 79553163480.18.929A795 Received: from mail-vs1-f45.google.com (mail-vs1-f45.google.com [209.85.217.45]) by imf13.hostedemail.com (Postfix) with ESMTP id C8FC22005F for ; Wed, 8 Jun 2022 00:20:59 +0000 (UTC) Received: by mail-vs1-f45.google.com with SMTP id x187so18282337vsb.0 for ; Tue, 07 Jun 2022 17:20:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=HcoQknaUmHCf5NNiXv3gVYmAHiag+xrcpWJjWHvdOZg=; b=El5lVxwUAtp3lYdRS44FJh/m5iar83RYu20Tydqw/TogWEAsetBxn8OJEoJ4NbUDYw 5SZ18TwrkYHP+tIY4Qi0UQ0ohj5187hpUqw5K/NApRm6tafb74zy30i3RhTxpSAyyEBA wekj3031y/gM+0TLAC59s856hYDsl+SwB5L67Axrvr3jh3I9KsROUiMYMY+x/wS5PRAV RWMZ6cEjDE9+gffEit8/6ip0ISJn8QzjSuKESMbHEI2nqlConWUAcRbFzJdvBlW5S8k2 iT7K2XjiuBzL3Ycg6O9EjbPmXlUXDCakvQpFZeTE9MsjPTcjY38elYLNAFjYPuRnmKIV oUMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=HcoQknaUmHCf5NNiXv3gVYmAHiag+xrcpWJjWHvdOZg=; b=IqyAqmYKbtkv6XYzTy66hKU+Csxbb840fjqj+Tr+gIquMEZCcOfsffUhxttza7KWSM T0AqqxkXJ2zuMR6hXqDmKo8u3J7sVoMTT0+saWW5urFfF97wPZr6Q7wJj6O35ZbMSg3d SvKrk2okJ344n5kPvIi5on8EVFkF4kLYtnmC6TxnYiTHM4uC9QS1CnQcxwi7hR9enUQb gaOKLthK66A9hrTGDFXS7ZPxV9I2urq7evhvIwF19Nkz2gso2UzbQgaX9y6tP0r7OCKb xglLk8sOsACTjdn1d5iyc+2aA4b8Ecty6Z7wzO+MdjyBt2v6H55Yq9FcLD+QZ3Pq5yom +fqg== X-Gm-Message-State: AOAM532+jgkYGGvtZrKbONQeVVGyyPjyeFEY4LZHVr7MFFRhuMZXq+kH 3h9l0qHKiGJZ2rIcHLAJMpS41eikP/6/Ykalup9cMpFDR3W79Q== X-Google-Smtp-Source: ABdhPJwR83OdverZPBwh+fyBA6W0CyLaD3ErYLyAqZbzFrJhTaO/fdBcoruij3Q+OjCh5eilVSpJLRhHQB+HdqCVSuE= X-Received: by 2002:a05:6102:1356:b0:34b:bc70:7b44 with SMTP id j22-20020a056102135600b0034bbc707b44mr6576079vsl.22.1654647658820; Tue, 07 Jun 2022 17:20:58 -0700 (PDT) MIME-Version: 1.0 References: <20220507015646.5377-1-hdanton@sina.com> In-Reply-To: From: Yu Zhao Date: Tue, 7 Jun 2022 18:20:22 -0600 Message-ID: Subject: Re: Alpha: rare random memory corruption/segfault in user space bisected To: Michael Cree Cc: Linux-MM , linux-kernel , Hillf Danton , Joonsoo Kim Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=El5lVxwU; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of yuzhao@google.com designates 209.85.217.45 as permitted sender) smtp.mailfrom=yuzhao@google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C8FC22005F X-Rspam-User: X-Stat-Signature: madyrh81fn8kxns5uphoa9wajz5xrx4j X-HE-Tag: 1654647659-751612 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, May 30, 2022 at 2:25 AM Michael Cree wrote: > > On Mon, May 23, 2022 at 02:56:12PM -0600, Yu Zhao wrote: > > On Wed, May 11, 2022 at 2:37 PM Michael Cree wrote: > > > > > > On Sat, May 07, 2022 at 11:27:15AM -0700, Yu Zhao wrote: > > > > On Fri, May 6, 2022 at 6:57 PM Hillf Danton wrote: > > > > > > > > > > On Sat, 7 May 2022 09:21:25 +1200 Michael Cree wrote: > > > > > > Alpha kernel has been exhibiting rare and random memory > > > > > > corruptions/segaults in user space since the 5.9.y kernel. First seen > > > > > > on the Debian Ports build daemon when running 5.10.y kernel resulting > > > > > > in the occasional (one or two a day) build failures with gcc ICEs either > > > > > > due to self detected corrupt memory structures or segfaults. Have been > > > > > > running 5.8.y kernel without such problems for over six months. > > > > > > > > > > > > Tried bisecting last year but went off track with incorrect good/bad > > > > > > determinations due to rare nature of bug. After trying a 5.16.y kernel > > > > > > early this year and seen the bug is still present retried the bisection > > > > > > and have got to: > > > > > > > > > > > > aae466b0052e1888edd1d7f473d4310d64936196 is the first bad commit > > > > > > commit aae466b0052e1888edd1d7f473d4310d64936196 > > > > > > Author: Joonsoo Kim > > > > > > Date: Tue Aug 11 18:30:50 2020 -0700 > > > > > > > > > > > > mm/swap: implement workingset detection for anonymous LRU > > > > > > > > This commit seems innocent to me. While not ruling out anything, i.e., > > > > this commit, compiler, qemu, userspace itself, etc., my wild guess is > > > > the problem is memory barrier related. Two lock/unlock pairs, which > > > > imply two full barriers, were removed. This is not a small deal on > > > > Alpha, since it imposes no constraints on cache coherency, AFAIK. > > > > > > > > Can you please try the attached patch on top of this commit? Thanks! > > > > > > Thanks, I have that running now for a day without any problem showing > > > up, but that's not long enough to be sure it has fixed the problem. Will > > > get back to you after another day or two of testing. > > > > Any luck? Thanks! > > Sorry for the delay in replying. Testing has taken longer due to an > unexpected hitch. The patch proved to be good but for a double check I > retested the above commit without the patch but it now won't fail which > calls into question whether aae466b0052e188 is truly the bad commit. I > have gone back to the prior bad commit in the bisection (25788738eb9c) > and it failed again confirming it is bad. So it looks like the first > bad commit is somewhere between aae466b0052e188 and 25788738eb9c (a > total of five commits inclusive, four if we take aae466b0052e188 as > good) and I am now building 471e78cc7687337abd1 and will test that. No worries. Thanks for the update. Were swap devices used when the ICEs happened? If so, 1) What kind of swap devices, e.g., zram, block device, etc.? 2) aae466b0052e188 might have made the kernel swap more frequently and thus the problem easier to reproduce. Assuming this is the case, then setting swappiness to 200 might help reproduce the problem.