From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.3 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5ED6FC2BA19 for ; Wed, 15 Apr 2020 11:09:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E154220732 for ; Wed, 15 Apr 2020 11:09:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vvcS/23m" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E154220732 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 33FB28E000B; Wed, 15 Apr 2020 07:09:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F1238E0001; Wed, 15 Apr 2020 07:09:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DF998E000B; Wed, 15 Apr 2020 07:09:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id 06B568E0001 for ; Wed, 15 Apr 2020 07:09:46 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BE6C3180AD802 for ; Wed, 15 Apr 2020 11:09:45 +0000 (UTC) X-FDA: 76709819130.30.trip95_8f64f0adf3762 X-HE-Tag: trip95_8f64f0adf3762 X-Filterd-Recvd-Size: 6300 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Wed, 15 Apr 2020 11:09:45 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id m67so16637043qke.12 for ; Wed, 15 Apr 2020 04:09:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=21hC42/qRxZDmqiUxEvzYQPrDZOJ/smRvBWWoE1k9hQ=; b=vvcS/23mH11jORzDN8BGcCfZvJeI826FrnG3u+asgYZjL616yu62DIXvN4L4mHRN8v L2mwkXiEwhvpvYYvranvuYpbFBlaCiS8JdywVkE8pyyDLAjZV2wrfEMNqxN5F6hA0JFM xPzIB9ihc+EQwh1lRf3VNQ4yqgX22zDAy2N0mLLHpo2AKkrhB2yb+1FqDaCjdl+fdhRa pzHxRoJ8HK6zqNP7CPtKx6+xSPvQ+EXt5RhNSKjPhGMfC0HC0Yv+a5M555lQPbKJ1eNL 5Gs9DoTHmumKvr5bql4j8pBdlxxsCWamObMWPWeyLIuHNTd+ixwU/w0jqHNzja8sHO9/ 9WPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=21hC42/qRxZDmqiUxEvzYQPrDZOJ/smRvBWWoE1k9hQ=; b=OqPmtIGwC3A5I6fIlc64FpNQJoKCc0IVdM2698wYU6E4aT00k/mkrqBfLiHFeMmXMD NXesoaKNhxV+7jZcx9sTfXelJ499SSsfEp5omOKVgD+NS/bwBrMNaHJxl4OGfJhIS1oi K4vK6InsgJ13Ff4G9bXzbHcjfC+Ia4ddOMPrK1r6Z+u3NLVWyg6hYaMvnrok6saPWgZq SShavvd0n/xNY4puv7xAjjkqGWMmhZICJV9rCJ47FKaghVQVcp9pBSii730jLbx5x1fs fN5tOMYqBkWn0pZ6oTqX7UL86fmhV53KoN7FfQsjTWzRQdGNYALT4dHThVw2/qQmAxa6 Oilw== X-Gm-Message-State: AGi0PuYxHUhN9hMRCCmT/065vlvBvAU+iQkv0EeufY49m4dUjIORemza wnQzPNQXwFN6IeMcAkSMTs4mxv8lsR4c4r9YYPrUkg== X-Google-Smtp-Source: APiQypJSkEadODH65efJa5blZDKvd61WTzbxiNRypvYGvovsgxcszhaNwzUEHkGlzlvJ9T6bppZZezWw5Pk7EPbYV2s= X-Received: by 2002:a37:8d86:: with SMTP id p128mr26303566qkd.250.1586948984176; Wed, 15 Apr 2020 04:09:44 -0700 (PDT) MIME-Version: 1.0 References: <20200408014010.80428-1-peterx@redhat.com> <20200408174732.bc448bbe41d190bfe5cc252e@linux-foundation.org> <20200410092951.6db32bfe@canb.auug.org.au> <7325374A-6072-44E4-85EE-F97FC7E8565F@lca.pw> <20200414192840.4yp3zqbe2tgtesve@xps.therub.org> In-Reply-To: <20200414192840.4yp3zqbe2tgtesve@xps.therub.org> From: Dmitry Vyukov Date: Wed, 15 Apr 2020 13:09:32 +0200 Message-ID: Subject: Re: [PATCH 0/2] mm: Two small fixes for recent syzbot reports To: Dan Rue Cc: Qian Cai , Linus Torvalds , Stephen Rothwell , Andrew Morton , Peter Xu , LKML , Linux-MM , Jens Axboe , Christoph Lameter , Johannes Weiner , syzkaller Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Apr 14, 2020 at 9:28 PM Dan Rue wrote: > > On Tue, Apr 14, 2020 at 01:12:50PM +0200, Dmitry Vyukov wrote: > > On Tue, Apr 14, 2020 at 12:06 AM Qian Cai wrote: > > > Well, there are other CI's beyond syzbot. > > > On the other hand, this makes me worry who is testing on linux-next every day. > > > > How do these use-after-free's and locking bugs get past the > > unit-testing systems (which syzbot is not) and remain unnoticed for so > > long?... > > syzbot uses the dumbest VMs (GCE), so everything it triggers during > > boot should be triggerable pretty much everywhere. > > It seems to be an action point for the testing systems. "Boot to ssh" > > is not the best criteria. Again if there is a LOCKDEP error, we are > > not catching any more LOCKDEP errors during subsequent testing. If > > there is a use-after-free, that's a serious error on its own and KASAN > > produces only 1 error by default as well. And as far as I understand, > > lots of kernel testing systems don't even enable KASAN, which is very > > wrong. > > I've talked to +Dan Rue re this few days ago. Hopefully LKFT will > > start catching these as part of unit testing. Which should help with > > syzbot testing as well. > > LKFT has recently added testing with KASAN enabled and improved the > kernel log parsing to catch more of this class of errors while > performing our regular functional testing. > > Incidentally, -next was also broken for us from March 25 through April 5 > due to a perf build failure[0], which eventually made itself all the way > down into v5.6 release and I believe the first two 5.6.x stable > releases. > > For -next, LKFT's gap is primarily reporting. We do build and run over > 30k tests on every -next daily release, but we send out issues manually > when we see them because triaging is still a manual effort. We're > working to build better automated reporting. If anyone is interested in > watching LKFT's -next results more closely (warning, it's a bit noisy), > please let me know. Watching the results at https://lkft.linaro.org > provides some overall health indications, but again, it gets pretty > difficult to figure out signal from noise once you start drilling down > without sufficient context of the system. What kind of failures and noise do you get? Is it flaky tests? I would assume build failures are ~0% flaky/noisy. And boot failures are maybe ~1% flaky/noisy due to some infra issues. I can't find any actual test failure logs in the UI. I've got to this page: https://qa-reports.linaro.org/lkft/linux-mainline-oe/build/v5.7-rc1-24-g8632e9b5645b/testrun/1363280/suite/kselftest/tests/ which seem to contain failed tests on mainline. But I still can't find the actual test failure logs. > Dan > > [0] https://lore.kernel.org/stable/CA+G9fYsZjmf34pQT1DeLN_DDwvxCWEkbzBfF0q2VERHb25dfZQ@mail.gmail.com/ > > -- > Linaro LKFT > https://lkft.linaro.org