From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63830C433DB for ; Thu, 14 Jan 2021 09:51:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF42A23A21 for ; Thu, 14 Jan 2021 09:51:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF42A23A21 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 034498D00CD; Thu, 14 Jan 2021 04:51:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F27CA8D008E; Thu, 14 Jan 2021 04:51:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E161E8D00CD; Thu, 14 Jan 2021 04:51:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id C84648D008E for ; Thu, 14 Jan 2021 04:51:48 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8C919824556B for ; Thu, 14 Jan 2021 09:51:48 +0000 (UTC) X-FDA: 77703913896.14.actor46_060f5c827525 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 6F77718229835 for ; Thu, 14 Jan 2021 09:51:48 +0000 (UTC) X-HE-Tag: actor46_060f5c827525 X-Filterd-Recvd-Size: 8875 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Thu, 14 Jan 2021 09:51:47 +0000 (UTC) Received: by mail-qk1-f182.google.com with SMTP id z11so6685487qkj.7 for ; Thu, 14 Jan 2021 01:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yxYQK6YSchj/5tz12GdK3iWKS925woovB4KSStjvLsw=; b=dsg6gCEha6NEUyRN9p08CkWxSkPALUSogf8aqNruAiqnDuuKmt641bkFtzqy6MjXdg AXK8e7MqD328P/iDK6IWSFzZTGFQKhtJjMRcWGxoeKkEUs+pCP9QE3CH/+HB1FWiZsE0 DALdXWC5xA8eEBGE8g50iPKMPX1jhcb//M/UFHJ07LSv7rRvsHjufFuU7U4tF8EPcqjp 19KeStMKA1XLW2IsGotHD/KavpW4hSjfxFtxDng76gk0++/cc+UBDW/jPhVMaDygmtqZ dPdfqgFxJkbRdS9HZfOodzcBuI4+7mViC5RQZD1si6vPb+R3eOHaNDZ00YCtDWIRsC0y 4zWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yxYQK6YSchj/5tz12GdK3iWKS925woovB4KSStjvLsw=; b=o9GSkoYWqzUjuzbX24g+V0r+OLazvYRi9pOuVRn3GCIvPZWqmZYt+eHlLUBhVnA0gf 2KEKmNtcdMhAFGann8YFEA+DMfqMZLKFcKmjdJgjzH0izNDwhVkwf//C6jbhy9oJ/EhL OtbC9eH+fU7kh0XmzfpxtihdpyNwDfaLIr9ER4RuvW1y+LqoxDzznWdv8df/eKBdfVP3 i1UQEMPAGtPsvReG0I70FRA81c/VgMqCOXEiLG4hAip48XqT9Fibu/b2oCfXgnk0QIkB 0tH+cRosxODSH2itFWrabCVop/ZgcqyVmbKbEtc03tHJ/hr66XTdHJJFkb4EGitrFYCT nxaA== X-Gm-Message-State: AOAM5310gx4ibc6dAGLI4r2uCjywlTHd3Q0VhZie/jJ7YlkOgOHoLUf8 yTJWfcg+FMAtuZnJ9dkopzEJJm0+YgA4rpnySd0NYQ== X-Google-Smtp-Source: ABdhPJzJYr2OFcAcpM5em2Bemn9ria+MS8gS8XOJONyd1jXl3RKfI4/K+dYO4Fe1ZrIuo8o+5ozg0uMrvLEeAPQK3EA= X-Received: by 2002:a37:a747:: with SMTP id q68mr6347636qke.352.1610617907047; Thu, 14 Jan 2021 01:51:47 -0800 (PST) MIME-Version: 1.0 References: <20210113091657.1456216-1-glider@google.com> <20210113091657.1456216-3-glider@google.com> <20210113160612.32f8b67494521ce23cc9cba5@linux-foundation.org> In-Reply-To: <20210113160612.32f8b67494521ce23cc9cba5@linux-foundation.org> From: Alexander Potapenko Date: Thu, 14 Jan 2021 10:51:34 +0100 Message-ID: Subject: Re: [PATCH 2/4] lib: add error_report_notify to collect debugging tools' reports To: Andrew Morton Cc: LKML , Marco Elver , Andrey Konovalov , Dmitry Vyukov , Ingo Molnar , Petr Mladek , Steven Rostedt , Sergey Senozhatsky , Linux Memory Management List Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jan 14, 2021 at 1:06 AM Andrew Morton wrote: > > On Wed, 13 Jan 2021 10:16:55 +0100 Alexander Potapenko wrote: > > > With the introduction of various production error-detection tools, such as > > MTE-based KASAN and KFENCE, the need arises to efficiently notify the > > userspace OS components about kernel errors. Currently, no facility exists > > to notify userspace about a kernel error from such bug-detection tools. > > The problem is obviously not restricted to the above bug detection tools, > > and applies to any error reporting mechanism that does not panic the > > kernel; this series, however, will only add support for KASAN and KFENCE > > reporting. > > > > All such error reports appear in the kernel log. But, when such errors > > occur, userspace would normally need to read the entire kernel log and > > parse the relevant errors. This is error prone and inefficient, as > > userspace needs to continuously monitor the kernel log for error messages. > > On certain devices, this is unfortunately not acceptable. Therefore, we > > need to revisit how reports are propagated to userspace. > > > > The library added, error_report_notify (CONFIG_ERROR_REPORT_NOTIFY), > > solves the above by using the error_report_start/error_report_end tracing > > events and exposing the last report and the total report count to the > > userspace via /sys/kernel/error_report/last_report and > > /sys/kernel/error_report/report_count. > > > > Userspace apps can call poll(POLLPRI) on those files to get notified about > > the new reports without having to watch dmesg in a loop. > > It would be nice to see some user-facing documentation for this, under > Documentation/. How to use it, what the shortcomings are, etc. Good point, will do. > For instance... what happens when userspace is slow reading > /sys/kernel/error_report/last_report? Does that file buffer multiple > reports? Does the previous one get overwritten? etc. Words on how > this obvious issue is handled... Yes, there can be issues with overwriting, and the recommended way to handle them would be to check the value in /sys/kernel/error_report/report_count before and after reading the report. > > --- a/lib/Kconfig.debug > > +++ b/lib/Kconfig.debug > > @@ -209,6 +209,20 @@ config DEBUG_BUGVERBOSE > > of the BUG call as well as the EIP and oops trace. This aids > > debugging but costs about 70-100K of memory. > > > > +config ERROR_REPORT_NOTIFY > > + bool "Expose memory error reports to the userspace" > > There's really nothing "memory" specific about this? Any kernel > subsystem could use it? Indeed. Perhaps it's better to emphasize "production" here, because users of debugging tools are more or less happy with dmesg output. > > > + depends on TRACING > > + help > > + When enabled, captures error reports from debugging tools (such as > > + KFENCE or KASAN) using console tracing, and exposes reports in > > + /sys/kernel/error_report/: the file last_report contains the last > > + report (with maximum report length of PAGE_SIZE), and report_count, > > + the total report count. > > + > > + Userspace programs can call poll(POLLPRI) on those files to get > > + notified about the new reports without having to watch dmesg in a > > + loop. > > So we have a whole new way of getting debug info out of the kernel. I > fear this will become a monster. And anticipating that, we should make > darn sure that the interface is right, and is extensible. Let me elaborate a bit on the problem we are trying to solve here. It is specific to Android, but other Linux-based systems may require something similar. There's a userspace daemon that collects kernel/userspace crashes from the device if its owner has opted into that, and we want that daemon to also collect non-fatal error reports. There are several issues with that: - there is currently no way to synchronously notify the userspace about an error, and without that the daemon will have to actively monitor the kernel log (or some other file, e.g. /proc/kernel/tainted for certain strings; - once we figure out there is an error report available, the daemon will have to find its beginning and end, and also filter out the lines that do not belong to that report; - this all requires letting that daemon see the whole dmesg output, which may contain sensitive data accidentally printed by the kernel. So, first of all, our solution had to provide some poll()-based interface to avoid reading files in a loop, and that interface should trigger every time an error report is printed. Adding ftrace tracepoints to every tool at the points where reports start/end is perhaps least invasive, and also allows multiple subscribers (plus free tracing!). Then, since the notification library was already in the business of trace probes, we thought it makes sense to capture the whole report, assuming that every dmesg line from the same task/cpu between the report probes belongs to the report. That drastically reduces the amount of data the userspace daemon has access to (only the report instead of the whole dmesg), and removes the need of active polling. A potential pinch-point is the report size, which cannot exceed 4K if we want it to live in sysfs. It turned out certain reports didn't fit under that limit when taken as-is, but stripping away the timestamps and task IDs printed by CONFIG_PRINTK_CALLER saved us 1.5-2K in those cases. Given that the information we expose is a subset of what dmesg provides, I wouldn't call it a "whole new way" though. Existing users will probably still stick to dmesg unless they want to be notified of errors.