From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E5DDC4332F for ; Mon, 30 Oct 2023 10:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87DC26B0195; Mon, 30 Oct 2023 06:10:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 82BBD6B0197; Mon, 30 Oct 2023 06:10:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71AEE6B0198; Mon, 30 Oct 2023 06:10:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 622046B0195 for ; Mon, 30 Oct 2023 06:10:39 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3A524B492E for ; Mon, 30 Oct 2023 10:10:39 +0000 (UTC) X-FDA: 81401708598.03.2C57AEC Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by imf12.hostedemail.com (Postfix) with ESMTP id 391D24001D for ; Mon, 30 Oct 2023 10:10:36 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=erHBJ4ME; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of dvyukov@google.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=dvyukov@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698660637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T3zgy30RJ+D4CBrJvlTQXpMKazTmgNjyYtW3+p1yMKE=; b=ofT8jKUeXkfZw/PcQiMpd3G6SElbvTavCDMdKHAcAwSB3EF3ajXQtDEQmfDI1SNoyPMYDq KbciPdPunchB0PfgKEpQY1iKjH1PeTL5VH9KrAdVubfLeINe9js91PD/xHk9DP1FvnKh1p O8BtbqYAqN2UVwQIXYcU+2DFnVOf7Hc= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=erHBJ4ME; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of dvyukov@google.com designates 209.85.128.46 as permitted sender) smtp.mailfrom=dvyukov@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698660637; a=rsa-sha256; cv=none; b=L6iZXIR37bake0sZIqW0yDfoQBbDHIF0l2zwuRAPpOvirGMBRgW+DQ9VAO4uHindyIrigh jUTAjdnLA9vkYc48b7jZdDQm/mhPQrYFXuJrRZnVzP4OSt99k6z/rvcqhCobUzm+9VhpGX zmwS2Hs69HaIQgGNxtc4xNXdWUPLt9g= Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4078fe6a063so77525e9.1 for ; Mon, 30 Oct 2023 03:10:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698660635; x=1699265435; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=T3zgy30RJ+D4CBrJvlTQXpMKazTmgNjyYtW3+p1yMKE=; b=erHBJ4MEER6f/dadyGxCD2wUwbT+46Qe4mUdCIi/EoIf8VCfX1koGT3AMJHdyYXQw/ oYQhV4OxQGkfnXqMSP7Uq2cDfP2a8VtK3WbFoDP/43T5hIIfwOp2Xhkvn2LVvpxrTJHu E49E/20OkBWyaAYZ4qcAWkDgJy3+RsRaPeYiUDUubAGCdB3+SURxKfhSZ4dlTa5b9L1d YlztCGWJ+Bt3z+Plf8eq9SzI+ePE2N4j8gz896K3lDKSzYm2XDbJ06CMeNNsV+zqHdHb jT11ViYPxwsJ3FXnZAgc02OpoQfvR5qZBmil1XNkR1Gpqk0w4/YDVHJsPW1fEmoWYYms Liig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698660635; x=1699265435; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=T3zgy30RJ+D4CBrJvlTQXpMKazTmgNjyYtW3+p1yMKE=; b=TB+Y3ojXyQio97YJyUCryqf2jaWvOz/VTgWAZoL7BbnGyYTtlhh78FH3XDEumzjJf4 PAlBjG22WGfBi1ivnlAMvWDQVcOSRXDqNZBoIhcJBArrQ60DE04Ws/qLzSjcDx90/tMt VbThNyItHWdc1JtZWVtbh8dj13TYRa6E7a9+MbE9e6678y+5+1Pb4RyPiUKSAbeWWS86 7GIIG/fS1CoR+pKmltwGZ2G+V8WbE70YH5INahByoCNs8QbyZw7TZKFh66bwHLEAdxJq W2EsxiQKmcKqzoOCAwSOhq2juG8f1IZIJfEtcjg/v/vhfW6zbTv5um64pwORdBoFqEgp TjaA== X-Gm-Message-State: AOJu0YzGkpXbfRIpSSrLdY60+YEJLrfhFmAxlqeLCrjHFfntMBFd0xEI A9YZdeBOz7spfRS8c89VCPhCJXzP4a5tiWMwX66HCA== X-Google-Smtp-Source: AGHT+IEx18gywoxoTCuRJl1RZx7vKGVb7LBy12SquWLHYEomOx12+8+oFG8e5WT78snujNUoacBz1xrMag4bb2IlSZE= X-Received: by 2002:a05:600c:1d17:b0:400:c6de:6a20 with SMTP id l23-20020a05600c1d1700b00400c6de6a20mr92585wms.3.1698660635350; Mon, 30 Oct 2023 03:10:35 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dmitry Vyukov Date: Mon, 30 Oct 2023 11:10:22 +0100 Message-ID: Subject: Re: [RFC] mm/kasan: Add Allocation, Free, Error timestamps to KASAN report To: Juntong Deng Cc: Andrey Konovalov , ryabinin.a.a@gmail.com, glider@google.com, vincenzo.frascino@arm.com, akpm@linux-foundation.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, "linux-kernel@vger.kernel.org" , "linux-kernel-mentees@lists.linuxfoundation.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 391D24001D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: mczoxpjnifzwo58jadg11so49kiwsfx8 X-HE-Tag: 1698660636-150102 X-HE-Meta: U2FsdGVkX19zM3IoTNeoSdGR7pGFXYhypoAh7V5yQN1SLDuv32cGUuUa+cq9S6zhRN0lt6Q9EKjzWi841C1i2ZYRVePDDlXq5k7oLi17RzrMt9LlR5DWmejA7NN2DtfRTOUbbtW1rY7MHAZiOulu3USK88gr+7o/9J+jPweO41KinyHDJjeNetxNNcWSjxuV5fCZN9U1Zwhu8pehaZJcPKLSYcbMC6dDqkphByWvuvhIbMkbq93TxsuR7KtBCd0c1xAnFMv49yevJSnNN4P/2l3hj2zohkw6culKjMA5HQSN4s+nHnxc3Pa03fOn9yKht8G76tzaFwaKnPFQm6CutJqZB3uzJXbXiQldF3D86ItFF3IFJBYV8sCpQIoiuptWnEqnPFyVGkk1YbCnwVOtfFypBgK4VUiGQROFnmjUnetAH7P2eFe0wjKrjFdoEmWtbeDMlWlmBWUQm4dCeWU1AY235BQjhrTioHs0BZ+zUv9fxNFe632nWoG9pBu8eEVuAxCtkM+iHo10E/16dQlFbXrHkRS0rqPzEFcXpeRx3yKrk31fTOjixThZMdgHcjrUtUFUHGX05il7xEaWzLh4u+YStS8dxROMwNbxhyOY0SiXdTuwOvZz84ZIwo/np2qC9mGe4AvufLdP8k4xmd+38HpI55C+hmv68zKgj1WAa2Vm1G+aUsze9yo83F7pA38BJEsW82gTVazYOrq1DA/zgADlSBv1/n7OSqHMWxQZ3CaoYDLhZVW35SVOkFBJz5X9Lz7TuAUcl6BY6ayOYI/+Yyw2e0s0vdI1aFV85HNp4De5Gfr8oI6EFk/Rz1xGHpaSv838tiQ1t7EAbjIgSmqnKxR//0nuwseprJxgX+9yPQGsR1qCB1CxasTKcNYFL3jybAwVvVugt2npuYkUa2dzCdU+7i3kRUY3yO6MPdLzMECcO7B0rWBoiA4+Z4h5TLK8QHIo/givKenSx0/+ksJ CpUILsl6 nlTeHL2lGnY6oQddJ8vcqw9knu76h3qo9nv8zA94KcAFoFyQC89QwhHW1+lzNvd0+cqyraTfvLp0LN5sh+JKuZ9UUkRlPPPJOJcPyozUYqZPL19aUD43eR4kFYAFlnPO8KjrRiqP8l/odnqz8wSa9HsNfRhCuwkqzUluH8gnJ9lflypskWHJS/FWJVwD/SNBIT0Fzn+DGBK3X/KVdn0ZmymzZ6et/iLLiD0+S75kN14eWPWtY+rQmqWSeGs3gfwfvx5W+MsPL2VeIjdF3dV9cm3c1yzM8MQrVX5tkY/S3N09fM5uivKtl4yTx4Slaco20eALwNWBjqvUCd7blxWYJNhBLm8TK9MmDZFtmMPQlOZi08sE7nT6r9sUedDgKoYR3k0N9EFF1t1h3KtC6OfqH9jrK8l/FmGOkUd0+Z6vnLgFj/cg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.014511, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 30 Oct 2023 at 10:28, Juntong Deng wrote= : > > On 2023/10/30 14:29, Dmitry Vyukov wrote: > > On Sun, 29 Oct 2023 at 10:05, Juntong Deng w= rote: > >> > >> On 2023/10/26 3:22, Andrey Konovalov wrote: > >>> On Tue, Oct 17, 2023 at 9:40=E2=80=AFPM Juntong Deng wrote: > >>>> > >>>> The idea came from the bug I was fixing recently, > >>>> 'KASAN: slab-use-after-free Read in tls_encrypt_done'. > >>>> > >>>> This bug is caused by subtle race condition, where the data structur= e > >>>> is freed early on another CPU, resulting in use-after-free. > >>>> > >>>> Like this bug, some of the use-after-free bugs are caused by race > >>>> condition, but it is not easy to quickly conclude that the cause of = the > >>>> use-after-free is race condition if only looking at the stack trace. > >>>> > >>>> I did not think this use-after-free was caused by race condition at = the > >>>> beginning, it took me some time to read the source code carefully an= d > >>>> think about it to determine that it was caused by race condition. > >>>> > >>>> By adding timestamps for Allocation, Free, and Error to the KASAN > >>>> report, it will be much easier to determine if use-after-free is > >>>> caused by race condition. > >>> > >>> An alternative would be to add the CPU number to the alloc/free stack > >>> traces. Something like: > >>> > >>> Allocated by task 42 on CPU 2: > >>> (stack trace) > >>> > >>> The bad access stack trace already prints the CPU number. > >> > >> Yes, that is a great idea and the CPU number would help a lot. > >> > >> But I think the CPU number cannot completely replace the free timestam= p, > >> because some freeing really should be done at another CPU. > >> > >> We need the free timestamp to help us distinguish whether it was freed > >> a long time ago or whether it was caused to be freed during the > >> current operation. > >> > >> I think both the CPU number and the timestamp should be displayed, mor= e > >> information would help us find the real cause of the error faster. > >> > >> Should I implement these features? > > > > Hi Juntong, > > > > There is also an aspect of memory consumption. KASAN headers increase > > the size of every heap object. So we tried to keep them as compact as > > possible. At some point CPU numbers and timestamps (IIRC) were already > > part of the header, but we removed them to shrink the header to 16 > > bytes. > > PID gives a good approximation of potential races. I usually look at > > PIDs to understand if it's a "plain old single-threaded > > use-after-free", or free and access happened in different threads. > > Re timestamps, I see you referenced a syzbot report. With syzkaller > > most timestamps will be very close even for non-racing case. > > So if this is added, this should be added at least under a separate con= fig. > > > > If you are looking for potential KASAN improvements, here is a good lis= t: > > https://bugzilla.kernel.org/buglist.cgi?bug_status=3D__open__&component= =3DSanitizers&list_id=3D1134168&product=3DMemory%20Management > > Hi Dmitry, > > I think PID cannot completely replace timestamp for reason similar to > CPU number, some frees really should be done in another thread, but it > is difficult for us to distinguish if it is a free that was done some > time ago, or under subtle race conditions. I agree it's not a complete replacement, it just does not consume additional memory. > As to whether most of the timestamps will be very close even for > non-racing case, this I am not sure, because I do not have > enough samples. > > I agree that these features should be in a separate config and > the user should be free to choose whether to enable them or not. > > We can divide KASAN into normal mode and depth mode. Normal mode > records only minimal critical information, while depth mode records > more potentially useful information. > > Also, honestly, I think a small amount of extra memory consumption > should not stop us from recording more information. > > Because if someone enables KASAN for debugging, then memory consumption > and performance are no longer his main concern. There are a number of debugging tools created with the "performance does not matter" attitude. They tend to be barely usable, not usable in wide scale testing, not usable in canaries, etc. All of sanitizers were created with lots of attention to performance, attention on the level of the most performance critical production code (sanitizer code is hotter than any production piece of code). That's what made them so widely used. Think of interactive uses, smaller devices, etc. Please let's keep this attitude.