From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6BFFC4332F for ; Tue, 31 Oct 2023 09:46:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E28DE6B02A9; Tue, 31 Oct 2023 05:46:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DD8F56B02AD; Tue, 31 Oct 2023 05:46:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC7936B02C3; Tue, 31 Oct 2023 05:46:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BD3326B02A9 for ; Tue, 31 Oct 2023 05:46:23 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9A07F40AA9 for ; Tue, 31 Oct 2023 09:46:23 +0000 (UTC) X-FDA: 81405276246.25.79C6859 Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by imf23.hostedemail.com (Postfix) with ESMTP id BC49F14000E for ; Tue, 31 Oct 2023 09:46:21 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FO7zkjtO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of dvyukov@google.com designates 209.85.208.42 as permitted sender) smtp.mailfrom=dvyukov@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698745581; a=rsa-sha256; cv=none; b=clWVUPOsZSD5/zv+a8f5Nv2Y/7hUOHfXM1MH6WtzNraE9PaszirIk7jfnw4QWEIAnSlh2k r/2szC6jjwXh0FN+B4mB9b4sMUerq8rSobur6/27X1BGibrv2HDjRcXVtI4QA8rzpJerP+ SCptM+QwklqEZEqIPRfjc/uHsG4qTqU= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FO7zkjtO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of dvyukov@google.com designates 209.85.208.42 as permitted sender) smtp.mailfrom=dvyukov@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698745581; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qTxwVslDLW3XYJPo7d6erwK3L6Z/wP0aOPPJa5+jpt4=; b=HTe5x6Iqwtvd5dHzh1BtXLdWY0ZlGcf7nx3Rb5IdVM/9KOgpRYzvZdBBY9YnKuqKYAlgEC AldfS5JbZvs1AqEQmd2+BZAr+WJmDA1YZd6QLLSIxpaSKJ8iQCM/tpnrMWrUWOHrtwJFcz jbRfbfXrIdHIchdnv54BgMUemgHithM= Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-54366bb1c02so7315a12.1 for ; Tue, 31 Oct 2023 02:46:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698745580; x=1699350380; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=qTxwVslDLW3XYJPo7d6erwK3L6Z/wP0aOPPJa5+jpt4=; b=FO7zkjtO5Kuplj+8uiwwFadL3yu9ztWw3OV951+q/vcMhpdqt8uXiuXPu8RasEwPyo 6ccGDvreHiabWf7auxcQs8ksz+3dkEEMLQNnMU9WutJNNT39QTjXZBHTYTdBW2ke/P7m yhlZoR7+laInsigoxA9UmZU02g0Bahl0g/KJ//KY2HYxcu6vNmDgoHWReW+aELYQkhTt CSmWvQkvONyTlwnr2PL8fVpPLF0XBo1uH0zSfU7cq87nhubjsq70M4Z9CHWSZKtppUgq +rR+4jv3U2SpZhKFIscOCirTziAxK8imR1OwKxm+/AxRT1KCvaaY4j/jA0OWhfFOdsq3 MY0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698745580; x=1699350380; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qTxwVslDLW3XYJPo7d6erwK3L6Z/wP0aOPPJa5+jpt4=; b=UXgojGJ4rqnOT+3H95JxWyC/9Yb+j/iFSD/2gXMS9R7H9eTQk3+fZg5VxMWDLEuvRi tXmA+6WkCtBK+WO+UGThhCPrjzxjrs4VvRaRuK+uAwOAq9HmKQTj9Ta9Fd8GkiGVwXWN T9TUbdXRZwldgsFypHe6BEFo1iI2UIJcTmr9gNrvz98OH2gmWQ52m+dOU40PkV5biQ+o iNHSuIvb+yy1NUZVnLQu7bVB1ctq71aDz7Vd+tjwl9L6+I3TkStVlmK3ys7RZcZrDb+/ 6s70C6kZCHwpQO01UrOko8tqjpkc6zgsPu2tuxDZri3xdc7SVRNsW1Zoxuhl8jIAKb20 6pgQ== X-Gm-Message-State: AOJu0YzN3jtyUbh0qBKavm+bhVMPR7dlZiMTxjNaFAeRkGIBURMY3Iea TktpNoTAlvUDEXuL2d2ljeSZlfehj8BvcvPZkVAuDA== X-Google-Smtp-Source: AGHT+IGN3mM2N+nqhUvVo2QBdU6T2kfKp/GcVG7fysFgfPGginW9R/mMZndp5Wgf1dH9dRq0NT6zzHwUQrAg7PXbjGQ= X-Received: by 2002:a50:d514:0:b0:543:3f97:aa0f with SMTP id u20-20020a50d514000000b005433f97aa0fmr77047edi.4.1698745579834; Tue, 31 Oct 2023 02:46:19 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dmitry Vyukov Date: Tue, 31 Oct 2023 10:46:07 +0100 Message-ID: Subject: Re: [RFC] mm/kasan: Add Allocation, Free, Error timestamps to KASAN report To: Juntong Deng Cc: Andrey Konovalov , ryabinin.a.a@gmail.com, glider@google.com, vincenzo.frascino@arm.com, akpm@linux-foundation.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, "linux-kernel@vger.kernel.org" , "linux-kernel-mentees@lists.linuxfoundation.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: BC49F14000E X-Stat-Signature: q5qb7i54s9bjpyhawjkzde674mki7586 X-HE-Tag: 1698745581-535836 X-HE-Meta: U2FsdGVkX1/CdeVNTGPHajmViHEUv9YK9VSUNhQq+GH69zNUiY+DFahzG/7Oa2PcUYmR4sHZ2qBZPeHlVEGF7GgcGS11psf/6JFKb6Mgth9onkAHUgOyc7iXeKRoqmthE30a1o0VabtCSsdhaMXVhzgtTCaAQiCUaXqlwDUY3CJIDHr5yOHrx+RuPjrkgSTJTXXY6MPnzytxPC/AY9W0Dk52Sl+0kzv79oULGKLu+CYIr+bGn0tvOBR8GwPSNX0sQy/e3xPN9ClDf/yhYQ/xeC+WL3GJPcdDt/wYvuXKVq9Uik0St/TPhpJh93Uekw/x4nilGq7fuI+mCT1sMJD6QbpSnRGk3c6fqeZr1tQdoZEdCHX2CYPChsUs/N/+HKb9gXIdoB0kv3qxS1wt66RF0p1g67kE9otvRCo6oIb9VqufsKYVz37LpTJqNe414m6F4IPTir0D4aCnEEkNAm+K9V9X/aa8L1ho5S3cwJTzuXS/mCTNR2+xvRF+lPtrhopkCaC63qnWmUvkBWrVoQfbp8kAcsW3EJL4Q4+NTRIEsy2Z6VUiRFRdFZxam0rM24Q+g/iuoKlb9WjD6iAuvTVjQ8aCy2QLBa831q5guibwSyu3o0+r23HWxb2JZGrRX7Euwp2xN1NKly5WiFZx1xCt3FhwT151cGIJ1N8uUJSH6oetm4C5AHb0WH0fCcnQgYD9YsgxVtWCFrfxB6tTm6y4ae4aIb3gFVkV4h53lVHxnAFwJV0uaGDjRXomzjK6/VEmTFOdmiIbjh8Pq0dAiwgRdtKI54WSATh53MNWnZXUeV7HrICKaoBh58v8tNe09ShmGz6HRkPgvUG8N+Rt9NGUsmLPcx6nEoc8LwTLIqnHwoG4kYqTYU4vIz0y3liyzbpWQrFHLYu8rV8yFnMRcPGd5p0yQNiqmjUdk2EZBdmP6OABP+mJ+M3G44tUv1IoQWqKC3dfV3DI5acSfVW5xuD NmJ29Q6s PjSBoCChNj/YuCffQUeDpcR7Hd1nRmlVQHFj7gr9wR3EGqPXyhp/6LuCpwHZGHxIh0NXMGQYT8eNPd1EkusryZcY+7iOrH0zn82BxtMi0SQ5h0m3gJiUf12KEnVHBqjRt5GgDlGIRekba/drK1xmdT51pZrfCrPcj+lxYYrd3DgCk61KiynGfRD9YLRqiOf+5kqomA+tvWJ0kiO3XGnuOLsiiuLWBsfUy1lJ7PDDLeKwlmElAaDIATJsYG1OTPNfgw6ZwevK9a99wglCTQKM/XmZJ/QbgNtBRjEV1zo5Jiq9DvLyRI35BX+RdNIWKfSjq05e5NoRz2iTuPNW+4ml1vwXvK2QA+j5RZCne19Opgix47yBF5OMrjblQNPjZ6/TXMowy7uFvL4UhCaJYyLGmK7dv+G/KcAJLl7CfJXqq1pXkwcE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.080018, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 30 Oct 2023 at 12:32, Juntong Deng wrote= : > > On 2023/10/30 18:10, Dmitry Vyukov wrote: > > On Mon, 30 Oct 2023 at 10:28, Juntong Deng w= rote: > >> > >> On 2023/10/30 14:29, Dmitry Vyukov wrote: > >>> On Sun, 29 Oct 2023 at 10:05, Juntong Deng = wrote: > >>>> > >>>> On 2023/10/26 3:22, Andrey Konovalov wrote: > >>>>> On Tue, Oct 17, 2023 at 9:40=E2=80=AFPM Juntong Deng wrote: > >>>>>> > >>>>>> The idea came from the bug I was fixing recently, > >>>>>> 'KASAN: slab-use-after-free Read in tls_encrypt_done'. > >>>>>> > >>>>>> This bug is caused by subtle race condition, where the data struct= ure > >>>>>> is freed early on another CPU, resulting in use-after-free. > >>>>>> > >>>>>> Like this bug, some of the use-after-free bugs are caused by race > >>>>>> condition, but it is not easy to quickly conclude that the cause o= f the > >>>>>> use-after-free is race condition if only looking at the stack trac= e. > >>>>>> > >>>>>> I did not think this use-after-free was caused by race condition a= t the > >>>>>> beginning, it took me some time to read the source code carefully = and > >>>>>> think about it to determine that it was caused by race condition. > >>>>>> > >>>>>> By adding timestamps for Allocation, Free, and Error to the KASAN > >>>>>> report, it will be much easier to determine if use-after-free is > >>>>>> caused by race condition. > >>>>> > >>>>> An alternative would be to add the CPU number to the alloc/free sta= ck > >>>>> traces. Something like: > >>>>> > >>>>> Allocated by task 42 on CPU 2: > >>>>> (stack trace) > >>>>> > >>>>> The bad access stack trace already prints the CPU number. > >>>> > >>>> Yes, that is a great idea and the CPU number would help a lot. > >>>> > >>>> But I think the CPU number cannot completely replace the free timest= amp, > >>>> because some freeing really should be done at another CPU. > >>>> > >>>> We need the free timestamp to help us distinguish whether it was fre= ed > >>>> a long time ago or whether it was caused to be freed during the > >>>> current operation. > >>>> > >>>> I think both the CPU number and the timestamp should be displayed, m= ore > >>>> information would help us find the real cause of the error faster. > >>>> > >>>> Should I implement these features? > >>> > >>> Hi Juntong, > >>> > >>> There is also an aspect of memory consumption. KASAN headers increase > >>> the size of every heap object. So we tried to keep them as compact as > >>> possible. At some point CPU numbers and timestamps (IIRC) were alread= y > >>> part of the header, but we removed them to shrink the header to 16 > >>> bytes. > >>> PID gives a good approximation of potential races. I usually look at > >>> PIDs to understand if it's a "plain old single-threaded > >>> use-after-free", or free and access happened in different threads. > >>> Re timestamps, I see you referenced a syzbot report. With syzkaller > >>> most timestamps will be very close even for non-racing case. > >>> So if this is added, this should be added at least under a separate c= onfig. > >>> > >>> If you are looking for potential KASAN improvements, here is a good l= ist: > >>> https://bugzilla.kernel.org/buglist.cgi?bug_status=3D__open__&compone= nt=3DSanitizers&list_id=3D1134168&product=3DMemory%20Management > >> > >> Hi Dmitry, > >> > >> I think PID cannot completely replace timestamp for reason similar to > >> CPU number, some frees really should be done in another thread, but it > >> is difficult for us to distinguish if it is a free that was done some > >> time ago, or under subtle race conditions. > > > > I agree it's not a complete replacement, it just does not consume > > additional memory. > > > >> As to whether most of the timestamps will be very close even for > >> non-racing case, this I am not sure, because I do not have > >> enough samples. > >> > >> I agree that these features should be in a separate config and > >> the user should be free to choose whether to enable them or not. > >> > >> We can divide KASAN into normal mode and depth mode. Normal mode > >> records only minimal critical information, while depth mode records > >> more potentially useful information. > >> > >> Also, honestly, I think a small amount of extra memory consumption > >> should not stop us from recording more information. > >> > >> Because if someone enables KASAN for debugging, then memory consumptio= n > >> and performance are no longer his main concern. > > > > There are a number of debugging tools created with the "performance > > does not matter" attitude. They tend to be barely usable, not usable > > in wide scale testing, not usable in canaries, etc. > > All of sanitizers were created with lots of attention to performance, > > attention on the level of the most performance critical production > > code (sanitizer code is hotter than any production piece of code). > > That's what made them so widely used. Think of interactive uses, > > smaller devices, etc. Please let's keep this attitude. > > Yes, I agree that debugging tools used at a wide scale need to have > more rigorous performance considerations. > > Do you think it is worth using the extra bytes to record more > information? If this is a user-configurable feature. If it's user-configurable, then it is OK.