From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14105C433F5 for ; Thu, 21 Apr 2022 10:04:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8924C6B0071; Thu, 21 Apr 2022 06:04:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8424A6B0073; Thu, 21 Apr 2022 06:04:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70A796B0074; Thu, 21 Apr 2022 06:04:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 637C96B0071 for ; Thu, 21 Apr 2022 06:04:31 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 40740122F39 for ; Thu, 21 Apr 2022 10:04:31 +0000 (UTC) X-FDA: 79380451542.22.51B0C1C Received: from mail-yb1-f181.google.com (mail-yb1-f181.google.com [209.85.219.181]) by imf30.hostedemail.com (Postfix) with ESMTP id F24E480029 for ; Thu, 21 Apr 2022 10:04:27 +0000 (UTC) Received: by mail-yb1-f181.google.com with SMTP id m132so7843295ybm.4 for ; Thu, 21 Apr 2022 03:04:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jYyPXLiEirgi7xpe02I4Hhb1NNjlliER4jXCeSNqs/s=; b=Ocgx+hoZE6a3M64LJTz4ptcOCzgEudqomqdLcyKVc8MDGYJahYlrgOGlDYsnNw4Jch hINmuvo0KHmq3SyB88dJB2khqbyXgVT6F5VCGUtcgnOwHRgjPxZte7ewfM6nrnjSaq3c b1lacfFWNldgL6hMrre1KyzeX3fSwFuFrNFdLL1+IGFUOneB0DqVtuPolAq8GjtuHtqE J81iCllWzoEY6afvhPmkZT7+Dx+a5fJFhXo6FBfLHQfokNGNQu2qaiEqh0F//nCWXM5C ZTuzBX5B788Z8oaJtmTnN3hvY02XoeQXyhsvUPAJ3DhFcpnmybuKHCp6Ly4crRjh4glG L6hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jYyPXLiEirgi7xpe02I4Hhb1NNjlliER4jXCeSNqs/s=; b=CL2ZEFeThryAMcCX32/ki1nFGJ2wM8K1Sve7SXL8J4LvAFC/brF5Al+pjhjA9j2P03 Rml8pVsZWhfDAMxtlfGRbIP+cEqkEOwTAweiA9TLcexDxJZdu2gV/elgVAVuWGsz4yqj X7CcEoFV95EdWa+OMl0kZg3AQWkm2j4AZsTiapEnXrUHIlamUk81zOP1ORUV41ZldCCc gt/F6XkRxDCVT0k6Hlz6xY2EDBcYBp3YPpLEMI8IlootScJyFVMOtMo3LDDyKQbP3CC6 jeXCfzPdEDLMOyTO6PFd25aXpZBlgE8wURjz5J+zXSTobchwQEFH7PH+/9Y33e9BJKuz EwRg== X-Gm-Message-State: AOAM53116UeivrYckortHhnZwq1k9xU8HWlLQ89DaNJZaUpCpY0gtd+1 woXQiIK9vTrVSKocm4LNGK3SCctBYRpaqDcGpTYYjA== X-Google-Smtp-Source: ABdhPJx8Be7TKOivuLQeNiDPlwn/JTtrgSwz8wBkjM5WWynPDdu643Ew9x7d4aqvfnIjehjx7f0vIVNNZGvKE9FHFGs= X-Received: by 2002:a25:bb0f:0:b0:61d:60f9:aaf9 with SMTP id z15-20020a25bb0f000000b0061d60f9aaf9mr23093358ybg.199.1650535469600; Thu, 21 Apr 2022 03:04:29 -0700 (PDT) MIME-Version: 1.0 References: <20220420104927.59056-1-huangshaobo6@huawei.com> In-Reply-To: <20220420104927.59056-1-huangshaobo6@huawei.com> From: Alexander Potapenko Date: Thu, 21 Apr 2022 12:03:53 +0200 Message-ID: Subject: Re: [PATCH] kfence: check kfence canary in panic and reboot To: Shaobo Huang Cc: Marco Elver , Dmitriy Vyukov , Andrew Morton , kasan-dev , Linux Memory Management List , LKML , young.liuyang@huawei.com, zengweilin@huawei.com, chenzefeng2@huawei.com, nixiaoming@huawei.com, wangbing6@huawei.com, wangfangpeng1@huawei.com, zhongjubin@huawei.com Content-Type: multipart/alternative; boundary="0000000000005b4af005dd273ce1" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F24E480029 X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ocgx+hoZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf30.hostedemail.com: domain of glider@google.com designates 209.85.219.181 as permitted sender) smtp.mailfrom=glider@google.com X-Stat-Signature: m9thfpxe1bqbs8hs88ewxhj7k7p7oqwn X-HE-Tag: 1650535467-624628 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000067, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --0000000000005b4af005dd273ce1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Apr 20, 2022 at 12:50 PM Shaobo Huang wrote: > From: huangshaobo > > when writing out of bounds to the red zone, it can only be detected at > kfree. However, there were many scenarios before kfree that caused this > out-of-bounds write to not be detected. Therefore, it is necessary to > provide a method for actively detecting out-of-bounds writing to the red > zone, so that users can actively detect, and can be detected in the > system reboot or panic. > > After having analyzed a couple of KFENCE memory corruption reports in the wild, I have doubts that this approach will be helpful. Note that KFENCE knows nothing about the memory access that performs the actual corruption. It's rather easy to investigate corruptions of short-living objects, e.g. those that are allocated and freed within the same function. In that case, one can examine the region of the code between these two events and try to understand what exactly caused the corruption. But for long-living objects checked at panic/reboot we'll effectively have only the allocation stack and will have to check all the places where the corrupted object was potentially used. Most of the time, such reports won't be actionable. > for example, if the application memory is out of bounds and written to > the red zone in the kfence object, the system suddenly panics, and the > following log can be seen during system reset: > BUG: KFENCE: memory corruption in atomic_notifier_call_chain+0x49/0x70 > > Corrupted memory at 0x(____ptrval____) [ ! ] (in kfence-#59): > atomic_notifier_call_chain+0x49/0x70 > panic+0x134/0x278 > sysrq_handle_crash+0x11/0x20 > __handle_sysrq+0x99/0x160 > write_sysrq_trigger+0x26/0x30 > proc_reg_write+0x51/0x70 > vfs_write+0xb6/0x290 > ksys_write+0x9c/0xd0 > __do_fast_syscall_32+0x67/0xe0 > do_fast_syscall_32+0x2f/0x70 > entry_SYSCALL_compat_after_hwframe+0x45/0x4d > > kfence-#59: > 0x(____ptrval____)-0x(____ptrval____),size=3D100,cache=3Dkmalloc-128 > allocated by task 77 on cpu 0 at 28.018073s: > 0xffffffffc007703d > do_one_initcall+0x3c/0x1e0 > do_init_module+0x46/0x1d8 > load_module+0x2397/0x2860 > __do_sys_init_module+0x160/0x190 > __do_fast_syscall_32+0x67/0xe0 > do_fast_syscall_32+0x2f/0x70 > entry_SYSCALL_compat_after_hwframe+0x45/0x4d > > Suggested-by: chenzefeng > Signed-off-by: huangshaobo > --- > mm/kfence/core.c | 28 ++++++++++++++++++++++++++++ > 1 file changed, 28 insertions(+) > > diff --git a/mm/kfence/core.c b/mm/kfence/core.c > index 9b2b5f56f4ae..85cc3ca4b71c 100644 > --- a/mm/kfence/core.c > +++ b/mm/kfence/core.c > @@ -29,6 +29,9 @@ > #include > #include > #include > +#include > +#include > +#include > > #include > > @@ -716,6 +719,29 @@ static const struct file_operations objects_fops =3D= { > .release =3D seq_release, > }; > > +static void kfence_check_all_canary(void) > +{ > + int i; > + > + for (i =3D 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++) { > + struct kfence_metadata *meta =3D &kfence_metadata[i]; > + > + if (meta->state =3D=3D KFENCE_OBJECT_ALLOCATED) > + for_each_canary(meta, check_canary_byte); > + } > +} > + > +static int kfence_check_canary_callback(struct notifier_block *nb, > + unsigned long reason, void *arg) > +{ > + kfence_check_all_canary(); > + return NOTIFY_OK; > +} > + > +static struct notifier_block kfence_check_canary_notifier =3D { > + .notifier_call =3D kfence_check_canary_callback, > +}; > + > static int __init kfence_debugfs_init(void) > { > struct dentry *kfence_dir =3D debugfs_create_dir("kfence", NULL); > @@ -806,6 +832,8 @@ static void kfence_init_enable(void) > > WRITE_ONCE(kfence_enabled, true); > queue_delayed_work(system_unbound_wq, &kfence_timer, 0); > + register_reboot_notifier(&kfence_check_canary_notifier); > + atomic_notifier_chain_register(&panic_notifier_list, > &kfence_check_canary_notifier); > > pr_info("initialized - using %lu bytes for %d objects at > 0x%p-0x%p\n", KFENCE_POOL_SIZE, > CONFIG_KFENCE_NUM_OBJECTS, (void *)__kfence_pool, > -- > 2.12.3 > > --=20 Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Stra=C3=9Fe, 33 80636 M=C3=BCnchen Gesch=C3=A4ftsf=C3=BChrer: Paul Manicle, Liana Sebastian Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Diese E-Mail ist vertraulich. Falls Sie diese f=C3=A4lschlicherweise erhalt= en haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, l=C3=B6schen Sie alle Kopien und Anh=C3=A4nge davon und lassen Sie mich bit= te wissen, dass die E-Mail an die falsche Person gesendet wurde. This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. --0000000000005b4af005dd273ce1 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Apr 20, 2022 at 12:50 PM Shao= bo Huang <huangshaobo6@huawei= .com> wrote:
From: huangshaobo <huangshaobo6@huawei.com>

when writing out of bounds to the red zone, it can only be detected at
kfree. However, there were many scenarios before kfree that caused this
out-of-bounds write to not be detected. Therefore, it is necessary to
provide a method for actively detecting out-of-bounds writing to the red zone, so that users can actively detect, and can be detected in the
system reboot or panic.


After having analyzed a couple of KFEN= CE memory corruption reports in the wild, I have doubts that this approach = will be helpful.

Note that KFENCE knows nothing about th= e memory access that performs the actual corruption.

It's rather easy to investigate corruptions of short-living objects,= e.g. those that are allocated and freed within the same function. In that = case, one can examine the region of the code between these two events and t= ry to understand what exactly caused the corruption.

But for long-living objects checked at panic/reboot we'll effectivel= y have only the allocation stack and will have to check all the places wher= e the corrupted object was potentially used.
Most of the time, su= ch reports won't be actionable.
=C2=A0
for example, if the application memory is out of bounds and written to
the red zone in the kfence object, the system suddenly panics, and the
following log can be seen during system reset:
BUG: KFENCE: memory corruption in atomic_notifier_call_chain+0x49/0x70

Corrupted memory at 0x(____ptrval____) [ ! ] (in kfence-#59):
=C2=A0atomic_notifier_call_chain+0x49/0x70
=C2=A0panic+0x134/0x278
=C2=A0sysrq_handle_crash+0x11/0x20
=C2=A0__handle_sysrq+0x99/0x160
=C2=A0write_sysrq_trigger+0x26/0x30
=C2=A0proc_reg_write+0x51/0x70
=C2=A0vfs_write+0xb6/0x290
=C2=A0ksys_write+0x9c/0xd0
=C2=A0__do_fast_syscall_32+0x67/0xe0
=C2=A0do_fast_syscall_32+0x2f/0x70
=C2=A0entry_SYSCALL_compat_after_hwframe+0x45/0x4d

kfence-#59: 0x(____ptrval____)-0x(____ptrval____),size=3D100,cache=3Dkmallo= c-128
=C2=A0allocated by task 77 on cpu 0 at 28.018073s:
=C2=A00xffffffffc007703d
=C2=A0do_one_initcall+0x3c/0x1e0
=C2=A0do_init_module+0x46/0x1d8
=C2=A0load_module+0x2397/0x2860
=C2=A0__do_sys_init_module+0x160/0x190
=C2=A0__do_fast_syscall_32+0x67/0xe0
=C2=A0do_fast_syscall_32+0x2f/0x70
=C2=A0entry_SYSCALL_compat_after_hwframe+0x45/0x4d

Suggested-by: chenzefeng <chenzefeng2@huawei.com>
Signed-off-by: huangshaobo <huangshaobo6@huawei.com>
---
=C2=A0mm/kfence/core.c | 28 ++++++++++++++++++++++++++++
=C2=A01 file changed, 28 insertions(+)

diff --git a/mm/kfence/core.c b/mm/kfence/core.c
index 9b2b5f56f4ae..85cc3ca4b71c 100644
--- a/mm/kfence/core.c
+++ b/mm/kfence/core.c
@@ -29,6 +29,9 @@
=C2=A0#include <linux/slab.h>
=C2=A0#include <linux/spinlock.h>
=C2=A0#include <linux/string.h>
+#include <linux/notifier.h>
+#include <linux/reboot.h>
+#include <linux/panic_notifier.h>

=C2=A0#include <asm/kfence.h>

@@ -716,6 +719,29 @@ static const struct file_operations objects_fops =3D {=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 .release =3D seq_release,
=C2=A0};

+static void kfence_check_all_canary(void)
+{
+=C2=A0 =C2=A0 =C2=A0 =C2=A0int i;
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0for (i =3D 0; i < CONFIG_KFENCE_NUM_OBJECTS;= i++) {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0struct kfence_metad= ata *meta =3D &kfence_metadata[i];
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (meta->state = =3D=3D KFENCE_OBJECT_ALLOCATED)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0for_each_canary(meta, check_canary_byte);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0}
+}
+
+static int kfence_check_canary_callback(struct notifier_block *nb,
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0unsigned = long reason, void *arg)
+{
+=C2=A0 =C2=A0 =C2=A0 =C2=A0kfence_check_all_canary();
+=C2=A0 =C2=A0 =C2=A0 =C2=A0return NOTIFY_OK;
+}
+
+static struct notifier_block kfence_check_canary_notifier =3D {
+=C2=A0 =C2=A0 =C2=A0 =C2=A0.notifier_call =3D kfence_check_canary_callback= ,
+};
+
=C2=A0static int __init kfence_debugfs_init(void)
=C2=A0{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 struct dentry *kfence_dir =3D debugfs_create_di= r("kfence", NULL);
@@ -806,6 +832,8 @@ static void kfence_init_enable(void)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 WRITE_ONCE(kfence_enabled, true);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 queue_delayed_work(system_unbound_wq, &kfen= ce_timer, 0);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0register_reboot_notifier(&kfence_check_cana= ry_notifier);
+=C2=A0 =C2=A0 =C2=A0 =C2=A0atomic_notifier_chain_register(&panic_notif= ier_list, &kfence_check_canary_notifier);

=C2=A0 =C2=A0 =C2=A0 =C2=A0 pr_info("initialized - using %lu bytes for= %d objects at 0x%p-0x%p\n", KFENCE_POOL_SIZE,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 CONFIG_KFENCE_NUM_O= BJECTS, (void *)__kfence_pool,
--
2.12.3



--
Alexander Potapenko
Software= Engineer

Google Germany GmbH
Erika-Mann-Stra=C3=9Fe, 33
80636= M=C3=BCnchen

Gesch=C3=A4ftsf=C3=BChrer: Paul Manicle, Liana Sebasti= an
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellsch= aft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese f=C3=A4ls= chlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jeman= d anderes weiter, l=C3=B6schen Sie alle Kopien und Anh=C3=A4nge davon und l= assen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet= wurde.


This e-mail is confidential. If you received this commun= ication by mistake, please don't forward it to anyone else, please eras= e all copies and attachments, and please let me know that it has gone to th= e wrong person.
--0000000000005b4af005dd273ce1--