From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E900C761A6 for ; Tue, 4 Apr 2023 09:25:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB1326B0071; Tue, 4 Apr 2023 05:25:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A60E96B0074; Tue, 4 Apr 2023 05:25:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 903426B0075; Tue, 4 Apr 2023 05:25:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7F5186B0071 for ; Tue, 4 Apr 2023 05:25:57 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4A2871A0771 for ; Tue, 4 Apr 2023 09:25:57 +0000 (UTC) X-FDA: 80643176754.10.9A521B6 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf14.hostedemail.com (Postfix) with ESMTP id 68CF2100025 for ; Tue, 4 Apr 2023 09:25:55 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ha74+lmM; spf=pass (imf14.hostedemail.com: domain of glider@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=glider@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680600355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zj3a2MYNbkCFssTfjIF9gvNI8FfTYBgR4BuPPz6nhXc=; b=IHX8RwZlyAUlHhhNlSIzGZrsUx2l1wtIkeewoxfa0oLTE+jKdltGngJ8tayu6V1gHddFHr ehkLov86CSvGy8VDYxPk/fhOjx6r3tdQ6KUlHsyrsRODB3k/7KQI/ssj3sz2Mi8OQSEjoH HNzMeAsDSQAY3kolnUABrVfUbJukOa8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Ha74+lmM; spf=pass (imf14.hostedemail.com: domain of glider@google.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=glider@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680600355; a=rsa-sha256; cv=none; b=gGh5HJRAR+/VRZIxUm0X0/aILImfFvybH9V10dc2XLJR3BkOBH8qB+oD6wtG0ge79V/uFw TA7lhu2Q099lSWs4iLxx5wYr2efc9L7tjGytpr1++cYFgPxkXAVUJ3Yc2e25zgyzTys3Sl kUFFqCU8K6TIJHBPKZWgkWO1wPTaVdg= Received: by mail-yb1-f177.google.com with SMTP id n125so37833154ybg.7 for ; Tue, 04 Apr 2023 02:25:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680600354; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=zj3a2MYNbkCFssTfjIF9gvNI8FfTYBgR4BuPPz6nhXc=; b=Ha74+lmMeivYh1u52QynjkWeG4fZIvyvw5HbjOIl67pwttzzwsJRqoDCmgvBST0gSd t4xqbw/tOMXsDwNloimeFslTRB23JNDJBtfWisNxDkyEP89qW+oSLRsr85GoQZtR4O8c q/uiXk0IkCZNR6pRJGaP+c1VdbBqUlUNY1WueQTEQNxedWErjfokrxUpyowfwo1zorEJ dUz7cIBeevTrkBs1ry0oFrc3zTGoGKtN4RDuiDR8QWguHCTA0rK2Pb+pP+mHgSOoRIJa /EIsXmc34JWIO/CyUzV1+Dy78Svopna5l8JFjcMlvUMPszjQkck4DtS6hT+0PnKNa9TG wmgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680600354; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zj3a2MYNbkCFssTfjIF9gvNI8FfTYBgR4BuPPz6nhXc=; b=lilqSefsXOhwvY1sJqpnRt1b+Qop3cCn54jwvz+fyaevczxd9xSmZDkMLnzjrKmdyW 6u8v3L1Ebgk8Z5FifMxsRTEDrtrBXo9l0TK3X03HbzYe3aJmceSm9YJMsHbEsduNCmxA e9wm48ZfePwbUdPj1K3meeAYOOiNhX60shKBYXt8Wj3QPdhMaJAx4JJQwVbYNiDdH4oq yzZZjocd8ypTT64r2TO5AONLN5xYik6T0lvp34lkCza7mpXPWS4TwTV+GRd92p1wvG36 KGJAas076+w42cmgGJEuAMLQr00wLfbTa8Gek2/9IyX2tpFAULf1l95IrvDg5mN5RC/P DlDA== X-Gm-Message-State: AAQBX9f39odFa5e5j9rOuTMgSwyU9NES3DpTPxisuW0kbTlAsU5VXAgb VZw+4IZ+lYN9rMLBMGrrxrHr0EA5TVr0tsSGOx5/rw== X-Google-Smtp-Source: AKy350bUM9iWWLmPvON6nK7eho1YZvuR/HHQO7bShb1xj8yL6HbUpBmDvFZoTxGfH3EhHB31k2ylPOisedybJpfhqlM= X-Received: by 2002:a25:2605:0:b0:b87:8580:ee37 with SMTP id m5-20020a252605000000b00b878580ee37mr2158593ybm.60.1680600354405; Tue, 04 Apr 2023 02:25:54 -0700 (PDT) MIME-Version: 1.0 References: <20230403122738.6006-1-zhangpeng.00@bytedance.com> In-Reply-To: <20230403122738.6006-1-zhangpeng.00@bytedance.com> From: Alexander Potapenko Date: Tue, 4 Apr 2023 11:25:17 +0200 Message-ID: Subject: Re: [PATCH v2] mm: kfence: Improve the performance of __kfence_alloc() and __kfence_free() To: Peng Zhang Cc: elver@google.com, dvyukov@google.com, akpm@linux-foundation.org, kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: s5p5mpggky9naxo84kfhwjfeix5p8wzo X-Rspamd-Queue-Id: 68CF2100025 X-HE-Tag: 1680600355-330232 X-HE-Meta: U2FsdGVkX18K3X+9Veu3vL6MfvNa4zsEdou5KrvTCfr1GG99jbOCFQmpnkFyLmDsPhWqwCVPWWMHW4JQG+V5hCGaSegcnujZSomBagdTxryBWWHJnjtQqw1L4SIz0IT2l8gbiMxSMQ+OGr7XB2lJuSlprzaftQkrlPDhak4SOzW192O5g3ni27VHpOqzdU5WZxcAuOQkDP+x2isJ2G4p5WxrWIvqFwXUmE0Wx8grOe5g8ZAEvfPe+2Ps9ezXGnC5z2mASUt/CaSgJzrMxykj1V0AQHAlSl3glliA85HWEFofi4DqUDFxcLts/MhCJ+cA2eCVRnyNFpviLnfS1Q4g1jBE7erYtF9bejWduOuWElau3DURFyU+Z70AIR1LHOqmsnyImwRaMW1ess92PEmIhUDFY2HyF7rw/cz+/fJVARjt3Oyn1HBp7dlyLIYwJ/gitHorfaRWOa9ehR+yCW9FRq9kjw/VB6vNCl2qKQyBEQZKZEvCPrCaA9fntWAwkm7BVXesFqlW/J66WsTe+mS4fPB5LnJDGFV8gk5hkupZlRQNiUfJ8bciGicWmgOch7X9in5l23iTerV+i57F2VIpusiWk+7BxJdZreU24StBIVRk313sUEZf5+8toYAI7inH/SC6ShKvpcUQsH22FUWEAS9Pg7wIf+xHWpfTAXpXPP95ND5p4M1e826nyhBsl31qfc/fMTw2FbgJF9P34AwTab7y0kr4kR0yio1CZ0RhSebN9kFmZo5QfKsFQQlukHA/Ztshfsq3iJmG9qU9t+RdKjEHeuinDSUkR2Xc1vBGhqevaU1Lqkf5YIG/Q4LUtn0DNdv/qXLTfGsG9OIMKGlQPrWxNWjJ9RzreaEW5m1NElR3EaxB82j/Iy1pmxkBdC+WWc6poDIR+amBwFZoNFeA6g+MpCVSdTOIn3WMYDvm2gSNKg4ADaw4A5iLNbLISHtoPNLHijbaIXh9N3McVqF M8GnEsSn o5quAdRmfRLHpUO4GYAXSfMiNzc5X06YcGaO8753z13wMNtSBrq9cqBpAkONhqdMU4v/kypmBdP+I6xowX4lhxr6n3YNCfpBGwIduOMdSsXncZcvGBjO5j8DzkNfpmj+IZ0v/d19VpVR6OYX184DljR7tyL5Pu+mOE8xqpGnAKGr5ZRrgKaD2poAI8GVtsQnr/lYmKiKLfbsT7YgDSj+aTDoBYtPKq1V4bTizFvnxWu3c8XtS3Lf9DQFuOORf3FNaaRLXp8HS0eXwMjN/bVUE2pfNrtMsdkpc2rKB39I2z244nuzdR4+EyvjhMsZ6IquK4IFfZJTkE7m7nV8eiPGZAZ/FET5EBFGPCpHO+Eh2B4XI4pBG/TVjRik8zee+iN1E3rK5yG2ztWr+lC2TwrqRZJxxoC0NJgDflU55hjujHxYU47/LPQuxHMTSbSzw3vD5ivRqzOImXXBGlqk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > +static inline void check_canary(const struct kfence_metadata *meta) > +{ > + const unsigned long pageaddr =3D ALIGN_DOWN(meta->addr, PAGE_SIZE= ); > + unsigned long addr =3D pageaddr; > > /* > - * We'll iterate over each canary byte per-side until fn() return= s > - * false. However, we'll still iterate over the canary bytes to t= he > + * We'll iterate over each canary byte per-side until a corrupted= byte > + * is found. However, we'll still iterate over the canary bytes t= o the > * right of the object even if there was an error in the canary b= ytes to > * the left of the object. Specifically, if check_canary_byte() > * generates an error, showing both sides might give more clues a= s to > @@ -339,16 +348,35 @@ static __always_inline void for_each_canary(const s= truct kfence_metadata *meta, > */ > > /* Apply to left of object. */ > - for (addr =3D pageaddr; addr < meta->addr; addr++) { > - if (!fn((u8 *)addr)) > + for (; meta->addr - addr >=3D sizeof(u64); addr +=3D sizeof(u64))= { > + if (unlikely(*((u64 *)addr) !=3D KFENCE_CANARY_PATTERN_U6= 4)) > break; > } I am confused. Right now this loop either runs from pageaddr to meta_addr if there's no corruption, or breaks at the first corrupted byte. Regardless of that, we are applying check_canary_byte() to every byte of that range in the following loop. Shouldn't the two be nested, like in the case of the canary bytes to the right of the object? > > - /* Apply to right of object. */ > - for (addr =3D meta->addr + meta->size; addr < pageaddr + PAGE_SIZ= E; addr++) { > - if (!fn((u8 *)addr)) > + /* > + * If the canary is corrupted in a certain 64 bytes, or the canar= y > + * memory cannot be completely covered by multiple consecutive 64= bytes, > + * it needs to be checked one by one. > + */ > + for (; addr < meta->addr; addr++) { > + if (unlikely(!check_canary_byte((u8 *)addr))) > break; > } > + > + /* Apply to right of object. */ > + for (addr =3D meta->addr + meta->size; addr % sizeof(u64) !=3D 0;= addr++) { > + if (unlikely(!check_canary_byte((u8 *)addr))) > + return; > + } > + for (; addr - pageaddr < PAGE_SIZE; addr +=3D sizeof(u64)) { > + if (unlikely(*((u64 *)addr) !=3D KFENCE_CANARY_PATTERN_U6= 4)) { > + > + for (; addr - pageaddr < PAGE_SIZE; addr++) { > + if (!check_canary_byte((u8 *)addr)) > + return; > + } > + } > + } > } > > static void *kfence_guarded_alloc(struct kmem_cache *cache, size_t size,= gfp_t gfp, > @@ -434,7 +462,7 @@ static void *kfence_guarded_alloc(struct kmem_cache *= cache, size_t size, gfp_t g > #endif > > /* Memory initialization. */ > - for_each_canary(meta, set_canary_byte); > + set_canary(meta); > > /* > * We check slab_want_init_on_alloc() ourselves, rather than lett= ing > @@ -495,7 +523,7 @@ static void kfence_guarded_free(void *addr, struct kf= ence_metadata *meta, bool z > alloc_covered_add(meta->alloc_stack_hash, -1); > > /* Check canary bytes for memory corruption. */ > - for_each_canary(meta, check_canary_byte); > + check_canary(meta); > > /* > * Clear memory if init-on-free is set. While we protect the page= , the > @@ -751,7 +779,7 @@ static void kfence_check_all_canary(void) > struct kfence_metadata *meta =3D &kfence_metadata[i]; > > if (meta->state =3D=3D KFENCE_OBJECT_ALLOCATED) > - for_each_canary(meta, check_canary_byte); > + check_canary(meta); > } > } > > diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h > index 600f2e2431d6..2aafc46a4aaf 100644 > --- a/mm/kfence/kfence.h > +++ b/mm/kfence/kfence.h > @@ -21,7 +21,15 @@ > * lower 3 bits of the address, to detect memory corruptions with higher > * probability, where similar constants are used. > */ > -#define KFENCE_CANARY_PATTERN(addr) ((u8)0xaa ^ (u8)((unsigned long)(add= r) & 0x7)) > +#define KFENCE_CANARY_PATTERN_U8(addr) ((u8)0xaa ^ (u8)((unsigned long)(= addr) & 0x7)) > + > +/* > + * Define a continuous 8-byte canary starting from a multiple of 8. The = canary > + * of each byte is only related to the lowest three bits of its address,= so the > + * canary of every 8 bytes is the same. 64-bit memory can be filled and = checked > + * at a time instead of byte by byte to improve performance. > + */ > +#define KFENCE_CANARY_PATTERN_U64 ((u64)0xaaaaaaaaaaaaaaaa ^ (u64)(0x070= 6050403020100)) > > /* Maximum stack depth for reports. */ > #define KFENCE_STACK_DEPTH 64 > diff --git a/mm/kfence/report.c b/mm/kfence/report.c > index 60205f1257ef..197430a5be4a 100644 > --- a/mm/kfence/report.c > +++ b/mm/kfence/report.c > @@ -168,7 +168,7 @@ static void print_diff_canary(unsigned long address, = size_t bytes_to_show, > > pr_cont("["); > for (cur =3D (const u8 *)address; cur < end; cur++) { > - if (*cur =3D=3D KFENCE_CANARY_PATTERN(cur)) > + if (*cur =3D=3D KFENCE_CANARY_PATTERN_U8(cur)) > pr_cont(" ."); > else if (no_hash_pointers) > pr_cont(" 0x%02x", *cur); > -- > 2.20.1 > > -- > You received this message because you are subscribed to the Google Groups= "kasan-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an= email to kasan-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgi= d/kasan-dev/20230403122738.6006-1-zhangpeng.00%40bytedance.com. -- Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Stra=C3=9Fe, 33 80636 M=C3=BCnchen Gesch=C3=A4ftsf=C3=BChrer: Paul Manicle, Liana Sebastian Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg