From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52060C43331 for ; Thu, 2 Apr 2020 04:04:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 10513206E9 for ; Thu, 2 Apr 2020 04:04:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="RZ9AV8Sj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10513206E9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A35628E0007; Thu, 2 Apr 2020 00:04:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E5B28E000D; Thu, 2 Apr 2020 00:04:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92BCB8E0007; Thu, 2 Apr 2020 00:04:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0125.hostedemail.com [216.40.44.125]) by kanga.kvack.org (Postfix) with ESMTP id 77D258E0007 for ; Thu, 2 Apr 2020 00:04:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3876A180AD811 for ; Thu, 2 Apr 2020 04:04:29 +0000 (UTC) X-FDA: 76661573058.07.wish64_2722b90b5474a X-HE-Tag: wish64_2722b90b5474a X-Filterd-Recvd-Size: 4235 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Thu, 2 Apr 2020 04:04:28 +0000 (UTC) Received: from localhost.localdomain (c-73-231-172-41.hsd1.ca.comcast.net [73.231.172.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A7EAC20747; Thu, 2 Apr 2020 04:04:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1585800268; bh=LZ4STEQjcXU311BkZV8n7kVf0npH8u4GNX5JVB2QMVQ=; h=Date:From:To:Subject:In-Reply-To:From; b=RZ9AV8SjfMrbZrTE4DvGhObQHVSEU9HF6VWhsxj7YI3YEBpRlSOFyYHFsWNWZZpch zhPEbM5zK3rTABFhchffHgCzPsq/r1AeFdH9u4Do0ZOYvfeKC1dbnjXcfY72xCTdKW M2DUvMFlCaIaJTq5kj3biV71Go0fDykCICbUBF7M= Date: Wed, 01 Apr 2020 21:04:27 -0700 From: Andrew Morton To: akpm@linux-foundation.org, cl@linux.com, iamjoonsoo.kim@lge.com, keescook@chromium.org, linux-mm@kvack.org, mm-commits@vger.kernel.org, penberg@kernel.org, rientjes@google.com, silvio.cesare@gmail.com, torvalds@linux-foundation.org, vnik@duasynt.com Subject: [patch 025/155] slub: relocate freelist pointer to middle of object Message-ID: <20200402040427.WyxceElzI%akpm@linux-foundation.org> In-Reply-To: <20200401210155.09e3b9742e1c6e732f5a7250@linux-foundation.org> User-Agent: s-nail v14.8.16 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Kees Cook Subject: slub: relocate freelist pointer to middle of object In a recent discussion[1] with Vitaly Nikolenko and Silvio Cesare, it became clear that moving the freelist pointer away from the edge of allocations would likely improve the overall defensive posture of the inline freelist pointer. My benchmarks show no meaningful change to performance (they seem to show it being faster), so this looks like a reasonable change to make. Instead of having the freelist pointer at the very beginning of an allocation (offset 0) or at the very end of an allocation (effectively offset -sizeof(void *) from the next allocation), move it away from the edges of the allocation and into the middle. This provides some protection against small-sized neighboring overflows (or underflows), for which the freelist pointer is commonly the target. (Large or well controlled overwrites are much more likely to attack live object contents, instead of attempting freelist corruption.) The vaunted kernel build benchmark, across 5 runs. Before: Mean: 250.05 Std Dev: 1.85 and after, which appears mysteriously faster: Mean: 247.13 Std Dev: 0.76 Attempts at running "sysbench --test=memory" show the change to be well in the noise (sysbench seems to be pretty unstable here -- it's not really measuring allocation). Hackbench is more allocation-heavy, and while the std dev is above the difference, it looks like may manifest as an improvement as well: 20 runs of "hackbench -g 20 -l 1000", before: Mean: 36.322 Std Dev: 0.577 and after: Mean: 36.056 Std Dev: 0.598 [1] https://twitter.com/vnik5287/status/1235113523098685440 Link: http://lkml.kernel.org/r/202003051624.AAAC9AECC@keescook Signed-off-by: Kees Cook Acked-by: Christoph Lameter Cc: Vitaly Nikolenko Cc: Silvio Cesare Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Signed-off-by: Andrew Morton --- mm/slub.c | 7 +++++++ 1 file changed, 7 insertions(+) --- a/mm/slub.c~slub-relocate-freelist-pointer-to-middle-of-object +++ a/mm/slub.c @@ -3581,6 +3581,13 @@ static int calculate_sizes(struct kmem_c */ s->offset = size; size += sizeof(void *); + } else if (size > sizeof(void *)) { + /* + * Store freelist pointer near middle of object to keep + * it away from the edges of the object to avoid small + * sized over/underflows from neighboring allocations. + */ + s->offset = ALIGN(size / 2, sizeof(void *)); } #ifdef CONFIG_SLUB_DEBUG _