From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11AF0C433EF for ; Mon, 11 Oct 2021 07:21:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A705960EC0 for ; Mon, 11 Oct 2021 07:21:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A705960EC0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 483596B006C; Mon, 11 Oct 2021 03:21:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40B4A900002; Mon, 11 Oct 2021 03:21:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ACAC6B0073; Mon, 11 Oct 2021 03:21:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0213.hostedemail.com [216.40.44.213]) by kanga.kvack.org (Postfix) with ESMTP id 1CDD56B006C for ; Mon, 11 Oct 2021 03:21:04 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D31402FDEF for ; Mon, 11 Oct 2021 07:21:03 +0000 (UTC) X-FDA: 78683310006.28.D17EC4A Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf27.hostedemail.com (Postfix) with ESMTP id 52DA67008017 for ; Mon, 11 Oct 2021 07:21:03 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C690A22034; Mon, 11 Oct 2021 07:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1633936861; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUVi+DxairWD+VkQf6dH9WAeeHTskkzQRsj/8RNh5ZQ=; b=e+sA/i1b0HA9+DgdWnHb+992RjMBeptf4qC24uRPj20INlmN1iLK8xqG4NLBEzv9C8+G/0 mi0Zqdh8Kwcdoc/Jz6JELeM9FCVOFO+udoUGj1+cr58UCyxx2xN7DczCCDZZ9vjB7c/0rG pI2ArVYPSh9AedbIuZTiXYUojOPkh9M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1633936861; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VUVi+DxairWD+VkQf6dH9WAeeHTskkzQRsj/8RNh5ZQ=; b=AeGVcYTz5CK7qE3jozxGrnPapF50VMSkVj76p2omY9m/4t+XiiB7a5DNHES7CQo9+hM3iD AUk8G6L7+f7G4gDA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A5C5813BAF; Mon, 11 Oct 2021 07:21:01 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id muCxJ93lY2ElWAAAMHmgww (envelope-from ); Mon, 11 Oct 2021 07:21:01 +0000 Message-ID: <904b6e72-cc2e-2e4d-5601-dacab734bf15@suse.cz> Date: Mon, 11 Oct 2021 09:21:01 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.2.0 Content-Language: en-US To: David Rientjes , Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , Pekka Enberg , Joonsoo Kim , Andrew Morton References: <20211008133602.4963-1-42.hyeyoo@gmail.com> <30a76d87-e0af-3eec-d095-d87e898b31cf@google.com> From: Vlastimil Babka Subject: Re: [PATCH] mm, slub: Use prefetchw instead of prefetch In-Reply-To: <30a76d87-e0af-3eec-d095-d87e898b31cf@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 52DA67008017 X-Stat-Signature: adfqdfffyckzo895chajhs6dhbsdd6pd Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="e+sA/i1b"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=AeGVcYTz; dmarc=none; spf=pass (imf27.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1633936863-493513 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/11/21 00:49, David Rientjes wrote: > On Fri, 8 Oct 2021, Hyeonggon Yoo wrote: > >> It's certain that an object will be not only read, but also >> written after allocation. >> > > Why is it certain? I think perhaps what you meant to say is that if we > are doing any prefetching here, then access will benefit from prefetchw > instead of prefetch. But it's not "certain" that allocated memory will be > accessed at all. I think the primary reason there's a prefetch is freelist traversal. The cacheline we prefetch will be read during the next allocation, so if we expect there to be one soon, prefetch might help. That the freepointer is part of object itself and thus the cache line will be probably accessed also after the allocation, is secondary. Yeah this might help some workloads, but perhaps hurt others - these things might look obvious in theory but be rather unpredictable in practice. At least some hackbench results would help... >> Use prefetchw instead of prefetchw. On supported architecture > > If we're using prefetchw instead of prefetchw, I think the diff would be > 0 lines changed :) > >> like x86, it helps to invalidate cache line when the object exists >> in other processors' cache. >> >> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> >> --- >> mm/slub.c | 7 +++---- >> 1 file changed, 3 insertions(+), 4 deletions(-) >> >> diff --git a/mm/slub.c b/mm/slub.c >> index 3d2025f7163b..2aca7523165e 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object) >> return freelist_dereference(s, object + s->offset); >> } >> >> -static void prefetch_freepointer(const struct kmem_cache *s, void *object) >> +static void prefetchw_freepointer(const struct kmem_cache *s, void *object) I wouldn't rename the function itself, unless we have both variants for different situations (we don't). That it uses prefetchw() is internal detail at this point. >> { >> - prefetch(object + s->offset); >> + prefetchw(object + s->offset); >> } >> >> static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) >> @@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, >> note_cmpxchg_failure("slab_alloc", s, tid); >> goto redo; >> } >> - prefetch_freepointer(s, next_object); >> + prefetchw_freepointer(s, next_object); >> stat(s, ALLOC_FASTPATH); >> } >> - >> maybe_wipe_obj_freeptr(s, object); >> init = slab_want_init_on_alloc(gfpflags, s); >> >> -- >> 2.27.0 >> >> >