From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFCD8C433F5 for ; Mon, 11 Oct 2021 07:23:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 55F1660EFE for ; Mon, 11 Oct 2021 07:23:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 55F1660EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E1C8A6B0071; Mon, 11 Oct 2021 03:23:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DCCB9900002; Mon, 11 Oct 2021 03:23:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C94096B0073; Mon, 11 Oct 2021 03:23:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id B7F8B6B0071 for ; Mon, 11 Oct 2021 03:23:41 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6E09A2C69B for ; Mon, 11 Oct 2021 07:23:41 +0000 (UTC) X-FDA: 78683316642.14.F77EC45 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf13.hostedemail.com (Postfix) with ESMTP id 28E6C103E780 for ; Mon, 11 Oct 2021 07:23:41 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id ns7-20020a17090b250700b001a0937b87b7so1163468pjb.1 for ; Mon, 11 Oct 2021 00:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=+wjcgXWzQZD7HaXDjp6jzo5agAgGWRJse11x4Bu0/cM=; b=cPUgApwliW+Cbo9e9tFlVhVKvscJY5IcH9fd2lqUHM7RciAKMxgnYuFrsYcnifkTvn pU5RudlReApPsGGYMn5Ay0PnPzXQMULaQOjadcgWzFmgcguoOVR2uXUOu9GdL0euBCYu tY2yYy1UDlNuZzreD2I8Tu7wQmghfBYFEGKVfXC6LlpNoiNyoknVGDsUKiuwbYV5x8+j SfIfVhrGzIZXUSn+VZ1nRzlW4jiSJWHTEmB/BHv+BI9/NXh+mc5jSFg6Bwmb4DjLostI D3fBD3jrbpNv5n5M4n6HqoI8ZEpfPPTVZPf+igpMw70mZGfzoZ1sa3onO820rajL7EDu RwgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=+wjcgXWzQZD7HaXDjp6jzo5agAgGWRJse11x4Bu0/cM=; b=H6a0tu90XFAxxS8+FJRl3E4R9nb8RspmnZ41k3RaZYtftrRkC6lIMHVnO5XlyI9pUe TDrnLtO0vCuQRmnG6OWggEl5hPQrJykluAf+nORUES2xkeaXtBmK6K81GY/fZSz27vyp AzXXDQ6qdKUodahSn5rS2NbzQMbytvNDW95+PSHyefjkOa8ouqqp0OsFGYTogWxgIZMg Oo+QxBtqDQ68ejkAvsY5FhzXKOMgmzJQBLvV5E7+x59pLybggklCLEu3hVbCjvkAMsn2 pkMiiBPb2nwJC0SaGgfosIjhtg7rHforIHpPlAplA6stUpd9gAgi8tdJLiIHBWy7lTjI /tPA== X-Gm-Message-State: AOAM530gYHfIU+sbQxW22VnTw5FSGi0etwnPnMrWlkpQG3mpWmlR/XDU N5d6gW1siEf9nyEfWiOv/P0= X-Google-Smtp-Source: ABdhPJw06fGR1fY3AjB1GwEyJNne3+wG6k5M7it+RndbaWuqoTE5RZNIXH2CIlIPGJ834ALpnVH4Wg== X-Received: by 2002:a17:902:70cb:b0:13e:91f3:641a with SMTP id l11-20020a17090270cb00b0013e91f3641amr22833719plt.13.1633937019980; Mon, 11 Oct 2021 00:23:39 -0700 (PDT) Received: from kvm.asia-northeast3-a.c.our-ratio-313919.internal (24.151.64.34.bc.googleusercontent.com. [34.64.151.24]) by smtp.gmail.com with ESMTPSA id t1sm3818890pfe.51.2021.10.11.00.23.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Oct 2021 00:23:39 -0700 (PDT) Date: Mon, 11 Oct 2021 07:23:35 +0000 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: David Rientjes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , Pekka Enberg , Joonsoo Kim , Andrew Morton , Vlastimil Babka Subject: Re: [PATCH] mm, slub: Use prefetchw instead of prefetch Message-ID: <20211011072335.GA63489@kvm.asia-northeast3-a.c.our-ratio-313919.internal> References: <20211008133602.4963-1-42.hyeyoo@gmail.com> <30a76d87-e0af-3eec-d095-d87e898b31cf@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <30a76d87-e0af-3eec-d095-d87e898b31cf@google.com> X-Rspamd-Queue-Id: 28E6C103E780 X-Stat-Signature: xp1dkrp7gyh1qsfs9nhznoi7kbpfxnws Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=cPUgApwl; spf=pass (imf13.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1633937021-849863 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, Oct 10, 2021 at 03:49:07PM -0700, David Rientjes wrote: > On Fri, 8 Oct 2021, Hyeonggon Yoo wrote: > > > It's certain that an object will be not only read, but also > > written after allocation. > > > > Why is it certain? I think perhaps what you meant to say is that if we > are doing any prefetching here, then access will benefit from prefetchw > instead of prefetch. But it's not "certain" that allocated memory will be > accessed at all. > Blame my english skill :( When I wrote I thought it was ok, but it was unclear. Thank you for pointing them! What I meant was "When accessing an object, it must be written before read. So There's no situation that caller only reads an object and does not write. Thus it's better to use prefetchw instead of prefetch.". Let's rephrase: commit 0ad9500e16fe ("slub: prefetch next freelist pointer in slab_alloc()") introduced prefetch_freepointer() because when other cpu(s) freed objects into a page that current cpu owns, the freelist link is hot on cpu(s) which freed objects and possibly very cold on current cpu. But if freelist link chain is hot on cpu(s) which freed objects, it's better to invalidate that chain because they're not going to access again within a short time. So use prefetchw instead of prefetch. On supported architectures like x86, it invalidates other copied instances of a cache line when prefetching it. > > Use prefetchw instead of prefetchw. On supported architecture > > If we're using prefetchw instead of prefetchw, I think the diff would be > 0 lines changed :) > That was my typo. thankfully Andrew fixed that. > > like x86, it helps to invalidate cache line when the object exists > > in other processors' cache. > > > > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> > > --- > > mm/slub.c | 7 +++---- > > 1 file changed, 3 insertions(+), 4 deletions(-) > > > > diff --git a/mm/slub.c b/mm/slub.c > > index 3d2025f7163b..2aca7523165e 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object) > > return freelist_dereference(s, object + s->offset); > > } > > > > -static void prefetch_freepointer(const struct kmem_cache *s, void *object) > > +static void prefetchw_freepointer(const struct kmem_cache *s, void *object) > > { > > - prefetch(object + s->offset); > > + prefetchw(object + s->offset); > > } > > > > static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) > > @@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, > > note_cmpxchg_failure("slab_alloc", s, tid); > > goto redo; > > } > > - prefetch_freepointer(s, next_object); > > + prefetchw_freepointer(s, next_object); > > stat(s, ALLOC_FASTPATH); > > } > > - > > maybe_wipe_obj_freeptr(s, object); > > init = slab_want_init_on_alloc(gfpflags, s); > > > > -- > > 2.27.0 > > > >