From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 015D6C433F5 for ; Mon, 11 Oct 2021 07:13:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3585260EBB for ; Mon, 11 Oct 2021 07:13:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3585260EBB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=gentwo.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id CC9876B006C; Mon, 11 Oct 2021 03:13:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C5287900003; Mon, 11 Oct 2021 03:13:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4146900002; Mon, 11 Oct 2021 03:13:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id A1C916B006C for ; Mon, 11 Oct 2021 03:13:55 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 63D90181AEF21 for ; Mon, 11 Oct 2021 07:13:55 +0000 (UTC) X-FDA: 78683292030.01.864D1A7 Received: from gentwo.de (vmi485042.contaboserver.net [161.97.139.209]) by imf19.hostedemail.com (Postfix) with ESMTP id 3DE9BB001CD1 for ; Mon, 11 Oct 2021 07:13:54 +0000 (UTC) Received: by gentwo.de (Postfix, from userid 1001) id 180B9B0025F; Mon, 11 Oct 2021 09:13:52 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by gentwo.de (Postfix) with ESMTP id 15A60B00100; Mon, 11 Oct 2021 09:13:52 +0200 (CEST) Date: Mon, 11 Oct 2021 09:13:52 +0200 (CEST) From: Christoph Lameter To: Hyeonggon Yoo <42.hyeyoo@gmail.com> cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka Subject: Re: [RFC] Some questions and an idea on SLUB/SLAB In-Reply-To: <20211009001903.GA3285@kvm.asia-northeast3-a.c.our-ratio-313919.internal> Message-ID: References: <20211009001903.GA3285@kvm.asia-northeast3-a.c.our-ratio-313919.internal> User-Agent: Alpine 2.22 (DEB 394 2020-01-19) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3DE9BB001CD1 X-Stat-Signature: 4k6ofenq6drya6c5g6teza6b8j5wzfgh Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf19.hostedemail.com: domain of cl@gentwo.de designates 161.97.139.209 as permitted sender) smtp.mailfrom=cl@gentwo.de X-HE-Tag: 1633936434-176230 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sat, 9 Oct 2021, Hyeonggon Yoo wrote: > - Is there a reason that SLUB does not implement cache coloring? > it will help utilizing hardware cache. Especially in block layer, > they are literally *squeezing* its performance now. Well as Matthew says: The high associativity of caches and the execution of other code path seems to make this not useful anymore. I am sure you can find a benchmark that shows some benefit. But please realize that in real-life the OS must perform work. This means that multiple other code paths are executed that affect cache use and placement of data in cache lines. > - In SLAB, do we really need to flush queues every few seconds? > (per cpu queue and shared queue). Flushing alien caches makes > sense, but flushing queues seems reducing it's fastpath. > But yeah, we need to reclaim memory. can we just defer this? The queues are designed to track cache hot objects (See the Bonwick paper). After a while the cachelines will be used for other purposes and no longer reflect what is in the caches. That is why they need to be expired. > - I don't like SLAB's per-node cache coloring, because L1 cache > isn't shared between cpus. For now, cpus in same node are sharing > its colour_next - but we can do better. This differs based on the cpu architecture in use. SLAB has an ideal model of how caches work and keeps objects cache hot based on that. In real life the cpu architecture differs from what SLAB things how caches operate. > what about splitting some per-cpu variables into kmem_cache_cpu > like SLUB? I think cpu_cache, colour (and colour_next), > alloc{hit,miss}, and free{hit,miss} can be per-cpu variables. That would in turn increase memory use and potentially the cache footprint of the hot paths.