From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09883C433F5 for ; Wed, 13 Oct 2021 03:44:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AE8FB60273 for ; Wed, 13 Oct 2021 03:44:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org AE8FB60273 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 04A41900002; Tue, 12 Oct 2021 23:44:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3C1A6B0071; Tue, 12 Oct 2021 23:44:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E03BB900002; Tue, 12 Oct 2021 23:44:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id D2B816B006C for ; Tue, 12 Oct 2021 23:44:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 975851839B99A for ; Wed, 13 Oct 2021 03:44:55 +0000 (UTC) X-FDA: 78690022950.28.200A8E1 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) by imf03.hostedemail.com (Postfix) with ESMTP id 3FB88300009B for ; Wed, 13 Oct 2021 03:44:55 +0000 (UTC) Received: by mail-pg1-f177.google.com with SMTP id g184so1001608pgc.6 for ; Tue, 12 Oct 2021 20:44:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=NPYgrbf9559QlllyAfgrfZzaAR2G3z6MzzMdSacYVDk=; b=qwTDO27q1hmN0i8E5OebUw3Fy9WikHhwEqwRQbWYg4iX56IgsvfOa6W/qc7Vx8NqX2 06Ifc3jggJa20HFM5uC8YZzDvrfDLkoul9BRvEQUSp91LPxehHsAKYUDvaREEkGsJO4E akWnsEFPMol2kigezE/0fUWBPt+FKXxgD+rE2dwNuT5F5FF+kywYzcBVaok940z8IMhY ZWf7aSlBlqvtG6+8oLeK7J8yWgqbBLNzZWnmCYShdL509863gKLKTezy7bhMzn5okr/F m2oeGfO9E7f5ueN5PT+5rF9GHTEMKjyILFwzbJSB8NkbKCpm8ezO/fMa5D9GWfWS08wM niAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=NPYgrbf9559QlllyAfgrfZzaAR2G3z6MzzMdSacYVDk=; b=uHb4EWAPtCTBcuwkYrmTRs2uj3TSdnqv4DR2SK0KFheEUv4B5zo/RcR+KcjpGL6HHc 6J5T3AujCQpL5caWbb6WIxNEW3DiNF6NXE/XQwugIjC9ccXRa5W7FVMYyiRW3qjcGMYw Ntp01ltiNjK5nGm8R1Frj999O9EQtqZ5aFfq8tJPgJL+rHPEinYdMWpckqAmIdpQX7Zz cWRtOZBOObj3MRn+XfRgjiCTY60nI4weAsMR4Mp/gFVbigV1ZLiM2PlJOVSD5KvzVoQQ iV152Eznn7E4lhXNyOL4IoVQRzOg4U6nN2WmifHeTbgGrqDwEnQiuTjq+aJrXHJ0Sf4c rBGQ== X-Gm-Message-State: AOAM533t3WNNQhQXMaYzFS1jUP0VcaauzpSzx8UNebzPqRM7+QjG1X/i uiT0NLjramJrO5VeDoV0Ohc= X-Google-Smtp-Source: ABdhPJxvkBaMJi8cGjDOfMO7F/noVX+3tEWo5xolEbY8oKD+sSwF8+K5HXuaD6vx5S8NX8ty8nJ6OA== X-Received: by 2002:a05:6a00:8d0:b0:44c:26e6:1c13 with SMTP id s16-20020a056a0008d000b0044c26e61c13mr35896493pfu.28.1634096694122; Tue, 12 Oct 2021 20:44:54 -0700 (PDT) Received: from kvm.asia-northeast3-a.c.our-ratio-313919.internal (24.151.64.34.bc.googleusercontent.com. [34.64.151.24]) by smtp.gmail.com with ESMTPSA id m11sm12888291pga.27.2021.10.12.20.44.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Oct 2021 20:44:53 -0700 (PDT) Date: Wed, 13 Oct 2021 03:44:49 +0000 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Christoph Lameter Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka Subject: Re: [RFC] Some questions and an idea on SLUB/SLAB Message-ID: <20211013034449.GA118049@kvm.asia-northeast3-a.c.our-ratio-313919.internal> References: <20211009001903.GA3285@kvm.asia-northeast3-a.c.our-ratio-313919.internal> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3FB88300009B X-Stat-Signature: kigb6diphh9znbsiz4eo5wsg6qe3rsdn Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=qwTDO27q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.177 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com X-HE-Tag: 1634096695-898554 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello Christoph, thank you for answering. On Mon, Oct 11, 2021 at 09:13:52AM +0200, Christoph Lameter wrote: > On Sat, 9 Oct 2021, Hyeonggon Yoo wrote: > > > - Is there a reason that SLUB does not implement cache coloring? > > it will help utilizing hardware cache. Especially in block layer, > > they are literally *squeezing* its performance now. > > Well as Matthew says: The high associativity of caches it seems not useful on my both machines (4-way / 8-way set associative) too. > and the execution > of other code path seems to make this not useful anymore. > > I am sure you can find a benchmark that shows some benefit. But please > realize that in real-life the OS must perform work. This means that > multiple other code paths are executed that affect cache use and placement > of data in cache lines. > cache coloring can make benchmark results better. But as slab uses more cache lines - that reduces other code paths' cache line. Did I get right? > > > - In SLAB, do we really need to flush queues every few seconds? > > (per cpu queue and shared queue). Flushing alien caches makes > > sense, but flushing queues seems reducing it's fastpath. > > But yeah, we need to reclaim memory. can we just defer this? > > The queues are designed to track cache hot objects (See the Bonwick > paper). After a while the cachelines will be used for other purposes and > no longer reflect what is in the caches. That is why they need to be > expired. I've read Bonwick paper but I thought expiring was need for reclaiming memory. maybe I got it wrong.. I should read it again. > > > > - I don't like SLAB's per-node cache coloring, because L1 cache > > isn't shared between cpus. For now, cpus in same node are sharing > > its colour_next - but we can do better. > > This differs based on the cpu architecture in use. SLAB has an ideal model > of how caches work and keeps objects cache hot based on that. In real life > the cpu architecture differs from what SLAB things how caches operate. > So the point is, As cache hierarchy differs based on architecture, assuming cpus have both unique cache per cpu, and shared cache among cpus can misfit in some architectures. > > what about splitting some per-cpu variables into kmem_cache_cpu > > like SLUB? I think cpu_cache, colour (and colour_next), > > alloc{hit,miss}, and free{hit,miss} can be per-cpu variables. > > That would in turn increase memory use and potentially the cache footprint > of the hot paths. > I thought splitting percpu data was need for coloring but it isn't useful. So that's unnecessary cost. Thanks, Hyeonggon.