From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38405C433F5 for ; Tue, 22 Feb 2022 10:23:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 888FE8D0002; Tue, 22 Feb 2022 05:23:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 811528D0001; Tue, 22 Feb 2022 05:23:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B4748D0002; Tue, 22 Feb 2022 05:23:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 5538A8D0001 for ; Tue, 22 Feb 2022 05:23:57 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1942922B6C for ; Tue, 22 Feb 2022 10:23:57 +0000 (UTC) X-FDA: 79170030114.01.E5C2B44 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf12.hostedemail.com (Postfix) with ESMTP id 6AAD04000F for ; Tue, 22 Feb 2022 10:23:56 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id g1so11552666pfv.1 for ; Tue, 22 Feb 2022 02:23:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=waRnTntaRcV123jQKR8XgkkGD1L2jU2jFxjpiAyOhmg=; b=P5kvT78Fw+5eP0y9W2JqmsqE1Ac1D/++OXV9m1caFdOFBjWArMgpIODIuam6dtBmW9 KxPq1KtNcMnZ/v0YfqUQwfYTblgfnuhRydVgNuckQeIxaDy1erbjqY2BJ22YZYtn0E+f Iug0LLP5NeEILOh6C6aqh/8WuwTDRd68+jImRZ3ptcKwq5BQeOckkqRD+drLRBBP5qlx svlRjhxxENKLJ4TTdi7W5+UO/fgo1zuVNNNA1903w89WlmjjyNbWxpe+b8xVLwJhYMqL PmXgSGCwzGo2McJbaheM5aCb/bVtlIT6j/uj3UnKYPw+2ynLs3xWb7SyQuhovJdVW+Lg i/dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=waRnTntaRcV123jQKR8XgkkGD1L2jU2jFxjpiAyOhmg=; b=ezJdxre4GQA6OXgvtQqvoEQBEWnd6r+2alyWLfkmkAty6pLQq65jgxlx3Ll8QgnFwX KAHjni83uPIKA9NFrw2HTtqLLEiMr8QGuecBMiSWaMnHcJiHIECiQUqn+kYqeM0ZabQI 3iRHh2VCJTMBBae28W5VNcowKYzIr9HC/C31xzXw54jaMJvwzBTaznJf8lA0+hWHaz9i +gO7N8kCi4UsAsp7zSVRecEHvnJF+47zM4aRJe4ItmB+auqYyXMZ3XHIqHT7PXebFHSU jSrpixuL4BlMTka5TIcRrQekqEUdruYoQFgQGrtKHSG3IwQch+S62H96QwQCu8itkQ55 uwKQ== X-Gm-Message-State: AOAM533PqrGQhkA3mQEmH7+qArJHQBk9Zo1SevfQRVKQGsKJeZ0Ewzld 3pcBArhILE0iZEH4VCD34qA= X-Google-Smtp-Source: ABdhPJwa2gfr9rLZSyl1W0w1g+3NYe3Mj9+0qKsh04iXECHtKqHRio7VaAj+zxgrOMPmAJtajZmE3A== X-Received: by 2002:a05:6a00:2301:b0:4e1:5842:48d7 with SMTP id h1-20020a056a00230100b004e1584248d7mr24382374pfh.14.1645525435537; Tue, 22 Feb 2022 02:23:55 -0800 (PST) Received: from ip-172-31-19-208.ap-northeast-1.compute.internal (ec2-18-181-137-102.ap-northeast-1.compute.amazonaws.com. [18.181.137.102]) by smtp.gmail.com with ESMTPSA id s19sm16541936pfu.34.2022.02.22.02.23.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Feb 2022 02:23:55 -0800 (PST) Date: Tue, 22 Feb 2022 10:23:51 +0000 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Vasily Averin Cc: Linux MM , Andrew Morton , kernel@openvz.org Subject: Re: slabinfo shows incorrect active_objs ??? Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=P5kvT78F; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com X-Rspam-User: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6AAD04000F X-Stat-Signature: au11dgf8mwdm7sbifhoahkn61fopeccp X-HE-Tag: 1645525436-442669 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 22, 2022 at 12:22:02PM +0300, Vasily Averin wrote: > Dear all, > > I've found that /proc/slabinfo shows inadequate numbers of in-use slab objects. > it assumes that all objects stored in cpu caches are always 100% in use. > > for example: > slabinfo shows that all 20 objects are in use. > > [root@fc34-vvs linux]# uname -a > Linux fc34-vvs.sw.ru 5.17.0-rc3+ #42 SMP PREEMPT Mon Feb 21 20:14:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux > > [root@fc34-vvs linux]# cat /proc/slabinfo > slabinfo - version: 2.1 > # name : tunables : slabdata > ... > kmalloc-cg-8k 20 20 8192 4 8 : tunables 0 0 0 : slabdata 5 5 0 > > At the same time crash said that only 2 objects are in use. > > crash> kmem -s kmalloc-cg-8k > CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME > ffff8f4840043b00 8192 2 20 5 32k kmalloc-cg-8k > > And this looks like true, see kmem -S output below. > > Is it a bug or perhaps a well-known feature that I missed? > This is not a bug.. TL;DR: Some of slabs are locked by CPU and SLUB does not accurately account them. (I guess its purpose is to reduce cacheline usages in fastpath?) There are 3 status of slab. A slab can be: 1) a cpu slab or 2) cpu partial slabs: They're locked (frozen) by cpu to avoid per-node locking overhead. 3) node partial slabs: They exist in node's partial slab list. Any CPU can take them using spinlock. Usually SLUB takes slab from node partial slabs when there is no cpu slab and cpu partial slabs. Only free objects in node partial slabs are calculated as free objects. > Numbers are counted in mm/slub.c, see below, > but count_partial() doe not includes free objects of cpu caches > > Moreover adequate statistic is not showed in any other interfaces too > /sys/kerenl/slab/ read cpu slab caches but does not output these numbers. > > Thank you, > Vasily Averin > > #ifdef CONFIG_SLUB_DEBUG > void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo) > { > unsigned long nr_slabs = 0; > unsigned long nr_objs = 0; > unsigned long nr_free = 0; > int node; > struct kmem_cache_node *n; > > for_each_kmem_cache_node(s, node, n) { > nr_slabs += node_nr_slabs(n); > nr_objs += node_nr_objs(n); > nr_free += count_partial(n, count_free); > } > > sinfo->active_objs = nr_objs - nr_free; This code assumes all objects in frozen slabs (cpu slab or cpu partial slabs) are not free. With current implementation of SLUB, there is no easy way to accurately account objects in cpu slab and cpu partial slabs' objects because SLUB sets slab->inuse = slab->objects when taking slab from node partial slabs. They are accounted again only when going back to node partial slabs. (deactivated). Thanks, Hyeonggon > sinfo->num_objs = nr_objs; > > > > crash> kmem -S kmalloc-cg-8k > CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME > ffff8f4840043b00 8192 2 20 5 32k kmalloc-cg-8k > CPU 0 KMEM_CACHE_CPU: > ffff8f4b58236360 > CPU 0 SLAB: > (empty) > CPU 0 PARTIAL: > (empty) > CPU 1 KMEM_CACHE_CPU: > ffff8f4b58276360 > CPU 1 SLAB: > SLAB MEMORY NODE TOTAL ALLOCATED FREE > ffffed3f842af400 ffff8f484abd0000 0 4 1 3 > FREE / [ALLOCATED] > ffff8f484abd0000 (cpu 1 cache) > ffff8f484abd2000 (cpu 1 cache) > ffff8f484abd4000 (cpu 1 cache) > [ffff8f484abd6000] > CPU 1 PARTIAL: > (empty) > CPU 2 KMEM_CACHE_CPU: > ffff8f4b582b6360 > CPU 2 SLAB: > (empty) > CPU 2 PARTIAL: > (empty) > CPU 3 KMEM_CACHE_CPU: > ffff8f4b582f6360 > CPU 3 SLAB: > SLAB MEMORY NODE TOTAL ALLOCATED FREE > ffffed3f842ce600 ffff8f484b398000 0 4 0 4 > FREE / [ALLOCATED] > ffff8f484b398000 (cpu 3 cache) > ffff8f484b39a000 (cpu 3 cache) > ffff8f484b39c000 (cpu 3 cache) > ffff8f484b39e000 (cpu 3 cache) > CPU 3 PARTIAL: > (empty) > CPU 4 KMEM_CACHE_CPU: > ffff8f4b58336360 > CPU 4 SLAB: > SLAB MEMORY NODE TOTAL ALLOCATED FREE > ffffed3f8418c200 ffff8f4846308000 0 4 0 4 > FREE / [ALLOCATED] > ffff8f4846308000 (cpu 4 cache) > ffff8f484630a000 (cpu 4 cache) > ffff8f484630c000 (cpu 4 cache) > ffff8f484630e000 (cpu 4 cache) > CPU 4 PARTIAL: > (empty) > CPU 5 KMEM_CACHE_CPU: > ffff8f4b58376360 > CPU 5 SLAB: > (empty) > CPU 5 PARTIAL: > (empty) > CPU 6 KMEM_CACHE_CPU: > ffff8f4b583b6360 > CPU 6 SLAB: > SLAB MEMORY NODE TOTAL ALLOCATED FREE > ffffed3f8412d000 ffff8f4844b40000 0 4 0 4 > FREE / [ALLOCATED] > ffff8f4844b40000 (cpu 6 cache) > ffff8f4844b42000 (cpu 6 cache) > ffff8f4844b44000 (cpu 6 cache) > ffff8f4844b46000 (cpu 6 cache) > CPU 6 PARTIAL: > (empty) > CPU 7 KMEM_CACHE_CPU: > ffff8f4b583f6360 > CPU 7 SLAB: > SLAB MEMORY NODE TOTAL ALLOCATED FREE > ffffed3f84124000 ffff8f4844900000 0 4 1 3 > FREE / [ALLOCATED] > ffff8f4844900000 (cpu 7 cache) > ffff8f4844902000 (cpu 7 cache) > [ffff8f4844904000] > ffff8f4844906000 (cpu 7 cache) > CPU 7 PARTIAL: > (empty) > KMEM_CACHE_NODE NODE SLABS PARTIAL PER-CPU > ffff8f48400416c0 0 5 0 5 > NODE 0 PARTIAL: > (empty) > NODE 0 FULL: > (not tracked) > > >