From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3328C636D6 for ; Sat, 4 Feb 2023 02:01:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 330756B0073; Fri, 3 Feb 2023 21:01:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B9226B0075; Fri, 3 Feb 2023 21:01:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 159716B0078; Fri, 3 Feb 2023 21:01:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 033C66B0073 for ; Fri, 3 Feb 2023 21:01:20 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B8A7EAB816 for ; Sat, 4 Feb 2023 02:01:19 +0000 (UTC) X-FDA: 80427957078.21.624B20F Received: from mail-pf1-f171.google.com (mail-pf1-f171.google.com [209.85.210.171]) by imf01.hostedemail.com (Postfix) with ESMTP id D932740024 for ; Sat, 4 Feb 2023 02:01:17 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ozNcmix2; spf=pass (imf01.hostedemail.com: domain of john.fastabend@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=john.fastabend@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675476077; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MR1UMa9NQuAw0hKniXicFJam+Vk7pdmWl7enSJwds8I=; b=QIISQ0IqbulkK4gREXI3u7zlzo4xFEvjzpU+KUqvFVBm2E+T3c4rkO4O+Fnuh4lM/d9j9A B/7vTloNX+KI8TA/6rK5f8qm2QDnv0QKnl07Lc4l/Kk4qI6mpq7uCrXiWZq4wwwSDPa34V AavPW8VcsO+BI6U2BjfxlsRvqvln1ww= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ozNcmix2; spf=pass (imf01.hostedemail.com: domain of john.fastabend@gmail.com designates 209.85.210.171 as permitted sender) smtp.mailfrom=john.fastabend@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675476077; a=rsa-sha256; cv=none; b=7lZv6/ggJKlutNzJOXw/dkTzLtP5tnqv5jtsXTvTcEQb4JNaJLl/HfYeKQWZRCvuZPZoUX Qz7N75JAPrTLZM9undJ7jMEUnr4LsZXAKyja39clV54j1zklgBowdXnhlVEIqUyhBLC5a9 /DDtimaHMvkxpYBxitL0zIyHCEiPs5w= Received: by mail-pf1-f171.google.com with SMTP id ay1so4967684pfb.7 for ; Fri, 03 Feb 2023 18:01:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=MR1UMa9NQuAw0hKniXicFJam+Vk7pdmWl7enSJwds8I=; b=ozNcmix20AdZk6KwMw/XZ8HelBBBmc0ozY0lywRcac8w/cUP01VmmTaFrvVWFM+/FP kE6nS2nKshxAwa8ZCz8E/+TIwsglxSfrtksE3eECAjrVyIGCDpjZMdy2OEt3jaXE2FA8 B16VJu0m175moKuDNBp4SSBO6LOyjFow+4m+l3AfURWTGeLRKCIVpaNJN3UTEokA5Re5 kgZ0bZQlRASzR4dScx8Zt/8qmkExoR+EQs4Z18bAHOIp4picQJHaX5rEyedQD/fx8ANo gMZ/92Iu86wbyIFkinwNujU7uHdL2hQGiua+jcStIuUI1VCSukYB8VuKc5Hg5280c/yE V1Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=MR1UMa9NQuAw0hKniXicFJam+Vk7pdmWl7enSJwds8I=; b=TTp4b1AX9pqS4+5B+f5Yo5vX3UlKw+KgeUmr6NuQOnFPE6XYrzeh/DryffIeLLZvcp U0zOZzR3rpDFog+fN6mU07cg7t7TMxzKKx9bWaz6sdSebXNKFLEpQCM8holKZmQwM9EO NHidtleIKQg7rEMhAGXGk35KBRgl93ynmOazHtzOGTgbpQxj3Ew/omiPye7o8xJlhKYl xWB/y1d9TALf3Kn4ucdHpWKBSBcgsf8Bz1At6FMrbmrbwwEP+/WqKUQ31SG13ntvr20i K4a9/gKT+XuBxKOyPnWSZrbv44PgTw8k3EiFws/wsgC51kN2sQB6KV/YYOTjPoWVnIl3 2t/g== X-Gm-Message-State: AO0yUKUE1RAXEdmxWL3MXGuLheB38EiG/+yqF3Z0zh6HpUj1CnTOlHfR 8YLv9jcW6pPOEK2bw5OYAxE= X-Google-Smtp-Source: AK7set/dEMYMVPk3JtMRcJTMcchyyjHAzFA4l7Y4yBvskaazinjOuyGfquntbpzbCaMTLZGomTaxQw== X-Received: by 2002:a05:6a00:23d5:b0:58d:be61:4859 with SMTP id g21-20020a056a0023d500b0058dbe614859mr14370246pfc.11.1675476076637; Fri, 03 Feb 2023 18:01:16 -0800 (PST) Received: from localhost ([98.97.116.12]) by smtp.gmail.com with ESMTPSA id s67-20020a625e46000000b00589c467ed88sm2526143pfb.69.2023.02.03.18.01.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Feb 2023 18:01:15 -0800 (PST) Date: Fri, 03 Feb 2023 18:01:13 -0800 From: John Fastabend To: Yafang Shao , ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, kafai@fb.com, songliubraving@fb.com, yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, tj@kernel.org, dennis@kernel.org, cl@linux.com, akpm@linux-foundation.org, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, vbabka@suse.cz, urezki@gmail.com Cc: linux-mm@kvack.org, bpf@vger.kernel.org, Yafang Shao Message-ID: <63ddbc69ef50f_6bb1520813@john.notmuch> In-Reply-To: <20230202014158.19616-8-laoar.shao@gmail.com> References: <20230202014158.19616-1-laoar.shao@gmail.com> <20230202014158.19616-8-laoar.shao@gmail.com> Subject: RE: [PATCH bpf-next 7/7] bpf: hashtab memory usage Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: fp43qe8cqosao3ytg5z5sh8p9cbzf7gw X-Rspam-User: X-Rspamd-Queue-Id: D932740024 X-Rspamd-Server: rspam06 X-HE-Tag: 1675476077-903570 X-HE-Meta: U2FsdGVkX1/WQE3QC/qYHBcqGyNsuif7Xf2BuetNDhSbe3UUPOchNtNAUpXCK0e+RUoHy5JY4BDRlGKvEayK0M0R59C7wjOPkbLn+Q4C4Hx+jmUgiA24d7FsNXgWHtvcNDPZNvCmoKMgBLD391vRCR7s01pSUnHYEKZ3cDbnKfKprFd1F1Nw6JHzKm2BDtAl0TW5ZFkeX7eoMhymCfn1yC/w65C1cnT6sRiWyMrDD2hf8q/nsCPIinwNZiS0UGYE937s4OSOVucgYTRjvHz4W8AXS5+avhD+V39p2cK1CUACeyhhEnmcspAu3vKAABUcypa6fIkklEAIXaFYwIPw8fIEApdYC8cjwoax4sscfI7QlERtpoyl/3iylsNwQjxsdPajKemVuOFb9w3zRqKohrmCru0sWVdUeWpv3NuhvuGIXW3agxRfS71fbPSf2LoWNW9TDluUilzkQ+uRq/MgH9oNe04I91S62Srg9jy2eFs5XsdLBc2KQxNlD5GERDcuXy9IThKhFqailyzK4ej3W6LMzNJ3rX9GBHWKMapEqVQbRqXJ7nZ6kpDbBexilhZk3FhY0rxgIUefy0suvPoV/qq9ILRfb86zz5slzZ0xLDcqogmtSEVSvJktE9Dcsqe8oor35Mb2KDKZmG1nPyp4NgN6UHloy0pjdDISpoya3wXULOIkawCtLXLJRGaMo5sCTMSmxomnTRwToPDU2hTmFojrnB2opvT8pLNSZv+hKGxnfYH6L+l130jf4hy1LP2N2QaF9DcZFNTLp4kpsfJnP4ajQhzl/ubHxfkA/UyZoRi/rJfEVEhXQGgDZ6w4QiEiUmTJzht+xjmfJFYthaHu6l/fghIXYouwBzw3IhBd55LG5JssOlDij5euvOxiaRCcWdkFS9ykdvWu4F0BM7wisSiHzWkO3FrpsTg4Lv+cbsshBvih8avleREF0Ml9snA0A5afLzsgM+uYFPJZpwn Q1/WIMDL aAd5B7wFAL7Cla3ZQoqnxOv4JzBlTyAcSulyGw0CGBKE9HZ1QegQo5f9b/ZYHE0BlZ/OwqL0R2uwcVaHP3SmFbi28IGhmeTg+rU409BCbgT3m84Mp+xlXnRM9cONoGHemp3iRbFrtbG4GOgwQODPoUgqyLD2zh0L/LcxDWibugmGnBeiQhGz+BUEGknSeTuj3+grVq8L6/vMQcKee6GTexxoOwq76j9Zcc/f5c3BDZ+qx6+BnaWlVc0QbG1/206JJ0U9Cshb3Bm9rJwnUSR4Xy9w8EItt2y61JxnKk+vigiqy4IZGyHtldXuHaNp2qHB/p55TBauqep82BePg9SWGIgkkOIr4xTIyAY3z5kILhDq+Ejr2xkIGYsCHlib5hksXbyPeGHLCxjwbzhgsa4ntSEYaDiQ/q43P+PTHJr8xDhhvgqOAclyv1+Ze0yR3GcEXF5UynSoQWRH4i7J0t+ZOH34O+ixHXmGATyBdq+KkZGxIUOHpPDp1b7J/N/ZrmiD3rJCg/Zl8dlE7umQTDcsFfaIX8txMXyRjTjgf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000024, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Yafang Shao wrote: > Get htab memory usage from the htab pointers we have allocated. Some > small pointers are ignored as their size are quite small compared with > the total size. > > The result as follows, > - before this change > 1: hash name count_map flags 0x0 <<<< prealloc > key 16B value 24B max_entries 1048576 memlock 41943040B > 2: hash name count_map flags 0x1 <<<< non prealloc, fully set > key 16B value 24B max_entries 1048576 memlock 41943040B > 3: hash name count_map flags 0x1 <<<< non prealloc, non set > key 16B value 24B max_entries 1048576 memlock 41943040B > > The memlock is always a fixed number whatever it is preallocated or > not, and whatever the allocated elements number is. > > - after this change > 1: hash name count_map flags 0x0 <<<< prealloc > key 16B value 24B max_entries 1048576 memlock 109064464B > 2: hash name count_map flags 0x1 <<<< non prealloc, fully set > key 16B value 24B max_entries 1048576 memlock 117464320B > 3: hash name count_map flags 0x1 <<<< non prealloc, non set > key 16B value 24B max_entries 1048576 memlock 16797952B > > The memlock now is hashtab actually allocated. > > At worst, the difference can be 10x, for example, > - before this change > 4: hash name count_map flags 0x0 > key 4B value 4B max_entries 1048576 memlock 8388608B > > - after this change > 4: hash name count_map flags 0x0 > key 4B value 4B max_entries 1048576 memlock 83898640B > This walks the entire map and buckets to get the size? Inside a rcu critical section as well :/ it seems. What am I missing, if you know how many elements are added (which you can track on map updates) how come we can't just calculate the memory size directly from this? Thanks, John