From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDF04C77B7F for ; Thu, 11 May 2023 07:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 493B76B0075; Thu, 11 May 2023 03:50:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 442A96B0078; Thu, 11 May 2023 03:50:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30A016B007B; Thu, 11 May 2023 03:50:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 219CF6B0075 for ; Thu, 11 May 2023 03:50:48 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D8C5E1C74A4 for ; Thu, 11 May 2023 07:50:47 +0000 (UTC) X-FDA: 80777202534.28.91D260B Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by imf13.hostedemail.com (Postfix) with ESMTP id EB65020005 for ; Thu, 11 May 2023 07:50:44 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TpTvOo69; spf=pass (imf13.hostedemail.com: domain of edumazet@google.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=edumazet@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1683791445; a=rsa-sha256; cv=none; b=kH+Zd1SWUmrDBvjbz9igZZYCt+gS2a2U+9ajFGMO84ep+zNXwz8AkHSic+XPryt4dvd5B2 vZQS9ZksxD3WmCOfIT5Wnps20dOHjN42VXfnMMiBJtw9VXZE/rlM1zZ8Q8LIhf2m/B71k+ eYCJxmIo1Dc07x9wdRp1DC0EyjL0bkM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TpTvOo69; spf=pass (imf13.hostedemail.com: domain of edumazet@google.com designates 209.85.128.52 as permitted sender) smtp.mailfrom=edumazet@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1683791445; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ybwF8qJFWAbvIvu+SC1YZHKFpjOW6AAAy4k7z/ztuq8=; b=prXVCJFRIwDvgq318FAURg5WlECuSUY7sGoBFvX2+ZSmlAjLuCFDm2/ZVYPbAEuERQ8g9a zqHWJQxb7vl8wYr6oGMdDjn8CE2BlLrYU1R7D4dtADAuHyDc2mdMVnuTFdqFXSB99qMTMa manNmsyyOYWnqaLRaTnblLQidpwPjvA= Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-3f4234f67feso178925e9.0 for ; Thu, 11 May 2023 00:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683791443; x=1686383443; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ybwF8qJFWAbvIvu+SC1YZHKFpjOW6AAAy4k7z/ztuq8=; b=TpTvOo69AQQX6ckCLnMGs2C1Ya6+FtQsvhiEZukT0WOHLPa07jBGET3X+nbAMr1oII R527Pl+B48yYnKtX27XeGJfugo+z5bSUH/sqfd0dIjeUa2XBeE4EFu2rts/BQl3iVIHv zqBO4Z1XEHk40zwW6plDhl55btXvYaBAbhZBcW+ElbgJjm1k8mQoq42qjt14t2ZS4HNz zVt5v/M/buMkXDJYdtgaXtOOYZDXh4FaSoDQTV5QSDy5hstp+KI+/WouYOz7vbw6Jjxb bqYabyihtnyc6iqaudJsjdtgibb89uBOxRZNM3ODJMMgZSAsSYA8FuO80G5H2zzEAbwR 9l4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683791443; x=1686383443; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ybwF8qJFWAbvIvu+SC1YZHKFpjOW6AAAy4k7z/ztuq8=; b=bRQuD/xB3PDdxpDElt3pvw98XrTgiSVgwWoJ01T0iVqpnij3RqfEEgoTxqYmdDjzaK hwr3RRfrZi6+NaJEIZWCUwZEFW8ZQKqJRNel8zeUP0Yak/3tvBflWcwjRFawZiiifkkr 8i8gLu/K1SoffX55Ru76zFwmVVP56Rqs1tmgVTLvvlnL4cciDYHzDmQF1m2t8DRS73nr p8Xxa5zhYv1JupCmEbbbGEHfldM8EuzHNskopt32a+TAlYamDmPO0mgeX9L4yyqeoHDK G5OIMQ8J61JF80cvcrtkc6vFdzk4GoLjBt+8Jn5A89HlFRBc3YewIQR7zRel2t8/L+wb c5+g== X-Gm-Message-State: AC+VfDwroqQ64QGIXzQbTSSh9D7bRH9sLwxGQklZZ++otISTzjFZf7Yw OoCFLJ/q4JafSg/c7VX2GdGOkb6BAsL+iPcTp+v/kA== X-Google-Smtp-Source: ACHHUZ6TXJOO2RDr71ETxHJyDOo5G2bNSt1XGb8zhwZ5Dinr0Sv66xDTHCGvLoBiO697s04PMDeyCI7fzks/g6or/OE= X-Received: by 2002:a05:600c:1c8b:b0:3f1:664a:9a52 with SMTP id k11-20020a05600c1c8b00b003f1664a9a52mr81948wms.7.1683791443152; Thu, 11 May 2023 00:50:43 -0700 (PDT) MIME-Version: 1.0 References: <20230508020801.10702-1-cathy.zhang@intel.com> <20230508020801.10702-2-cathy.zhang@intel.com> <3887b08ac0e55e27a24d2f66afcfff1961ed9b13.camel@redhat.com> In-Reply-To: From: Eric Dumazet Date: Thu, 11 May 2023 09:50:30 +0200 Message-ID: Subject: Re: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a proper size To: "Zhang, Cathy" Cc: Shakeel Butt , Linux MM , Cgroups , Paolo Abeni , "davem@davemloft.net" , "kuba@kernel.org" , "Brandeburg, Jesse" , "Srinivas, Suresh" , "Chen, Tim C" , "You, Lizhen" , "eric.dumazet@gmail.com" , "netdev@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: EB65020005 X-Stat-Signature: j87dky5unoi6j8yy8e4dicr8kazgqc8y X-HE-Tag: 1683791444-124188 X-HE-Meta: U2FsdGVkX1+IkY/CI26pmlBi+up06WfZIjUC9LO4C8TJ1ZvnKV4aMAxbwfl6kHNOowH281DUagjqCLXegl34J32QcKuzCQMtaENBwiQBi0yQvBLAUK/zK8GDrrlK+65YgoAJwtsjGqyNaz6kQ55mhRs/A/rLyAEhJC51TR7LDlLNM5PzA7J3K3IPE7NAmN03lmieZFDR8PqOBfVBMXIwnMVBpWGzAZNVg1VvRDBbQpi/1Bv0TXzQM+Wnh9K//n80IPdSeNx4+zbYwQGkZDxHJGnAQqB0JMqw6e9mgQSCXkxuWdh240oaa52FDLO4XaarlxKD2abCM9IDD4ACTfEOGvibMvX9H3rq1qeAK7wxDUYExYtXtDPVdMMZ4oUe89Bf3ZxO2iLFEDoEY5OuAdYWH4l4k+iQlV/XxNM6zSdzr6gEQ3PHtqrseXnqd4xZCkXsdCcxPclR8DOAZIhMlKwL88Awdm9guxel5cfrWchqGyx/flSIKQy7cKDw5nYT+XED4bsWVBcBjt/7dNe0k3z9s5zUn6ZtmHPJSeuJfwvs50W4CmjlEn0yeW7AhzcR+otg39hel77SqKSx7LQYpio/YqPR3kObtx7iiO0T2kPv7Lqt3bxcb4gmPnK79+HTi9BSWANfpY0xWdmUNGVStChZ2HYMbPkH2vFLrpwLay7pvMGGCicjorFossIyRfgGARzkJtmRJtZfyup0sPoQtLgtRM86iSDL/Pr+jvrorGp6rkFpJx3166Vs2u/KqEK2TbPTYiMh24Eq5vlEqwPy0rdHu9gd1XERsWQMOfXxnX89gxMuc3kY9YAV3l3SJClCbLtJly+bM7UO0+pjIc/ciMpIIaJ+pu0CMCoqtJUoCmqqnQP3j6kgQq5cGGwY1MbTQRcfiDk80yaFEOtUA3rTkHFmTwyrGy/gSavL6g8IXtx2mT3RNCB+A6YBvUhJvms6nNFszRml/D8t4pVOjacEcGf 2Wv6xjr9 3dbJZBFLebVTAR12mZ75sUhsXzLGDTgkXvJK63xCU2VNqr1GcRTqxAfDyGK4922IdCTjuY6b7uUXLZso2NitVMgO4dTDTYyF8+nmu/EiVug4SZnzx+P0kStNCFC7e48ayspRYmUMTKuIkbQtqlDcwa53EFdzWwXs5W0/aF1lB4Vp9hfpOJ8yqZtalCQZiHFVxNOqE0X/iQdAlwe4KGRCBZp10R2OuSJbSyjoUaaKa/L0LqIxjCs+zYKuc4rDGq8zumcGP+rggzjKYqRubi4lYgeXLAKdnPnO3CsOuFin5kXXHF92aZt3zoqosvJqIXbiQWZxiO10aMe1FFa0Ar78+ILQUApwSJExgWKFEbJzv1LoGbo+xLfkSQeAAd8oC+ggaHin6xNQDrP8SFjlFUKx9F8S/40qUEFcvhDeg0SGI3gX67SyjoWK6z+hdCjPi4wTVqQhvyekd3yll1MkMQaRJEQu+BUmlLPQzUIImeaSFxCHhljVD2ZlmcBVSvCmOSk44luUbxp7pyW92Is6+GcCXerYn1KaU+51yknkVXH6fotoyID5uK+NBw/871TiMTALp82M9E2JbOKw8w/8lo5z2yDOb1Dm+YG3AGJyWfDFox9IyQaQNTo16R6UUj182QxOJXRwvcuK1KnoWWm0czuxt016pMw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, May 11, 2023 at 9:00=E2=80=AFAM Zhang, Cathy wrote: > > > > > -----Original Message----- > > From: Zhang, Cathy > > Sent: Thursday, May 11, 2023 8:53 AM > > To: Shakeel Butt > > Cc: Eric Dumazet ; Linux MM > mm@kvack.org>; Cgroups ; Paolo Abeni > > ; davem@davemloft.net; kuba@kernel.org; > > Brandeburg, Jesse ; Srinivas, Suresh > > ; Chen, Tim C ; You, > > Lizhen ; eric.dumazet@gmail.com; > > netdev@vger.kernel.org > > Subject: RE: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a p= roper > > size > > > > > > > > > -----Original Message----- > > > From: Shakeel Butt > > > Sent: Thursday, May 11, 2023 3:00 AM > > > To: Zhang, Cathy > > > Cc: Eric Dumazet ; Linux MM > > mm@kvack.org>; Cgroups ; Paolo Abeni > > > ; davem@davemloft.net; kuba@kernel.org; > > Brandeburg, > > > Jesse ; Srinivas, Suresh > > > ; Chen, Tim C ; You, > > > Lizhen ; eric.dumazet@gmail.com; > > > netdev@vger.kernel.org > > > Subject: Re: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a > > > proper size > > > > > > On Wed, May 10, 2023 at 9:09=E2=80=AFAM Zhang, Cathy > > > wrote: > > > > > > > > > > > [...] > > > > > > > > > > > > > > Have you tried to increase batch sizes ? > > > > > > > > > > > > I jus picked up 256 and 1024 for a try, but no help, the > > > > > > overhead still > > > exists. > > > > > > > > > > This makes no sense at all. > > > > > > > > Eric, > > > > > > > > I added a pr_info in try_charge_memcg() to print nr_pages if > > > > nr_pages > > > > >=3D MEMCG_CHARGE_BATCH, except it prints 64 during the initializat= ion > > > > of instances, there is no other output during the running. That > > > > means nr_pages is not over 64, I guess that might be the reason why > > > > to increase MEMCG_CHARGE_BATCH doesn't affect this case. > > > > > > > > > > I am assuming you increased MEMCG_CHARGE_BATCH to 256 and 1024 > > but > > > that did not help. To me that just means there is a different > > > bottleneck in the memcg charging codepath. Can you please share the > > > perf profile? Please note that memcg charging does a lot of other > > > things as well like updating memcg stats and checking (and enforcing) > > > memory.high even if you have not set memory.high. > > > > Thanks Shakeel! I will check more details on what you mentioned. We use > > "sudo perf top -p $(docker inspect -f '{{.State.Pid}}' memcached_2)" to > > monitor one of those instances, and also use "sudo perf top" to check t= he > > overhead from system wide. > > Here is the annotate output of perf top for the three memcg hot paths: > > Showing cycles for page_counter_try_charge > Events Pcnt (>=3D5%) > Percent | Source code & Disassembly of elf for cycles (543288 sampl= es, percent: local period) > -------------------------------------------------------------------------= -------------------------- > 0.00 : ffffffff8141388d: mov %r12,%rax > 76.82 : ffffffff81413890: lock xadd %rax,(%rbx) > 22.10 : ffffffff81413895: lea (%r12,%rax,1),%r15 > > > Showing cycles for page_counter_cancel > Events Pcnt (>=3D5%) > Percent | Source code & Disassembly of elf for cycles (1004744 samp= les, percent: local period) > -------------------------------------------------------------------------= --------------------------- > : 160 return i + xadd(&v->counter, i); > 77.42 : ffffffff81413759: lock xadd %rax,(%rdi) > 22.34 : ffffffff8141375e: sub %rsi,%rax > > > Showing cycles for try_charge_memcg > Events Pcnt (>=3D5%) > Percent | Source code & Disassembly of elf for cycles (256531 sampl= es, percent: local period) > -------------------------------------------------------------------------= -------------------------- > : 22 return __READ_ONCE((v)->counter); > 77.53 : ffffffff8141df86: mov 0x100(%r13),%rdx > : 2826 READ_ONCE(memcg->memory.high); > 19.45 : ffffffff8141df8d: mov 0x190(%r13),%rcx This is rephrasing the info you gave earlier ? 16.77% [kernel] [k] page_counter_try_charge 16.56% [kernel] [k] page_counter_cancel 15.65% [kernel] [k] try_charge_memcg What matters here is a call graph. perf record -a -g sleep 5 # While the test is running perf report --no-children --stdio What precise kernel are you using btw ?