From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8F95C77B7C for ; Thu, 11 May 2023 16:24:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BD256B0075; Thu, 11 May 2023 12:24:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26CE86B0078; Thu, 11 May 2023 12:24:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1358E6B007B; Thu, 11 May 2023 12:24:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F3ACA6B0075 for ; Thu, 11 May 2023 12:24:05 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8DC19160D95 for ; Thu, 11 May 2023 16:24:05 +0000 (UTC) X-FDA: 80778496050.21.AC494CF Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf13.hostedemail.com (Postfix) with ESMTP id 8ED5220011 for ; Thu, 11 May 2023 16:24:02 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="VTb18c/6"; spf=pass (imf13.hostedemail.com: domain of shakeelb@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1683822243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XB+NFYBcznFooDN9O0xTNgaLjM4RUCXNttuSZwxlS94=; b=KeOSSQANTc3mugwiVdaZ2S71ikFoiFje3eP3HBrgT3lXX/ubzcdX8V4y+8qVlblDSpbmYx pMQPSBSU6kCclfslXAtrtE04HR6o5aTCWlActqo0Y5X9bnI00vM9oV4qLoJJ8JjwGsPODJ sk0hUNbDtjEnyjNEFUgbViR4VilDLUQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1683822243; a=rsa-sha256; cv=none; b=S9de23FIj7Q3ABpJWkcsTwd+TnYYUzf9JemrKIomEd+1CvOZAFqAM68bog/fCkUJ6egb6V AdQESXK6+u4XiMv3wjIUwQZktRiZwnIJNJ/1EOE2U07/xyf7upNoyy/UMW51HkFIwbF81i ldSpWYbCkSTSwxnCGMg/2y6IRPT5Il4= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="VTb18c/6"; spf=pass (imf13.hostedemail.com: domain of shakeelb@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-3f38a9918d1so779521cf.1 for ; Thu, 11 May 2023 09:24:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683822242; x=1686414242; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XB+NFYBcznFooDN9O0xTNgaLjM4RUCXNttuSZwxlS94=; b=VTb18c/6+iJuVDVsNI1M/SqfD5jiDm60zxPGBzJOSf/UvuUnJiHFM3aNLflAMCOG/M gyRRuU++w5cV3LOE36hexMFH6NQXlRscSX4T0svjYDh2tzlqA5bMycbUOrtm2Nd5cwu5 HJ816jt33I57CiwU4WZuLqtdyXzOQ6BH9gG/rFt/u+dyK5krjkoimIrGDVbJgpAcTJ4a xVaeixXqkpl0y+0kVyGtwg2WU3jWlyq/Lq1NvMYGc2+Q0rQ7Ur2KmRIqBaYtypvEfsfy KarovuSrmCsLU5sP2lItzMMyn+xLLxkTq9uzoSz14yBsABOZvv+lg2Knvs3zz8BXoqRE jdcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683822242; x=1686414242; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XB+NFYBcznFooDN9O0xTNgaLjM4RUCXNttuSZwxlS94=; b=UvAdLHCfLMGkTqqrhxyGLrpC4Ga7B6hn9t/jBuGOMmWWAliKUOsl77qTmk28oJHuOH NL+65t4yKyfSigit3oyIWJJLJJw302ByFiasFef++rlULkL1+SDicRWK8UWqJOKRghaC OZuml24D9XZJ8+A00gTzyDOYv+KYvhHpu++29qVKTCr7PZrjz3PRhdBUc6rDwnn5UHKA z49ImqQcCl71B6dSlqVUEs8M17dzUbNpa8FN3S0r3nxXTKoo+RbRqIzT6JL9V0VtZhWT oE8/GurK8fQ2kkcpNAFv+GznszstxU8A/GefhDvx/RGADt9fuiT2WkFKuj/wKP84/fm4 J5Xw== X-Gm-Message-State: AC+VfDwIEWN+eQ+yTkczXC06kThkBz8rNPP8WXDj0JfD7FHyiBy60/El 2X2Qb/S0HENQBMnfQaIhvrBp/1fPmoGHLJmLqLpDng== X-Google-Smtp-Source: ACHHUZ4ypJsIVRgm/X1RPCQnNGSAat3RCJCSqFVFnzhyYvYyVTQlt776R5RQo0XDBgfSBASnJ5rFzkziL4sgClgoe+8= X-Received: by 2002:a05:622a:82:b0:3ef:404a:b291 with SMTP id o2-20020a05622a008200b003ef404ab291mr53343qtw.7.1683822242046; Thu, 11 May 2023 09:24:02 -0700 (PDT) MIME-Version: 1.0 References: <20230508020801.10702-1-cathy.zhang@intel.com> <20230508020801.10702-2-cathy.zhang@intel.com> <3887b08ac0e55e27a24d2f66afcfff1961ed9b13.camel@redhat.com> In-Reply-To: From: Shakeel Butt Date: Thu, 11 May 2023 09:23:50 -0700 Message-ID: Subject: Re: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a proper size To: "Zhang, Cathy" Cc: Eric Dumazet , Linux MM , Cgroups , Paolo Abeni , "davem@davemloft.net" , "kuba@kernel.org" , "Brandeburg, Jesse" , "Srinivas, Suresh" , "Chen, Tim C" , "You, Lizhen" , "eric.dumazet@gmail.com" , "netdev@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 8ED5220011 X-Stat-Signature: a1o6nehqgzimp8ag7uoqzox7t44jxdmt X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1683822242-406699 X-HE-Meta: U2FsdGVkX1/5VdTAgylGLu6KYssFT1rKoGAs3tyb04ea/QfyRrYVWCJJrShcUOWKOpcCnYRsXp1uZTubHPu1vtqpOS1j5jDoqHzrwqDK7fsP9wuZmW7SRkTJIuHc0xKeTYXkD4XzXBqMrbluCAHYfy2eojVUSXLsltdXRcF6vKJNT2NNcQxfGtoQW5FrHl5nWtR7Bp9/IXs7TaIdh/VZXk8PAZBWL+9CvrahwvkIu/te6pK7nbeaeLqS5sbHlXoHV1D6+ifAqt0iMELSsHCc24gKlqVv7C/bzsZfY8FdtHrkhcxHUoLYOrWVKni/4Ba4/0Ix/8sL1KgMnTZAjrGrRQEoXq6HfkdZzqFnRTfG0qi0TkKPFBQrWRqY/JYtZ0BvOGDf48a+gfT+1ZRA81U/BvNuh8z0blMrVwJqqy9Hez9ZPixhNvx2Uv7cdeG5WVF0XntOEGiCFzzpqYtx/Y6etmnWB+5uUzFdnOLWKCD/ZMiYqsvgKTuYokzHjQQ1zmyBHqruTGY6KTYQwLpa5TAhopAsLQfN5/ApeWIfDXewFZP86D2Tlsa+9f/rO1No9e1MgnqEeNJkr71FkZZo6nYYe3+s3CT7ogeSv6kPXZCrPOZmPQhF1Mv5FGTuboWlYGoN0CUzwz3nCz7fcfGZ4io/IRCU2SvRKSBeB9/jh+EtmT0AnMzonPOXjdbV0l+y5df41b2soyg3bSEjUPMisYEndXeuL9rKuE1OPQ+C6OdrjQMPn1XUUILKCAGP5n39WIMAZdHDSK6EN+CWJ1qUo0Wb6XlByYB4FPnJAYHXBUt50VLpJ8tbz4+HVCjWBH2XggrTyu18kwSlZSGLIcNzivolvKgOJ2SuyXDYQZYVZqoFR4ep1ghsxR1pnAWM+4llTg774W+0veQAm1Ml+SF2PlLwCo5GOz6sBrIqn95xB58L1v9y/2Gyhq+xZBvahGX54Nw13n+QM0O7ZQFLlwxGfaE YvpMg3Gf 9tTl+LzzbrtM2cu7MY2fx0RYL20WiwTONBOAcjO5fHxQ1LkFd+eAamhIS4H4K1OOYIMTTiVeAAwlWhONXj1mpD/qkiFim4QPLlEW6IlLoN3PS7BBJlWgkoU2o2MsY3iixjooBzp58Xd9W/Wq5IhSbXQm8tbef0fuua+6RiP1Au8cay4P25OOMNalsE7uyYZRugLZ1vbXUNN7ShuGf3MJ2RpscwBGWBd0eN2TT6TJ9sNzV725qr31wgINp1ka2Mxjhe2xOJXuvr4HS7kD2HyodyvE9Eudc4x3MNOInrMznHRujpHdBKssWNZEV70kXrk4uRXB99Q4cc298vfBTbO/6bTACY6HqhzIwN+68PpaWTf2eMXbpo8yf4YzKUq91LdkMVZg+n285g20SoE18hrH91yNl3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, May 11, 2023 at 2:27=E2=80=AFAM Zhang, Cathy wrote: > > > [...] > > Here is the output with the command you paste, it's from system wide, > I only show pieces of memcached records, and it seems to be a > callee -> caller stack trace: > > 9.02% mc-worker [kernel.vmlinux] [k] page_counter_t= ry_charge > | > --9.00%--page_counter_try_charge > | > --9.00%--try_charge_memcg > mem_cgroup_charge_skmem > | > --9.00%--__sk_mem_raise_allocated > __sk_mem_schedule > | > |--5.32%--tcp_try_rmem_sched= ule > | tcp_data_queue > | tcp_rcv_establish= ed > | tcp_v4_do_rcv > | tcp_v4_rcv > | ip_protocol_deliv= er_rcu > | ip_local_deliver_= finish > | ip_local_deliver > | ip_rcv > | __netif_receive_s= kb_one_core > | __netif_receive_s= kb > | process_backlog > | __napi_poll > | net_rx_action > | __do_softirq > | | > | --5.32%--do_soft= irq.part.0 > | __loca= l_bh_enable_ip > | __dev_= queue_xmit > | ip_fin= ish_output2 > | __ip_f= inish_output > | ip_fin= ish_output > | ip_out= put > | ip_loc= al_out > | __ip_q= ueue_xmit > | ip_que= ue_xmit > | __tcp_= transmit_skb > | tcp_wr= ite_xmit > | __tcp_= push_pending_frames > | tcp_pu= sh > | tcp_se= ndmsg_locked > | tcp_se= ndmsg > | inet_s= endmsg > | sock_s= endmsg > | ____sy= s_sendmsg > > 8.98% mc-worker [kernel.vmlinux] [k] page_counter_c= ancel > | > --8.97%--page_counter_cancel > | > --8.97%--page_counter_uncharge > drain_stock > __refill_stock > refill_stock > | > --8.91%--try_charge_memcg > mem_cgroup_charge_skmem > | > --8.91%--__sk_mem_raise_all= ocated > __sk_mem_schedule > | > |--5.41%--tcp_try= _rmem_schedule > | tcp_da= ta_queue > | tcp_rc= v_established > | tcp_v4= _do_rcv > | tcp_v4= _rcv > | ip_pro= tocol_deliver_rcu > | ip_loc= al_deliver_finish > | ip_loc= al_deliver > | ip_rcv > | __neti= f_receive_skb_one_core > | __neti= f_receive_skb > | proces= s_backlog > | __napi= _poll > | net_rx= _action > | __do_s= oftirq > | do_sof= tirq.part.0 > | __loca= l_bh_enable_ip > | __dev_= queue_xmit > | ip_fin= ish_output2 > | __ip_f= inish_output > | ip_fin= ish_output > | ip_out= put > | ip_loc= al_out > | __ip_q= ueue_xmit > | ip_que= ue_xmit > | __tcp_= transmit_skb > | tcp_wr= ite_xmit > | __tcp_= push_pending_frames > | tcp_pu= sh > | tcp_se= ndmsg_locked > | tcp_se= ndmsg > | inet_s= endmsg > > 8.78% mc-worker [kernel.vmlinux] [k] try_charge_mem= cg > | > --8.77%--try_charge_memcg > | > --8.76%--mem_cgroup_charge_skmem > | > --8.76%--__sk_mem_raise_allocated > __sk_mem_schedule > | > |--5.21%--tcp_try_rmem_sched= ule > | tcp_data_queue > | tcp_rcv_establish= ed > | tcp_v4_do_rcv > | | > | --5.21%--tcp_v4_= rcv > | ip_pro= tocol_deliver_rcu > | ip_loc= al_deliver_finish > | ip_loc= al_deliver > | ip_rcv > | __neti= f_receive_skb_one_core > | __neti= f_receive_skb > | proces= s_backlog > | __napi= _poll > | net_rx= _action > | __do_s= oftirq > | | > | --5.2= 1%--do_softirq.part.0 > | = __local_bh_enable_ip > | = __dev_queue_xmit > | = ip_finish_output2 > | = __ip_finish_output > | = ip_finish_output > | = ip_output > | = ip_local_out > | = __ip_queue_xmit > | = ip_queue_xmit > | = __tcp_transmit_skb > | = tcp_write_xmit > | = __tcp_push_pending_frames > | = tcp_push > | = tcp_sendmsg_locked > | = tcp_sendmsg > | = inet_sendmsg > | = sock_sendmsg > | = ____sys_sendmsg > | = ___sys_sendmsg > | = __sys_sendmsg > > > > I am suspecting we are doing a lot of charging for a specific memcg on one CPU (or a set of CPUs) and a lot of uncharging on the different CPU (or a different set of CPUs) and thus both of these code paths are hitting the slow path a lot. Eric, I remember we have an optimization in the networking stack that tries to free the memory on the same CPU where the allocation happened. Is that optimization enabled for this code path? Or maybe we should do something similar in memcg code (with the assumption that my suspicion is correct).