From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F272C71153 for ; Mon, 11 Sep 2023 20:01:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B76A56B02F1; Mon, 11 Sep 2023 16:01:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B261E6B02F2; Mon, 11 Sep 2023 16:01:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C75A6B02F3; Mon, 11 Sep 2023 16:01:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8B3006B02F1 for ; Mon, 11 Sep 2023 16:01:41 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3EAD21CA669 for ; Mon, 11 Sep 2023 20:01:41 +0000 (UTC) X-FDA: 81225386802.03.C90B2B2 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf21.hostedemail.com (Postfix) with ESMTP id 461C91C0028 for ; Mon, 11 Sep 2023 20:01:39 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p48ZpY5j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of weixugc@google.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=weixugc@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694462499; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rc/ITscuXzrOZNzjOX81ClkWsxPRiA/63di+UZB/Fss=; b=U6twHNf5RrZ4VIxDG6nX5GPAPKIjilIDEUVP3szMrFTWLXJ9kMisxHkrF7tUL0mjJWj3kw 8C5P8MqMJBEHfC1GNLaTLA4wTFCKmtHe08bqgOaJD4lwfvj/JqKfMPd+ufdYXJ6XqCUr0j tG1nzU6Oc8B/X7ZAlOYcj0g4/ey48Kc= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=p48ZpY5j; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of weixugc@google.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=weixugc@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694462499; a=rsa-sha256; cv=none; b=fw2Ir5pQgYa+0yun/JEf5xrole7mF+hPwE/yckjO+ZziwymyI7dxbv5hcASR1f+3u5AQh3 stQoQR7UA5vnvgg81ZQ2Msx3u+rv8gMaInE20ihzMboym75DoC0AwE9oHWtB2Xy4x3rHH2 T7371Kevdy5Qz0orr/3LWnOkgkCzgXc= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-31fa15f4cc6so1616293f8f.2 for ; Mon, 11 Sep 2023 13:01:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694462497; x=1695067297; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Rc/ITscuXzrOZNzjOX81ClkWsxPRiA/63di+UZB/Fss=; b=p48ZpY5jZaRDK7fHU3+gmmRwDfT5NCNoo+L8ZRrPQdV40BTEOunDkCGS93elTviT30 jGVTwQW+e6jiBTv0Mr4mwxT5arcsH3i3I/leK56gwK9fCXrHKgoRgJehZQyFmFHB8tae Wll5x5DL0hyN1r4uJPNgu8kc6IAtS+OxHDpsTj3DcBOTnz/lzbY0LHlzs5+D5T7xtllt dcAc7Fkuma5K1qVTrxZ20Ac4Dfpcet2ciibTbLBazrXJ/B02k/tMaVAgv4PL99OtNzO+ FRQSZtFALzs+aKnBpL2SE+/eZPDLtUW/i73D6D/AOLRq8Hgek935OvkdllQvzYkIRa+z YWaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694462497; x=1695067297; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Rc/ITscuXzrOZNzjOX81ClkWsxPRiA/63di+UZB/Fss=; b=mnuPDVnBFt/tYwi21wpD4zNhlxR6tYMNf1sdpd+lEeO1ksQxtAn2KS2etCwjijBhGn Xc9rh0e5bwSVSMmVKTejUNgGG9dEbdL1Gh201OnpZtp8frFX/uyFN0w/45Jdey1XYVJb rMEkiIVI2Q/VDLSazjrrDoJjSlZwzOG7BJQ3PrujNNqdJliYNs/r3kcK+3KumuIk9c5i 576CP2QSrwvGENFeuXPyaSfwbFxID9K16DdLxJ8X1xTwMM5Fwa3zl/yLayRt2+aP4WC/ 7ShEwXFjwr1j/Ixrvlvrzblnpob3Yzx5zD0C84Uo4rhRXTIFP5eNCLIBy6xrKKmQY4xA 8njw== X-Gm-Message-State: AOJu0YwZYvY7MwDrQyT5sFTN9QbBWsvJ7tnloRQMNzn9PBVGD05NiSL2 RU4DSAj4FftlwjkmrXux9SgVZqAxZjxJh/POFiDAkg== X-Google-Smtp-Source: AGHT+IFl+KdEdEZPM6dgoOAIbUh01xAe49DOxDPyclO9MjoDMPy6oI+PGBUbBzXjIZIb0Pzdpfqm23roWvQdVndSEEw= X-Received: by 2002:a5d:4f8c:0:b0:314:3e77:f210 with SMTP id d12-20020a5d4f8c000000b003143e77f210mr8531185wru.59.1694462497487; Mon, 11 Sep 2023 13:01:37 -0700 (PDT) MIME-Version: 1.0 References: <20230831165611.2610118-1-yosryahmed@google.com> <20230831165611.2610118-5-yosryahmed@google.com> In-Reply-To: From: Wei Xu Date: Mon, 11 Sep 2023 13:01:25 -0700 Message-ID: Subject: Re: [PATCH v4 4/4] mm: memcg: use non-unified stats flushing for userspace reads To: Michal Hocko Cc: Yosry Ahmed , Andrew Morton , Johannes Weiner , Roman Gushchin , Shakeel Butt , Muchun Song , Ivan Babrou , Tejun Heo , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Waiman Long , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Greg Thelen Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 461C91C0028 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: s4gwscwjjp3nwcmdo7o6t4bfnbiikhr8 X-HE-Tag: 1694462499-774194 X-HE-Meta: U2FsdGVkX18Cm49QEsk9xzsEjjU925xXWknWsjHcTtZOv+BlwZkCKRrN38eu6mhuGNOryoiYqxtM0UGEa9pOEBNS0uu7NDzA+DO5ReubSuC05aBngtsU2IOEKYqmtvW+4zdGmVBrikJR3P7dEOHBBAy7RpZFQAijBRehbDKz14oiD8xJlFJJJqtZc03LdOGVX+SlrPRRPHUVZNrF4zc67onJw0hHqUJOy1exxmQDHYO9du9kEX1Q0ZWZfuu79Mj9Ycu0iNu1yt7hfGlpxrqC8lHOQkcMidTtEgaVzs8eEZsKCREubobMCcdRn3icImxKD0mhJKugvfXgJu+DH8uvanbbLtXRgZ+7dVcgMgiqjh9ngWnL5A43WNKBWw184A3Zh0pXkK7BgHs3r8RDq30aME0TpnuvuvPjDlRiCZEo2pL4NL6TWZf7iv3Pnn+WKs/fcaPNHL8NdxdzcTtzlPfa7CHsnufWViWXvVrzhTPJSbYPoGvieClPvSUcfb+FYJ1xuSTHAkyB8ZPvHpCGNPh3W4IQoXjYPz2DdtPHj/2z1McLhPuXE0ObMtY/iGOeACS9CcmsQ2HjE6j8+HLrJ+K4+kQf0dcxrecUKB3BkT4vgG8sCNUs5ozkg7PAu3vzlArPtV3hkwWUasXRQwmjx+lMZDdF8oWt3Q1aPaoE0ddaePAN8jy1hh94oEu1V2mlMa7ymen9jhQOHPojtjICppn6MWLVfNGmvL7uZVHJBd+n+nO9Ldwh2Y0rJGFjkued7nqFHejl5p+fsltK01CY4JeyctOUGQTlHYZ9gjhcyXHP42tLMPkwUSu8OQ+y2ukxguh0U6jFxAwBg+uFVdzqGOsvzvyt+oxUm1xJlVCuu8RRB91JVRV4EKYcIfj/SHZ1S8XZ6PvcAUjbTez17sCWSBjgVBPJBcGpVtveboy1iaoWhFl4EuUnflJvQnBVj1v61gpGAJsd5g5vfDjSR1+1F71 qu9qcDkx iz/eWk9x7aA3qeo0Q5cxdmTgrNw5Xc6nr2sTut74hvH1QUT/o4tY9WH9byJZCURZQm1qRuvSlJRFNqN9Hxj46JATpHEc1XGgqgvi+3idOJhXhfF2lb6eXsVuBpv0ncmIVlPGAfKBCliKXDOCrPdQ6rwRwGeP+Eobo739Jqb/L7+WSUnfyZiy6u8OO1aIPIOHSCdREgMcqjFJSNJ6F9L5b5313nmDgQCB+7fS7L4pJRgJJVz/f0vKXXKnhPeFFovdaiFgnyYG8kzzPubKs5zcA3AIWsgpaU0lA8+Nfo4JE7f6bcg9GxMkeD780TS0aB4DUItYx X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Sep 11, 2023 at 12:34=E2=80=AFPM Michal Hocko wro= te: > > On Mon 11-09-23 12:15:24, Wei Xu wrote: > > On Mon, Sep 11, 2023 at 6:11=E2=80=AFAM Michal Hocko = wrote: > > > > > > On Thu 07-09-23 17:52:12, Wei Xu wrote: > > > [...] > > > > I tested this patch on a machine with 384 CPUs using a microbenchma= rk > > > > that spawns 10K threads, each reading its memory.stat every 100 > > > > milliseconds. > > > > > > This is rather extreme case but I wouldn't call it utterly insane > > > though. > > > > > > > Most of memory.stat reads take 5ms-10ms in kernel, with > > > > ~5% reads even exceeding 1 second. > > > > > > Just curious, what would numbers look like if the mutex is removed an= d > > > those threads would be condending on the existing spinlock with lock > > > dropping in place and removed. Would you be willing to give it a shot= ? > > > > Without the mutex and with the spinlock only, the common read latency > > of memory.stat is still 5ms-10ms in kernel. There are very few reads > > (<0.003%) going above 10ms and none more than 1 second. > > Is this with the existing spinlock dropping and same 10k potentially > contending readers? Yes, it is the same test (10K contending readers). The kernel change is to remove stats_user_flush_mutex from mem_cgroup_user_flush_stats() so that the concurrent mem_cgroup_user_flush_stats() requests directly contend on cgroup_rstat_lock in cgroup_rstat_flush(). > -- > Michal Hocko > SUSE Labs