From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D711EC3DA41 for ; Tue, 9 Jul 2024 04:57:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F6BE6B009B; Tue, 9 Jul 2024 00:57:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A6866B009D; Tue, 9 Jul 2024 00:57:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 194D86B009E; Tue, 9 Jul 2024 00:57:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F0A1B6B009B for ; Tue, 9 Jul 2024 00:57:53 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 93CC8C164D for ; Tue, 9 Jul 2024 04:57:53 +0000 (UTC) X-FDA: 82319006826.15.4A0DC32 Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by imf11.hostedemail.com (Postfix) with ESMTP id C045040018 for ; Tue, 9 Jul 2024 04:57:51 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=linux.microsoft.com header.s=default header.b=PK4rekXG; dmarc=pass (policy=none) header.from=linux.microsoft.com; spf=pass (imf11.hostedemail.com: domain of ssengar@linux.microsoft.com designates 13.77.154.182 as permitted sender) smtp.mailfrom=ssengar@linux.microsoft.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720501049; a=rsa-sha256; cv=none; b=HfZrpHYxr/YCUQUle+cHswkgUJq+ctAd/uFNN24vtFel9zlSqYPtdDLObkuuPnU1pS9YV+ sA5tK+0Ugcx45fG5hDurwsKxiyYMWl9U8hTRWLqQITnQxIFytJL0Bp5BHD5UyTS1WixFqX lLdTzcWsdTDSLEdjiNs4gNWj0Il+NJM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=linux.microsoft.com header.s=default header.b=PK4rekXG; dmarc=pass (policy=none) header.from=linux.microsoft.com; spf=pass (imf11.hostedemail.com: domain of ssengar@linux.microsoft.com designates 13.77.154.182 as permitted sender) smtp.mailfrom=ssengar@linux.microsoft.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720501049; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DYB+5c7dR2RtFbylwSCFyX/WQFMuYrmuD6DA/31T/pc=; b=25cHnK8rWhDSomBDS3NzeRrmmfLsgMhmnkOug/f/utwuicCTyUzvNES/w48+NA5dkqtF8o IPquLhAeW7/NOEpbJNz2nwXLWdkdWF6K4B2kOWDpnOtM6cdvHIT+ATs/Dkun6Ep8PkPnRi qaoA6aoOUWLrq6GaPWxyX2ACsZXTMnw= Received: by linux.microsoft.com (Postfix, from userid 1127) id 8972E20B7165; Mon, 8 Jul 2024 21:57:50 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 8972E20B7165 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.microsoft.com; s=default; t=1720501070; bh=DYB+5c7dR2RtFbylwSCFyX/WQFMuYrmuD6DA/31T/pc=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=PK4rekXG71/2Ylo70/z08bazi5PzlA9lHMnlpNc4/H/IKRWG9Qq+luTUxkPAQnzZe 4HdKCGEuZv45fEOFJE/x0+x4nalq0BA6MMnNcM49l/ZvIGBEk+SGgQGSVhDPxVgCKY IWP7DsWymT3PeGptE3K3B/7kaDMOeRa/EGMzqGXE= Date: Mon, 8 Jul 2024 21:57:50 -0700 From: Saurabh Singh Sengar To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, ssengar@microsoft.com, wei.liu@kernel.org Subject: Re: [PATCH] mm/vmstat: Defer the refresh_zone_stat_thresholds after all CPUs bringup Message-ID: <20240709045750.GA32083@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net> References: <1720169301-21002-1-git-send-email-ssengar@linux.microsoft.com> <20240705135911.4a6e38379ae95c3fc6bbe7e2@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240705135911.4a6e38379ae95c3fc6bbe7e2@linux-foundation.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Rspamd-Queue-Id: C045040018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1pt9fs9ch93dfuxc8xbwsootqrrsbwrb X-HE-Tag: 1720501071-408185 X-HE-Meta: U2FsdGVkX1+QDx4o7Rqxa4/FcDmuZN7hRUoYOQ8Daj4oX0x78nB4LZLJ3jBiSlC9vWBG1psX1qjpkjaWJEcN+YDrqVCFN6ArORDnOsr1xcXTr245ED3BfmzxRFBpoJe+S2aeqJL0LRW+hYP7ynkmZ3E5TkG+LA4bcOYa1h62TYypv1zf+gy8qmGGXp9Tb3HbUXSk07PPeZqWIjlujCYtJs8gG6JZERkpEqrSUnKEqNmNT/VzndeiANISQOx6kGfid7LhIfyX+i+MA6zYJxh4ATpxQ0yOWlVERL8gFqZwgZ47lkba5W/seHkTyDqerMwTAVYy1J26tTmT+K1ZrEWvAlRFEMPWE4YVOG7O/P+bQbSHP8VwrO7jmuarKgOQwjJ1PgYGodufaZ7nDTLhi3WdtihXErs6fIewlnFW0wNVRTE2Diix5m6b25JSYwjNhs9lBLja898/4eBxK3pE5JSwWbqCZWFdMaB3rpNLrpkNfHzCu3zsXFlN+N+cVDFonGihYT0z1GTiG96RhGYUlp8I0zNND5ij0pxpJO1tTtXup0XZQgJ8QkCwBytzDfyIy9cXL6eSbKTnxyz/jBZmcDRwwzpnqYDVr81Qiw/XHDVHwLaYsjbDDePN2kylll6kYM3SD6au6MLuO50+lrU4W/V8c6dQ0Q1fll5pwnetKCUB6Jf1KjUba/6XBFXIU8UDar3hGpcT8sxxyw7/qOYUIheJxQviwPVBK1K4KR8ZZAPdoE9lI4KNDK9xavwK9a8xAVPHGWHzFLPZBT0smsAZP4hsgwq2jG9v8ymfOxtD3esXtzJBpM6Lk0FVQePEaxTBM0qhczBXWAcjmznPHKO05unyDrDdyVETmB0/UtSKKA742nCIS9c1QgsbfwkbxhBI7vZ/CpdluWRqoUcZnfxR/GTQ2vtIcLACYvCa/ilalsbjiC2mdW97enShIbuaVAk8b05DS3QIBqChTMVSijiwoyY c6gpbwJL Qq2wT82oOd2fPDBteQ0cnTGTRHBkW5FqyzwMblsfrB+JsawhzgaCuJYeciAofGgUPMRWaTOPBoMvEk91FQRcs03AT4jOB4U6CiHCxXqFa/9+watf2Nyv5UcVrU6J0Hfm+GwczLWxZ2iChOcA+MGQmd7Dgohkdm+CLuVPxHQr2/PnmcSwPTF1nRF0db9pv76lQ7L0OudRo4T022IHqZxtw/TBEKHrTz0JWNddpel9U1CZypUvcBtHZ19drHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jul 05, 2024 at 01:59:11PM -0700, Andrew Morton wrote: > On Fri, 5 Jul 2024 01:48:21 -0700 Saurabh Sengar wrote: > > > refresh_zone_stat_thresholds function has two loops which is expensive for > > higher number of CPUs and NUMA nodes. > > > > Below is the rough estimation of total iterations done by these loops > > based on number of NUMA and CPUs. > > > > Total number of iterations: nCPU * 2 * Numa * mCPU > > Where: > > nCPU = total number of CPUs > > Numa = total number of NUMA nodes > > mCPU = mean value of total CPUs (e.g., 512 for 1024 total CPUs) > > > > For the system under test with 16 NUMA nodes and 1024 CPUs, this > > results in a substantial increase in the number of loop iterations > > during boot-up when NUMA is enabled: > > > > No NUMA = 1024*2*1*512 = 1,048,576 : Here refresh_zone_stat_thresholds > > takes around 224 ms total for all the CPUs in the system under test. > > 16 NUMA = 1024*2*16*512 = 16,777,216 : Here refresh_zone_stat_thresholds > > takes around 4.5 seconds total for all the CPUs in the system under test. > > Did you measure the overall before-and-after times? IOW, how much of > that 4.5s do we reclaim? This entire gain is accounted in over all boot processi time. Most of the Linux kernel boot process is sequential and doesn't take advantage of SMP. > > > Calling this for each CPU is expensive when there are large number > > of CPUs along with multiple NUMAs. Fix this by deferring > > refresh_zone_stat_thresholds to be called later at once when all the > > secondary CPUs are up. Also, register the DYN hooks to keep the > > existing hotplug functionality intact. > > > > Seems risky - we'll now have online CPUs which have unintialized data, > yes? What assurance do we have that this data won't be accessed? I understand that this data is only accessed by userspace tools, and they can only access it post late_initcall. Please let me know if there are any other cases, I will look to address them. > > Another approach might be to make the code a bit smarter - instead of > calculating thresholds for the whole world, we make incremental changes > to the existing thresholds on behalf of the new resource which just > became available? I agree, and I have spent good amount of time undertanding the calculation, but couldn't find any obvious way to code everything it does in incremental way. I would be happy to assist if you have any suggestions how to do this. - Saurabh