From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5AE2C7EE32 for ; Thu, 2 Mar 2023 19:18:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 22D076B0073; Thu, 2 Mar 2023 14:18:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1DD556B0074; Thu, 2 Mar 2023 14:18:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 07E176B0075; Thu, 2 Mar 2023 14:18:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id EC8846B0073 for ; Thu, 2 Mar 2023 14:18:25 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BF383A61BA for ; Thu, 2 Mar 2023 19:18:25 +0000 (UTC) X-FDA: 80524919370.24.E12D386 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id D60752000E for ; Thu, 2 Mar 2023 19:18:23 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WT5Guzt2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677784703; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HutLHX9XRz9zx0C4eNoPP5mJD8j0TMgh1/XjqloYWko=; b=Kdh5UsHaT5UJUSh9UGqRHzFvkTVcegXlI7A1tOQlQ/J6lgfIN2jcZpJuZZ45n77oXPVreR gFNPaKymjFrfatwfCiJXRlUkyY7HR6DBc6cFUUWmY8/uvyXd621WMjkQmLpZuZ2fxBrS+O aC2K9+MPc5mZOAv07klPiD1R6/Y8GRs= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WT5Guzt2; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of mtosatti@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=mtosatti@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677784703; a=rsa-sha256; cv=none; b=RcMZiQJ2IzaDNnOb36EaqokHwDeLSqKRRCBphU6mStp2/VVqs1nlAy+YVZR+tsJJMkhn2f 2wv0zGBKylSn8Nl0ayWHPt32F8fuBRbw292Z/JbaFP+Onr3wWPvx09WOrJKkJ+lPtHAqWo CZdSZFQY1d0m6/JUq6H5A4iwbSHAKjE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677784703; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=HutLHX9XRz9zx0C4eNoPP5mJD8j0TMgh1/XjqloYWko=; b=WT5Guzt2vZ1pvdCxxp6+pYnNg5mx1aZ93Tyb4Fx+LQch8Deh+WFBcc7+sLkmofHked0H8m 6PD+50E4600T3XJ7UAYBYTvkC9lepIYDpW+b9YP/68T+/Dr9hKTSyeuBl2cHdCe9+Qfu4Q TdIfnnNMKNS3zC4qLMkhKAGxbZcE5wk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-193-Fq8LOJqgMX2lSdYMJAEh5Q-1; Thu, 02 Mar 2023 14:18:19 -0500 X-MC-Unique: Fq8LOJqgMX2lSdYMJAEh5Q-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3820B811E6E; Thu, 2 Mar 2023 19:18:19 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0194A492C18; Thu, 2 Mar 2023 19:18:19 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 57FC4403F7260; Thu, 2 Mar 2023 15:56:58 -0300 (-03) Date: Thu, 2 Mar 2023 15:56:58 -0300 From: Marcelo Tosatti To: Peter Xu Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 01/11] mm/vmstat: remove remote node draining Message-ID: References: <20230209150150.380060673@redhat.com> <20230209153204.656996515@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Rspamd-Queue-Id: D60752000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ce574k88tp8rks4naytmdknnk45sccxb X-HE-Tag: 1677784703-533500 X-HE-Meta: U2FsdGVkX1/EnrT4miv/690aTgK+65ZfLa+XieUU+100eRwDFDfKi1CEctqy8SPTpcv9V/Xbke5rBmCS8z+eBoc4WHRVDOTpwLPSHcyVep/RvQCcZNNQHSWl4ZvArmHERfs/1+AJLIADS2kxtX2BbhLlZRQWEor58tVco7Ntn8ci0ZcrsH47njA9DIhTVlEMbDqgY3+u1VMthcBRca0FQBGx5QPhVKpE5XzRSyH5UmYWJEG2AbKrb0JRh9f1cJ0zGc888LCvCYvvF9AowaM3YnqvVplCu3quEa1bsQzSm/PlwRDdJQoLvlxQYWlrpevZ2L7B1aDTLM7SKAnZ7sGv2XcmsviM/BmsqTyq77IsjqJ0Cht5uKhie3wSV9XhS+FUryxbJ47bXZpwZ4zvXlCHGysUfHQSGgYYDND22P2pCRcvryotc9MrnI4sU2rvkV7PWrgG0UXuh7/g5GTUjvT2C69z9KyHBDglcqzCF/Bus/nzqTVtZ0rjaGpffn8TaSsg/lEUGUOI5mQh/d54bGV0zpHKCQ9iDidDvDeVU/7LMHkR5TM5yMaO3X4SC8Rh/SdA8GRTjA4v9n5zK0ZBJcTX+FTehBCi6cASrs1aLUbzzaeQrWfH1BMIprafbwtIhCRopPiKDI8TVkN+cfFmwhHicVOyXf3xBiAlxFP4bczWchpDdI3SR1NEN7k9BjNnjS3MkNUOB0mz7B/soFmfbah8/w5KfwQY2DlcMyTLSFUaudlctresmKS0pJMvc8rrQyJCxKjzAQLSKAahFSCHdWr1xqP6j0hZsInc4wX+azEYn5XfowCQoczRjC45HCw8HGiWBLdQdcUkmgLYEwRY+iyHglltIOoGavLEaXDhfFQ4q7ZH7TkOAXKCpMCDXbgcdz1K0Ho6TyTFflkSNz8bjN7M9JKEOsIVvkn+enXWnp3Wpuj+OIuD6kCCasywIhYpincxclc6udBredQ7r+NStQQ GIrstdAZ l9wxUuiWg8pN5guzV+rHDRHvQl/9v7ipyQA4IBeWknyCQ0wCmNZnRf1MlWhtLpzrHwNymnO2+U0tPiO3pdHd60YarZz2Jbhw5/WsR5Fp/B1BeamkNDu36I2sonAYKjBUv4eS78SUc/6RVPGo8PETTfludLvE0kS6o+j7uXyyN9AQWQaK3fU9DndNfky+JFS3vLSPKn6Hrm4C69ooiIzJltat5cejZdPRh+/y0gFkfgB6hVis= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 02, 2023 at 12:21:15PM -0500, Peter Xu wrote: > On Thu, Feb 09, 2023 at 12:01:51PM -0300, Marcelo Tosatti wrote: > > Draining of pages from the local pcp for a remote zone was necessary > > since: > > > > "Note that remote node draining is a somewhat esoteric feature that is > > required on large NUMA systems because otherwise significant portions > > of system memory can become trapped in pcp queues. The number of pcp is > > determined by the number of processors and nodes in a system. A system > > with 4 processors and 2 nodes has 8 pcps which is okay. But a system > > with 1024 processors and 512 nodes has 512k pcps with a high potential > > for large amount of memory being caught in them." > > How about mentioning more details on where does this come from? > > afaict it's from commit 4037d45 since 2007. > > So I digged that out mostly because I want to know why we did flush pcp at > all during vmstat update. It already sounds weird to me but I could have > been missing important details. > > The rational I had here is refresh_cpu_vm_stats(true) is mostly being > called by the shepherd afaict, while: > > (1) The frequency of that interval is defined as sysctl_stat_interval, > which has nothing yet to do with pcp pages but only stat at least in > the name of it, and, > > (2) vmstat_work is only queued if need_update() here: > > for_each_online_cpu(cpu) { > struct delayed_work *dw = &per_cpu(vmstat_work, cpu); > > if (!delayed_work_pending(dw) && need_update(cpu)) > queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); > > cond_resched(); > } > > need_update() tells us "we should flush vmstats", nothing it tells > about "we should flush pcp list".. > > I looked into the 2007 commit, besides what Marcelo quoted, I do see > there's a major benefit of reusing cache lines, quotting from the commit: > > Move the node draining so that is is done when the vm statistics > are updated. At that point we are already touching all the > cachelines with the pagesets of a processor. > > However I didn't see why it's rational to flush pcp list when vmstat needs > flushing either. I also don't know whether that "cacheline locality" hold > true or not, because I saw that the pcp page list is split from vmstats > since 2021: > > commit 28f836b6777b6f42dce068a40d83a891deaaca37 > Author: Mel Gorman > Date: Mon Jun 28 19:41:38 2021 -0700 > > mm/page_alloc: split per cpu page lists and zone stats > > I'm not even sure my A-b or R-b worth anything at all here, "A-b or R-b" ? I think your points are valid. Also, the fact that sysctl_stat_interval is large (a second or more), means that any cache locality concern is would be limited to that time span. > just offer > something I got from git archaeology so maybe helpful to readers and > reasoning to this patch. > The correctness of archaeology needs help from > others (Christoph and Gel?).. I would just say if there's anything useful > or correct may worth collect some into the commit log. Agreed, i forgot to include the commit id in the changelog. > So from what I can tell this patch makes sense. Thanks!