From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4AD2C6FA8E for ; Thu, 2 Mar 2023 17:21:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 543476B0074; Thu, 2 Mar 2023 12:21:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F33A6B0075; Thu, 2 Mar 2023 12:21:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 393D26B0078; Thu, 2 Mar 2023 12:21:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 246226B0074 for ; Thu, 2 Mar 2023 12:21:23 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EAE981C66FA for ; Thu, 2 Mar 2023 17:21:22 +0000 (UTC) X-FDA: 80524624404.15.3101E31 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf05.hostedemail.com (Postfix) with ESMTP id 3239D10001F for ; Thu, 2 Mar 2023 17:21:20 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U9Jo+sMN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677777680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Z19v7MXKb8AKOCqgrXYO60C645U6qWX+/Cu/AHRQzmQ=; b=eRhGFyRxR7j/ENHnAH2gQzfLrjGiIIhhfknSsGYlwQjbwMdy7xgN95amjgG0McUhfJniOX TAo2iW/XZLSoG2vgZUiM76p+2w3QGHppp8mqBpdEe2TaJ7wuz1KjuQVYEkv7zydG3g8+o/ K0p/jFZSdSsouQoa9g76puZcXhb5MFw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=U9Jo+sMN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677777680; a=rsa-sha256; cv=none; b=69HbvyMhhqDnPPK6e4m8YnDS+BSGe882FGntUtIuGxntm+PV14/OYuA6PnRVjEdFK0TdvX NzDMo72NdKfWIlU00f8KrLHT2tK6K/AxEUiCrkiPgDJfHk4zslvIZ1z9NQNWygHGRfxmPD mYEyBXXYTPveqihMKrbnlUW6OF72APs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677777679; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Z19v7MXKb8AKOCqgrXYO60C645U6qWX+/Cu/AHRQzmQ=; b=U9Jo+sMN4uU1SxdF8NZub4lNF+NL8X2GxMk0uyniaHScRA55NkD6fYwHb9bCRIdNCqX7DJ phsIkVFHdIclTHgO+wNP5cLe6buCbA44qj/GL+VP8v4W61rsa9AxLZOH7rZom07bNFttlf MgmlQn6si6P9fiYfR8ebBech651VHW4= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-425-dYjsuF9gNU2xpE5pFpyRQg-1; Thu, 02 Mar 2023 12:21:18 -0500 X-MC-Unique: dYjsuF9gNU2xpE5pFpyRQg-1 Received: by mail-qk1-f200.google.com with SMTP id x5-20020a05620a01e500b007428997e800so9934498qkn.10 for ; Thu, 02 Mar 2023 09:21:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677777678; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Z19v7MXKb8AKOCqgrXYO60C645U6qWX+/Cu/AHRQzmQ=; b=SQuSh327dKUII6TcFqKrLXoy0YhKJ8PecUDewSJP9pwY3no+HlbkdjsmXZ3O+tRAXr A2ia1cXIzJHaUmp6eNi3AhjhA21v98lmAahD/pIjgxiQ/FFbRQh0rv9RbAXqCsbrnxMg OrTcwZVnQC5RJgPe1THiLAs6E3EYZbh6GuU9BWZlVWkyJ2ndnZ2os2cSuoT3ZPzU7hm/ TVPCpjYpvCZkvSBPdf266ATnfur0r+j3FsN3ICtQL0+5WqERdvUKnCQFknLl6OEFCTf5 TK3vSbn/4R/cqDiJmHuFxs9vcCVS+ZY6BNelD13BK969E2p9Rtxh3pDCFlH3lkTXatu3 vIoA== X-Gm-Message-State: AO0yUKXJqlQ+qZZQtkjPP+X/iro8jB3sKMogiieWOXtpXbAyv8zaznZU zKiwm5wCujIxiPBbMDhwyNFRj8xN0GDC0Gw69+Ty2yTKtB/n025UsnWe2YfS20xU+McxtplYWgV h2X+DejAq60g= X-Received: by 2002:ac8:7e87:0:b0:3bf:f7d0:9ba9 with SMTP id w7-20020ac87e87000000b003bff7d09ba9mr11125215qtj.5.1677777678017; Thu, 02 Mar 2023 09:21:18 -0800 (PST) X-Google-Smtp-Source: AK7set9655JRdCvXNBMMODLkWTf3aygdgdxwgN+WSco8cUroxUBDyqAX41ub2H/ZnN7moPgkTzWHgQ== X-Received: by 2002:ac8:7e87:0:b0:3bf:f7d0:9ba9 with SMTP id w7-20020ac87e87000000b003bff7d09ba9mr11125185qtj.5.1677777677688; Thu, 02 Mar 2023 09:21:17 -0800 (PST) Received: from x1n (bras-base-aurron9127w-grc-56-70-30-145-63.dsl.bell.ca. [70.30.145.63]) by smtp.gmail.com with ESMTPSA id b13-20020ac8540d000000b003ba19e53e43sm63118qtq.25.2023.03.02.09.21.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Mar 2023 09:21:17 -0800 (PST) Date: Thu, 2 Mar 2023 12:21:15 -0500 From: Peter Xu To: Marcelo Tosatti Cc: Christoph Lameter , Aaron Tomlin , Frederic Weisbecker , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2 01/11] mm/vmstat: remove remote node draining Message-ID: References: <20230209150150.380060673@redhat.com> <20230209153204.656996515@redhat.com> MIME-Version: 1.0 In-Reply-To: <20230209153204.656996515@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3239D10001F X-Stat-Signature: 5bnf4n8jdp3jpg1pn3tomosiiddmfmm6 X-HE-Tag: 1677777680-540974 X-HE-Meta: U2FsdGVkX18qdeqqbt2EAGmgMFXBqEyxTwW1vnpVgrKb1tSBtVW74EeETuT7SEV02N+NHhw50hH5HFrmCwWKIS+p4hi+I73XOUmb+91qlQtFBmPGqg9SlZnu/dJKRIz83QaFZmE4VRaVZygJhuzsBJIZg/tLh7PpYx55p90GIERVpUgB5Ld/c/W9p0M/j82qgqX09R0piSuXC0/dGp0Mf0ReKs6yAUu/PDLFW/KPpLLLUTppe3bB50nUPD+YopE/0kjZr1mrtBjXa68JlXiGdVle8/auGkgTl3nYK2M4nuqtmqnhPLqCrSBdyZsq8F6YiNcNbybQCHfOxdJHf3X5l8IdQmEpGmPsMDcttK02qomVhxpDzovJnG/RVmkBBRq10qBWVecQqK14HXlOu36inwlN2FZqTt1DHI7hHt1tY4nT4oQ0PmUYiA5tS8+ClchGuwp7BQMaXZgbbjwnG79SO2li2ht2g4/oWNtDcjzpIL6QenHmY618Th9v1xmj6aP7D3mBWtvhbTzMxK0qFAfZoRbxPfwWBNN57cINt8US8jubhEbWA14od82cvUPRaeoe2f5NGTxrNNv3CzfjV4F02wd2qojPggDqsSwXg7BQkTOICBSEn/8Q7jAuEBlu1WgzIdxNfvGrl9z6LnkWbEyLnDZPlMEvgkhSjpN7A1vqKYzVScAAUdkFO6I4FPsikz0gBRAFD1HfIrvRd/qEbGVyZXNomBOa7XOfVpDDnjMR5cAzgOIOrsMPZrb9Semp4n0/5hyM2428z3QAUqYL0+dmpGTCDxpxXOtwTt49kDDAbqkHQAawM19wAp1zNAwcjiiW7IH9rRqwRhwFdLwWlFGa4zf1C55CP4UjAxTGEuvUmEiBh4xewl/JdPyaA/i0d4XUpAtJCarGhH7PudwZisOdnq1RhGspDQAHSBIdHrNrBJqfy2B+9JSNeDqdaCeI8+ZqpyBvy52T2ZfXvbKovK+ x0G7xRuT 6FPOv8vF/gUx3ntOxHJCv66lZPy+DtpdRdYj+Yj0arysODM6SNI4Xz3ZGFwzA409zh5HhzqvSXng96GD+yVpse0gXqoHumOxHNuKzXB2TzdR8U6ZnJYjDH5umpGs2JRXc2nGy+2HC9clfXZFGp2geiTp7jo2Tu2SKHfWZ4gzX6nDIvgU+/PVLxxq3mDW+WZ8mHe3JBxod4DDZ+tXCQc+Y9bCHAza6Y2RWCHtBYQEtPrjxjL9cQQFgqcyg1FgEaBE0jadPjpTgr0GsWg1crbc5uyMlx5J20xLinqcJoKSQJY0+9hh6l4iICVPJZAKnjCD7RVJagmT2ygQfa0xTy6nmcdBydMLV1Ly7qpvoYcRQT45S9qZKJENHSiF+clPSdGSo/WYYknG7KrCo2SZYFm67AgH2c2SyEA/FQAjs3S6pXuLdby8bSRk+/Mr7Gg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Feb 09, 2023 at 12:01:51PM -0300, Marcelo Tosatti wrote: > Draining of pages from the local pcp for a remote zone was necessary > since: > > "Note that remote node draining is a somewhat esoteric feature that is > required on large NUMA systems because otherwise significant portions > of system memory can become trapped in pcp queues. The number of pcp is > determined by the number of processors and nodes in a system. A system > with 4 processors and 2 nodes has 8 pcps which is okay. But a system > with 1024 processors and 512 nodes has 512k pcps with a high potential > for large amount of memory being caught in them." How about mentioning more details on where does this come from? afaict it's from commit 4037d45 since 2007. So I digged that out mostly because I want to know why we did flush pcp at all during vmstat update. It already sounds weird to me but I could have been missing important details. The rational I had here is refresh_cpu_vm_stats(true) is mostly being called by the shepherd afaict, while: (1) The frequency of that interval is defined as sysctl_stat_interval, which has nothing yet to do with pcp pages but only stat at least in the name of it, and, (2) vmstat_work is only queued if need_update() here: for_each_online_cpu(cpu) { struct delayed_work *dw = &per_cpu(vmstat_work, cpu); if (!delayed_work_pending(dw) && need_update(cpu)) queue_delayed_work_on(cpu, mm_percpu_wq, dw, 0); cond_resched(); } need_update() tells us "we should flush vmstats", nothing it tells about "we should flush pcp list".. I looked into the 2007 commit, besides what Marcelo quoted, I do see there's a major benefit of reusing cache lines, quotting from the commit: Move the node draining so that is is done when the vm statistics are updated. At that point we are already touching all the cachelines with the pagesets of a processor. However I didn't see why it's rational to flush pcp list when vmstat needs flushing either. I also don't know whether that "cacheline locality" hold true or not, because I saw that the pcp page list is split from vmstats since 2021: commit 28f836b6777b6f42dce068a40d83a891deaaca37 Author: Mel Gorman Date: Mon Jun 28 19:41:38 2021 -0700 mm/page_alloc: split per cpu page lists and zone stats I'm not even sure my A-b or R-b worth anything at all here, just offer something I got from git archaeology so maybe helpful to readers and reasoning to this patch. The correctness of archaeology needs help from others (Christoph and Gel?).. I would just say if there's anything useful or correct may worth collect some into the commit log. So from what I can tell this patch makes sense. -- Peter Xu