From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA678C43334 for ; Fri, 8 Jul 2022 08:23:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A9856B0075; Fri, 8 Jul 2022 04:23:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4320B8E0001; Fri, 8 Jul 2022 04:23:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D3346B007B; Fri, 8 Jul 2022 04:23:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1F45F6B0075 for ; Fri, 8 Jul 2022 04:23:28 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id DEB8C80FD6 for ; Fri, 8 Jul 2022 08:23:27 +0000 (UTC) X-FDA: 79663243254.11.C244B99 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf31.hostedemail.com (Postfix) with ESMTP id 6295F20045 for ; Fri, 8 Jul 2022 08:23:27 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id j3so10305525pfb.6 for ; Fri, 08 Jul 2022 01:23:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XuDZzz9tVnB8J2jQEjeJNVf5Aik6JfaRXz4bVLQK2Uk=; b=DIZH2h+N1eb2jQqowZGUpKwgUfWoCe3azh5guhmj8MDrC3dMQ3QvY4azmwaqEt1asq GHX6szMJe7+TQRsqaW0h1F1xH5/Vt3ZOxzJNknCLtw7qSFJ2qUP6Mqz83hJTSlP1HbOB 8tVQ64fDYFAZS8zGV8EpvERQDadFWuc3exmymJKUpAWvy9dUz/k/y7FjvJ/Mg02eoB93 SmehkHPQYRql9dwypeIAeoVFNVdwvI6Pwx+SqkLHswEmZT22De/evOfaz6cljvVTRzPS K4WLNbqWZ17kl86EY9XjxMWk7dLlqfuPkSJY6giWhrh9Cj3qaJBqcMNYhbFYVrTfrhK1 pJ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XuDZzz9tVnB8J2jQEjeJNVf5Aik6JfaRXz4bVLQK2Uk=; b=Tx3oXnnSZaSyVRnm+uo2p8CUqzfwUZxk5hPzT7QRnVPZNS6FwAxjRc/WgyOdamcxBf /0JSVNX412q6PopTYve099qcIH0wkAJrqr6C18mCQeVULrB/EmxBP7veMki+2Rh873Lx M0ZW5gq/gMik3Edqz1poIUfTHPH3j/Mb0+b9jVoOU8u8+vw9WKwtAGfoVYEPP3NyX2uv CW8kpCWwoOdt9SvbGpQONMwlazK69RAxYfMAsVwb7hxcMD/pE6aQY/T3lJuxyP+4JhJn r9kQa5BC4yUIm7QUfy7vOiPR3DMgKlyfL9wBYBQyktq5txic1x7wADhMkhTa6E4fYj/a rf9g== X-Gm-Message-State: AJIora+S4XegztikJalpu1al8LoiiiIJeUEkB5wlpbKM10opJnKCH0WE qG/Sz9t921jFUbD5XZMbc+F0Qw== X-Google-Smtp-Source: AGRyM1uDYINZFh863sauNGlXqipvFwUiPCbVj1rtxjbEOySlvPOW7DbzRdiKgLA6/RsZxnbuDo5Xmw== X-Received: by 2002:a65:6e4d:0:b0:411:c102:397e with SMTP id be13-20020a656e4d000000b00411c102397emr2297823pgb.271.1657268606508; Fri, 08 Jul 2022 01:23:26 -0700 (PDT) Received: from C02FT5A6MD6R.bytedance.net ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id x65-20020a636344000000b00412b1043f33sm3329291pgb.39.2022.07.08.01.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Jul 2022 01:23:25 -0700 (PDT) From: Gang Li To: mhocko@suse.com, akpm@linux-foundation.org, surenb@google.com Cc: hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, viro@zeniv.linux.org.uk, ebiederm@xmission.com, keescook@chromium.org, rostedt@goodmis.org, mingo@redhat.com, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, david@redhat.com, imbrenda@linux.ibm.com, adobriyan@gmail.com, yang.yang29@zte.com.cn, brauner@kernel.org, stephen.s.brennan@oracle.com, zhengqi.arch@bytedance.com, haolee.swjtu@gmail.com, xu.xin16@zte.com.cn, Liam.Howlett@Oracle.com, ohoono.kwon@samsung.com, peterx@redhat.com, arnd@arndb.de, shy828301@gmail.com, alex.sierra@amd.com, xianting.tian@linux.alibaba.com, willy@infradead.org, ccross@google.com, vbabka@suse.cz, sujiaxun@uniontech.com, sfr@canb.auug.org.au, vasily.averin@linux.dev, mgorman@suse.de, vvghjk1234@gmail.com, tglx@linutronix.de, luto@kernel.org, bigeasy@linutronix.de, fenghua.yu@intel.com, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-perf-users@vger.kernel.org, Gang Li Subject: [PATCH v2 4/5] mm: enable per numa node rss_stat count Date: Fri, 8 Jul 2022 16:21:28 +0800 Message-Id: <20220708082129.80115-5-ligang.bdlg@bytedance.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220708082129.80115-1-ligang.bdlg@bytedance.com> References: <20220708082129.80115-1-ligang.bdlg@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657268607; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XuDZzz9tVnB8J2jQEjeJNVf5Aik6JfaRXz4bVLQK2Uk=; b=0NKXTckAwMFuve6D/TgPqeXQYShBK17HFB1OBRpBZYpLCXQwPBDegoF7wX90wIfkUZ+7Sh 1eVIkJ3zD/A7SrKIYo2lxNKsiU9c/wk6rz9JN1Pue4ztReAcV7JnrN5S8XAiLsNU5q1JVH 83C4FguDx79a5Nb8xhwttieUzdpH1NU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657268607; a=rsa-sha256; cv=none; b=Q+atAswTXYROe97QKjT/e9lkvLQPDIQEJVuzXkS6hYfegYOm1bHJ4ywW8McKpi1pFzigP5 IpzBRynyQzZQYitvoxgIy/XD42elwsahaXEVh80upFis+vZ2n2nW8U6Nu80qfu6iUNW7Rx +bftwX+0vbS0/NCAliI/f+t46PCnVoY= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DIZH2h+N; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf31.hostedemail.com: domain of ligang.bdlg@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=ligang.bdlg@bytedance.com Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DIZH2h+N; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf31.hostedemail.com: domain of ligang.bdlg@bytedance.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=ligang.bdlg@bytedance.com X-Stat-Signature: jfk3unit4byn1yur774qmp7f5ewiqk86 X-Rspamd-Queue-Id: 6295F20045 X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1657268607-992003 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now we have all the infrastructure ready. Modify `*_mm_counter`, `sync_mm_rss`, `add_mm_counter_fast` and `add_mm_rss_vec` to enable per numa node rss_stat count. Signed-off-by: Gang Li --- include/linux/mm.h | 42 +++++++++++++++++++++++++++++++++++------- mm/memory.c | 20 ++++++++++++++++++-- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a7150ee7439c..4a8e10ebc729 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2028,8 +2028,18 @@ static inline bool get_user_page_fast_only(unsigned long addr, */ static inline unsigned long get_mm_counter(struct mm_struct *mm, int member, int node) { - long val = atomic_long_read(&mm->rss_stat.count[member]); + long val; + WARN_ON(node == NUMA_NO_NODE && member == MM_NO_TYPE); + + if (node == NUMA_NO_NODE) + val = atomic_long_read(&mm->rss_stat.count[member]); + else +#ifdef CONFIG_NUMA + val = atomic_long_read(&mm->rss_stat.numa_count[node]); +#else + val = 0; +#endif #ifdef SPLIT_RSS_COUNTING /* * counter is updated in asynchronous manner and may go to minus. @@ -2046,23 +2056,41 @@ void mm_trace_rss_stat(struct mm_struct *mm, int member, long member_count, int static inline void add_mm_counter(struct mm_struct *mm, int member, long value, int node) { - long count = atomic_long_add_return(value, &mm->rss_stat.count[member]); + long member_count = 0, numa_count = 0; - mm_trace_rss_stat(mm, member, count, NUMA_NO_NODE, 0, value); + if (member != MM_NO_TYPE) + member_count = atomic_long_add_return(value, &mm->rss_stat.count[member]); +#ifdef CONFIG_NUMA + if (node != NUMA_NO_NODE) + numa_count = atomic_long_add_return(value, &mm->rss_stat.numa_count[node]); +#endif + mm_trace_rss_stat(mm, member, member_count, node, numa_count, value); } static inline void inc_mm_counter(struct mm_struct *mm, int member, int node) { - long count = atomic_long_inc_return(&mm->rss_stat.count[member]); + long member_count = 0, numa_count = 0; - mm_trace_rss_stat(mm, member, count, NUMA_NO_NODE, 0, 1); + if (member != MM_NO_TYPE) + member_count = atomic_long_inc_return(&mm->rss_stat.count[member]); +#ifdef CONFIG_NUMA + if (node != NUMA_NO_NODE) + numa_count = atomic_long_inc_return(&mm->rss_stat.numa_count[node]); +#endif + mm_trace_rss_stat(mm, member, member_count, node, numa_count, 1); } static inline void dec_mm_counter(struct mm_struct *mm, int member, int node) { - long count = atomic_long_dec_return(&mm->rss_stat.count[member]); + long member_count = 0, numa_count = 0; - mm_trace_rss_stat(mm, member, count, NUMA_NO_NODE, 0, -1); + if (member != MM_NO_TYPE) + member_count = atomic_long_dec_return(&mm->rss_stat.count[member]); +#ifdef CONFIG_NUMA + if (node != NUMA_NO_NODE) + numa_count = atomic_long_dec_return(&mm->rss_stat.numa_count[node]); +#endif + mm_trace_rss_stat(mm, member, member_count, node, numa_count, -1); } /* Optimized variant when page is already known not to be PageAnon */ diff --git a/mm/memory.c b/mm/memory.c index b085f368ae11..66c8d10d36cc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -191,6 +191,14 @@ void sync_mm_rss(struct mm_struct *mm) current->rss_stat.count[i] = 0; } } +#ifdef CONFIG_NUMA + for_each_node(i) { + if (current->rss_stat.numa_count[i]) { + add_mm_counter(mm, MM_NO_TYPE, current->rss_stat.numa_count[i], i); + current->rss_stat.numa_count[i] = 0; + } + } +#endif current->rss_stat.events = 0; } @@ -198,9 +206,12 @@ static void add_mm_counter_fast(struct mm_struct *mm, int member, int val, int n { struct task_struct *task = current; - if (likely(task->mm == mm)) + if (likely(task->mm == mm)) { task->rss_stat.count[member] += val; - else +#ifdef CONFIG_NUMA + task->rss_stat.numa_count[node] += val; +#endif + } else add_mm_counter(mm, member, val, node); } #define inc_mm_counter_fast(mm, member, node) add_mm_counter_fast(mm, member, 1, node) @@ -520,6 +531,11 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss, int *numa_rss) for (i = 0; i < NR_MM_COUNTERS; i++) if (rss[i]) add_mm_counter(mm, i, rss[i], NUMA_NO_NODE); +#ifdef CONFIG_NUMA + for_each_node(i) + if (numa_rss[i] != 0) + add_mm_counter(mm, MM_NO_TYPE, numa_rss[i], i); +#endif } /* -- 2.20.1