From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0851AC433F5 for ; Thu, 7 Apr 2022 08:13:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 554376B0071; Thu, 7 Apr 2022 04:13:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DD816B0074; Thu, 7 Apr 2022 04:13:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 356B26B0075; Thu, 7 Apr 2022 04:13:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 229676B0071 for ; Thu, 7 Apr 2022 04:13:10 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id ED59E308C for ; Thu, 7 Apr 2022 08:12:59 +0000 (UTC) X-FDA: 79329367278.08.704F6B0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by imf05.hostedemail.com (Postfix) with ESMTP id 817B9100002 for ; Thu, 7 Apr 2022 08:12:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1649319178; x=1680855178; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=AjVgvE/DkaKviw97JGgJFD73gnHwnDL+e+OjQnruVKI=; b=PbJ+AtArdfJt3BSiYGZDhyPwo25zLk1E2Hqy/e1XWJEJmDh9BuyhpzPq KLu2hnbSEUdWHdGa4lzkCxzdmowkkPjkyPAwxuZGh9CAG+D1wE5HWE/3/ mWRqNgyyScIAxLZlStUqp6RAmEV+h7a23ngembY8aqLg8x8EA6bgekTir fVHhuAX4AY3t+jO2U2wgnCPRwzDGnj0Mx5Sf8k2ZULLzTMFKf40ustz8z wfcCPiCQ6783kbWYOBfHaDTNP4Tq6eJy8izkyNJwdC23EAsAirZ/zPs2l Dv2hQP8C2Xe27WXCsuEBWX3dWi1SiaPFkSr/+yimvpGBHFi5cKh7aOWnn w==; X-IronPort-AV: E=McAfee;i="6200,9189,10309"; a="261438184" X-IronPort-AV: E=Sophos;i="5.90,241,1643702400"; d="scan'208";a="261438184" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2022 01:12:56 -0700 X-IronPort-AV: E=Sophos;i="5.90,241,1643702400"; d="scan'208";a="722864004" Received: from ziqianlu-desk.sh.intel.com (HELO ziqianlu-nuc9qn) ([10.238.5.244]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Apr 2022 01:12:54 -0700 Date: Thu, 7 Apr 2022 16:13:00 +0800 From: Aaron Lu To: Yang Shi Cc: ying.huang@intel.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] mm: swap: determine swap device by using page nid Message-ID: References: <20220407020953.475626-1-shy828301@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220407020953.475626-1-shy828301@gmail.com> X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: oh4xg6nyikccq7qmx4p9e5nkrk6mig3w Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PbJ+AtAr; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf05.hostedemail.com: domain of aaron.lu@intel.com has no SPF policy when checking 192.55.52.115) smtp.mailfrom=aaron.lu@intel.com X-Rspamd-Queue-Id: 817B9100002 X-HE-Tag: 1649319178-389912 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 06, 2022 at 07:09:53PM -0700, Yang Shi wrote: > The swap devices are linked to per node priority lists, the swap device > closer to the node has higher priority on that node's priority list. > This is supposed to improve I/O latency, particularly for some fast > devices. But the current code gets nid by calling numa_node_id() which > actually returns the nid that the reclaimer is running on instead of the > nid that the page belongs to. > Right. > Pass the page's nid dow to get_swap_pages() in order to pick up the > right swap device. But it doesn't work for the swap slots cache which > is per cpu. We could skip swap slots cache if the current node is not > the page's node, but it may be overkilling. So keep using the current > node's swap slots cache. The issue was found by visual code inspection > so it is not sure how much improvement could be achieved due to lack of > suitable testing device. But anyway the current code does violate the > design. > I intentionally used the reclaimer's nid because I think when swapping out to a device, it is faster when the device is on the same node as the cpu. Anyway, I think I can make a test case where the workload allocates all its memory on the remote node and its workingset memory is larger then the available memory so swap is triggered, then we can see which way achieves better performance. Sounds reasonable to you? > Cc: Huang Ying > Signed-off-by: Yang Shi > --- > include/linux/swap.h | 3 ++- > mm/swap_slots.c | 7 ++++--- > mm/swapfile.c | 5 ++--- > 3 files changed, 8 insertions(+), 7 deletions(-) > > diff --git a/include/linux/swap.h b/include/linux/swap.h > index 27093b477c5f..e442cf6b61ea 100644 > --- a/include/linux/swap.h > +++ b/include/linux/swap.h > @@ -497,7 +497,8 @@ extern void si_swapinfo(struct sysinfo *); > extern swp_entry_t get_swap_page(struct page *page); > extern void put_swap_page(struct page *page, swp_entry_t entry); > extern swp_entry_t get_swap_page_of_type(int); > -extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size); > +extern int get_swap_pages(int n, swp_entry_t swp_entries[], int entry_size, > + int node); > extern int add_swap_count_continuation(swp_entry_t, gfp_t); > extern void swap_shmem_alloc(swp_entry_t); > extern int swap_duplicate(swp_entry_t); > diff --git a/mm/swap_slots.c b/mm/swap_slots.c > index 2b5531840583..a1c5cf6a4302 100644 > --- a/mm/swap_slots.c > +++ b/mm/swap_slots.c > @@ -264,7 +264,7 @@ static int refill_swap_slots_cache(struct swap_slots_cache *cache) > cache->cur = 0; > if (swap_slot_cache_active) > cache->nr = get_swap_pages(SWAP_SLOTS_CACHE_SIZE, > - cache->slots, 1); > + cache->slots, 1, numa_node_id()); > > return cache->nr; > } > @@ -305,12 +305,13 @@ swp_entry_t get_swap_page(struct page *page) > { > swp_entry_t entry; > struct swap_slots_cache *cache; > + int nid = page_to_nid(page); > > entry.val = 0; > > if (PageTransHuge(page)) { > if (IS_ENABLED(CONFIG_THP_SWAP)) > - get_swap_pages(1, &entry, HPAGE_PMD_NR); > + get_swap_pages(1, &entry, HPAGE_PMD_NR, nid); > goto out; > } > > @@ -342,7 +343,7 @@ swp_entry_t get_swap_page(struct page *page) > goto out; > } > > - get_swap_pages(1, &entry, 1); > + get_swap_pages(1, &entry, 1, nid); > out: > if (mem_cgroup_try_charge_swap(page, entry)) { > put_swap_page(page, entry); > diff --git a/mm/swapfile.c b/mm/swapfile.c > index 63c61f8b2611..151fffe0fd60 100644 > --- a/mm/swapfile.c > +++ b/mm/swapfile.c > @@ -1036,13 +1036,13 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) > swap_range_free(si, offset, SWAPFILE_CLUSTER); > } > > -int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size) > +int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size, > + int node) > { > unsigned long size = swap_entry_size(entry_size); > struct swap_info_struct *si, *next; > long avail_pgs; > int n_ret = 0; > - int node; > > /* Only single cluster request supported */ > WARN_ON_ONCE(n_goal > 1 && size == SWAPFILE_CLUSTER); > @@ -1060,7 +1060,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size) > atomic_long_sub(n_goal * size, &nr_swap_pages); > > start_over: > - node = numa_node_id(); > plist_for_each_entry_safe(si, next, &swap_avail_heads[node], avail_lists[node]) { > /* requeue si to after same-priority siblings */ > plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]); > -- > 2.26.3 > >