From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3478AC54E67 for ; Wed, 27 Mar 2024 06:32:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACF7C6B0093; Wed, 27 Mar 2024 02:32:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7FAA6B0096; Wed, 27 Mar 2024 02:32:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9489E6B0098; Wed, 27 Mar 2024 02:32:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 838016B0093 for ; Wed, 27 Mar 2024 02:32:43 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0377A1209BA for ; Wed, 27 Mar 2024 06:32:42 +0000 (UTC) X-FDA: 81941850606.14.E0BE92A Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.9]) by imf06.hostedemail.com (Postfix) with ESMTP id CA02C180005 for ; Wed, 27 Mar 2024 06:32:40 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FJuOFXia; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.9 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711521161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AMLpBtVxjldQbzkmE9qfhw96AiNErZ01vo5bfg//ROU=; b=LxhVsPdC158Afar27LPFyqp3dU5R0IdE6+GrPhlJxHACC09nLNfLTwqpsS6A2JmWUiX49/ ZqPAKHlNF60cP/AwtQsjuOvyVU6svnYOD9Fs6rfkYDy0dnsHvfi/UtDM+AP/Hzs4iw06kn 0ybT6cwOM0J5JI41h3R6YAFzZ75gam0= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=FJuOFXia; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.9 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711521161; a=rsa-sha256; cv=none; b=oX4OaEZxxAm69BCN6dbNM4N2t9oQec0UJPqUStD4kvb5i/7Nf0xJhsLTp4eLzGgPLfBZHU 8W0rmi5gNrRYrnM3O1we0Po9FzkUL4UnIZS5oQD/UGy8fiB4xbd5HJC1PmyS3IKMskHlae nOmV2d3UiSdXArVIULlB+evJ3K7UOXk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711521161; x=1743057161; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=BsWibFsPr6AJVwQBOI/XULxd/ZV584eWufvBF+1J3Z0=; b=FJuOFXiaNZAVlgLIwyTeMa4mPGl5sQyKtO2IzkdLgreU7JGaP5F0WGar 0Jf5e5jraq/I+5Gf9OMwhMZZM1enZGimJiPVDSbJfZCRWkufSa9c23chf Lhn3fgpWY7yGjJiAJOlYzP/+A64sDbEagN9d3IXXDL/pK/cvv6+UhODMJ WIxFPngTwKnDW7eXFOFwpeqf4Knc0aTx0+KYGoi9X+NEYfPbvynCHBvKN isrKREyZ+EwF4dQuN5Bfhp0xsEsEB66Hz38gl9pXjjOoBjyMNk3+rLtG+ Xcbehw3tZOQye2oL753BpffRuQUIT8dyuGd3kP/KtFvZh/OXC+rR6iKM3 A==; X-CSE-ConnectionGUID: SUxnG7R1RzKULfDdRmbiTA== X-CSE-MsgGUID: YLP/xFurTsqclJTc2gXLeg== X-IronPort-AV: E=McAfee;i="6600,9927,11025"; a="17333447" X-IronPort-AV: E=Sophos;i="6.07,158,1708416000"; d="scan'208";a="17333447" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Mar 2024 23:32:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,158,1708416000"; d="scan'208";a="16181127" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Mar 2024 23:32:35 -0700 From: "Huang, Ying" To: Kairui Song Cc: linux-mm@kvack.org, Kairui Song , Chris Li , Minchan Kim , Barry Song , Ryan Roberts , Yu Zhao , SeongJae Park , David Hildenbrand , Yosry Ahmed , Johannes Weiner , Matthew Wilcox , Nhat Pham , Chengming Zhou , Andrew Morton , linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 04/10] mm/swap: remove cache bypass swapin In-Reply-To: <20240326185032.72159-5-ryncsn@gmail.com> (Kairui Song's message of "Wed, 27 Mar 2024 02:50:26 +0800") References: <20240326185032.72159-1-ryncsn@gmail.com> <20240326185032.72159-5-ryncsn@gmail.com> Date: Wed, 27 Mar 2024 14:30:42 +0800 Message-ID: <87v858mbjh.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: CA02C180005 X-Rspam-User: X-Stat-Signature: 4eh8f7go8r4jcogtnyokw4u538ba1x91 X-Rspamd-Server: rspam01 X-HE-Tag: 1711521160-795784 X-HE-Meta: U2FsdGVkX18WqRnXNxUzKK98rK1cgvPadcem7YCSbgdWnckUQOFDkFRnZeP8dlY8YjxMY2CKYAhMD0z/wuoqjc9sBac6cruvN7NWQNPzDG24srIVIk9dp629aul42bcn97qlOZXWEJ6JYYF0plgUkoCgGiyfz+eHb73Q1iXXWAvn6uqePSieNWAKsORCuV+lcMmzB8LDHfAqP87R6+4gQ2NdZAEZMTuOD3fFs5hn5psm0e5oAdlE6m9UNwmzZEoW0xf93hQBY20PfaPcKTzuSUpApfF/1AC5B43zLlXxWSH7kUwhXg5FXD2X6rc12zip/jevT+mOotRoc2GjjIxl6xsAPcWupUWTp9xwGSKJwcArClQhEwbjlFjCXU8q+pMJlC/crTdg0cgN/XMaU1KrZpuexnyCtCbA+Ccvfl7AXJqjmw1T42swWi1BdoqdorxruGKbo6n6UkvqK8hH8Rq4vqIRp2Zse0mz3oOI7z3oelJwJIw+5pfO72OnXKx/AsRpCKVc++NnYDahQSkNR7a3xW52lkl6szTUZY9rYtho7aNwZoZcsoLHSbv/SQRF1OFdPHspCpLifTSoBtqn2Hl+BFJoFfU36isnx+n3BeninvvY89PPcO5VgpTp19pv0aqva7ejQq8wSQKdQymKNP1rZ5eW3ni+KdeNpA6K2Gutl7bQdVpH7LOjvUmcNDuUzA3c3M4rYZ8+YHXYPKZokLf9/0WwCKnYBb05m1+vnCdlBSGAZFVeenaabUVtrXnaT4Lm2PkJ+h2E+hm6mMP8YAAk+VfsV+k3jOjscrymSVZhFNGN+acuH7pvwBaaKUvkOe4tsFNTB1R+b4xii1QmxaGDp67nwp88TQftFyrT46ouvuUVwwZqYDQkRdr5dp04nmOKLuDVJ5hoWes/qTmaEp44v2zwdtNBwJqP/zkVpoJfKWXN9xvZB7gBmPh+hka12lRlNlTdc0X2/Xxt9Ur1dgo IngZ8BCc OtWnF0qlOcsNabEfmhiu3QJnxO53R4zncDjfcffZtOpjAMp3RVaUWsUY8+SEKSTVhrefj19d0n+Hc7vqSyrVLbmI7QSSdqyxKapmgVaozDeJVfw4OFm0s/qS/ErkvSAHfgdu+xN9Tu3LYSDwvsSEXw4xGNwi3CAzQ0Xch3eTGv17Meb0DJNtLfU2UhH6G8mbkbP1BJRs6ht3rTCIF+jRtMTvyHNmNeLN6fdWqSna4/vL2dRsM4dC4f18sdftu+1uQ1TJs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Kairui Song writes: > From: Kairui Song > > We used to have the cache bypass swapin path for better performance, > but by removing it, more optimization can be applied and have > an even better overall performance and less hackish. > > And these optimizations are not easily doable or not doable at all > without this. > > This patch simply removes it, and the performance will drop heavily > for simple swapin, things won't get this worse for real workloads > but still observable. Following commits will fix this and archive a > better performance. > > Swapout/in 30G zero pages from ZRAM (This mostly measures overhead > of swap path itself, because zero pages are not compressed but simply > recorded in ZRAM, and performance drops more as SWAP device is getting > full): > > Test result of sequential swapin/out: > > Before (us) After (us) > Swapout: 33619409 33624641 > Swapin: 32393771 41614858 (-28.4%) > Swapout (THP): 7817909 7795530 > Swapin (THP) : 32452387 41708471 (-28.4%) > > Signed-off-by: Kairui Song > --- > mm/memory.c | 18 ++++------------- > mm/swap.h | 10 +++++----- > mm/swap_state.c | 53 ++++++++++--------------------------------------- > mm/swapfile.c | 13 ------------ > 4 files changed, 19 insertions(+), 75 deletions(-) > > diff --git a/mm/memory.c b/mm/memory.c > index dfdb620a9123..357d239ee2f6 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -3932,7 +3932,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > struct page *page; > struct swap_info_struct *si = NULL; > rmap_t rmap_flags = RMAP_NONE; > - bool need_clear_cache = false; > bool exclusive = false; > swp_entry_t entry; > pte_t pte; > @@ -4000,14 +3999,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > if (!folio) { > if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && > __swap_count(entry) == 1) { > - /* skip swapcache and readahead */ > folio = swapin_direct(entry, GFP_HIGHUSER_MOVABLE, vmf); > - if (PTR_ERR(folio) == -EBUSY) > - goto out; > - need_clear_cache = true; > } else { > folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf); > - swapcache = folio; > } > > if (!folio) { > @@ -4023,6 +4017,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > goto unlock; > } > > + swapcache = folio; > page = folio_file_page(folio, swp_offset(entry)); > > /* Had to read the page from swap area: Major fault */ > @@ -4187,7 +4182,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > vmf->orig_pte = pte; > > /* ksm created a completely new copy */ > - if (unlikely(folio != swapcache && swapcache)) { > + if (unlikely(folio != swapcache)) { > folio_add_new_anon_rmap(folio, vma, vmf->address); > folio_add_lru_vma(folio, vma); > } else { > @@ -4201,7 +4196,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); > > folio_unlock(folio); > - if (folio != swapcache && swapcache) { > + if (folio != swapcache) { > /* > * Hold the lock to avoid the swap entry to be reused > * until we take the PT lock for the pte_same() check > @@ -4227,9 +4222,6 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > if (vmf->pte) > pte_unmap_unlock(vmf->pte, vmf->ptl); > out: > - /* Clear the swap cache pin for direct swapin after PTL unlock */ > - if (need_clear_cache) > - swapcache_clear(si, entry); > if (si) > put_swap_device(si); > return ret; > @@ -4240,12 +4232,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) > folio_unlock(folio); > out_release: > folio_put(folio); > - if (folio != swapcache && swapcache) { > + if (folio != swapcache) { > folio_unlock(swapcache); > folio_put(swapcache); > } > - if (need_clear_cache) > - swapcache_clear(si, entry); > if (si) > put_swap_device(si); > return ret; > diff --git a/mm/swap.h b/mm/swap.h > index aee134907a70..ac9573b03432 100644 > --- a/mm/swap.h > +++ b/mm/swap.h > @@ -41,7 +41,6 @@ void __delete_from_swap_cache(struct folio *folio, > void delete_from_swap_cache(struct folio *folio); > void clear_shadow_from_swap_cache(int type, unsigned long begin, > unsigned long end); > -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry); > struct folio *swap_cache_get_folio(swp_entry_t entry, > struct vm_area_struct *vma, unsigned long addr); > struct folio *filemap_get_incore_folio(struct address_space *mapping, > @@ -100,14 +99,15 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask, > { > return NULL; > } > - > -static inline int swap_writepage(struct page *p, struct writeback_control *wbc) > +static inline struct folio *swapin_direct(swp_entry_t entry, gfp_t flag, > + struct vm_fault *vmf); > { > - return 0; > + return NULL; > } > > -static inline void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) > +static inline int swap_writepage(struct page *p, struct writeback_control *wbc) > { > + return 0; > } > > static inline struct folio *swap_cache_get_folio(swp_entry_t entry, > diff --git a/mm/swap_state.c b/mm/swap_state.c > index 2a9c6bdff5ea..49ef6250f676 100644 > --- a/mm/swap_state.c > +++ b/mm/swap_state.c > @@ -880,61 +880,28 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, > } > > /** > - * swapin_direct - swap in folios skipping swap cache and readahead > + * swapin_direct - swap in folios skipping readahead > * @entry: swap entry of this memory > * @gfp_mask: memory allocation flags > * @vmf: fault information > * > - * Returns the struct folio for entry and addr after the swap entry is read > - * in. > + * Returns the folio for entry after it is read in. > */ > struct folio *swapin_direct(swp_entry_t entry, gfp_t gfp_mask, > struct vm_fault *vmf) > { > - struct vm_area_struct *vma = vmf->vma; > + struct mempolicy *mpol; > struct folio *folio; > - void *shadow = NULL; > - > - /* > - * Prevent parallel swapin from proceeding with > - * the cache flag. Otherwise, another thread may > - * finish swapin first, free the entry, and swapout > - * reusing the same entry. It's undetectable as > - * pte_same() returns true due to entry reuse. > - */ > - if (swapcache_prepare(entry)) { > - /* Relax a bit to prevent rapid repeated page faults */ > - schedule_timeout_uninterruptible(1); > - return ERR_PTR(-EBUSY); > - } > - > - /* skip swapcache */ > - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, > - vma, vmf->address, false); > - if (folio) { > - __folio_set_locked(folio); > - __folio_set_swapbacked(folio); > - > - if (mem_cgroup_swapin_charge_folio(folio, > - vma->vm_mm, GFP_KERNEL, > - entry)) { > - folio_unlock(folio); > - folio_put(folio); > - return NULL; > - } > - mem_cgroup_swapin_uncharge_swap(entry); > - > - shadow = get_shadow_from_swap_cache(entry); > - if (shadow) > - workingset_refault(folio, shadow); > + bool page_allocated; > + pgoff_t ilx; > > - folio_add_lru(folio); > + mpol = get_vma_policy(vmf->vma, vmf->address, 0, &ilx); > + folio = __read_swap_cache_async(entry, gfp_mask, mpol, ilx, > + &page_allocated, false); > + mpol_cond_put(mpol); > > - /* To provide entry to swap_read_folio() */ > - folio->swap = entry; > + if (page_allocated) > swap_read_folio(folio, true, NULL); > - folio->private = NULL; > - } > > return folio; > } This looks similar as read_swap_cache_async(). Can we merge them? And, we should avoid to readahead in swapin_readahead() or swap_vma_readahead() for SWP_SYNCHRONOUS_IO anyway. So, it appears that we can change and use swapin_readahead() directly? > diff --git a/mm/swapfile.c b/mm/swapfile.c > index 4dd894395a0f..ae8d3aa05df7 100644 > --- a/mm/swapfile.c > +++ b/mm/swapfile.c > @@ -3389,19 +3389,6 @@ int swapcache_prepare(swp_entry_t entry) > return __swap_duplicate(entry, SWAP_HAS_CACHE); > } > > -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) > -{ > - struct swap_cluster_info *ci; > - unsigned long offset = swp_offset(entry); > - unsigned char usage; > - > - ci = lock_cluster_or_swap_info(si, offset); > - usage = __swap_entry_free_locked(si, offset, SWAP_HAS_CACHE); > - unlock_cluster_or_swap_info(si, ci); > - if (!usage) > - free_swap_slot(entry); > -} > - > struct swap_info_struct *swp_swap_info(swp_entry_t entry) > { > return swap_type_to_swap_info(swp_type(entry)); -- Best Regards, Huang, Ying