From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7DA59CDB474 for ; Fri, 20 Oct 2023 03:44:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AF283800C7; Thu, 19 Oct 2023 23:44:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AA0EC8D0003; Thu, 19 Oct 2023 23:44:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96908800C7; Thu, 19 Oct 2023 23:44:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 840168D0003 for ; Thu, 19 Oct 2023 23:44:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 56E6C14053C for ; Fri, 20 Oct 2023 03:44:31 +0000 (UTC) X-FDA: 81364447542.15.22B1C0E Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by imf29.hostedemail.com (Postfix) with ESMTP id 17D13120008 for ; Fri, 20 Oct 2023 03:44:28 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CXD0sUF3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697773469; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OSqJHYYQ7J70ZoiwPQnZL0fVH5rrZ/zDIMOXEhegfbg=; b=BmBFKPmYnq/wdapFu7CHfJ82GNQWVfs3TV10wHSmVzdoqfs3G7jBM7BpjhHRfHY3OrNc3p yGSDDqO1Myp2nkcZ8PyMAFDswVIGswTbwfqemGVh5RJS0EwmY9t5YL1or0Yq6QYc5CNSlt qlynghV6MarFB9ub9TQPzId4vg/w7S0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=CXD0sUF3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697773469; a=rsa-sha256; cv=none; b=B3AS5qGXzAPNn34GqFeTBuhMaeHfaCcp/WPGx+44Q+dd7a9pkHxJcW+HtbQll2c9jEPBq8 CSSfD+olFhUSS1Iu1hWK7/ombblKjPbMG/XhI6moL3/1Ad2nhN1L/KOzmvgcvYL3A4Digw rWSaedTEWxcNZEgJIner0+iKMjAoo0s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697773469; x=1729309469; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=3u/M+++aMmtOx+k1r32N3fr+/0cRroxzeAeIKLLmXRo=; b=CXD0sUF3XrPg91LZE8B9q9758b7G9XBY1eUAH2UFeC3XNG1Hc11RdK1U fJoOMoao8zdUuVToUZGHZFBH3wBiJq+bWEvUjdbmceZZXrZA9rewZmWj/ Xklh7g+18DqnNmYu1Cly6BKJlqOwrQLhTzLPdF13aqrXCUrkpVKwA8uiH SKPVghE314Vo/JFH3VssyjDVFKqzp9M45iwIDHoK7hHgA68FJexbKXtVL KWLfvXTIs1+JH6uIgrJNIkqFaPXRSPg3QhXJgZixQcspDu91KCSmtkOPE /WR7o9UtdnV9d2OelxrJwHGecexS7rUp2Rws118MqZ7DWB9T80r1jt7Kj Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10868"; a="365759842" X-IronPort-AV: E=Sophos;i="6.03,238,1694761200"; d="scan'208";a="365759842" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Oct 2023 20:44:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,238,1694761200"; d="scan'208";a="4978682" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Oct 2023 20:43:15 -0700 From: "Huang, Ying" To: Baolin Wang Cc: , , , , , , , Subject: Re: [PATCH v2] mm: migrate: record the mlocked page status to remove unnecessary lru drain In-Reply-To: (Baolin Wang's message of "Fri, 20 Oct 2023 11:33:47 +0800") References: Date: Fri, 20 Oct 2023 11:42:22 +0800 Message-ID: <87jzriez8h.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 17D13120008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: nj9a5ug7xztuxcfeku9jzjsi6i4ex8oh X-HE-Tag: 1697773468-391305 X-HE-Meta: U2FsdGVkX19j7wzKdOCjhqLjydRhxvpXSdFXBDIOoLSvSBaX7yp2gxe7Eg8ZQDSdy7Ztx5Rh1g0TvSkHrxt4FqPWq47eIR+gfBM73dj7zKx0e1cmdLC7cNa8QOyRAsCXeEmVxKQ6XN5NNLQN5UDKAAhtmhEAj13z+R8Maythr+w299jO9WSlNYGa2d6Gr6Vom803vDIWXvJyW3ZVcgZbQNO71RM/g9SC46AchHeLpzdw4K+XcaeeNpqXNysGRYP5IDjpGJsxa5zLTu6Iv4bo/B0P+Hqy9xOJL1h62SupCuPD7ZaZZUXuvFcBnYPLETlWu21hT3XI4dAHrF+zWYM1lM23TtLfnWd0TiNQKFd0ek9K5AzJ1tuIJ0dXTRgjCT40FTSp6MCvkC0uBEMcEUUx1KQ6QUjm0+ylgjQcBWjrl7gwqJ5OIkqKBQAeTwsuHwMqC7aHJwVodggxazTgCHRNo7neYHBUqg3rrcvgZGV7bQ726A2HYRhgbO9E26edId97S3YbYz9pM7gg3KIZ18nhEPJ/sbpuzpWBQfBKs+1yCdwzM492+688hMvLxViDyo+l8T0sS/gbdRDPfg4gU7Wu60Gj7k6+KYXRBhSaS8zzdGaLutt5ZeMgQXkWs2zuWcdefFHAjlfsK6Z58zqRWnaK7iEQLExbo574pBtfZZ1lWRFPJgDHdrlqOQTVdNpbY4M4tk5ahmuxxqtG47cosR7MF3jGHuCD8pG/whXZucmP6EJy82pn7uFPLG+Ttmiu3+dFj3YD/NH05hEK/zScAYd4tFOKvXx7HQn2PYI7LM+Uisvg2sLC4RLk2NMOcsV/0hGnHOVgkhgGxCAQ2G150o5sv5PfQ13ijugzH1bWXopDZcp5wkxkbKUgG3+Zl3GLpbEuE+YyMrjjfP5eTjbVyVtphUXDx25urjq6Y2AJWpB5OozLTDOEsjAR96lQCDSSWcNXqou80Zz6Hml9ewSN4Ox jek1Tr4/ znBrK5CZLbUM88BlWP4sVC2LEJG+zqE0rrbMxAubMlpLGVoKq3Tk3KgtvsY5thP1cA7RmHMUABKa86NyVc71vfZMk1xBPaL2w4Sz8ccMn1Bcx0HBdOqp+kfHZ28lqfg+pHECSI9U/0/6Mfhy0DEJrYAAhUrW+N281BBy+aT9ieedIWPg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Baolin Wang writes: > When doing compaction, I found the lru_add_drain() is an obvious hotspot > when migrating pages. The distribution of this hotspot is as follows: > - 18.75% compact_zone > - 17.39% migrate_pages > - 13.79% migrate_pages_batch > - 11.66% migrate_folio_move > - 7.02% lru_add_drain > + 7.02% lru_add_drain_cpu > + 3.00% move_to_new_folio > 1.23% rmap_walk > + 1.92% migrate_folio_unmap > + 3.20% migrate_pages_sync > + 0.90% isolate_migratepages > > The lru_add_drain() was added by commit c3096e6782b7 ("mm/migrate: > __unmap_and_move() push good newpage to LRU") to drain the newpage to LRU > immediately, to help to build up the correct newpage->mlock_count in > remove_migration_ptes() for mlocked pages. However, if there are no mlocked > pages are migrating, then we can avoid this lru drain operation, especailly > for the heavy concurrent scenarios. > > So we can record the source pages' mlocked status in migrate_folio_unmap(), > and only drain the lru list when the mlocked status is set in migrate_folio_move(). > In addition, the page was already isolated from lru when migrating, so checking > the mlocked status is stable by folio_test_mlocked() in migrate_folio_unmap(). > > After this patch, I can see the hotpot of the lru_add_drain() is gone: > - 9.41% migrate_pages_batch > - 6.15% migrate_folio_move > - 3.64% move_to_new_folio > + 1.80% migrate_folio_extra > + 1.70% buffer_migrate_folio > + 1.41% rmap_walk > + 0.62% folio_add_lru > + 3.07% migrate_folio_unmap > > Meanwhile, the compaction latency shows some improvements when running > thpscale: > base patched > Amean fault-both-1 1131.22 ( 0.00%) 1112.55 * 1.65%* > Amean fault-both-3 2489.75 ( 0.00%) 2324.15 * 6.65%* > Amean fault-both-5 3257.37 ( 0.00%) 3183.18 * 2.28%* > Amean fault-both-7 4257.99 ( 0.00%) 4079.04 * 4.20%* > Amean fault-both-12 6614.02 ( 0.00%) 6075.60 * 8.14%* > Amean fault-both-18 10607.78 ( 0.00%) 8978.86 * 15.36%* > Amean fault-both-24 14911.65 ( 0.00%) 11619.55 * 22.08%* > Amean fault-both-30 14954.67 ( 0.00%) 14925.66 * 0.19%* > Amean fault-both-32 16654.87 ( 0.00%) 15580.31 * 6.45%* > > Signed-off-by: Baolin Wang > --- > Chages from v1: > - Use separate flags in __migrate_folio_record() to avoid to pack flags > in each call site per Ying. > --- > mm/migrate.c | 47 +++++++++++++++++++++++++++++++++++------------ > 1 file changed, 35 insertions(+), 12 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 125194f5af0f..fac96139dbba 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1027,22 +1027,39 @@ union migration_ptr { > struct anon_vma *anon_vma; > struct address_space *mapping; > }; > + > +enum { > + PAGE_WAS_MAPPED = 1 << 0, PAGE_WAS_MAPPED = BIT(0) ? > + PAGE_WAS_MLOCKED = 1 << 1, > +}; > + > static void __migrate_folio_record(struct folio *dst, > - unsigned long page_was_mapped, > + unsigned int page_was_mapped, > + unsigned int page_was_mlocked, > struct anon_vma *anon_vma) > { > union migration_ptr ptr = { .anon_vma = anon_vma }; > + unsigned long page_flags = 0; page_flags wasn't a good name, it can be confused with page->flags. May be something like "page_attrs"? > + > + if (page_was_mapped) > + page_flags |= PAGE_WAS_MAPPED; > + if (page_was_mlocked) > + page_flags |= PAGE_WAS_MLOCKED; > dst->mapping = ptr.mapping; > - dst->private = (void *)page_was_mapped; > + dst->private = (void *)page_flags; > } > > static void __migrate_folio_extract(struct folio *dst, > int *page_was_mappedp, > + int *page_was_mlocked, Better to use the same naming convention. Either both have "p" suffix, or both not. Otherwise looks good to me. > struct anon_vma **anon_vmap) > { > union migration_ptr ptr = { .mapping = dst->mapping }; > + unsigned long page_flags = (unsigned long)dst->private; > + > *anon_vmap = ptr.anon_vma; > - *page_was_mappedp = (unsigned long)dst->private; > + *page_was_mappedp = page_flags & PAGE_WAS_MAPPED ? 1 : 0; > + *page_was_mlocked = page_flags & PAGE_WAS_MLOCKED ? 1 : 0; > dst->mapping = NULL; > dst->private = NULL; > } > @@ -1103,7 +1120,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > { > struct folio *dst; > int rc = -EAGAIN; > - int page_was_mapped = 0; > + int page_was_mapped = 0, page_was_mlocked = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > bool locked = false; > @@ -1157,6 +1174,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > folio_lock(src); > } > locked = true; > + page_was_mlocked = folio_test_mlocked(src); > > if (folio_test_writeback(src)) { > /* > @@ -1206,7 +1224,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > dst_locked = true; > > if (unlikely(!is_lru)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, page_was_mapped, > + page_was_mlocked, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1236,7 +1255,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > } > > if (!folio_mapped(src)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, page_was_mapped, > + page_was_mlocked, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1261,12 +1281,13 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > struct list_head *ret) > { > int rc; > - int page_was_mapped = 0; > + int page_was_mapped = 0, page_was_mlocked = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > struct list_head *prev; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > + __migrate_folio_extract(dst, &page_was_mapped, > + &page_was_mlocked, &anon_vma); > prev = dst->lru.prev; > list_del(&dst->lru); > > @@ -1287,7 +1308,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > * isolated from the unevictable LRU: but this case is the easiest. > */ > folio_add_lru(dst); > - if (page_was_mapped) > + if (page_was_mlocked) > lru_add_drain(); > > if (page_was_mapped) > @@ -1322,7 +1343,8 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > */ > if (rc == -EAGAIN) { > list_add(&dst->lru, prev); > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, page_was_mapped, > + page_was_mlocked, anon_vma); > return rc; > } > > @@ -1799,10 +1821,11 @@ static int migrate_pages_batch(struct list_head *from, > dst = list_first_entry(&dst_folios, struct folio, lru); > dst2 = list_next_entry(dst, lru); > list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { > - int page_was_mapped = 0; > + int page_was_mapped = 0, page_was_mlocked = 0; > struct anon_vma *anon_vma = NULL; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > + __migrate_folio_extract(dst, &page_was_mapped, > + &page_was_mlocked, &anon_vma); > migrate_folio_undo_src(folio, page_was_mapped, anon_vma, > true, ret_folios); > list_del(&dst->lru); -- Best Regards, Huang, Ying