From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 550D4C07545 for ; Mon, 23 Oct 2023 00:56:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D82C6B0092; Sun, 22 Oct 2023 20:56:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 789CF6B0095; Sun, 22 Oct 2023 20:56:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 676896B0096; Sun, 22 Oct 2023 20:56:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5813D6B0092 for ; Sun, 22 Oct 2023 20:56:57 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 27EB11A02F0 for ; Mon, 23 Oct 2023 00:56:57 +0000 (UTC) X-FDA: 81374911674.08.405CBE2 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by imf28.hostedemail.com (Postfix) with ESMTP id 7B3E3C0004 for ; Mon, 23 Oct 2023 00:56:54 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g48Fx0ip; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698022615; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hGtTM2evG7LAmBs24FIv+w85m5VGRFDz5JECdSlJyQM=; b=NqGR56tnupU27Jjt48T/4HAv9oc1kTmJV6T5E3AHOZnyQ9CXIO9HDqeARyqxTnxBUtSgQ+ kEquY8N5+W30Gk74Ijs1h1Oa8gYms3ECfzZ1WVVsJ6LNXdg7wuKNfiUg8dkMsMfOrrR0Zm MNOGQPq1KZxxINvGjJACoG3xnLGBtP0= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=g48Fx0ip; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698022615; a=rsa-sha256; cv=none; b=jbuBFjfq5RmdOtig1fmJ+oN5NDOMMsMHM+2laEhkMz2RGkka3U68z7l4iUBCnEl2KiyLSi ICkvMy5FgzO2Jslhv+Sosr0VFA1/alQvczu5jORCUJZznUZ1oSoiXoosNWxBSijkjcXDeg xDJWSuV7qDLPHTC0mICs6VAw0paJik4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698022614; x=1729558614; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=hS8mjo8Zr5a5TG8vn4Z2xY74poRaGE5x1DCwvuN39kk=; b=g48Fx0ipObxifGlIlibZBxYzVQEriKrEXBI3+3X+ZihRxQKZvwdQBa9T WDEWLFaKgE1XOOGK1qHmSIBe95ShJtUuZ6m+msRg4p4wdATpNvjzB0OIR O07PouvaJ9ELM17YBzce7ZryWgcbevF853mOF9nej2FTlHl9WYc8EEf0v S9L8d6C9cvy4gl32HqVdOccX2fsRTCMjm/A1CO1psP+hqZg2o42eD1EbA M711fUILPxr6z6WvGZ4NGvMrqVdspT/vT7Tr3nHX8zG+BoMyisZrwE9fV y+8GnK7Gu0CGbEWk8zKz88gVHHWfhyQ3IIYxguUS/lpSW4V0PrcV54kvh w==; X-IronPort-AV: E=McAfee;i="6600,9927,10871"; a="371811454" X-IronPort-AV: E=Sophos;i="6.03,244,1694761200"; d="scan'208";a="371811454" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2023 17:56:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10871"; a="823806134" X-IronPort-AV: E=Sophos;i="6.03,244,1694761200"; d="scan'208";a="823806134" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2023 17:56:50 -0700 From: "Huang, Ying" To: Baolin Wang Cc: , , , , , , , Subject: Re: [PATCH v3] mm: migrate: record the mlocked page status to remove unnecessary lru drain In-Reply-To: <06e9153a7a4850352ec36602df3a3a844de45698.1697859741.git.baolin.wang@linux.alibaba.com> (Baolin Wang's message of "Sat, 21 Oct 2023 12:33:22 +0800") References: <06e9153a7a4850352ec36602df3a3a844de45698.1697859741.git.baolin.wang@linux.alibaba.com> Date: Mon, 23 Oct 2023 08:54:47 +0800 Message-ID: <877cnef99k.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 7B3E3C0004 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: t9q9omn8oo7urx64kkxk7u7g5yhzmgq6 X-HE-Tag: 1698022614-106324 X-HE-Meta: U2FsdGVkX18tpfV8+QJgumR4R5fXxIdCrnFQLxyKrYS5EgqyaDY3zeBet0uFxy1bQzZSB3/YgQQFvCBjPmerRcgX97Lrnd7MUuPRxC4AS+mLxJoQCeq87uFsALKWim8LOG1a16+vhQuqz+bgw2vqU57s0Xp0RrGhurUrAqJoo4fAVH4r0KsiMeyfN+Yu+X6rReqlhXa4JKQvo6K/KbWk5GR5BZheoUwgziK+SnvxgHpmcUWWIESQThrEbC/iIRaQcdAKi1BMp1kPXocS9gBkFGnRSh1cgrp8bULGh1nOosd42uPLoJe0dhjE9xBLFgnxd4gpxAIDrjsPi2XTmgTrwbXy5ONx8rKU4QgQ3/vbIbofq4d3Lkjgi046WvwukO9x1BsNKKND1rSPaxnwsxequ7q2dzveskKNbEq/inrZHoUP2Yb5G9OQGFdwkIJWmJsI54bCndGDpUVRO0dF8viyKiUXD8K8i/+bp4MAH5ogebnjIIS+Wi3gFjogWGYvCNb6taJL/T5Pkhv0lQcr/VDLT0uVsY0fa78/Sgu6l8k7klkpzdx0nlxxWVFhUjTIqtnwuo3e5bMcYkCQPTjA5Cdt573Ibq0WFRIInbYyit9xJwdlvWaIvsGG2aS4jBKTX0EsHOTn9mzt98XwwttOxqsxjnZhYaVFcrV4Jab//BqVNAcYwwo2YUOOSnU/yEZScNNI+zXO7VmcpuVXh6iFw4y2LUu/SWygzLTNLtkB6tzuUZ/K8nr5kK4pIVULJrG/socFYWypfjaNUp4HovmpYkVj+c2E7Q5QejqgUnF+wVJ/T3dt43VQ7TpSw2AFgVPslaP6cfT4yUOQtMo/A1QL5kRw1Db0abuVr2DbI+Zu4FnK2IsoHKLS46whlHo18IrKZ3+d1+ebXqcgEkIYaO9NSbyR/EfuukCefMpQqr7LPJkR4dy7Srtyct3WszyM9BIRL92H1CjEgZ4qrBkazCxZSdm aCE16Ir+ ZKpHBVExtyZpcRI5FzNUwKn+KswpUuAUwus9oF/a6+ZDTSNw4trqkFPkgrHha/8FgHwYZMmgnjMyzvAqPhyaM4MSIVSNIZOTSah9nPBFRzxw4oiQDJQQS6hlwGqlsF0qrgowMIUSw3nbFPNmDwnotS4ku9evLHGo7ZfEL2huleiG0CDCvLA6MRcud/tr05c4wiOCW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Baolin Wang writes: > When doing compaction, I found the lru_add_drain() is an obvious hotspot > when migrating pages. The distribution of this hotspot is as follows: > - 18.75% compact_zone > - 17.39% migrate_pages > - 13.79% migrate_pages_batch > - 11.66% migrate_folio_move > - 7.02% lru_add_drain > + 7.02% lru_add_drain_cpu > + 3.00% move_to_new_folio > 1.23% rmap_walk > + 1.92% migrate_folio_unmap > + 3.20% migrate_pages_sync > + 0.90% isolate_migratepages > > The lru_add_drain() was added by commit c3096e6782b7 ("mm/migrate: > __unmap_and_move() push good newpage to LRU") to drain the newpage to LRU > immediately, to help to build up the correct newpage->mlock_count in > remove_migration_ptes() for mlocked pages. However, if there are no mlocked > pages are migrating, then we can avoid this lru drain operation, especailly > for the heavy concurrent scenarios. > > So we can record the source pages' mlocked status in migrate_folio_unmap(), > and only drain the lru list when the mlocked status is set in migrate_folio_move(). > In addition, the page was already isolated from lru when migrating, so checking > the mlocked status is stable by folio_test_mlocked() in migrate_folio_unmap(). > > After this patch, I can see the hotpot of the lru_add_drain() is gone: > - 9.41% migrate_pages_batch > - 6.15% migrate_folio_move > - 3.64% move_to_new_folio > + 1.80% migrate_folio_extra > + 1.70% buffer_migrate_folio > + 1.41% rmap_walk > + 0.62% folio_add_lru > + 3.07% migrate_folio_unmap > > Meanwhile, the compaction latency shows some improvements when running > thpscale: > base patched > Amean fault-both-1 1131.22 ( 0.00%) 1112.55 * 1.65%* > Amean fault-both-3 2489.75 ( 0.00%) 2324.15 * 6.65%* > Amean fault-both-5 3257.37 ( 0.00%) 3183.18 * 2.28%* > Amean fault-both-7 4257.99 ( 0.00%) 4079.04 * 4.20%* > Amean fault-both-12 6614.02 ( 0.00%) 6075.60 * 8.14%* > Amean fault-both-18 10607.78 ( 0.00%) 8978.86 * 15.36%* > Amean fault-both-24 14911.65 ( 0.00%) 11619.55 * 22.08%* > Amean fault-both-30 14954.67 ( 0.00%) 14925.66 * 0.19%* > Amean fault-both-32 16654.87 ( 0.00%) 15580.31 * 6.45%* > > Signed-off-by: Baolin Wang LGTM, Thanks! Reviewed-by: "Huang, Ying" > --- > Changes from v2: > - Use BIT() to define macros. > - Simplify handing page_was_mapped and page_was_mlocked variables with > introducing old_page_state variable. > > Changes from v1: > - Use separate flags in __migrate_folio_record() to avoid to pack flags > in each call site per Ying. > --- > mm/migrate.c | 48 +++++++++++++++++++++++++++++------------------- > 1 file changed, 29 insertions(+), 19 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 125194f5af0f..35a88334bb3c 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1027,22 +1027,28 @@ union migration_ptr { > struct anon_vma *anon_vma; > struct address_space *mapping; > }; > + > +enum { > + PAGE_WAS_MAPPED = BIT(0), > + PAGE_WAS_MLOCKED = BIT(1), > +}; > + > static void __migrate_folio_record(struct folio *dst, > - unsigned long page_was_mapped, > + unsigned long old_page_state, > struct anon_vma *anon_vma) > { > union migration_ptr ptr = { .anon_vma = anon_vma }; > dst->mapping = ptr.mapping; > - dst->private = (void *)page_was_mapped; > + dst->private = (void *)old_page_state; > } > > static void __migrate_folio_extract(struct folio *dst, > - int *page_was_mappedp, > + int *old_page_state, > struct anon_vma **anon_vmap) > { > union migration_ptr ptr = { .mapping = dst->mapping }; > *anon_vmap = ptr.anon_vma; > - *page_was_mappedp = (unsigned long)dst->private; > + *old_page_state = (unsigned long)dst->private; > dst->mapping = NULL; > dst->private = NULL; > } > @@ -1103,7 +1109,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > { > struct folio *dst; > int rc = -EAGAIN; > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > bool locked = false; > @@ -1157,6 +1163,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > folio_lock(src); > } > locked = true; > + if (folio_test_mlocked(src)) > + old_page_state |= PAGE_WAS_MLOCKED; > > if (folio_test_writeback(src)) { > /* > @@ -1206,7 +1214,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > dst_locked = true; > > if (unlikely(!is_lru)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1232,11 +1240,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > VM_BUG_ON_FOLIO(folio_test_anon(src) && > !folio_test_ksm(src) && !anon_vma, src); > try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); > - page_was_mapped = 1; > + old_page_state |= PAGE_WAS_MAPPED; > } > > if (!folio_mapped(src)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1248,7 +1256,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > if (rc == -EAGAIN) > ret = NULL; > > - migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); > + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, locked, ret); > migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private); > > return rc; > @@ -1261,12 +1270,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > struct list_head *ret) > { > int rc; > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > struct list_head *prev; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > + __migrate_folio_extract(dst, &old_page_state, &anon_vma); > prev = dst->lru.prev; > list_del(&dst->lru); > > @@ -1287,10 +1296,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > * isolated from the unevictable LRU: but this case is the easiest. > */ > folio_add_lru(dst); > - if (page_was_mapped) > + if (old_page_state & PAGE_WAS_MLOCKED) > lru_add_drain(); > > - if (page_was_mapped) > + if (old_page_state & PAGE_WAS_MAPPED) > remove_migration_ptes(src, dst, false); > > out_unlock_both: > @@ -1322,11 +1331,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > */ > if (rc == -EAGAIN) { > list_add(&dst->lru, prev); > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return rc; > } > > - migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); > + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, true, ret); > migrate_folio_undo_dst(dst, true, put_new_folio, private); > > return rc; > @@ -1799,12 +1809,12 @@ static int migrate_pages_batch(struct list_head *from, > dst = list_first_entry(&dst_folios, struct folio, lru); > dst2 = list_next_entry(dst, lru); > list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > - migrate_folio_undo_src(folio, page_was_mapped, anon_vma, > - true, ret_folios); > + __migrate_folio_extract(dst, &old_page_state, &anon_vma); > + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, true, ret_folios); > list_del(&dst->lru); > migrate_folio_undo_dst(dst, true, put_new_folio, private); > dst = dst2;