From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB737C05027 for ; Wed, 8 Feb 2023 11:36:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 33EBD6B0071; Wed, 8 Feb 2023 06:36:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C8136B0072; Wed, 8 Feb 2023 06:36:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 141AC6B0073; Wed, 8 Feb 2023 06:36:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F1C196B0071 for ; Wed, 8 Feb 2023 06:36:25 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BBC65C0CD6 for ; Wed, 8 Feb 2023 11:36:25 +0000 (UTC) X-FDA: 80443921530.06.6DFF337 Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf06.hostedemail.com (Postfix) with ESMTP id 8851418001B for ; Wed, 8 Feb 2023 11:36:23 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PSv6Vy52; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675856183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q5+p8Xb87CpVbCqVfgrGK2bBmhNTCNaI57YzxDHpmXU=; b=zDyKkc7QCLVr3UqyAJlgSr09D5zSuqrRN1o+QmxfZqeniayzgsbM386PrfToZB6EW6No3L GLqehrw3r3eUFkpV7t9abtDRq/oMkUM7eRdp22VLHbaTKtOuolWFEpVJjJ/RE6Cmg0FVAP kbpCQ8GKlTWSkOMJeKyz81cmR2EpxgE= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PSv6Vy52; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675856183; a=rsa-sha256; cv=none; b=yCC6hh3YdXyqj4DXvZb0oYLwtJjBn6G8dwsXbVBRjL7qzKtU5iJVnwmn50SV6AJ3Z8ZbYu tdSRFjpmY7LAELMjZcY8UwXFkyFeZd2tK9yuP29ipFQpbCOJdZuewBgNSBQnkNlXXBODXv 3WWgeouDpjrQI3UtSpIEsiw8lr79auQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675856183; x=1707392183; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=8U1cq6u571MOpBdauFJ+N72lphdj+XDpLYcB04WZZqM=; b=PSv6Vy528gOXVe5Afgem+snqCcgyDeiB2um3DZXwL0hDP/q8FAs5luD5 AFH/69aE5dsSzz7raZneEx+uXBHbBWYUdhSgRo1QpgaTHn0MBhDU8T8vM ORqM1Tt+CUlqSfKMetyMHaApP7lkYn97Ywe2Jp8fo/LcbAIkLDkzpHgYi jfCixK6V2e23rnweiGFEYpkAmV7PX6aKFcUmLVVqyEX7QFQraR0dFwNA3 TasxsjfvH9QKpAkrQuV8fqu6/2SDi1tP26hnDcud3nNcYv1ePBRMMa4YO r20R3Ht3//exE47qtRjCc1FYSjMExcHL5VyQedUWwxfXUhwFEzMcUKR3/ Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10614"; a="415995797" X-IronPort-AV: E=Sophos;i="5.97,280,1669104000"; d="scan'208";a="415995797" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2023 03:36:03 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10614"; a="841149099" X-IronPort-AV: E=Sophos;i="5.97,280,1669104000"; d="scan'208";a="841149099" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2023 03:35:59 -0800 From: "Huang, Ying" To: haoxin Cc: Andrew Morton , , , Baolin Wang , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: Re: [PATCH -v4 2/9] migrate_pages: separate hugetlb folios migration References: <20230206063313.635011-1-ying.huang@intel.com> <20230206063313.635011-3-ying.huang@intel.com> <840f6621-5a43-9cc2-6620-6d149ea712cd@linux.alibaba.com> Date: Wed, 08 Feb 2023 19:35:10 +0800 In-Reply-To: <840f6621-5a43-9cc2-6620-6d149ea712cd@linux.alibaba.com> (haoxin's message of "Wed, 8 Feb 2023 00:42:11 +0800") Message-ID: <874jrwgzkh.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 8851418001B X-Stat-Signature: xm8d87ieo8zjpy7zqgwmdm5613qqo9xx X-HE-Tag: 1675856183-284599 X-HE-Meta: U2FsdGVkX19ecj8JXk+ayyOskeUJN4SCDRUwp5AMRuYlKPIk7S6F9vI4etmcoRorIJJ2HA2lciXZjlHldbIEanw0Kna4C6chSn6GlmnXHGS+bp27tUNFF1N4MKkT84g6dh3pQlyy5Iekf+UG2106PALuthFJxTvEdY8VMojeiBYh43h+MaF/8VSyEn5umMw2j2vHMZOie54nU4D7UnEgbY57cPy3fbb0EHEf981IUVIq+NiCrMrQDex5ATNEERtdViaDbCBB8YG/7F3gPWW4ysD67GgXhrIaLmOcaZuOYxLFyRsyxQXE8v3SwwF6S1MfmdpUeSf+a3jizZuI2EUgHIQWID0GnzbI5VD4dxsESCjwcYx1i9YxT2P3YV2y7HBVu7bS+DBN5AFMYy/xuH1SxPYdBiyP0ehmv8qPtawzIzJMi2LA4YUMeob5NPHBM/pf29dmHWJu9d/KtFlUbgtjmyXX0tN1NKQBZeGXIi2a837SYkG9TEfL/iyqVdnFX8RoaFgNpXLxEbwTIZgr9z/6xIygGuzk3KvFHWsShOCoRFztG9MzvUWB+hB7+dT1qa9hQskvRtw8uQU8g2iaBrA7/Dq7b5WlS3pwVrGxgCB80gzGgVEwPmHwQcxzv7Ou/JZk8pL6BzEJJrN+aCXFKSyZm7kQK4ZxQOliU1WBuP2qzKRYze50yayUp/RuiKOmZgZnZ/9PzLX+vNuOjNq7vPaog7M2ZZKwSzzgwYjZkrTIBQQhKB9MR5EHYnN5ll8mnkT30LipsTX8DC5x0BAMe2ws4EAy/ZgyBl5uMXRKB/26akjuWTTWiP2aGtrL4lilG/pMAPitIBBLPgcJCLebmrE/nnuL7xUL59rYwWB+qYAxH0QJYWcedRdSTTirGgqavB4GeZ+t1GI7rhbvYWazGeE+fHErO5OlwsgyWOV/Cy0sUX+ad0BxuV2wHmBMqvvWDRbd1hLtR86DkbGCi43uDMM 7WaEeBlj /icvxiXxZsJg5Q+VIjkkpIWjjKLo4InF0D1woPkTCpSZqGYd0aJEMeNs7monoaw1zSm3HpIFWUFmLnaVrotjciZsvt/nYe4/8iPg5FHUsfsQI2tpMYe9MEXyPS9T7evo6iL/huF1vyGBK34GlOY6yM/Lrn6/7QeRfF2jlDfYzQlO2vCUqMk9mnHCXIYVd7U+GKeAXoRlfjsQaQmFf6qliGXgVf2JTFt32d8lFbIiW1v6GTvqAl3yw0ik2MfivDf6c22S3giL3zso27yfOZgNHGie9mEDWYJFubNagdbJgfy/yDJ+HW2sZHUffHMtrU3BT5kUAerx5LTZosw4NPeXRUdvd4sLD1LqeZDFZUTaEzS6irS8RW9oURxm9gLngsPjDXhy6PHSQfLFZB1kYwRfLxp2od58OAT/ZUc0nCqdpCxPabkRgp9+ciNyHtrWyypYfJIzRCYrAwZG8UXBCA0nKkVotZWyMedAS9VaA4XL/SfBMyyH9ncvEJfHihm6XGvYkwlGGh/f5pPFi0fQhVvVhAEFHKIckN2tMGeGE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: haoxin writes: > ( 2023/2/6 H2:33, Huang Ying S: >> This is a preparation patch to batch the folio unmapping and moving >> for the non-hugetlb folios. Based on that we can batch the TLB >> shootdown during the folio migration and make it possible to use some >> hardware accelerator for the folio copying. >> >> In this patch the hugetlb folios and non-hugetlb folios migration is >> separated in migrate_pages() to make it easy to change the non-hugetlb >> folios migration implementation. >> >> Signed-off-by: "Huang, Ying" >> Reviewed-by: Baolin Wang >> Cc: Zi Yan >> Cc: Yang Shi >> Cc: Oscar Salvador >> Cc: Matthew Wilcox >> Cc: Bharata B Rao >> Cc: Alistair Popple >> Cc: haoxin >> Cc: Minchan Kim >> Cc: Mike Kravetz >> Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> >> --- >> mm/migrate.c | 141 +++++++++++++++++++++++++++++++++++++++++++-------- >> 1 file changed, 119 insertions(+), 22 deletions(-) >> >> diff --git a/mm/migrate.c b/mm/migrate.c >> index ef388a9e4747..be7f37523463 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -1396,6 +1396,8 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f >> return rc; >> } >> +#define NR_MAX_MIGRATE_PAGES_RETRY 10 >> + >> struct migrate_pages_stats { >> int nr_succeeded; /* Normal and large folios migrated successfully, in >> units of base pages */ >> @@ -1406,6 +1408,95 @@ struct migrate_pages_stats { >> int nr_thp_split; /* THP split before migrating */ >> }; >> +/* >> + * Returns the number of hugetlb folios that were not migrated, or an error code >> + * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable >> + * any more because the list has become empty or no retryable hugetlb folios >> + * exist any more. It is caller's responsibility to call putback_movable_pages() >> + * only if ret != 0. >> + */ >> +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, >> + free_page_t put_new_page, unsigned long private, >> + enum migrate_mode mode, int reason, >> + struct migrate_pages_stats *stats, >> + struct list_head *ret_folios) >> +{ >> + int retry = 1; >> + int nr_failed = 0; >> + int nr_retry_pages = 0; >> + int pass = 0; >> + struct folio *folio, *folio2; >> + int rc, nr_pages; >> + >> + for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) { >> + retry = 0; >> + nr_retry_pages = 0; >> + >> + list_for_each_entry_safe(folio, folio2, from, lru) { >> + if (!folio_test_hugetlb(folio)) >> + continue; >> + >> + nr_pages = folio_nr_pages(folio); >> + >> + cond_resched(); > Just curious, why put cond_resched() here, it makes > "nr_pages = folio_nr_pages(folio)" looks Separately with below > codes. This is the original behavior. Per my understanding, this can reduce the schedule latency of the page migration. >> + >> + rc = unmap_and_move_huge_page(get_new_page, >> + put_new_page, private, >> + &folio->page, pass > 2, mode, >> + reason, ret_folios); >> + /* >> + * The rules are: >> + * Success: hugetlb folio will be put back >> + * -EAGAIN: stay on the from list >> + * -ENOMEM: stay on the from list >> + * -ENOSYS: stay on the from list >> + * Other errno: put on ret_folios list >> + */ >> + switch(rc) { >> + case -ENOSYS: >> + /* Hugetlb migration is unsupported */ >> + nr_failed++; >> + stats->nr_failed_pages += nr_pages; >> + list_move_tail(&folio->lru, ret_folios); >> + break; >> + case -ENOMEM: >> + /* >> + * When memory is low, don't bother to try to migrate >> + * other folios, just exit. >> + */ >> + stats->nr_failed_pages += nr_pages + nr_retry_pages; >> + return -ENOMEM; >> + case -EAGAIN: >> + retry++; >> + nr_retry_pages += nr_pages; >> + break; >> + case MIGRATEPAGE_SUCCESS: >> + stats->nr_succeeded += nr_pages; >> + break; >> + default: >> + /* >> + * Permanent failure (-EBUSY, etc.): >> + * unlike -EAGAIN case, the failed folio is >> + * removed from migration folio list and not >> + * retried in the next outer loop. >> + */ >> + nr_failed++; >> + stats->nr_failed_pages += nr_pages; >> + break; >> + } >> + } >> + } >> + /* >> + * nr_failed is number of hugetlb folios failed to be migrated. After >> + * NR_MAX_MIGRATE_PAGES_RETRY attempts, give up and count retried hugetlb >> + * folios as failed. >> + */ >> + nr_failed += retry; >> + stats->nr_failed_pages += nr_retry_pages; >> + >> + return nr_failed; >> +} >> + >> /* >> * migrate_pages - migrate the folios specified in a list, to the free folios >> * supplied as the target for the page migration >> @@ -1422,10 +1513,10 @@ struct migrate_pages_stats { >> * @ret_succeeded: Set to the number of folios migrated successfully if >> * the caller passes a non-NULL pointer. >> * >> - * The function returns after 10 attempts or if no folios are movable any more >> - * because the list has become empty or no retryable folios exist any more. >> - * It is caller's responsibility to call putback_movable_pages() to return folios >> - * to the LRU or free list only if ret != 0. >> + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios >> + * are movable any more because the list has become empty or no retryable folios >> + * exist any more. It is caller's responsibility to call putback_movable_pages() >> + * only if ret != 0. >> * >> * Returns the number of {normal folio, large folio, hugetlb} that were not >> * migrated, or an error code. The number of large folio splits will be >> @@ -1439,7 +1530,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> int retry = 1; >> int large_retry = 1; >> int thp_retry = 1; >> - int nr_failed = 0; >> + int nr_failed; >> int nr_retry_pages = 0; >> int nr_large_failed = 0; >> int pass = 0; >> @@ -1456,38 +1547,45 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> trace_mm_migrate_pages_start(mode, reason); >> memset(&stats, 0, sizeof(stats)); >> + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, >> + &stats, &ret_folios); >> + if (rc < 0) >> + goto out; >> + nr_failed = rc; >> + >> split_folio_migration: >> - for (pass = 0; pass < 10 && (retry || large_retry); pass++) { >> + for (pass = 0; >> + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); >> + pass++) { >> retry = 0; >> large_retry = 0; >> thp_retry = 0; >> nr_retry_pages = 0; >> list_for_each_entry_safe(folio, folio2, from, lru) { >> + /* Retried hugetlb folios will be kept in list */ >> + if (folio_test_hugetlb(folio)) { >> + list_move_tail(&folio->lru, &ret_folios); >> + continue; >> + } >> + >> /* >> * Large folio statistics is based on the source large >> * folio. Capture required information that might get >> * lost during migration. >> */ >> - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); >> + is_large = folio_test_large(folio); >> is_thp = is_large && folio_test_pmd_mappable(folio); >> nr_pages = folio_nr_pages(folio); >> + >> cond_resched(); >> - if (folio_test_hugetlb(folio)) >> - rc = unmap_and_move_huge_page(get_new_page, >> - put_new_page, private, >> - &folio->page, pass > 2, mode, >> - reason, >> - &ret_folios); >> - else >> - rc = unmap_and_move(get_new_page, put_new_page, >> - private, folio, pass > 2, mode, >> - reason, &ret_folios); >> + rc = unmap_and_move(get_new_page, put_new_page, >> + private, folio, pass > 2, mode, >> + reason, &ret_folios); >> /* >> * The rules are: >> - * Success: non hugetlb folio will be freed, hugetlb >> - * folio will be put back >> + * Success: folio will be freed >> * -EAGAIN: stay on the from list >> * -ENOMEM: stay on the from list >> * -ENOSYS: stay on the from list >> @@ -1514,7 +1612,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> stats.nr_thp_split += is_thp; >> break; >> } >> - /* Hugetlb migration is unsupported */ >> } else if (!no_split_folio_counting) { >> nr_failed++; >> } >> @@ -1608,8 +1705,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> */ >> if (!list_empty(&split_folios)) { >> /* >> - * Move non-migrated folios (after 10 retries) to ret_folios >> - * to avoid migrating them again. >> + * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY >> + * retries) to ret_folios to avoid migrating them again. >> */ >> list_splice_init(from, &ret_folios); >> list_splice_init(&split_folios, from); > Reviewed-by: Xin Hao Thanks! Best Regards, Huang, Ying