From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8431EC64EC4 for ; Wed, 1 Mar 2023 03:08:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFE286B0071; Tue, 28 Feb 2023 22:08:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EAE9B6B0072; Tue, 28 Feb 2023 22:08:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D75CF6B0073; Tue, 28 Feb 2023 22:08:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C3CBB6B0071 for ; Tue, 28 Feb 2023 22:08:28 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8E224AA9F9 for ; Wed, 1 Mar 2023 03:08:28 +0000 (UTC) X-FDA: 80518846296.12.E34E143 Received: from out30-111.freemail.mail.aliyun.com (out30-111.freemail.mail.aliyun.com [115.124.30.111]) by imf04.hostedemail.com (Postfix) with ESMTP id 8D0F140007 for ; Wed, 1 Mar 2023 03:08:25 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf04.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.111 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677640107; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lfxaUdcNumSEUI39JYHXFBeo7SJbnvrbFYd5mLVAttg=; b=QaNHbOuT+P0Z5Gr2loMKbdSpI6BFJy/gH8KUGMb5g1egFdq327lMhvrbBOmb9TXgoCRSVx 2TSylubPX45tQdVDJVF9balWt0ILI+NTIX3V+4nND+IrRQNjSTdoY4JVdGNm1ofVcKnN3N 57Jj6jZnX0KHYoWnV5P4p1oy4Os6WmQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf04.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.111 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677640107; a=rsa-sha256; cv=none; b=YpNfgUN+Z1Cf5fOrIqbXMeAo4SeDhmjlSNgXs6rsP7gzo7z1MNnzfXp2SuESAX40fux1xZ Lg2UqwagifLUkVXrftJUEkBwL71jdBj7yfr1XmsWv9/Q6BsFv+R1ikCWrVdQ7UK21LuG3B xUDHt9lZbVwHXiu71Jjsnzoda0wqftw= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0VcmORIu_1677640100; Received: from 30.97.48.59(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VcmORIu_1677640100) by smtp.aliyun-inc.com; Wed, 01 Mar 2023 11:08:21 +0800 Message-ID: Date: Wed, 1 Mar 2023 11:08:26 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.7.2 Subject: Re: [PATCH 3/3] migrate_pages: try migrate in batch asynchronously firstly To: Huang Ying , Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Hugh Dickins , "Xu, Pengfei" , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Matthew Wilcox , Mike Kravetz References: <20230224141145.96814-1-ying.huang@intel.com> <20230224141145.96814-4-ying.huang@intel.com> From: Baolin Wang In-Reply-To: <20230224141145.96814-4-ying.huang@intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 8D0F140007 X-Stat-Signature: eokdu3xafskzaey4x1m1b189xz1f83qy X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1677640105-782697 X-HE-Meta: U2FsdGVkX1/zt4xxT/l9uNuQADMZPh2ihn6A3wWw6HVZ3Q3WdwnyNudJbLZOUzCsi0mVbdSXnTBvURKfQPidTGJCkunBLuRflXpLbH1HGyuOgYBUfz9qjFjKJjW5HtFitE7/GDgjGG7VJifu+UzqewiE1JhZnT8HSRrYVM+89XU0LCf15iYdub8dT/bMzD4fTwcnRmwqHO/BFXBPfDsTV29GPofRU9cJyy8PNCKEVEP/REqIIEOvEhJLVg+n0lk9umgrrv3yOAHqhP7F8nkHeI3yBwaUEJLHWv9qHUhdsxoKuwU8nbYwh+1SDnjZvOB11gudSrRr/v2VsSEIkCVcvCc9g7vRJ+H/FC+K1v0k3zplkibWW1IoJCEm/c3/att+OfnuoJvNp6HcEYULkFmOEjSadIXonGcf25t/HR9+qvgh0jaq7cypHMrxJ0vS6Lhbv0YD951QyX4LykC9qdHkpnJe2Vjbu+2aiev5Otd/EL9bbHQYbavZLVizhRfzwhsolnZg33W0ghgvkFqTYPlxqo33tygUbDdHbuHzPS0mTY8+Z1Tf4SfPQIvfPOXffGjSs5Lnlkl991WUC6nSYIhbnBusKNGcAJPDanhk7UgVaZeNT0TRVEbvUVulWA0yj0gPlsWvaEPWgJ/eCAO2agsQTWnhWHhbUdQl9NxM2mBmqs+/ydJvmlaGAd3yT/YNMRPO1To8FASjiwXHJ1t9n+csJ1mIV0yT2mG3eQtNvV3ef3ECg1q/4Lx9WAJPb4+2fex+B0hKYcEOCyGe9a0v4/6cRoAbcBODrtmtEUtTXED9JcgqstgUT72VzNoTFj4gSpnpj5iTPze3qmqDP74ukeapVJarQXU3JsK7nb/4t3jmpPSgZr+aLfCoaVR1Ej5iXDdIsPg8LxTqOWOSnkVPyDZ0ODWd/mMVMV5NQYqC6HjHbtcdCLSfW4RUkRnE9cNqUR8IFvdV7JOE1ZJWF1z5ApM Qtr4FSSO m1F1HFNRhloLEfQ3DTUy5EIdGicLmr+IBgCh3y2Nnom8SJeB+kNTPbfNz321Z3I919lP8KX751bD/gzfNKfYlJP2HPFBJ/a0qLH+y23M28qaCQghw/5XvQZWuHalbgv6EVCeUIrWFM0iE5bH4oSL0IW1a/dGZqaLkLn0CHCqwj1Dxa43CTOrz6GsG4poNAe6Q0EWrwi02Rxc9tC+JJ0hmB2WfL9lpezbUwiZoOrXkXczN0eniWQGUL5fufk514XySvBYo3N6PF8Ke02/cxHWTSZDY2dfs5Lvnvm+SmlqGeVSxuXJ7QI6KdfNMG1/tSuXL+H9vlDkM4q/ZVG2mGEH/F6mqoJtNq5rsK3v+CgfRUXoyOB3xZS/tLFhSl8Ne0D9w5an4V05rPAm8RSyb4iTqee5GMxjRwyiC/yPp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/24/2023 10:11 PM, Huang Ying wrote: > When we have locked more than one folios, we cannot wait the lock or > bit (e.g., page lock, buffer head lock, writeback bit) synchronously. > Otherwise deadlock may be triggered. This make it hard to batch the > synchronous migration directly. > > This patch re-enables batching synchronous migration via trying to > migrate in batch asynchronously firstly. And any folios that are > failed to be migrated asynchronously will be migrated synchronously > one by one. > > Test shows that this can restore the TLB flushing batching performance > for synchronous migration effectively. > > Signed-off-by: "Huang, Ying" > Cc: Hugh Dickins > Cc: "Xu, Pengfei" > Cc: Christoph Hellwig > Cc: Stefan Roesch > Cc: Tejun Heo > Cc: Xin Hao > Cc: Zi Yan > Cc: Yang Shi > Cc: Baolin Wang > Cc: Matthew Wilcox > Cc: Mike Kravetz > --- > mm/migrate.c | 65 ++++++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 55 insertions(+), 10 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 91198b487e49..c17ce5ee8d92 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1843,6 +1843,51 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, > return rc; > } > > +static int migrate_pages_sync(struct list_head *from, new_page_t get_new_page, > + free_page_t put_new_page, unsigned long private, > + enum migrate_mode mode, int reason, struct list_head *ret_folios, > + struct list_head *split_folios, struct migrate_pages_stats *stats) > +{ > + int rc, nr_failed = 0; > + LIST_HEAD(folios); > + struct migrate_pages_stats astats; > + > + memset(&astats, 0, sizeof(astats)); > + /* Try to migrate in batch with MIGRATE_ASYNC mode firstly */ > + rc = migrate_pages_batch(from, get_new_page, put_new_page, private, MIGRATE_ASYNC, > + reason, &folios, split_folios, &astats, > + NR_MAX_MIGRATE_PAGES_RETRY); > + stats->nr_succeeded += astats.nr_succeeded; > + stats->nr_thp_succeeded += astats.nr_thp_succeeded; > + stats->nr_thp_split += astats.nr_thp_split; > + if (rc < 0) { > + stats->nr_failed_pages += astats.nr_failed_pages; > + stats->nr_thp_failed += astats.nr_thp_failed; > + list_splice_tail(&folios, ret_folios); > + return rc; > + } > + stats->nr_thp_failed += astats.nr_thp_split; > + nr_failed += astats.nr_thp_split; > + /* > + * Fall back to migrate all failed folios one by one synchronously. All > + * failed folios except split THPs will be retried, so their failure > + * isn't counted > + */ > + list_splice_tail_init(&folios, from); > + while (!list_empty(from)) { > + list_move(from->next, &folios); > + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, > + private, mode, reason, ret_folios, > + split_folios, stats, NR_MAX_MIGRATE_PAGES_RETRY); > + list_splice_tail_init(&folios, ret_folios); > + if (rc < 0) > + return rc; > + nr_failed += rc; > + } > + > + return nr_failed; > +} > + > /* > * migrate_pages - migrate the folios specified in a list, to the free folios > * supplied as the target for the page migration > @@ -1874,7 +1919,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > enum migrate_mode mode, int reason, unsigned int *ret_succeeded) > { > int rc, rc_gather; > - int nr_pages, batch; > + int nr_pages; > struct folio *folio, *folio2; > LIST_HEAD(folios); > LIST_HEAD(ret_folios); > @@ -1890,10 +1935,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > if (rc_gather < 0) > goto out; > > - if (mode == MIGRATE_ASYNC) > - batch = NR_MAX_BATCHED_MIGRATION; > - else > - batch = 1; > again: > nr_pages = 0; > list_for_each_entry_safe(folio, folio2, from, lru) { > @@ -1904,16 +1945,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > } > > nr_pages += folio_nr_pages(folio); > - if (nr_pages >= batch) > + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) > break; > } > - if (nr_pages >= batch) > + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) > list_cut_before(&folios, from, &folio2->lru); > else > list_splice_init(from, &folios); > - rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, > - mode, reason, &ret_folios, &split_folios, &stats, > - NR_MAX_MIGRATE_PAGES_RETRY); > + if (mode == MIGRATE_ASYNC) > + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, > + mode, reason, &ret_folios, &split_folios, &stats, > + NR_MAX_MIGRATE_PAGES_RETRY); > + else > + rc = migrate_pages_sync(&folios, get_new_page, put_new_page, private, > + mode, reason, &ret_folios, &split_folios, &stats); For split folios, it seems also reasonable to use migrate_pages_sync() instead of always using fixed MIGRATE_ASYNC mode? > list_splice_tail_init(&folios, &ret_folios); > if (rc < 0) { > rc_gather = rc;