From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E091CC636CC for ; Wed, 8 Feb 2023 11:04:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37D256B0071; Wed, 8 Feb 2023 06:04:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 32D6F6B0072; Wed, 8 Feb 2023 06:04:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F5716B0073; Wed, 8 Feb 2023 06:04:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0E7776B0071 for ; Wed, 8 Feb 2023 06:04:44 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C0D8F120E0B for ; Wed, 8 Feb 2023 11:04:43 +0000 (UTC) X-FDA: 80443841646.28.58FA758 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf29.hostedemail.com (Postfix) with ESMTP id B928B12001A for ; Wed, 8 Feb 2023 11:04:40 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675854281; a=rsa-sha256; cv=none; b=EziPxtWfYMSHAZPpv7L/bdoQfsRsBC9hfkDlhDjwPW4+Xg9am5JTzR1TN9oxNiFCIzXWFV j1OFBqgY9ITO7i/e5M2QkJ9EOcSMlrF8XLLrwkrAQO4i+LbvG8oBlYe0Jwwi5HaJy/1clj US+OiTNejFepYxQvWacUA8/DE2dEwXs= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of jonathan.cameron@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=jonathan.cameron@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675854281; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vew/Qk6IGJAG+nHp45xSfiqd5WXinpAY5gAeDhzxVDY=; b=z9kntZz0FZSF5+mRXeDmDQaIkh9TNjDI21YFkHWOVutc3+uXokyKJjGR4vKYe02pC+LO9p H7Mf1EubqHfHGHX1fsnmBT4rV6pg0x1SgEQ8HricOPfqgthTnIP4jGlnYbGgDuMFlHfo8k ggEwQbPuYzqXPukfSgxVolpKpVc3r0U= Received: from lhrpeml500005.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4PBcV738t6z67j7Z; Wed, 8 Feb 2023 19:00:31 +0800 (CST) Received: from localhost (10.202.227.76) by lhrpeml500005.china.huawei.com (7.191.163.240) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.17; Wed, 8 Feb 2023 11:04:33 +0000 Date: Wed, 8 Feb 2023 11:04:32 +0000 From: Jonathan Cameron To: haoxin CC: Huang Ying , Andrew Morton , , , Zi Yan , Yang Shi , Baolin Wang , "Oscar Salvador" , Matthew Wilcox , Bharata B Rao , Alistair Popple , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Subject: Re: [PATCH -v4 0/9] migrate_pages(): batch TLB flushing Message-ID: <20230208110432.000011b5@Huawei.com> In-Reply-To: <2b15a6c8-67a4-8d93-09a4-cdc9f09e6b78@linux.alibaba.com> References: <20230206063313.635011-1-ying.huang@intel.com> <2b15a6c8-67a4-8d93-09a4-cdc9f09e6b78@linux.alibaba.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 4.1.0 (GTK 3.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.202.227.76] X-ClientProxiedBy: lhrpeml500003.china.huawei.com (7.191.162.67) To lhrpeml500005.china.huawei.com (7.191.163.240) X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Queue-Id: B928B12001A X-Rspamd-Server: rspam01 X-Stat-Signature: z1nwowau3mue99fjeqz93gny3xjby539 X-HE-Tag: 1675854280-268427 X-HE-Meta: U2FsdGVkX18dfGNRqNqhRh0Q5LbHWZzTEKEfsYKogPIKjJrLe8fcr7nEC1Vi07ovUEoOJ/g0ley+a346sRNNofa/d9Y/rwrWMhGUtj0QPEcBA+7Iosqd+E1H8SVSOxisA1gLXfZCqVOhYltDKw8ducL/dAp0SqNV+NmuWDLCnzpgnGeWUpZVGHmik5AlrttLKbGbMBFXHWvUd044Nkv29zFP0VRYQB+rfmEozml0hkwPti0NMpcWgjf9eSe2aWu/XdAGsorD2gfSkO3eoiEiPCSCWBblL2XaFJokdUCyvv18SRaeaeztPEudQ0giuOduvtcGK23BIznERloY9iEZ4+HArSjoY8khdY856u9+v94vDJt6NRp8L0vVcLZAHWR35hixHY1Los/q7EpB1cvnnM0QeDyOytJrzlnqY5XUiKb7wNW3CO1D91FBdfij0IP+HjgUFP5BE8Bp/l8xUZEDQBkfcgKJfxr/r7/UcVcqnXNfemWTzb0pgKuXICVSaRsLFqNougK5ApfhiUDbJ6VWkQWY9bUi7IyVooBcmkDWlMVVFcB8wWTWsJfuIRrV/lgLGvyUTJWiVK/Cw5Tk854h9BtNsb5+PfYSgwuoK4azLDdwQIv4R74vKmUkZsC+WifOxjhqmii8T1Oz6SwGMKF3/9GX64FDKpGIM1gh5HwcragyPqh2F/zm6r02phKcYKiaUwOuZP3u5mv5/fheQGg8evYy3Dk2xOQnjhcr5GIc2XSfFktngDxS7HMkrAUum4PKeNZVV3GLGN3lWXwUSi/el/Q9EifkqqPzWbxV2vC4ZDcXLAHMqARmaAFJEYJ+VegMIpmA4Eci3WRSCXJXup+rDbOER2NUZAqYT0jw7VmBk/4vfIXjQI9NhZoicJecgV0OWJj3J2RS3JJmwTMixnhGgbuFFyDAV7f2w8iuUFa+pbLVCmQsNiM8R8Qpfz+KXkFwhyrvdt5hnRd8YetntN1 MK73v7bF 3jNLnogyvLjUpph0MGEv88deUp9p3W3utllPMZFHsDAJYwIDtioSIe9sXqIXt1O4mkKp5OKBRqmha/IySRgi4tJ9j9fj14hYNyZuH9nT2qoLZm6bWHGEm/6p2+xXkKwXgD4qTYL/TdLQ4O/txGRSupMkRJmavJ3LCABpUpdtnrrMQCG04zNsZGZY9f+IWFkRWdcD2LQ9/jy7A52Ha5i6H73TFpx47icyLQYFLqwmy4VrVURBXW3/LG2cVUU/zhCBqD/YtJn5xO54oSFYUkkfXQptofH5ENxgKRAAfRb47j+tZji+7OB4b+vJECvf8BvcoCjAJrvD6fcNpzMImOfuS1CqfhBi3r/fLeDPMHxTxIwjrAajPyRH1L0xQU4kY7axorPc0q5ZttQchJJzZyaVaCE/baFpWH8z4xcgLxht4THQpn0rnOcxmG7ixmhOJBUqsPM0YoH/YFzH31nQWsqpUX1EsHw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 8 Feb 2023 14:27:32 +0800 haoxin wrote: > =E5=9C=A8 2023/2/8 =E4=B8=8B=E5=8D=882:21, haoxin =E5=86=99=E9=81=93: > > > > On my arm64 server with 128 cores, 2 numa nodes. > > > > I used memhog as benchmark : > > > > =C2=A0=C2=A0=C2=A0 numactl -m -C 5 memhog -r100000 1G > > =20 > =C2=A0=C2=A0=C2=A0 Do a fix, numactl -m 0 -C 5 memhog -r100000 1G Nice results - thanks for sharing. Just to confirm, is this with this tlb batching on arm64 patch set? https://lore.kernel.org/linux-arm-kernel/20221117082648.47526-1-yangyicong@= huawei.com/ I think that still hasn't been applied upstream. Jonathan >=20 > > The test result as below: > > > > =C2=A0With this patch: > > > > =C2=A0=C2=A0=C2=A0 #time migratepages 8490 0 1 > > > > =C2=A0=C2=A0=C2=A0 real 0m1.161s > > > > =C2=A0=C2=A0=C2=A0 user 0m0.000s > > > > =C2=A0=C2=A0=C2=A0 sys 0m1.161s > > > > without this patch: > > > > =C2=A0=C2=A0=C2=A0 #time migratepages 8460 0 1 > > > > =C2=A0=C2=A0=C2=A0 real 0m2.068s > > > > =C2=A0=C2=A0=C2=A0 user 0m0.001s > > > > =C2=A0=C2=A0=C2=A0 sys 0m2.068s > > > > So you can see the migration performance improvement about *+78%* > > > > This is the perf record info. > > > > w/o > > +=C2=A0=C2=A0 51.07%=C2=A0=C2=A0=C2=A0=C2=A0 0.09%=C2=A0 migratepages= =C2=A0 [kernel.kallsyms]=C2=A0 [k]=20 > > migrate_folio_extra > > +=C2=A0=C2=A0 42.43%=C2=A0=C2=A0=C2=A0=C2=A0 0.04%=C2=A0 migratepages= =C2=A0 [kernel.kallsyms]=C2=A0 [k] folio_copy > > +=C2=A0=C2=A0 42.34%=C2=A0=C2=A0=C2=A0 42.34%=C2=A0 migratepages=C2=A0 = [kernel.kallsyms]=C2=A0 [k] __pi_copy_page > > +=C2=A0=C2=A0 33.99%=C2=A0=C2=A0=C2=A0=C2=A0 0.09%=C2=A0 migratepages= =C2=A0 [kernel.kallsyms]=C2=A0 [k] rmap_walk_anon > > +=C2=A0=C2=A0 32.35%=C2=A0=C2=A0=C2=A0=C2=A0 0.04%=C2=A0 migratepages= =C2=A0 [kernel.kallsyms]=C2=A0 [k] try_to_migrate > > *+=C2=A0=C2=A0 27.78%=C2=A0=C2=A0=C2=A0 27.78%=C2=A0 migratepages=C2=A0= [kernel.kallsyms]=C2=A0 [k]=20 > > ptep_clear_flush * > > +=C2=A0=C2=A0=C2=A0 8.19%=C2=A0=C2=A0=C2=A0=C2=A0 6.64%=C2=A0 migratepa= ges=C2=A0 [kernel.kallsyms]=C2=A0 [k]=20 > > folio_migrate_flagsmigrati_tlb_flush > > > > w/ this patch > > +=C2=A0=C2=A0 18.57%=C2=A0=C2=A0=C2=A0=C2=A0 0.13%=C2=A0 migratepages= =C2=A0=C2=A0=C2=A0=C2=A0 [kernel.kallsyms]=C2=A0=C2=A0 [k]=20 > > migrate_pages > > +=C2=A0=C2=A0 18.23%=C2=A0=C2=A0=C2=A0=C2=A0 0.07%=C2=A0 migratepages= =C2=A0=C2=A0=C2=A0=C2=A0 [kernel.kallsyms]=C2=A0=C2=A0 [k]=20 > > migrate_pages_batch > > +=C2=A0=C2=A0 16.29%=C2=A0=C2=A0=C2=A0=C2=A0 0.13%=C2=A0 migratepages= =C2=A0=C2=A0=C2=A0=C2=A0 [kernel.kallsyms]=C2=A0=C2=A0 [k]=20 > > migrate_folio_move > > +=C2=A0=C2=A0 12.73%=C2=A0=C2=A0=C2=A0=C2=A0 0.10%=C2=A0 migratepages= =C2=A0=C2=A0=C2=A0=C2=A0 [kernel.kallsyms]=C2=A0=C2=A0 [k]=20 > > move_to_new_folio > > +=C2=A0=C2=A0 12.52%=C2=A0=C2=A0=C2=A0=C2=A0 0.06%=C2=A0 migratepages= =C2=A0=C2=A0=C2=A0=C2=A0 [kernel.kallsyms]=C2=A0=C2=A0 [k]=20 > > migrate_folio_extra > > > > Therefore, this patch helps improve performance in page migration > > > > > > So,=C2=A0 you can add Tested-by: Xin Hao > > > > > > =E5=9C=A8 2023/2/6 =E4=B8=8B=E5=8D=882:33, Huang Ying =E5=86=99=E9=81= =93: =20 > >> From: "Huang, Ying" > >> > >> Now, migrate_pages() migrate folios one by one, like the fake code as > >> follows, > >> > >> for each folio > >> unmap > >> flush TLB > >> copy > >> restore map > >> > >> If multiple folios are passed to migrate_pages(), there are > >> opportunities to batch the TLB flushing and copying. That is, we can > >> change the code to something as follows, > >> > >> for each folio > >> unmap > >> for each folio > >> flush TLB > >> for each folio > >> copy > >> for each folio > >> restore map > >> > >> The total number of TLB flushing IPI can be reduced considerably. And > >> we may use some hardware accelerator such as DSA to accelerate the > >> folio copying. > >> > >> So in this patch, we refactor the migrate_pages() implementation and > >> implement the TLB flushing batching. Base on this, hardware > >> accelerated folio copying can be implemented. > >> > >> If too many folios are passed to migrate_pages(), in the naive batched > >> implementation, we may unmap too many folios at the same time. The > >> possibility for a task to wait for the migrated folios to be mapped > >> again increases. So the latency may be hurt. To deal with this > >> issue, the max number of folios be unmapped in batch is restricted to > >> no more than HPAGE_PMD_NR in the unit of page. That is, the influence > >> is at the same level of THP migration. > >> > >> We use the following test to measure the performance impact of the > >> patchset, > >> > >> On a 2-socket Intel server, > >> > >> - Run pmbench memory accessing benchmark > >> > >> - Run `migratepages` to migrate pages of pmbench between node 0 and > >> node 1 back and forth. > >> > >> With the patch, the TLB flushing IPI reduces 99.1% during the test and > >> the number of pages migrated successfully per second increases 291.7%. > >> > >> This patchset is based on v6.2-rc4. > >> > >> Changes: > >> > >> v4: > >> > >> - Fixed another bug about non-LRU folio migration. Thanks Hyeonggon! > >> > >> v3: > >> > >> - Rebased on v6.2-rc4 > >> > >> - Fixed a bug about non-LRU folio migration. Thanks Mike! > >> > >> - Fixed some comments. Thanks Baolin! > >> > >> - Collected reviewed-by. > >> > >> v2: > >> > >> - Rebased on v6.2-rc3 > >> > >> - Fixed type force cast warning. Thanks Kees! > >> > >> - Added more comments and cleaned up the code. Thanks Andrew, Zi, Ali= stair, Dan! > >> > >> - Collected reviewed-by. > >> > >> from rfc to v1: > >> > >> - Rebased on v6.2-rc1 > >> > >> - Fix the deadlock issue caused by locking multiple pages synchronously > >> per Alistair's comments. Thanks! > >> > >> - Fix the autonumabench panic per Rao's comments and fix. Thanks! > >> > >> - Other minor fixes per comments. Thanks! > >> > >> Best Regards, > >> Huang, Ying =20