From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C8EC48BC4 for ; Tue, 20 Feb 2024 08:03:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 45A3A6B0092; Tue, 20 Feb 2024 03:03:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E3BB6B0095; Tue, 20 Feb 2024 03:03:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25F226B0096; Tue, 20 Feb 2024 03:03:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 107B26B0092 for ; Tue, 20 Feb 2024 03:03:06 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D7159A074E for ; Tue, 20 Feb 2024 08:03:05 +0000 (UTC) X-FDA: 81811441530.04.1A8B4D1 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.16]) by imf13.hostedemail.com (Postfix) with ESMTP id B0C8520016 for ; Tue, 20 Feb 2024 08:03:03 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Gr+edJz3; spf=pass (imf13.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.16 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708416184; a=rsa-sha256; cv=none; b=z1VPm0Usaj1IlVe0Gqs+kwozbzvXoZGftAKeZWgEx+ghSkZRRqVfUAXB+Tc3ytfIXX93WC X72orWcIH/tDmJpMiDnZZX9ta1vGomrS8TI3Kk7KwP3Y8M3i7PDGOJ7meIAE1bRBmfM4QX yaYEqCpn3WeZvNpTCCkDSIq/OHeV38E= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Gr+edJz3; spf=pass (imf13.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.16 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708416184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WxQ6YogDNmAzMEZL1VnykgiW0tKzHbOSneNCH4+jLnY=; b=uRsw2jc/Eqwt2If9vNVAn06kawksZG4CTAIbQJJnHsc6jCY2xjajo21HMxNTW/sSIhpr09 g9g1N4F0NUZBEe0OqXUYLddKT2xmx99F4I5ldbUiqqLFHw+WCuKuEcoZ9CUBokVzIKqPRS Tdyuo5THe8JhuTeyCWhAf9vzL/xaJqY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1708416184; x=1739952184; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=U6BfJQ3lOKorfCB2xQScU7DGVW3UzgWVQwNkzflFuAA=; b=Gr+edJz3AGphGS128roybHp44N00VvTawg+yDj6s2ACcgd4vVERSM1+E ktO7eFDI0DcB85DErP/X0MqKMZQvMux6EnsfZWuydj87ZOfq3dNG3jPbY Gq3HKJLeEW9SoEHn8HTKGd+aCZedAx3YfCQgmAs2oodmH3sX4qE6L/CE/ 3HjdAc4ZAVMHfQrMb/l8LwDsivqXA7tNCxL1U1xQuyMuN03xvWmndRAke OoWCamY/9BJTOC5Q54T6zhW7Fp6P/PWSrKggMnN+fzwqBVCg8v8UQQr3b vBbXMvdGnqN5xQFI+VSQihXtBu8Z1pg8B8MeqkTzJxPeuCbgqzF955u2q g==; X-IronPort-AV: E=McAfee;i="6600,9927,10989"; a="2965042" X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="2965042" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by fmvoesa110.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Feb 2024 00:03:02 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.06,172,1705392000"; d="scan'208";a="4605179" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Feb 2024 00:02:56 -0800 From: "Huang, Ying" To: Aneesh Kumar K.V Cc: Donet Tom , Michal Hocko , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dave Hansen , Mel Gorman , Feng Tang , Andrea Arcangeli , Peter Zijlstra , Ingo Molnar , Rik van Riel , Johannes Weiner , Matthew Wilcox , Mike Kravetz , Vlastimil Babka , Dan Williams , Hugh Dickins , Kefeng Wang , Suren Baghdasaryan Subject: Re: [PATCH 3/3] mm/numa_balancing:Allow migrate on protnone reference with MPOL_PREFERRED_MANY policy In-Reply-To: <87v86jzifo.fsf@kernel.org> (Aneesh Kumar K. V.'s message of "Tue, 20 Feb 2024 13:16:51 +0530") References: <9c3f7b743477560d1c5b12b8c111a584a2cc92ee.1708097962.git.donettom@linux.ibm.com> <8d7737208bd24e754dc7a538a3f7f02de84f1f72.1708097962.git.donettom@linux.ibm.com> <87bk8bprpr.fsf@yhuang6-desk2.ccr.corp.intel.com> <87y1bfoayd.fsf@yhuang6-desk2.ccr.corp.intel.com> <87v86jzifo.fsf@kernel.org> Date: Tue, 20 Feb 2024 16:01:01 +0800 Message-ID: <87plwro98i.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B0C8520016 X-Stat-Signature: casm1uzfco9t64mdg5re1c93p6g76wfk X-Rspam-User: X-HE-Tag: 1708416183-596798 X-HE-Meta: U2FsdGVkX186B5OB2rWcvObf93FNgPSQw7H9NW+I/uPW/0X6c7awRGRadN3jz1wtw50BIO5ci2F799aG5j0VrVbgp3EeAJA03G4ZYFNd+qlgV6Jm+0gR7O2gH7f5Re4pIEXBxpdCb3ZqFpCOUawXo2mXHPn7WRQSzg+XUhVtSgDKw1GBt49SOCGK7VfL9UkCvkvK279EDjfkt+6ADgm2QjbyRwAtKs72dXg+K2BQKBXtH/e+Rn6ZlwsTx7lcamjGJ8CQDZwtDvz6abCh+2SRqOA3vigksbALxOCPNWTTogF1W+9f2KCUf+Jz2K8rSkB1p8kF57V7y8K9dQmDTO8c63fVGOTumA2Rvjgd7KOMPCJSi6PPzt67moW+VXXRx2+5SMYQDCsBpCSEz+VXjcrIArZBZ9VgasMUkFuMWLy7iavt32bR8/TVidkXkQB2e1E0X+ST0OoFWlWEj4SsPl8HmS7k3v9bM4G8tMkSV/+syNOg5XTzhzoYrNw4dJFHQiqkAB2HSBMsmd26nY7dqd5WP+kkd09zn8T1qnvIHZb+TGxpfqI3PDAptZ4LesIaEurPpz7NAwfPzKREZBbc3eGRxbaHs8882Ys8LYjaGcaQ0+wHaVMZTxYCZv4JLa9zKtY4VogLXghAV1f8GRXXnWBY7djjHN7NPTeGWbxojXyy3PGb4fqDTdBSlkrbA47f69XoAIGoB9S2RZnOKZUhFeJAMFsaEHJ7tepUG/jqY9tzs7NO1Y6pSzRJ74wz8caLYI+cBB0yBn5VcoHgW3WS5nNoHPmXP6jDqnhOl2RHK3BPXAiDrYyKAWct2NAAex/2I+qFAkZImB8WIDjeyWc7HTchtufmTS+QVUUWyDElZf/c3hO0nVSdszFTs9mypD4TcWtrgQaNBFdEsiZNh1TlQ9OvBEJo4lHvle/MTOKlQIaWvHJ8d5JkEcgjEjaAIabixn6TGHW3AvXTxbUynEyq447 Q/i6flsJ TNNMnQftjuTmjK/UZzObQnKlJUZnh4NI4O+Zlm/5oKI4BYQx3HcVLmETuWwyYqR9nQ3CWPYX+Mc7N4bgPxkbaodS8YcWA3jI3lg/jHaupb7u9FMjNFSppsLJH8xIi/97AKOuGTIjM9yf+X+RaMmi//X+UzyAVG9Av436wvHOqLbAVtm4N3iI4aORBMNnu/bCSlRuxuR/ZRnrGWzY91e+L6FaicPK2ILb0/yIfB/1YVdyUVijYB8VGpo8177sejUOMuSEC7gnvpIVNtF5nKagGBNISKsVFdiSsS+CzBp0qbh6H3wrPpXZIcv+Kt7bhBh6+fINiHCl0kADuM7tPURYsvhByM6ZsNne3OGHI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Aneesh Kumar K.V writes: > "Huang, Ying" writes: > >> "Aneesh Kumar K.V" writes: >> >>> On 2/20/24 12:06 PM, Huang, Ying wrote: >>>> Donet Tom writes: >>>>=20 >>>>> On 2/19/24 17:37, Michal Hocko wrote: >>>>>> On Sat 17-02-24 01:31:35, Donet Tom wrote: >>>>>>> commit bda420b98505 ("numa balancing: migrate on fault among multip= le bound >>>>>>> nodes") added support for migrate on protnone reference with MPOL_B= IND >>>>>>> memory policy. This allowed numa fault migration when the executing= node >>>>>>> is part of the policy mask for MPOL_BIND. This patch extends migrat= ion >>>>>>> support to MPOL_PREFERRED_MANY policy. >>>>>>> >>>>>>> Currently, we cannot specify MPOL_PREFERRED_MANY with the mempolicy= flag >>>>>>> MPOL_F_NUMA_BALANCING. This causes issues when we want to use >>>>>>> NUMA_BALANCING_MEMORY_TIERING. To effectively use the slow memory t= ier, >>>>>>> the kernel should not allocate pages from the slower memory tier via >>>>>>> allocation control zonelist fallback. Instead, we should move cold = pages >>>>>>> from the faster memory node via memory demotion. For a page allocat= ion, >>>>>>> kswapd is only woken up after we try to allocate pages from all nod= es in >>>>>>> the allocation zone list. This implies that, without using memory >>>>>>> policies, we will end up allocating hot pages in the slower memory = tier. >>>>>>> >>>>>>> MPOL_PREFERRED_MANY was added by commit b27abaccf8e8 ("mm/mempolicy= : add >>>>>>> MPOL_PREFERRED_MANY for multiple preferred nodes") to allow better >>>>>>> allocation control when we have memory tiers in the system. With >>>>>>> MPOL_PREFERRED_MANY, the user can use a policy node mask consisting= only >>>>>>> of faster memory nodes. When we fail to allocate pages from the fas= ter >>>>>>> memory node, kswapd would be woken up, allowing demotion of cold pa= ges >>>>>>> to slower memory nodes. >>>>>>> >>>>>>> With the current kernel, such usage of memory policies implies we c= an't >>>>>>> do page promotion from a slower memory tier to a faster memory tier >>>>>>> using numa fault. This patch fixes this issue. >>>>>>> >>>>>>> For MPOL_PREFERRED_MANY, if the executing node is in the policy node >>>>>>> mask, we allow numa migration to the executing nodes. If the execut= ing >>>>>>> node is not in the policy node mask but the folio is already alloca= ted >>>>>>> based on policy preference (the folio node is in the policy node ma= sk), >>>>>>> we don't allow numa migration. If both the executing node and folio= node >>>>>>> are outside the policy node mask, we allow numa migration to the >>>>>>> executing nodes. >>>>>> The feature makes sense to me. How has this been tested? Do you have= any >>>>>> numbers to present? >>>>> >>>>> Hi Michal >>>>> >>>>> I have a test program which allocate memory on a specified node and >>>>> trigger the promotion or migration (Keep accessing the pages). >>>>> >>>>> Without this patch if we set MPOL_PREFERRED_MANY promotion or migrati= on was not happening >>>>> with this patch I could see pages are getting migrated or promoted. >>>>> >>>>> My system has 2 CPU+DRAM node (Tier 1) and 1 PMEM node(Tier 2). Below >>>>> are my test results. >>>>> >>>>> In below table N0 and N1 are Tier1 Nodes. N6 is the Tier2 Node. >>>>> Exec_Node is the execution node, Policy is the nodes in nodemask and >>>>> "Curr Location Pages" is the node where pages present before migration >>>>> or promotion start. >>>>> >>>>> Tests Results >>>>> ------------------ >>>>> Scenario 1:=C2=A0 if the executing node is in the policy node mask >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >>>>> Exec_Node=C2=A0=C2=A0=C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 Curr Location Pages Observations >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N1 = N6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = N1 Pages Migrated from N1 to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 N0 N1 N6=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 N6 = Pages Promoted from N6 to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 N0 N1=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 = N1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 Pages M= igrated from N1 to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 N0 N1=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 = =C2=A0N6 =C2=A0 =C2=A0 Pages Promoted from N6 to N0 >>>>> >>>>> Scenario 2: If the folio node is in policy node mask and Exec node no= t in policy=C2=A0 node mask >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >>>>> Exec_Node=C2=A0=C2=A0=C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 Curr Location Pages=C2=A0=C2=A0=C2=A0 =C2=A0 Observations >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 N1 N6=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N1 = Pages are not Migrating to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N1 N6= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N6= Pages are not migration to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0N= 1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 =C2=A0N1 =C2=A0 =C2=A0 Pages are not Migrating to N0 >>>>> >>>>> Scenario 3: both the folio node and executing node are outside the po= licy nodemask >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >>>>> Exec_Node=C2=A0 =C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 Curr Location Pages=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Observati= ons >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 N1=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N6 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0 Page= s Promoted from N6 to N0 >>>>> N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0N6 N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0Pages Migrated from N1 to N0 >>>>> >>>>=20 >>>> Please use some benchmarks (e.g., redis + memtier) and show the >>>> proc-vmstat stats and benchamrk score. >>> >>> >>> Without this change numa fault migration is not supported with MPOL_PRE= FERRED_MANY >>> policy. So there is no performance comparison with and without patch. W= .r.t effectiveness of numa >>> fault migration, that is a different topic from this patch >> >> IIUC, the goal of the patch is to optimize performance, right? If so, >> the benchmark score will help justify the change. >> > > The objective is to enable the use of the MPOL_PREFERRED_MANY policy, > which is essential for the correct functioning of memory demotion in > conjunction with memory promotion. Once we can use memory promotion, we > should be able to observe the same benefits as those provided by numa > fault memory promotion. The actual benefit of numa fault migration is > dependent on various factors such as the speed of the slower memory > device, the access pattern of the application, etc. We are discussing > its effectiveness and how to improve numa fault overhead in other > forums. However, we believe that this discussion should not hinder the > merging of this patch. > > This change is similar to commit bda420b98505 ("numa balancing: migrate > on fault among multiple bound nodes") We provide the performance data in the description of that commit :-) -- Best Regards, Huang, Ying