From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79E6CC54E58 for ; Mon, 25 Mar 2024 02:51:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1A476B0082; Sun, 24 Mar 2024 22:51:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA2EC6B0083; Sun, 24 Mar 2024 22:51:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1C996B0085; Sun, 24 Mar 2024 22:51:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8B34C6B0082 for ; Sun, 24 Mar 2024 22:51:01 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 557EC40777 for ; Mon, 25 Mar 2024 02:51:01 +0000 (UTC) X-FDA: 81934034322.05.E25301A Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf27.hostedemail.com (Postfix) with ESMTP id C9B7E40002 for ; Mon, 25 Mar 2024 02:50:58 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PeQKnxii; spf=pass (imf27.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711335059; a=rsa-sha256; cv=none; b=zBJFD1FUrgyQTwMzbWghvAyF5gTILtCGcEfFxjOXmNwz5wnt44N1m91mKCHdzEZ9bOSoUY /oTkgaTnfUtyeQ1EiF+3KPN4Eo37mLM4MAkQTvt+9U8wHEaQqLWhUEBnNLKAdcCpBKB5je LGjq8mYz63wVrh7URlvvB53+GZLBBJY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=PeQKnxii; spf=pass (imf27.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711335059; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sLxzYhXSlaEWABuGJ+dP9abTPX+Otro8XjOZlPvH+fs=; b=cAtbJlLwgiHEpx8Ov4+rAolxhKlT9EAUW9Yjw+GksjXkXyRZtMvrM5nbJraIPpBg8499Vz GCJPDlsWDBz09gtb74Wqa5VHgWQ+0xBvscFZ1UokcGloWjznLqSqVBGCXg8v7uf545sv7v Yvi30s1ZIeld9FYhmsDU1ip9dc3rHEo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711335059; x=1742871059; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=KtDX9c5FQBgaoOD7h5ZI/Q8jd/0Y1KkqpSUWOSPHmPU=; b=PeQKnxii9CyXAQeZK/g/WQ1oEE90CpgpWAxjjRcCrcERm66HV7C+83Nd AkRDSDCfbWzFKaJ0bXhmc6sBHPw4usNxxecUTp+JC760mkPr1SkfjktHZ CoFpLhaWATyakwZ5Q6PHMPU6LkHhxyxZAZcnUT7RV1Ofv2H0JhRoqX2tn ZN6/qwfvYrNNBk+d5YsOWKtEoRz+uebbuFHTROE9v33GRmqOd8snJwpKy kw6eSp1wtkur+/ypTdUnaVwIps1uxVTbjtbT6AmapyIIB8D8Y/P4/PE0K VcqnxuITd+i13kNvmbSy3Z0ZTQvjNipgMFHgG/tqoVSNQcu9g86vJKSgP A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="10113202" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="10113202" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:50:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15358991" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa010-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:50:52 -0700 From: "Huang, Ying" To: Donet Tom Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Aneesh Kumar , Michal Hocko , Dave Hansen , Mel Gorman , Feng Tang , Andrea Arcangeli , Peter Zijlstra , Ingo Molnar , Rik van Riel , Johannes Weiner , Matthew Wilcox , Vlastimil Babka , Dan Williams , Hugh Dickins , Kefeng Wang , Suren Baghdasaryan Subject: Re: [PATCH v3 2/2] mm/numa_balancing:Allow migrate on protnone reference with MPOL_PREFERRED_MANY policy In-Reply-To: (Donet Tom's message of "Fri, 22 Mar 2024 15:35:58 +0530") References: <87h6gyr7jf.fsf@yhuang6-desk2.ccr.corp.intel.com> Date: Mon, 25 Mar 2024 10:48:58 +0800 Message-ID: <875xxbqb51.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C9B7E40002 X-Stat-Signature: t8abd47jrtd3bymn4cd94qodoia7fzjk X-Rspam-User: X-HE-Tag: 1711335058-13676 X-HE-Meta: U2FsdGVkX1++xDlVW67InifDG5H3U73L9+r4qWHeG9+Gw/TrD3wzRxmOHohM9VmESSydracbjIkZIKYI2IIwVCiISHNM8YQJzIrgbR/bMDIbxUvwyiuC+q+RPWAzTKLCMV4PevcqFwSZGbpqfR86tlIo00uEMG4nU1rNuiqRA/mjWfI61OO7cn3htj5akhYpIFah7Fk0OTXrYpa20DpkpEYMIQAmxTsMdqrB8b1fY5PNJh059H8k3djlO9V7CJljmx9KQ8uVdATOyJnRJ5mP8cgYWLe3z4DMNbPYjdqXeeP6efxwB6A4nUre9MjNIiJ+NVGpL8FAfdON1CBfXbs89mTpat9qIQ0qDm9srHvB5uGGBkVDTkWSYwl/YXfQ6Hln65gt3vorsBwPVRukk55DJ+twhYx2UQ3g4J3M+vU10OgPhV4HFOa3uiXWC19gEHumJhXE6kkwBXKE75HT66cPbD2eeQbpR+CK9wVLqogDycJ/3cKYOXUzW1u1RMbxT/XQFvvr6TNyaLNShhuDSDdSSDL0k0NfBmuwzJHXpSeaiyDELmYlITEbFLtK4/iNGpw87xvC5ECncIMWOCmTOD6hAJCjao62MzmwUHTRUr1k/PJpb5lgWzcc56K4ksbf/YJvAWqt5BfoSAl+8EvsI4T1NKtQ1hBs55oVwVDbNmBeRFpKVYFwSqsn9+kQpu4Wemh+mRAY6ssbMZ4t7Yp8+jptHx53eseneRIkBTplOGs5kw1yn8/OFRSXisstxfzspy8EL3DqbNZbCpJ00gTkmL6Kg8Mrh1y1KmN0oc28BZB1qm0wY0HZ4/IR8GXNFKadhkJIL6qvBk9ANJqsXAy3Gc85zNCr/BCNGUQD/pTE4UcXbMYJWqYWe3vYaSxlyXkh+/OmeksrcaFRHRbpu/ir+jAFPx/puCiEFiRgp9PIW8ydU2xB3Rm0kxuG/+mNvAjvNIZBXhsBcxlhlsCqiGGKafD vAaD9fu0 3xkwSdkrJCbirO+0Zp49rotwwTkAgTABzTF9Yo4DpMY1c27mOx9AJR7H+GiB/wWN3b7KVAdRE+mo6FXY9A9mBATLS9iz8hoKjnhl7LkwuhNyXUdMaL0oHRC2mluRBlMDyUgQWUa3GSVJfzVnVZQwEz5rmHol86X90lOKwVD69QkHQLC9apyY+zSO8lFPy/dmEH2ANnQ4LO40s3l0fmJKdQ8Bl8H5FKMzEQrPl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Donet Tom writes: > On 3/22/24 14:02, Huang, Ying wrote: >> Donet Tom writes: >> >>> commit bda420b98505 ("numa balancing: migrate on fault among multiple b= ound >>> nodes") added support for migrate on protnone reference with MPOL_BIND >>> memory policy. This allowed numa fault migration when the executing node >>> is part of the policy mask for MPOL_BIND. This patch extends migration >>> support to MPOL_PREFERRED_MANY policy. >>> >>> Currently, we cannot specify MPOL_PREFERRED_MANY with the mempolicy flag >>> MPOL_F_NUMA_BALANCING. This causes issues when we want to use >>> NUMA_BALANCING_MEMORY_TIERING. To effectively use the slow memory tier, >>> the kernel should not allocate pages from the slower memory tier via >>> allocation control zonelist fallback. Instead, we should move cold pages >>> from the faster memory node via memory demotion. For a page allocation, >>> kswapd is only woken up after we try to allocate pages from all nodes in >>> the allocation zone list. This implies that, without using memory >>> policies, we will end up allocating hot pages in the slower memory tier. >>> >>> MPOL_PREFERRED_MANY was added by commit b27abaccf8e8 ("mm/mempolicy: add >>> MPOL_PREFERRED_MANY for multiple preferred nodes") to allow better >>> allocation control when we have memory tiers in the system. With >>> MPOL_PREFERRED_MANY, the user can use a policy node mask consisting only >>> of faster memory nodes. When we fail to allocate pages from the faster >>> memory node, kswapd would be woken up, allowing demotion of cold pages >>> to slower memory nodes. >>> >>> With the current kernel, such usage of memory policies implies we can't >>> do page promotion from a slower memory tier to a faster memory tier >>> using numa fault. This patch fixes this issue. >>> >>> For MPOL_PREFERRED_MANY, if the executing node is in the policy node >>> mask, we allow numa migration to the executing nodes. If the executing >>> node is not in the policy node mask, we do not allow numa migration. >> Can we provide more information about this? I suggest to use an >> example, for instance, pages may be distributed among multiple sockets >> unexpectedly. > > Thank you for your suggestion. However, this commit message explains all = the scenarios. Yes. The commit message is correct and covers many cases. What I suggested is to describe why we do that? An examples can not covers all possibility, but it is easy to be understood. For example, something as below? For example, on a 2-sockets system, there are N0, N1, N2 in socket 0, N3 in socket 1. N0, N1, N3 have fast memory and CPU, while N2 has slow memory and no CPU. For a workload, we may use MPOL_PREFERRED_MANY with nodemask with N0 and N1 set because the workload runs on CPUs of socket 0 at most times. Then, even if the workload runs on CPUs of N3 occasionally, we will not try to migrate the workload pages from N2 to N3 because users may want to avoid cross-socket access as much as possible in the long term. > For example, Consider a system with 3 numa nodes (N0,N1 and N6). > N0 and N1 are tier1 DRAM nodes=C2=A0 and N6 is tier 2 PMEM node. > > Scenario 1: The process is executing on N1, > If the executing node is in the policy node mask, > Curr Loc Pages - The numa node where page present(folio node) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > Process=C2=A0=C2=A0=C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Curr Loc Pages=C2=A0=C2=A0=C2=A0 =C2=A0 Observat= ions > -------------------------------------------------------------------------= ---------- > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N1 N6= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Pages Migrated from N0 to N1 > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N1 N6= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 N6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Pages Migrated from N6 to N1 > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N1=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0N1=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0 Pages Migrated from N1 to N6 Pages are not Migrating ? > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N1=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0N6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Pages Migrated= from N6 to N1 > -------------------------------------------------------------------------= ----------- > Scenario 2:=C2=A0 The process is executing on N1, > If the executing node is NOT in the policy node mask, > Curr Loc Pages - The numa node where page present(folio node) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D > Process=C2=A0=C2=A0=C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 C= urr Loc Pages=C2=A0=C2=A0=C2=A0 Observations > -------------------------------------------------------------------------= ---------- > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N= 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N= 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0Pages are not Migrating > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0 N= 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N= 6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0Pages are not migration, > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Pages are not Migrating > -------------------------------------------------------------------------= ----------- > > Scenario 3: The process is executing on N1, > If the executing node and folio nodes are=C2=A0 NOT in the po= licy node mask, > Curr Loc Pages - The numa node where page present (folio node) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Thread=C2=A0=C2=A0=C2=A0 Policy=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Curr = Loc Pages=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Observations > -------------------------------------------------------------------------= ----------- > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0N6=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Pages are not Migrating > N1=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 N6=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0N0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Pages are not Migrating > -------------------------------------------------------------------------= ----------- > > We can conclude that even if the pages are distributed among multiple soc= kets, > if the executing node is in the policy node mask, we allow numa migration= to the > executing nodes. If the executing node is not in the policy node mask, > we do not allow numa migration. > [snip] -- Best Regards, Huang, Ying