From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 702B5C77B6F for ; Wed, 12 Apr 2023 15:26:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E915E6B0074; Wed, 12 Apr 2023 11:26:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E41126B0078; Wed, 12 Apr 2023 11:26:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE250900003; Wed, 12 Apr 2023 11:26:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BE5556B0074 for ; Wed, 12 Apr 2023 11:26:37 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8B3471C2D51 for ; Wed, 12 Apr 2023 15:26:37 +0000 (UTC) X-FDA: 80673116034.18.8A84A6A Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2074.outbound.protection.outlook.com [40.107.93.74]) by imf24.hostedemail.com (Postfix) with ESMTP id 24F83180008 for ; Wed, 12 Apr 2023 15:26:32 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=H9Kr8GZ6; spf=none (imf24.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.93.74) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681313193; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CSs5XGFPjmoIFUmSJc857gbTWVpZEm/ycVLSVId3Rjw=; b=BomdrLnOnfVmUECsMLa8RNJMD7SprLedBj94OJWnqgwLFiu/A+ZZpqJVfpmv0WItkOuw9A ebebwr4Qz+sarBRIyti91UYZ+LQWKcCAo49Q/rSOxH630V+6ql2cDtLBm96BzBGn3mDmaB bg4OALdRS2JHAQrn0nU9NCfthde7www= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=H9Kr8GZ6; spf=none (imf24.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.93.74) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1681313193; a=rsa-sha256; cv=pass; b=klZCdIXWk9A2ef96GCGsU5Nvak64fVwKDQcOWxdyEzMdwzFLayLYf51Bwip1aUk9wwhJ5n AmFt7SuD0L+2vk+1HvwzOgvr0BVa7+2CtKzVRRT7DQGIW4QkD+1Xo7QVBy0xDINyZfbVs5 10K/Uobi3YBsLeNq6o+eC3P8XmYs63U= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MX1RD087s0e6Ihf8MdOHi8nrXzrdoqpEKHzE4LfqpwfvtPOXnd4RXFNtGn6VzDBBr7h10qFyUtQv8FmKigYL/0WWa3kHcRu/1D5WL+sV2Ey3Q0DMBDI8vbGUFOq23gymfxG5/oIzhQSOu6Ztpf466CfhToBMbFBIP9h5cSuq1tkC8Z1swugbneHSPQqo0EWSyPBdYS+IVgOWSX77BcqvhaSsJ/HRoV6TJqN/8GGSAeeULbyUgpwLugqxveyyMZQt7PQgc2zbqUH8TvL6HGJHCP/37Tn9vxKvb7VKVrlWtR2HN/kvm6CxAbjyczjI3OOc5YRloRhjnNZu6TQu7Jff7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CSs5XGFPjmoIFUmSJc857gbTWVpZEm/ycVLSVId3Rjw=; b=B8u8xNphE1gEjTx5UaKnMfu9YjH/KmydD2Mo3aMHiBgBE8GH21sunFPKrnvnu/NjuUF53NP3ntYqF4o10Y3RoFuN0RdceqjUUxSKSaRYDZDkSeWD2KAOtj8zJlP9ajovceQe1NtZ0BhUjokqvSqbY9WBbINjAMSftoWxpm0VtsOgEVpnUArLWVEPbxvQO16X4UNUQM33Aa3EIkqVJJl41AVbbDOeNJ4LXK14pLya8AZHHRMZkzZzn5xZ4VqAzCAuTtueIifnHF6eFrl+GJXwlraSPgVpot7MippvnVIbngmKQjJbpIjoWM/EeuLw3brja6ajRvNHsG6E1tQI8hK7eA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CSs5XGFPjmoIFUmSJc857gbTWVpZEm/ycVLSVId3Rjw=; b=H9Kr8GZ6kkl0UWIr+Z+dOm2bp4HgMGxBpEj2s+Lt1GMWyF9C4VqiJCfkH9Sto1WdcsgLkKSiYur90MLBy2cdRni5jedKKHXCF5FNoLz/yzTiVw9yymCUSRYf+3cRHebLtSI3ASpHHblBorI1wcX+8qSe/4IA45ehEa7AedGHf/w= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by MW4PR17MB4777.namprd17.prod.outlook.com (2603:10b6:303:10a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6298.30; Wed, 12 Apr 2023 15:26:28 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7b97:62c3:4602:b47a]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::7b97:62c3:4602:b47a%6]) with mapi id 15.20.6277.036; Wed, 12 Apr 2023 15:26:28 +0000 Date: Wed, 12 Apr 2023 11:26:24 -0400 From: Gregory Price To: David Hildenbrand Cc: "Huang, Ying" , Dragan Stancevic , lsf-pc@lists.linux-foundation.org, nil-migration@lists.linux.dev, linux-cxl@vger.kernel.org, linux-mm@kvack.org Subject: Re: [LSF/MM/BPF TOPIC] BoF VM =?utf-8?Q?li?= =?utf-8?Q?ve_migration_over_CXL_memory=E2=80=8B?= Message-ID: References: <5d1156eb-02ae-a6cc-54bb-db3df3ca0597@stancevic.com> <87v8i22abl.fsf@yhuang6-desk2.ccr.corp.intel.com> <87bkjtzu7e.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: SJ0PR03CA0084.namprd03.prod.outlook.com (2603:10b6:a03:331::29) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|MW4PR17MB4777:EE_ X-MS-Office365-Filtering-Correlation-Id: e016d97d-855d-4e75-3c4f-08db3b6a490d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bM3u5fWC0VViSjMN0EhxpY3xgPtuspKtkN43ozYDcwEb1aGJ3PFtVhYweHi0lVIFABcSWikNPi+eCsOKS6Ovjx3+kahhkBEJJpnshthMs1Jd+G1qA61VR9LdNW/44khGp2zJfRVwJqUwfylSAvK5fToauZXNfkK4EqEvt/dOL3NlSGeSZbbOcEP+aCAFuuFOQmUzIKo7oDFarVoJmV+fuQVlVPZCwlkOV9c/xi4oClyKVfUBbt9KnZv6oTfAHR3Jitg4R0WBNdIu7+U7B/Yn+SlFE1j2iQvmT0UgAuXHyb4Go6FL7RvX/UMLzW3DsXZOTYWza+ngQJKsoJoZo7G88fjpfZgkcG/1He6zzk8TLih3o/0x/mfvxmLtwY63WCBIFE0ZHUyHUgsB3cciF4BIcjsZbZht2/+S+3YkcSonY3F7XPY8ibZCcl76BV5Ea49otXdMnmMuRNnejC0c7/n2sa46RfVQjPGubqI8VMSdoBF/jHsqbaB/nKZEQthYrKIPokFPMnodc0qPp9cz7VEJxWvIg2TqVOl/uV3anuewlIOU+wl+GzCO6BDDvnyNw2vL X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(136003)(39850400004)(396003)(376002)(346002)(366004)(451199021)(2616005)(6486002)(6666004)(478600001)(186003)(26005)(54906003)(6506007)(6512007)(53546011)(44832011)(2906002)(36756003)(38100700002)(5660300002)(6916009)(66946007)(4326008)(41300700001)(66556008)(66476007)(8936002)(316002)(86362001)(66899021);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zESlUvZf8vsH9q5gq/YgW64eGsUDFKNNFMcBmAUXYRR04Dfstg/YlXXoENT7?= =?us-ascii?Q?OYgnmz+eutyMETCN/NCLK4mKHoL92Q2nDWbSAH4PdyBUagS4+AzH4QZRGmyo?= =?us-ascii?Q?6aI6W/LBOnYwXpF0ztu8333rUqNw7yImRKTpY2GgvNtjBBko/m3/UYfNvnl8?= =?us-ascii?Q?BL0Tz7swW5Wrd9G6VEFFxpDJb3BdRqpS0Q7uNmDE19kvnpHRCYOHh9jkebMh?= =?us-ascii?Q?nAPzhLzYv5eRW7Y2sPcD1CCj6d/+fzmclW2mEbT0JrbMdrEXJnifgvwKPBWz?= =?us-ascii?Q?/qNoJNqRnDbea6KpyK9bXTYtOrRzqwWwZzxbd/D2SyxEcZLGeVIWK8EVC+w2?= =?us-ascii?Q?L4OtTFL47MWC45MwhZ29kPGuwdlOzGNIogZJ86MBo0MHMLlz5a/hS5NiYpHW?= =?us-ascii?Q?whAe9MSj1We5IvQO8F5DMlC7rkaLgjbVlQkFOidHO2bF6OGSw4nI53+ARPpC?= =?us-ascii?Q?YBGzRNTPRxrclTLwzcadbilLeLI4nkymeOuahFneDETjQ7MN2dIfTbGPm4In?= =?us-ascii?Q?/eL5zz660X6wOtSdzT5P3XNeX5xWLnGx9qPP4QB46fO1yLBrXv9xfASKQGta?= =?us-ascii?Q?PC8Vfke/Qh8fKK8btItWOJcX0dGTCz9+B5B2MpliAjJz1p4LsTvBZnHkxbaB?= =?us-ascii?Q?w0IwvdatBQ4J0i/A0kZejd8iIMJCi3PMClBhGs29o8yYzyErn6JMqV1kyI1l?= =?us-ascii?Q?naH4INjq1BOStAN8yfAsUyYMXIEu7Aw2Uyz2GYaPz3s86COt1i/WaBBp4Naj?= =?us-ascii?Q?XJ2KJlTeeEqg/gZQQb/uANfympNNy9jWtDIPQsd2KevpFjkptEphtKStYHGE?= =?us-ascii?Q?Bq6eP+lM1YjN/Msfl0M1qaeW8pJCqJ6fBmY5wyVK85x0mgRgFKplDo4Qq89m?= =?us-ascii?Q?zmQuN8rz80CRqDAUAJ5nx53/ojClGkNyRa0LJyEKJSQBQNtD8SxbfFt9877y?= =?us-ascii?Q?AqQsuWbtybRrYCANdmhjtv08mXmlyt9bFNJRiXDxXy9WDnuHFDh40syzpShQ?= =?us-ascii?Q?q03QWteL2Ql5pUaqYgCGzMqunKKdHcxpIShBVW21YZBSXH6mCpsEWcUCzutI?= =?us-ascii?Q?ygZnSaeB5tDGMYvDh0RGY2FCklDl2YLX3bighXDl/zS3ZgrEwe+eswYDfYZA?= =?us-ascii?Q?01bNa15liYOkG0KRiEmoBic0k9HZC8EydVFybtc0f/hB+6618qlMmzI4WVYx?= =?us-ascii?Q?YxtTtQ01JEaT6ToZyc8zoGLQtZsKDypMa6jcOxwhp3HLgbpU90PGncsoJorN?= =?us-ascii?Q?DE7AUX0/obBhGN6IFskq/GopwArRwsD8wUcBpLE4bZurwaPAd96eqGGfVNOB?= =?us-ascii?Q?2bQ8Mt6w0plMVPf3C1txRxGipPy/A5OcCnrF55IQTWOy9aMYIZekf7GXGk3a?= =?us-ascii?Q?q5lvbVfJniJH6w9U6Bzi+RP/e9C9XLD7+jh1z4ifmp0GTLNm7iG9jHdeOaUU?= =?us-ascii?Q?HMcM2jY/A7jlRwkUWbB4nUF8kzbhffdr8KhZAHvXnRdpX7HpvLtP/mP97EzQ?= =?us-ascii?Q?FWzDDQkfWRkV++jIHS/wqLlUbmYkpNLa0cC6eZsfQixA3DVX4G5lEL1CleVh?= =?us-ascii?Q?UShKDT2dO6ic4BHNc+43qLa5+pzpb/1e5qGBDEEqA/XNPfINOMPZDQd/c7kr?= =?us-ascii?Q?NA=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: e016d97d-855d-4e75-3c4f-08db3b6a490d X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2023 15:26:28.3974 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: WFgDb3rgIgjr/jCqmnNoW0gJQcGH4VfyHfxoUZw13rB5ftRHWsQxdNZakRbcv84L0t3eV175ndDV+9+0uNnfKbwVnM/fd6d604ryRF8/fyQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR17MB4777 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 24F83180008 X-Stat-Signature: bk5jegrypnde3fh3euxd1wwpmgcrpcgh X-HE-Tag: 1681313192-109685 X-HE-Meta: U2FsdGVkX19T9b4P3+Li0iKujYc7xhZq2A8wJ1nOLCNCL8PtjvJZDGdhxdVSbOpI8aw7KSIO5dySLDSnVUPTUXNsoT2iqTcmechJzfXsCOwS4YSRp0NB3KnlCWtXvRz3iwKITpGKD5tkC2EtnTS32X01DSPU9K6MYzFdPZ0iou/JgOAR/cG1Ei65+Qhr3OApV6AoW7MLhQr3CP0FCSIyK92VnT0XymR5r5WMeISRKrbeeVmd/fLprpGIj2u1sHlcyi0w57iUiv8nVqFCJnl+H7hY2CjFvnQdPC35DyS5PgFUSKIj+xfAJgCXDMcQCClZv/Q+0NFjwM71i71dMLJg45MaL3OjX288kXE4mRPgYdBNO/EzYzmzQpIlP7FHItPRW9JtsEFJsMC9G60mB0BB9QR3Ehsb3jd0KQdQGvyny/J1dlMjRafzK5TLtpvDPutHHutujHb1hb9l08nyODdbB6SUZEWF1za0OyhUIFrhVl45NDxRhR94PMHGmD4j1ipw7Mg95Gb3rYeSl5YkNTr40nB53/GWSwD28kkdDijfjUIE96rr3cAhoNs5jcWcwnE5DUIWf9dgnk3JSx8UyN25DrC7l57XLSI1kMe/PMCcojpD93wx2bwH/yyNB2fH5LFf2BELTngUqEVz5ADm4Mtj6FI6V3F4PCiLH8wDbPntTHNStv9497oU0Vf4u/fYD2ttnbgIexy53ok9A9xFvLh+6p9/O5ZwRnOYAFrqyoqVnuj67Pd36+anWK+dc4PdimG0OgCT5TSCWOy85PoEKCSErS6dlolsTOyFS9W/6cy3S5uJ3zDwl2z8DPEAHo33bJqNs4nQSrMz88qX6PDna1h90074aZfQO/q8qzC/rq4ZrCEgf8rOcqzf1aryB0iq/dZUHMz75IAFQvKTV6egpQnHWM3zuleoAr+TjzFk+zBHsJb/lNF5Tf+dsOe+u/xx57g0FTSWdGzQTp7cwaax+kA D2dIWLr1 ICKyuzfMCk88f4qzuynAXUgRkV2ecMRDBSz+1bkFJYj8NbbQCYb39tsrFYdMxCFRmfoSBvsqEmmRXkOOuAhs5OR5VdqafkNcSbWIK/o1+Jww+yUsotgH7JZxvCbDWFfWRiDAI9I8mLtU5HkJo4AaD7p6WNExtuY7F+Sf8Dm224P//3bEoLvXzhSIDlzyWhiWw1ZfLjIzE01uiWGBRBXvgywfUymyh4zcpwuE41JGhfTpr1c5NdxG/2JKvKlZd5glnXiK/zvov8nu6pQa26jh7E+eNVPjrs+04PohwhO3XN5Wqu/OnAJdi07kI8w2e5sKF30u/zvsmdo4lq8w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.293044, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Apr 12, 2023 at 10:38:04AM +0200, David Hildenbrand wrote: > On 12.04.23 04:54, Huang, Ying wrote: > > Gregory Price writes: > > > > > On Tue, Apr 11, 2023 at 02:37:50PM +0800, Huang, Ying wrote: > > > > Gregory Price writes: > > > > > > > > [snip] > > > > > > > > > 2. During the migration process, the memory needs to be forced not to be > > > > > migrated to another node by other means (tiering software, swap, > > > > > etc). The obvious way of doing this would be to migrate and > > > > > temporarily pin the page... but going back to problem #1 we see that > > > > > ZONE_MOVABLE and Pinning are mutually exclusive. So that's > > > > > troublesome. > > > > > > > > Can we use memory policy (cpusets, mbind(), set_mempolicy(), etc.) to > > > > avoid move pages out of CXL.mem node? Now, there are gaps in tiering, > > > > but I think it is fixable. > > > > > > > > Best Regards, > > > > Huang, Ying > > > > > > > > [snip] > > > > > > That feels like a hack/bodge rather than a proper solution to me. > > > > > > Maybe this is an affirmative argument for the creation of an EXMEM > > > zone. > > > > Let's start with requirements. What is the requirements for a new zone > > type? > > I'm stills scratching my head regarding this. I keep hearing all different > kind of statements that just add more confusions "we want it to be > hotunpluggable" "we want to allow for long-term pinning memory" "but we > still want it to be movable" "we want to place some unmovable allocations on > it". Huh? > > Just to clarify: ZONE_MOVABLE allows for pinning. It just doesn't allow for > long-term pinning of memory. > I apologize for the confusion, this is my fault. I had assumed that since dax regions can't be pinned, subsequent nodes backed by a dax device could not be pinned. In testing this, this is not the case. Re: long-term pinning, can you be more explicit as to what is considered long-term? Minutes? hours? days? etc If a migration operation is considered short term, then pinning VM memory during migration deals with this issue cleanly. So walking back my statement - give my testing, i don't believe there's a reason for a new zone. > For good reason, because long-term pinning of memory is just the worst > (memory waste, fragmentation, overcommit) and instead of finding new ways to > *avoid* long-term pinnings, we're coming up with advanced concepts to > work-around the fundamental property of long-term pinnings. > > We want all memory to be long-term pinnable and we want all memory to be > movable/hotunpluggable. That's not going to work. > > If you'd ask me today, my prediction is that ZONE_EXMEM is not going to > happen. > > -- > Thanks, > > David / dhildenb >