From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5955C02198 for ; Wed, 12 Feb 2025 15:23:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7985F6B0088; Wed, 12 Feb 2025 10:23:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 747CF6B008A; Wed, 12 Feb 2025 10:23:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59A296B008C; Wed, 12 Feb 2025 10:23:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 361386B0088 for ; Wed, 12 Feb 2025 10:23:47 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CC04D1221F6 for ; Wed, 12 Feb 2025 15:23:46 +0000 (UTC) X-FDA: 83111662452.24.EC879AB Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2075.outbound.protection.outlook.com [40.107.223.75]) by imf10.hostedemail.com (Postfix) with ESMTP id DBE86C0014 for ; Wed, 12 Feb 2025 15:23:43 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="KDNs/pCE"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf10.hostedemail.com: domain of jgg@nvidia.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1739373824; a=rsa-sha256; cv=pass; b=opfThIVANLwwy1FJWC5YYLR37kiSHOXd7+8vYGhmaV/vaQ541pcMOPRTuKcQdEcfKAMDcr XSuAuSIG97twyPK4+dSZ1QV+cRYkG+yqMJpuhnbC9kfiqJlq1y5MMVRyR7Rmzu6RNLFujQ qdxN1rH9QhLnHs2m3R6jN7237NiXy4k= ARC-Authentication-Results: i=2; imf10.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="KDNs/pCE"; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf10.hostedemail.com: domain of jgg@nvidia.com designates 40.107.223.75 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739373824; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OS4kBGgLsWLc6mLv3rvLLJcdQ9LwKsqBWtsrT9w6dYU=; b=iCaTWoY33GjSdFvvnt5vwkJnD0x/JikXp5VhZKzEUU2QbVgWQhWFwopxl2M6ukxVwK0R/Z kiYtCV/2Emn9HrKz+koHXvDGBfZgl38a/amaFfM0Q8iMnKg1iT80G6fYIRKXiOQCKNR8e7 GsNabL5efxUnmNSGM8NfHm1zZMejOfI= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CscaezITUFe2C/m+x/A1YkHugOTS6imFzoZHVXO1y3WxcEnEoaXLOc8ksrIIzzKHArao6oBAS0Kp44aaDwcLBFnMS7WuUJvmL57hem1xxeaMguNPodjXv80zRH34lX1Z/rjRvxkpDOLxNexjf3UHsW2YioDPSSjwDZlrcsUrbgdeak/AMYRM2S45ixQYMEgEfNMYfEtMbuu7fNRBq3kq9c0jg+eyicL5WVjKZrDZa7LC0KNzsDgp/MhTWS+T9DgGxqd08h9m9gZJvD7gC7k9GynnMF49/OktT61GmWA4pXeCRDX4RV2skXh+z2QBroTAT7Yn9Ano62iGa8gWSUw01Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OS4kBGgLsWLc6mLv3rvLLJcdQ9LwKsqBWtsrT9w6dYU=; b=fOMqA51YkNRa1KbeRAbUQF+BhaIbAq5Yjx/jQnk1wRZ10q0QXRUmewpxLvXx8eCyLMceVUZw2ewtbjfNKUSHojcfB6KC0PkRknV3sYTw3b9xaLNpX+ZqUzE+rpHsoB4NH/HnAehNm5tCqJop4qWdHOq5ir83fVYFQvvVVD5HZI0QNX16QCkgel33zKct4fCzRqDSLIDkTqRojzolvABZ1N3cim5vN7dcN8INFbLtbvYMBOb3KNbmsbgLNh0amZY6bav3WAzOxZ/UikpQcoC/SbslZHquIXgI+AGOpG4dFuERfoQK0aCFQMaMsdHCqIMAp8bQ26UTsLLEOVGh7OSnxg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OS4kBGgLsWLc6mLv3rvLLJcdQ9LwKsqBWtsrT9w6dYU=; b=KDNs/pCEqsVFzL+mPKgucZy3kRTBFIJT8FcQuFS9XyyufE/YfKe/RxLr5a3+UUyDskyUQfx7u2y5mbHXnEOL2oJ8tMZxJY7f7WsYgojXvmyXqXtUXXRERlIytSrIShpNk3PFz2Yu8r73cjDVgZC/cX0HY6dvn0n0q2a76eu3Lm2fG+kYmoFYdGr4ojUL7Sxsq0MsM00T8yomV4qRrbvPhUXZVJ7A77t9VKedPwTN4m5+gf1KxXrWAaQpjk/jVKXIsmNpK6SlV6CpnrKPya7xJub6lQFJ2kXW9+1EIEXQlCDQX9ts6RAj4h/q2sjXb7y2a1ooW8WpEkAnqcdlNbkMHg== Received: from MW6PR12MB8663.namprd12.prod.outlook.com (2603:10b6:303:240::9) by PH0PR12MB7838.namprd12.prod.outlook.com (2603:10b6:510:287::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8422.18; Wed, 12 Feb 2025 15:23:38 +0000 Received: from MW6PR12MB8663.namprd12.prod.outlook.com ([fe80::594:5be3:34d:77f]) by MW6PR12MB8663.namprd12.prod.outlook.com ([fe80::594:5be3:34d:77f%2]) with mapi id 15.20.8422.015; Wed, 12 Feb 2025 15:23:38 +0000 Date: Wed, 12 Feb 2025 11:23:36 -0400 From: Jason Gunthorpe To: Pasha Tatashin Cc: Mike Rapoport , linux-kernel@vger.kernel.org, Alexander Graf , Andrew Morton , Andy Lutomirski , Anthony Yznaga , Arnd Bergmann , Ashish Kalra , Benjamin Herrenschmidt , Borislav Petkov , Catalin Marinas , Dave Hansen , David Woodhouse , Eric Biederman , Ingo Molnar , James Gowans , Jonathan Corbet , Krzysztof Kozlowski , Mark Rutland , Paolo Bonzini , "H. Peter Anvin" , Peter Zijlstra , Pratyush Yadav , Rob Herring , Rob Herring , Saravana Kannan , Stanislav Kinsburskii , Steven Rostedt , Thomas Gleixner , Tom Lendacky , Usama Arif , Will Deacon , devicetree@vger.kernel.org, kexec@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Subject: Re: [PATCH v4 05/14] kexec: Add Kexec HandOver (KHO) generation helpers Message-ID: <20250212152336.GA3848889@nvidia.com> References: <20250206132754.2596694-1-rppt@kernel.org> <20250206132754.2596694-6-rppt@kernel.org> <20250210202220.GC3765641@nvidia.com> <20250211124943.GC3754072@nvidia.com> <20250211163720.GH3754072@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250211163720.GH3754072@nvidia.com> X-ClientProxiedBy: BL1PR13CA0303.namprd13.prod.outlook.com (2603:10b6:208:2c1::8) To MW6PR12MB8663.namprd12.prod.outlook.com (2603:10b6:303:240::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MW6PR12MB8663:EE_|PH0PR12MB7838:EE_ X-MS-Office365-Filtering-Correlation-Id: 3e1adb2b-b545-49aa-b5e8-08dd4b79398f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|376014|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?4r6B+LlKtbfYogL+ZfFaeoTILB3a12Hhb1BkGIjQ1BGr8Ju3R1Xr/cHKWA5A?= =?us-ascii?Q?n17IgOcsEgG9g5q7xMP8aXR69gYqD/G71LPLLnqEXXphi/hhKqU7UXN1teVA?= =?us-ascii?Q?K2Uf1DzX+ZZ7sGK3tfe0abQarkFdbeW+GWqIlU340unnPwtyLNpdEc64sf+S?= =?us-ascii?Q?j3uEFF66MJli8O5TOjhL2uXXxvlrmqNKaIJTUJAqsEzR0geYpVPB7LC9ZKZI?= =?us-ascii?Q?fYORl0RIlrKVeimLsgloEnmMLlfBVf382FU5Ky2iHwQ0vG2hyfOXulNrugkj?= =?us-ascii?Q?XhSImHSVwdfooRhVGOmeQn49u7pQ6FPZ7G4U4EkU2sWTYlHGpgrQq6GpjbIx?= =?us-ascii?Q?SwCVSdgBGsMGQrgs2hkG8Pw/MowiMMCw+ezvIVekqCcaIi470BLslnbuHiRc?= =?us-ascii?Q?lKG4NRLASwhqBPKyB99MQ8WOjCSfHS35k8m7fk1Sg6Xv/T4vFApHyNtUzl8f?= =?us-ascii?Q?9bVQkhGSRku0Bw/u+BlmqsHx5sm/D37vbhwac6Vedw/nUCEz+xG78Ks0z4MN?= =?us-ascii?Q?uZVMZdxYddw5LAVm/pctw+QPqnbU5ei3/OFTr/38R64+R/F1HEOFt9S1WVAC?= =?us-ascii?Q?Y2ljUJ1hiXyivi8WCrl4++8x03wpO1jDFOEB5oShhYZHhdZnNS3XqywM2lHa?= =?us-ascii?Q?jyRKEVDRL5rUDaJ6LPjZ2MCjuysgfWvrYH1RdauT4519AlBVVbbdd6xJ89qr?= =?us-ascii?Q?O6j8Yv7lJJw6jWyyvRm8ojsLPqDeec83yT6r4SE+kmhXz9VL0OgOKXs8dLQ2?= =?us-ascii?Q?MkW1y9Jh1vaXtp9OS/olOZ1v+FxkiKk7wFU4BayT44XdGS2ZyEX8mwKJ6/U2?= =?us-ascii?Q?MsSKUGv1yQ0wfWHFC7cjQrJaHAwf8jVWRfqkbJcOOMjKJilhPA0Yv3E+NOkW?= =?us-ascii?Q?rDUrGGJ3TbVq3j35I3BrSTnboTZNqKLYx3nY0LChblv0gGJBz/nEVxwHf75+?= =?us-ascii?Q?V+rOVk10GDYuLL5QiInO+3tW1UU53gEWNvjQZ90Jyk5G6YMXlRBxMh4WbPGC?= =?us-ascii?Q?nfHErFDtybSirV1CXQVJwsOJkJhbRa0Vwatt1w1ZdHHZXQxmrwSygcg1pMNa?= =?us-ascii?Q?6srHnJa7/yZgEfFKHC62c9MKJyzJN+crw9NuKbcqt8XCNcTvtEd5ojQrwEiS?= =?us-ascii?Q?FvlioIr6CkCZpbyapSc8pb2lMLRbnMCUT10ISzy6F6OBc5TfGYXmdHnLOhhD?= =?us-ascii?Q?gtvTNvegQUNTFurXw0d/W/SgkA6Hes10Cb/KGZ9Pip7eeJdHPsFmwBC+bCMD?= =?us-ascii?Q?S2W6Z1HKAhRticJUH62/nLx0k/HbNP+uSSBchGBkUdKfBoxw1gK0PUP1SUVG?= =?us-ascii?Q?x+Ay6Y0uYOl27nUdXnIu2cfKkxg1FLgxNgQjx1yI5sib4wPEIvaDBZULmXtS?= =?us-ascii?Q?9wcsUXWdT3GBleHIBoQunStrd0hv?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MW6PR12MB8663.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(366016)(376014)(7416014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?6zG97Ny/NhkvT+DrgpRZ6S8HhSZcRruwbxceshPdoKYIqy98+y7XyybHQzR/?= =?us-ascii?Q?Kx3jbeOnx5WuYkSVhTZvOROOBdLPi5sC6+L6vT4gPiQ4M3WphtpapVE8YqsE?= =?us-ascii?Q?jym5dSl40QEKo2nbh7R7nT05tfKSyfkMZr3BQODE0PONXAnRLq0yJheNaKci?= =?us-ascii?Q?bop0V+EXjPGUk4Q0UFXPiit4TFKq3qQPUhnYtq+5mRLD8srPdA3SVd+/EwQt?= =?us-ascii?Q?iCmzyILMuSGZtgQDX485VvQ02wSKj/vh6Duhye76o8ScbSNtgktpcZndvmj3?= =?us-ascii?Q?X/wlqwrKH93bMFyxHz/HpjJlFnsvCmHgC7Ydbv11GzvRBnbfi83cr9nNCQex?= =?us-ascii?Q?IyPQ6RIEyfWgmbQa+mlDH8JX+rHOPEZirlKUF5DyX2oqFMAkHzJPc2Y07LV0?= =?us-ascii?Q?vMQP2lOmudEZIHLuXC0qJnyQQdgFnDixOmXrGylk91epd719742OvGKNzwID?= =?us-ascii?Q?aiW6XZ6VCC7tJmFCXgXLT37hbkfg6EQ+Cc520WwQ1emqR3hFzj96dHAaVOkY?= =?us-ascii?Q?YJyZvhbm8PNqdDbKU6TYUOhqXwpkKxWsX6uzj62dXPsZ1jXH/JOe2/pai7ys?= =?us-ascii?Q?HdIBhued8bM8+XzE277ClD1bYjBmb8lDU/24Y+AyXF13xETf0yEC9OEKWnad?= =?us-ascii?Q?lTZL+3lVd0tTF6kNVQgPsj6U4I8sh/bp2J1eIvCaVmZg2zIwIKs/qOa+4ybM?= =?us-ascii?Q?cS4TM0/znPoByrQSkXDsjeQ4Pi8QNghBOSAg1XLWURCb5SdSWMvP3h73euod?= =?us-ascii?Q?kpzZVnH7vHjxp2HeYBNV9xOPMqBqraevyhncOsfhKaFYjl38eATVrxkRMnCk?= =?us-ascii?Q?7GVWXqG1SMMMjHXU/9fp0g+UHPbsKuT2lFB5FskFlMBmOpJFciIzasiol0cC?= =?us-ascii?Q?TW+TZXcu7pk65fF8CTtD/RHG59Q3q9yCiIMebXXzGqt19kSqyglS8wcU85oX?= =?us-ascii?Q?UUXNIwdyyfn6kGUKI+AQ32PvYFzA7vVhOKhEkx8vR8465nZz5HnBm2umyv72?= =?us-ascii?Q?LGs/w53NYgJOImdpx8XwZCVeThmnY2ko7cN9kfuB/ZHdm1EePb4qHy3HOBGC?= =?us-ascii?Q?joU+gkltwY/H4oPKGMxSbe1YBBXNkXnaULqvg7ymRFSV0QF+qlGCuvLiOnDX?= =?us-ascii?Q?IZ5S1SPqGYV05L8SoiPLPgBZUpLiCBS9RaXo/sv2eKjZO0eTUPXFn91yx5zZ?= =?us-ascii?Q?bCyfMkV170VkaHCf1p2m5Ll7nqEe3FBy/fwF2B3nGnPSyS4KGXN35b1Sd7z6?= =?us-ascii?Q?uW58Une0RiHRkZjdz5GiClOTw1+rnd9bVzxH1cNoNC9hCI3mnTWw9IcCkkqB?= =?us-ascii?Q?hzdE7OzTeK6PdNfVeFA9rPmLavhIKyKwhSWwxY/DJjv1Ozpy3cMJGM5jxH+f?= =?us-ascii?Q?StpG4WT+pOLkQcYSLENamdJKnzNwdQnu7+ntXfbmrvPrSUBPmMuZKqTrKUA3?= =?us-ascii?Q?JT/BsK3eGvYMzLk+bmSp97dh3UVpg5DDbwZmDI/7p1PZsy7vIWLOBvMOzMcB?= =?us-ascii?Q?wKFOLpCdkEk0XLyd8lKmvXWkzqNRCAxIDiTSQ5QezbwNlDfsoDMd6TxNMd7O?= =?us-ascii?Q?riA76Pd8PSSWuzawOg8=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3e1adb2b-b545-49aa-b5e8-08dd4b79398f X-MS-Exchange-CrossTenant-AuthSource: MW6PR12MB8663.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Feb 2025 15:23:38.8210 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rQc344HRnWLI//BxrMm/7uXzCAy94KJvhtiSQuYTmPaKDwDBU7ud4HX4yU+W1ecf X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7838 X-Rspamd-Queue-Id: DBE86C0014 X-Stat-Signature: tdttekndxygsy7586zichm7c7uesyuy5 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739373823-4635 X-HE-Meta: U2FsdGVkX1/xjCJ6IU+PwiIHl66IfwRXn2BvjQ+ichdrXD9y6thiurI5SwvInKb6NCoZuYr2bEM7O0CyoN/tSCvcx9GJztdl3Vm5e8jJbAWipEaLWDKP6SN6O5rWeMt1fwiTCn8Xw7sxAbm6o0L2C9C/FIniFG1GH5qfvKJg5jQVnXM5d5MjP00awJbZIGy0RFN/R/E/u3oOinxVkmAYCUlixgFW8Xe54bPg4MI+UN6t7vVtDFeINzGOxENuV/lDf88wZbGqXVxf7abbmsC96y1UK1FjgBLKS62J+YwWJRYpkGX3rJ+givxVjPmisL+1J6Id0wktTv30B6u7PbMy9ZMVcby0Bgrub2CaKanUcZa2hGtpCqmQgfBZPl+U4tBJrTTOXbtwXsfozvA3ZsV/v1wDIO1d08dYKqsFOZYMfyk3KvHKoItAcQmICAOMZR5vXHq8UV1eRrogJzGOz1rUEz5JZ+wjr1zE7l7tnC3gPTd7yNJ6cvXMc6N1DN/V3D6Sek1XBcdIcFh86sRiJyPaE9GR1b9PSp7aA9rvu96sVB+wMrXpIY/xbPBrJtXqwRr1uvB99BpR0hAhjGUI354E4//hJ6Kt2nGNsNbS61RQ70R+BXP/Qjg+cDeC0+U6Y7HnQ0HPqxi37I47+v9LQDyw/1o0YviPrTaPoYz/nDePKA8OEn+4sIc+CqSK9Py9CF0bFNU/aP+HWhmTqZ4gD43ZlZbx8FZqy25awfTRHeUiHXUjR4k6pG/V0p64DLa8jbLK79qZV3jUnJBZmIviWHS9Sov2jp0tFnsMDK1GjDU2O4CggK7wg2kzng0mrKH50Rcilzy3NoWkNpGMyzsN7D610BOWxMQFfiJ7A8S6nv66EBiFphcu/2SOPVk3+//UNmh0lsFNf0IoqSk5yLQiWiXAe3jbw9ZaPYtsCg1iyiseZgVI2QoS+2TqZzVH6eb2NFeZT0Ef0paG+m8/KUq1xs7 RA7hr9eE RI8KhVWuNZI9WgeclbYL3pw0paIYf3//JofspOyCbPwUOYoAcpG2LGLgDFEWg7RaAkZIsRbp5Ll5vVFUiDZOama7ViBKB+8lsALdICRGny02UfrHbTs6APHntHVsfxOgc+JmHEsKWsT+TAQ0VsCZPycmBkljUZ9cILezHa4J4DW1FfaQWINiIJF2r2N6zSz1HjhebH29aUhY/SB8g44DveQ6Q4iCRXcT30UADNpdTEgjiIQC/BnumH4MJCkYpw0J9Ps6KmqG65GukEMzJMNg+RsfNsruCRIH0aNuYPFET8OJ8Io7i3AvQNn/cOOuQleYuXNBMZsy3EZ3aOqW+4ilrmRGU/H7P5Bwp5eQUx4+K6dy1Bi3fKd5oHvj8d3LdzVNTBIelPPftTf9iCRvgMERfCLQkAfCE7GeYSTsWUJVUtQbnkJggd7zYhKCBQA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 11, 2025 at 12:37:20PM -0400, Jason Gunthorpe wrote: > To do that you need to preserve folios as the basic primitive. I made a small sketch of what I suggest. I imagine the FDT schema for this would look something like this: /dts-v1/; / { compatible = "linux-kho,v1"; phys-addr-size = 64; void-p-size = 64; preserved-folio-map = ; // The per "driver" storage instance@1 {..}; instance@2 {..}; }; I think this is alot better than what is in this series. It uses much less memory when there are alot of allocation, it supports any order folios, it is efficient for 1G guestmemfd folios, and it only needs a few bytes in the FDT. It could preserve and restore the high order folio struct page folding (HVO). The use cases I'm imagining for drivers would be pushing gigabytes of memory into this preservation mechanism. It needs to be scalable! This also illustrates my point that I don't think FDT is a good representation to use exclusively. This in-memory structure is much better and faster than trying to represent the same information embedded directly into the FDT. I imagine this to be the general pattern that drivers will want to use. A few bytes in the FDT pointing at a scalable in-memory structure for the bulk of the data. /* * Keep track of folio memory that is to be preserved across KHO. * * This is designed with the idea that the system will have alot of memory, eg * 1TB, and the majority of it will be ~1G folios assigned to a hugetlb/etc * being used to back guest memory. This would leave a smaller amount of memory, * eg 16G, reserved for the hypervisor to use. The pages to preserve across KHO * would be randomly distributed over the hypervisor memory. The hypervisor * memory is not required to be contiguous. * * This approach is fully incremental, as the serialization progresses folios * can continue be aggregated to the tracker. The final step, immediately prior * to kexec would serialize the xarray information into a linked list for the * successor kernel to parse. * * The serializing side uses two levels of xarrays to manage chunks of per-order * 512 byte bitmaps. For instance the entire 1G order of a 1TB system would fit * inside a single 512 byte bitmap. For order 0 allocations each bitmap will * cover 16M of address space. Thus, for 16G of hypervisor memory at most 512K * of bitmap memory will be needed for order 0. */ struct kho_mem_track { /* Points to kho_mem_phys, each order gets its own bitmap tree */ struct xarray orders; }; struct kho_mem_phys { /* * Points to kho_mem_phys_bits, a sparse bitmap array. Each bit is sized * to order. */ struct xarray phys_bits; }; #define PRESERVE_BITS (512 * 8) struct kho_mem_phys_bits { DECLARE_BITMAP(preserve, PRESERVE_BITS) }; static void * xa_load_or_alloc(struct xarray *xa, unsigned long index, size_t elmsz) { void *elm; void *res; elm = xa_load(xa, index); if (elm) return elm; elm = kzalloc(elmsz, GFP_KERNEL); if (!elm) return ERR_PTR(-ENOMEM); res = xa_cmpxchg(xa, elmsz, NULL, elm, GFP_KERNEL); if (xa_is_err(res)) { kfree(elm); return ERR_PTR(xa_err(res)); }; if (res != NULL) { kfree(elm); return res; } return elm; } /* * Record that the entire folio under virt is preserved across KHO. virt must * have come from alloc_pages/folio_alloc or similar and point to the first page * of the folio. The order will be preserved as well. */ int kho_preserve_folio(struct kho_mem_track *tracker, void *virt) { struct folio *folio = virt_to_folio(virt); unsigned int order = folio_order(folio); phys_addr_t phys = virt_to_phys(virt); struct kho_mem_phys_bits *bits; struct kho_mem_phys *physxa; might_sleep(); physxa = xa_load_or_alloc(&tracker->orders, order, sizeof(*physxa)); if (IS_ERR(physxa)) return PTR_ERR(physxa); phys >>= PAGE_SHIFT + order; static_assert(sizeof(phys_addr_t) <= sizeof(unsigned long)); bits = xa_load_or_alloc(&physxa->phys_bits, phys / PRESERVE_BITS, sizeof(*bits)); set_bit(phys % PRESERVE_BITS, bits->preserve); return 0; } #define KHOSER_PTR(type) union {phys_addr_t phys; type ptr;} #define KHOSER_STORE_PTR(dest, val) \ ({ \ (dest).phys = virt_to_phys(val); \ typecheck(typeof((dest).ptr), val); \ }) #define KHOSER_LOAD_PTR(src) ((typeof((src).ptr))(phys_to_virt((src).phys))) struct khoser_mem_bitmap_ptr { phys_addr_t phys_start; KHOSER_PTR(struct kho_mem_phys_bits *) bitmap; }; struct khoser_mem_chunk { unsigned int order; unsigned int num_elms; KHOSER_PTR(struct khoser_mem_chunk *) next; struct khoser_mem_bitmap_ptr bitmaps[(PAGE_SIZE - 16) / sizeof(struct khoser_mem_bitmap_ptr)]; }; static_assert(sizeof(struct khoser_mem_chunk) == PAGE_SIZE); static int new_chunk(struct khoser_mem_chunk **cur_chunk) { struct khoser_mem_chunk *chunk; chunk = kzalloc(sizeof(*chunk), GFP_KERNEL); if (!chunk) return -ENOMEM; if (*cur_chunk) KHOSER_STORE_PTR((*cur_chunk)->next, chunk); *cur_chunk = chunk; return 0; } /* * Record all the bitmaps in a linked list of pages for the next kernel to * process. Each chunk holds bitmaps of the same order and each block of bitmaps * starts at a given physical address. This allows the bitmaps to be sparse. The * xarray is used to store them in a tree while building up the data structure, * but the KHO successor kernel only needs to process them once in order. * * All of this memory is normal kmalloc() memory and is not marked for * preservation. The successor kernel will remain isolated to the scratch space * until it completes processing this list. Once processed all the memory * storing these ranges will be marked as free. */ int kho_serialize(struct kho_mem_track *tracker, phys_addr_t *fdt_value) { struct khoser_mem_chunk *first_chunk = NULL; struct khoser_mem_chunk *chunk = NULL; struct kho_mem_phys *physxa; unsigned long order; int ret; xa_for_each(&tracker->orders, order, physxa) { struct kho_mem_phys_bits *bits; unsigned long phys; ret = new_chunk(&chunk); if (ret) goto err_free; if (!first_chunk) first_chunk = chunk; chunk->order = order; xa_for_each(&physxa->phys_bits, phys, bits) { struct khoser_mem_bitmap_ptr *elm; if (chunk->num_elms == ARRAY_SIZE(chunk->bitmaps)) { ret = new_chunk(&chunk); if (ret) goto err_free; } elm = &chunk->bitmaps[chunk->num_elms]; chunk->num_elms++; elm->phys_start = phys << (order + PAGE_SIZE); KHOSER_STORE_PTR(elm->bitmap, bits); } } *fdt_value = virt_to_phys(first_chunk); return 0; err_free: chunk = first_chunk; while (chunk) { struct khoser_mem_chunk *tmp = chunk; chunk = KHOSER_LOAD_PTR(chunk->next); kfree(tmp); } return ret; } static void preserve_bitmap(unsigned int order, struct khoser_mem_bitmap_ptr *elm) { struct kho_mem_phys_bits *bitmap = KHOSER_LOAD_PTR(elm->bitmap); unsigned int bit; for_each_set_bit(bit, bitmap->preserve, PRESERVE_BITS) { phys_addr_t phys = elm->phys_start + (bit << (order + PAGE_SHIFT)); // Do the struct page stuff.. } } void kho_deserialize(phys_addr_t fdt_value) { struct khoser_mem_chunk *chunk = phys_to_virt(fdt_value); while (chunk) { unsigned int i; for (i = 0; i != chunk->num_elms; i++) preserve_bitmap(chunk->order, chunk->bitmaps[i]); chunk = KHOSER_LOAD_PTR(chunk->next); } }