From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5564DC0218C for ; Sat, 25 Jan 2025 09:53:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 67C342800A1; Sat, 25 Jan 2025 04:53:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 62C1E6B01B3; Sat, 25 Jan 2025 04:53:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51B642800A1; Sat, 25 Jan 2025 04:53:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3317C6B01B1 for ; Sat, 25 Jan 2025 04:53:50 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 854434B5B2 for ; Sat, 25 Jan 2025 09:53:49 +0000 (UTC) X-FDA: 83045512578.03.0F35978 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf15.hostedemail.com (Postfix) with ESMTP id C16DDA000F for ; Sat, 25 Jan 2025 09:53:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ELSNsvYk; spf=pass (imf15.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737798827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q6/Xo2aT6sk0E7DsNstlAuvx5w8XF0lUST+ZFZc1+ac=; b=OfvSoFsWJb9Sad9iRcoR3WqG79KdGqSEgasm28oj6OMdmGQtVxEFykix+cQeI3ywUFmrzc GyZqD0wiWWt5P8dSW5xyzGJgkmTa71dSmmTl+eqU94Nn3Nez5migmZR4pRLv62h6MciRk+ XEwU4+wYyoKcI1os6pSh4f/QXAZR9Wk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737798827; a=rsa-sha256; cv=none; b=U4OQPkFmiL/5vr7R1gO6PmMj69GYVvy3R+kbmO0qBGi0tHKMJ6SxwZsQz8/rX5xkpzuc95 sNhpXJgIyvcoRbuMFjsIR8AT8zu51geeR5VYU7Din7aSiunGYgHEbManWl5VZ0oJeh1W53 buqy5LevvwWGeaaNiZjQI0jZ7QrymMY= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ELSNsvYk; spf=pass (imf15.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 613225C5874; Sat, 25 Jan 2025 09:53:06 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A72EEC4CED6; Sat, 25 Jan 2025 09:53:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737798826; bh=lqrXw+4YBsiJEQLLRMjnYWtnF9HyNnnhBjTuQ3GXdvU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=ELSNsvYkvgY6q4xj1opmDB+ho5ZFUjGUV0s7VzkSot9agWoKkhOleVFP9CeBl2T4A eCCBqYumPTcghy16ojtP/wl7eJuw0sPn8Ywh/bKZSoyn1Cfc4ghhiTwGkCrXXLtDSQ 1DlkNxfGJ+wyjHa1hN034/Rww+GtXn9AkINPvYoIpabfHowdo83155Y5zCU8BD5h04 UEHAv4EFFWyKs1192H0yPjh+HYFByw+DY7Z0jGO7Tk+rrQbdgS9uwlIdlw6JZH0ecu OG5Yz9LsnFIwajR2aGuIjf90B8WgspJ5Ocbhh2cp+J1PCw2D3giP8O1Yl3FzRupm64 ICS/o0l09aleg== Date: Sat, 25 Jan 2025 11:53:33 +0200 From: Mike Rapoport To: Pasha Tatashin Cc: David Rientjes , Jason Gunthorpe , lsf-pc@lists.linux-foundation.org, Alexander Graf , "Gowans, James" , linux-mm@kvack.org Subject: Re: [LSF/MM/BPF TOPIC] memory persistence over kexec Message-ID: References: <20250120141427.GK674319@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: dbp973wa3nuo8n75p557rbfzfw3cobbe X-Rspamd-Queue-Id: C16DDA000F X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1737798827-637950 X-HE-Meta: U2FsdGVkX19e7dwgH4vRIlxckUeeaZOaZpmBFbv2xniLlsfWxoA+In1uDqFN0mRkHHX2jrBgiVUjGNCBIOfkjwg8Uq+hXxJFJZOiaeQRXqxoLKKL7zcWen6fzPPnPgtxaZdqTlsOi8IW/oR811wZFBGEPC5nl01jI96+2RsTPYw9YowS9zk6e5Xp2egUX+pH9FzRrMZFtDSQfBiJirDbzB+Dj0tEL2Vt/66eHY4uOAwKvG88GtWENxLU+x834D4wmPtFOoSoMzSnwgL3LWxCYXIG5AkYIGQtajhry1UauNrNfAFPZweLF321T8Io3sS+JoYSVvo0mexwLn8Wqx+bPPU2rtVWz9RsmX3H76QQMBhDCrjypZ3oMBmFD0KCmasbwJo2NydUEOJuk2slZ79dgVsiUhpTmfo5Fysffvc2AZKv4APbK7E7hwjiDtUytjTtjd5y0hq4ELTSJ4fTkY3lQ/AEhOzRITzfoQQs7j0MgvJgx+CMVMizDHvVZxq/LH+4eGIwYMqrd8QASTHRA2sxQuWcVOOuDu09Hl8UcWODPW95bSmSR2nEKrF/yZZUcEwP4LSWqD3U8JHfTY2w3gGFhonNGMpf/ldIU74V6ANwJRGixR4M2zByGMO0hlT8IEffxAl6VRVo+gwNty23WksESEn6893j4cQhIGyaRJCJ3MW94mXodIBXnXT8JLp+Vl3OihRqD6qTuLKVGQNw+DY2TdsQsxjBulD2+Ev0A5a3AczXjT/ZYpooRSdK1cgZ04R+JqfHKxLQC6y5nsf//q0Uh9wtQdpVOGns6kqg4zhykbYC73/Zu3W54vpq8yD7/M96zUvsTmspQtS2Q/bc2bps1ofIkN6HNQU6jJ/R0m9viAzfVAKDSDv07AZGHMj3yiMC8JTs+ZnY2k/YJ0Y2LouF50ZuE9uPrux8/1vOsOs050lJ6UL1K0qtsUbmbYGwjc9YRXIcWWbaTGoQ+Eva6/8 xCgVgIvH Vp2O931FyuWu1gzzwVWxP13ZS0S+EhIwZK2ymi2vIucB+NAv4qhCpGskR/SeLvY/9MJdi1CtH/iXlkGmI5nR9FHzO+f+YKL3Jjb+3SRz4EuWJ9R1hQhMVAQXh4hOJFzSQyT4pQGZY6Un8onqFL8Q922hiJoA8s3+nvXk9fAj3ay6EA4XiKoHOJosLiBpI2sArCR0CtAAg4U+QMrCG4V/5jcrJDR5+RgkGJYOaBXFuZiz2YMSkfU54XIaY8YArkaWLJITu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Pasha, On Wed, Jan 22, 2025 at 06:30:22PM -0500, Pasha Tatashin wrote: > > > On Mon, Jan 20, 2025 at 09:54:15AM +0200, Mike Rapoport wrote: > > > > Hi, > > > > > > > > I'd like to discuss memory persistence across kexec. > > > > > > Hi Mike, > > I'm very interested in this topic and can contribute both presenting > and implementing changes upstream. We're planning on using KHO in our > kernel at Google but there are some limitations for our use case that > I believe can be addressed. > > Limitations: > > 1. Serialization callbacks are called by KHO during the activation > phase in series. In most cases different device drivers are > independent, the serialization can be parallelized. > > 2. Once the serialization callbacks are done, the device tree data > cannot be altered and drivers cannot add more data into the device > tree (except limited modification where drivers can remember the exact > node that was created and modify some properties, but that is too > limited). > This is bad because we have use cases where we need to save buffered > data (not memory locations) into the device tree at some late stage > before jumping to the new kernel. The device tree data cannot be altered because at kexec load time it is appended to kexec image and that image cannot be altered without a new kexec load. > 3. KHO requires devices to be serialized before > kexec_file_load()/kexec_load(), which means that load becomes part of > the VM blackout window, if KHO is used for hypervisor live update > scenarios this is a very bad limitation. KHO data has to be a part of kexec image and the way kexec works now there is no way to add anything to kexec image after kexec load. To be able to serialize the state closer to kexec reboot we'd need to change the way kexec images are created, regardless of what data format we use to pass the data between kernels. > 4. KHO activation should not really be needed, there should be two > phases: old KHO tree passed from the old kernel, and once it is fully > consumed, new KHO tree that can be updated at any time by devices that > is going to be passed to the next kernel during next reboot (kexec or > firmware that is aware of KHO...), instead of activation there should > be a user driver phase shift from old tree to new tree, once that is > done drivers can start serialize at will. If I understand you correctly, it's up driver to decide when to update the data that should be passed to the new kernel? Again, for now it's kexec limitation that kexec image cannot be altered between load and exec. Still, it's not clear to me how drivers could decide when they need to do the updates. > As David mentioned there is going to be a hypervisor live update > bi-weekly meeting, where we can discuss this. Sure :) > Pasha -- Sincerely yours, Mike.