From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E40FC02181 for ; Mon, 20 Jan 2025 19:42:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63EFD6B0085; Mon, 20 Jan 2025 14:42:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EF326B0088; Mon, 20 Jan 2025 14:42:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48FF36B0089; Mon, 20 Jan 2025 14:42:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2A4D96B0085 for ; Mon, 20 Jan 2025 14:42:47 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CA9531C688B for ; Mon, 20 Jan 2025 19:42:46 +0000 (UTC) X-FDA: 83028852732.06.E53596F Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf25.hostedemail.com (Postfix) with ESMTP id E68CFA0010 for ; Mon, 20 Jan 2025 19:42:44 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G8MVBNIg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of rientjes@google.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737402165; a=rsa-sha256; cv=none; b=jYc/K6Ekmn49d+xiRwVT3qjCuyM44epqRVTsDITdqYtqgHtraYFJ+qcycH/fLYEJolkCNC Oes0CApPbsfxlCy6DfCdzKi7EoSy2ClccxLNRMvhrBCwqn+Kr7IQAhuxzeLbxWCfeCvmP4 Em+5lvABB2hYwv25bJfYymvfaAqe8AU= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G8MVBNIg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of rientjes@google.com designates 209.85.214.170 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737402165; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=StDSi/MEwUDEazmdARfhi3CgQ/tryIVXwtgxGSXsBBw=; b=UsNsEX5uC9dXgNSjtlrJin7xD9CXnyPgVmGrcTfCbEghwFMm13fOs+fiyu/1HVqCCafs+B lnRT8+trCdZ7REE/qPA01qiq5DOfes0/75lk1N/2y6lADMbPIONmeCa3dqM1eUzLtrFcXQ qFqXFu1vh+GV7IO18m4K4m/7im/PQr4= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-219f6ca9a81so228115ad.1 for ; Mon, 20 Jan 2025 11:42:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737402164; x=1738006964; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=StDSi/MEwUDEazmdARfhi3CgQ/tryIVXwtgxGSXsBBw=; b=G8MVBNIgd503w1xgzZjaVuFIDNzFbZew7X8xenfyxayWCdE3kwA7BPOUXHd5ZJF1RD NZEm1Y8BBpZtPFVSVPAhCMnMMjImuVQho5Jy1SiHr3Pkzs4LvKBwBma3C/Uba8HJYF5t FED19Jzv1S3rM5HSCfyT8hmPVhjhcgwMonnNdvubbUM89Bu02s0rbsYbx2M1eruV3Mgg +4ZmQKgFKPoL6UGTHgsBug6wEhMEOZfqPa+TgANiXVsDMhGHdH2XR9/rkBd4sFNC2Wqm 3xLYtFlMetGwDUJvHJaBMjc+eznTRQLpve7jkMvkGbZaiGbLTrwzfSmLOHoQ+DysZSRa GROQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737402164; x=1738006964; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=StDSi/MEwUDEazmdARfhi3CgQ/tryIVXwtgxGSXsBBw=; b=xUq4/988KFUXCw/XPjErtxswfrJQSV+I1wsslqyyC6zf75zclTDCbftLWDsOvsvZ2s 8EDZ6dp17j1qJhf0mIl9Cp3lsQWM/FR2Z+O+LbKQHYjn0P0bjWTrk7WCi61nsYm2P/4u vMPAWtoXPyQrmlthOTH8OlkRhitViIuAp5gJFU27xnNiFlHhTUibgO4ERyolEjESwSAn NrCSAZqdGZPw3GcfdbEV5UcdP0kcobz9ZBxBn+ndy5N+vt/iM/YMcMD0kA0gOhS59ZvI OId8/EikkFplt5HpnYWlSe1nBp4oJKJFCYkPYjLCtaIqI0QA40h8kwM1f8sAvj5rSycp wnpA== X-Forwarded-Encrypted: i=1; AJvYcCUKU4EZ5T79D9bskL/V2dbqT5LH1tGHmBLV9EHR3XVPMmItmf3nJqsPAMMek7lwq0SQt2siRhpeKw==@kvack.org X-Gm-Message-State: AOJu0Yzki/Mjy7XywvxvL5wTKPHObPJALt35TAxz71m/xWaXS1Uov4JB 2QuiRc/HAbKyfXbKvICENolMKAUsLBBaWzngduUZBSwDttq2sFkGlaUbSuTsXhAodq0yHTfmDMV +PQ== X-Gm-Gg: ASbGnctak93IDP7n3EN2iviQu385akFYCKVytWk3sqyoYql6LCG1taopJwidGWy8THu iVbU+OmXaW/GrMdS/653DfpDq/dz9hHom5rIxK3pXmitW2P2gcVEfrUrfe+RGC2+hvNCoHN7s5P PpAU62hgSbj5tqjaeL8KbuzikPrTMq3OmLYY6O0xPTKbs1U0AGrxgYRbzJ0JpoTOxFnMbRPYUtg 7q9q7ZD+KVGJPmV64cNe2ApwhU7UxbwNOeE/omXVI6dyWLc67X6fr3DSV/XIbUjZl+Zo+0c+y9X K4BbmkG2b0oUAlQTS7Ldt2cf X-Google-Smtp-Source: AGHT+IHTDv2VqirfEuHWoNffXPxvNHeunVMuxveP50OdVnkv3wzi1epZ8dk9CUnu8Z/9xvMAGedlDg== X-Received: by 2002:a17:903:31d6:b0:21c:e29:b20d with SMTP id d9443c01a7336-21c61735d35mr4078745ad.3.1737402162519; Mon, 20 Jan 2025 11:42:42 -0800 (PST) Received: from [2620:0:1008:15:1620:99d0:894d:83f7] ([2620:0:1008:15:1620:99d0:894d:83f7]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21c2ceb9d97sm64039455ad.95.2025.01.20.11.42.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jan 2025 11:42:41 -0800 (PST) Date: Mon, 20 Jan 2025 11:42:41 -0800 (PST) From: David Rientjes To: Jason Gunthorpe cc: Mike Rapoport , lsf-pc@lists.linux-foundation.org, Alexander Graf , "Gowans, James" , linux-mm@kvack.org, Pasha Tatashin Subject: Re: [LSF/MM/BPF TOPIC] memory persistence over kexec In-Reply-To: <20250120141427.GK674319@ziepe.ca> Message-ID: References: <20250120141427.GK674319@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: E68CFA0010 X-Stat-Signature: foq34sb51k7z9mtuearu6qc88gxcowr9 X-HE-Tag: 1737402164-261878 X-HE-Meta: U2FsdGVkX1/NMUXvjqbz3Y5M9aeAo8m51BoR0w0z7WA6mGYjNpOsw2B5RICxQwrVlQUnOvdUJB81G2klhOYVfxOh7JNGEOVD0zzasO+loGkMKVA4MUiQ3i0kGt4EOvxb9I7VPaU7LZhewXB41dDd7KmJ8E/bKbPHseq4Oc/U1F4RAXqnL7XFYWsqhOqlhZXSYrcRkxGTM8Nf6Q6fR1DPoZ1sbs1o+iB5GJF3IPKkYKGDX/grOva9AXJ3zVNP94pV+J9MNutn0rcjduF2Wu6GGyKTPhmAEg6mfcGtCvgBYmVaKmCTGcZSqJo+oUglgdEdsC3yrQay0T8v8VtgUUh9A/3ln4toOHegEyPwZG2PAtttGRPZR9iBNJlIoGLsd99UNF25V6Pkv2SJJCYU6yyzfsv5diIFWQ8BXxctTn6l5m3ig7jTSToz6RBH6HvV7ZxGt6BYLZHJgboUzjvLmJKIanuRqFLVho9j2rs3JZZBrkxXry3gt4Fe4bB6r+SC1+IT1lW/pYVQFW+07fYYh4TjC+Hus1y1bmAQSZbl5OaeAqfb+EUR0D8YD1ZSpsTuyE3+BD1C/3N2V5yLfccP60n4BH+FpX/zYZ8aRA/eMKEHAvpY8Wmbu/LrT2qRJSrBbO+J9UaeK9jFfSRXLCMcexGzUTFlJEZQ04U9/2g2Fp7qrLOrBCoSNqf6tUU9p07ulAPU9qaZdydEcI4XxFZtKAUpx+fOmbmmSTM/KkUogeljLHleOY+Kij2RBIIeGt+kggXVhhndF2I+FEM6g1p8gK+eITpbDlBvoOl8enZM/n8DHwVT3ZUQSO5nCGZfTxcgcKh3BnUBdul0bHPUCPCuk1B80guLdwmLcvfMPYbrPYLVvNTH6jC3+WPtIACBLVZcD/HZp2jFe9LL2B5Yzw0cgSmZ9YLzE9ARuoCUTdC4muYLEjZTFMjM4xK4N9D86L5i6Sarbg7BMDlOxx6zFBNmbVu CKlukNyF nG1xL99GcalldyzNGFEBcwcH3B7VE/oyjsRPNMxShQQQGQj/1WtH/VJpUNkZGA0b3xwdoMm98mujHlU2Aw3aNm41CfVNverVbM5w0v0//2MPw17m/jNqm1y33Gbv2d6hpwSL3Z1EP4/uTtbgqpzk601ZYMQnxUxb54LOt9oaVMVeefXWjow5uDrBgqP1JN5ZGnJeULpWX/BTFjPfmrpmlYSBxKfSyjgy/J4ZkXDRXSx/qzAu70t31Mei2MpD1NJo6vzh94OTr+dXrClncmLoSR75LTVkPCghY92Im9ud1sA8zqUZTyDM0I8LhX6c+kLX8G9SibhfzpiyHH7kKKLcSjQ0Us4RsfEO3bNa+kdqYXbVmzhwWh4KxtcacaHayyIXja3xV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000008, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 20 Jan 2025, Jason Gunthorpe wrote: > On Mon, Jan 20, 2025 at 09:54:15AM +0200, Mike Rapoport wrote: > > Hi, > > > > I'd like to discuss memory persistence across kexec. > > > > Currently there is ongoing work on Kexec HandOver (KHO) [1] that allows > > serialization and deserialization of kernel data as well as preserving > > arbitrary memory ranges across kexec. > > > > In addition, KHO keeps a physically contiguous memory regions that are > > guaranteed to not have any memory that KHO would preserve, but still can be > > used by the system. The kexeced kernel bootstraps itself using those > > regions and sets all handed over memory as in use. KHO users then can > > recover their state from the preserved data. This includes memory > > reservations, where the user can either discard or claim reservations. > > > > KHO can be used as the base layer for implementation of persistence-aware > > memory allocator and persistent in-memory filesystem. > > > > Aside from status update on KHO progress there are a few topics that I would > > like to discuss: > > * Is it feasible and desirable to enable KHO support in tmpfs and hugetlbfs? This is a very timely discussion since the last Linux MM Alignment Session on the topic since some use cases, at least for tmpfs, have emerged. Not necessarily a requirement, but more out of convenience. > > * Or is it better to implement yet another in-memory filesystem dedicated > > for persistence? > > * What is the best way to ensure that the memory we want to persist is not > > scattered all over the place? > > There is alot of talk about taking *drivers* and having them survive > kexec, meaning the driver has to put alot of its state into KHO and > then get it back out again. > > I've been hoping for a model where a driver can be told to "go to KHO" > and the KHO code can be largely contained in the driver and regulated > to recording the driver state. This implies the state may be > fragmented all over memory. > This sounds fantastic if it is doable! > The other direction is that the driver has to start up in some special > KHO mode and KHO becomes invasive on all driver paths to use special > KHO allocations. This seems like a PITA. > > You can see this difference just in the discussion around the iommu > serialization where one idea was to have KHO be an integral (and > invasive!) part of the page table operations from time zero vs some > later serialization at kexec time. > > Regardless, I'm interested in this discussion to bring some > concreteness about how drivers work.. > +1, I'm also interested in this discussion. As previously mentioned[1], we'll also start a biweekly on hypervisor live update to accelerate progress. The first instance of that meeting will be next week, Monday, January 27 at 8am PST (UTC-8). Calendar invites will go out later today for everybody on that email thread, if anybody else is interested in attending on a regular basis please email me. Hoping this can be leveraged as well to build up to LSF/MM/BPF. [1] https://lore.kernel.org/kexec/2908e4ab-abc4-ddd0-b191-fe820856cfb4@google.com/T/#u