From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F78BC4828D for ; Mon, 5 Feb 2024 17:42:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 178876B0087; Mon, 5 Feb 2024 12:42:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 128D06B0089; Mon, 5 Feb 2024 12:42:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F33286B008A; Mon, 5 Feb 2024 12:42:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E23B96B0087 for ; Mon, 5 Feb 2024 12:42:43 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B741D1C1449 for ; Mon, 5 Feb 2024 17:42:43 +0000 (UTC) X-FDA: 81758470206.15.955F9A5 Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf01.hostedemail.com (Postfix) with ESMTP id 9C0C140026 for ; Mon, 5 Feb 2024 17:42:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Iy1y3+6B; dmarc=none; spf=pass (imf01.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.180 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707154960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8HE4Ga8BSdBO6M6GmqVFa+HwwrrcZFVY3gTfyQML7uA=; b=wi9o7OyPqsAk53YaUevbhbOgwPGpt9SmGm7js3HEMvLtqIxP5QQ0OyOZlJDMQChk0s75ep tZNTYoFd/LAft5rJ8lvOiuKszV1leBKrP/WBoZ9kZOz56wKaKGsCfAx0j0lC3YVhzZmUYY I80YPqEYXbdnYv9POozWwxuF02RWVXE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Iy1y3+6B; dmarc=none; spf=pass (imf01.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.180 as permitted sender) smtp.mailfrom=jgg@ziepe.ca ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707154960; a=rsa-sha256; cv=none; b=6CdHtzFFJtMxTpwgU1IAlWYNKpp2i84X2aYnBTN+ULgsEHdMqHegQjepOMnw/lH77Uwgm3 70ycp/zWwcWsshxC5+hifoior58slMG6prIlO2JDPZisaX0RzLwGaCqyXPXGXBMLhIJ4pi bcl0LUlh7YhLXLN/PuUXBrB+dGr3cVs= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-42c2998d3a3so5468301cf.3 for ; Mon, 05 Feb 2024 09:42:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1707154960; x=1707759760; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=8HE4Ga8BSdBO6M6GmqVFa+HwwrrcZFVY3gTfyQML7uA=; b=Iy1y3+6BfRatK76Waf1zuXnhAp1bWf4xt49EkeUbmlaQ7zl93YyAIa3AXCwYvDj/Am abp3SWA3UMJ9ioYIG8wk6DuxVDZs9U3qhq6eSieCVXAadJN1vDJSIFiihGdrfXHnVSHe JFNwAD6pBzlrsyVrlKAv1jt77ZHvzOufg1cs3O3+OBMszfGbBXtoGQDtKgHBy4FR4fhs YkXH+tVID8GMi7rheN/yWn2V/7zMB+fG12IU8Unz1H59MVC13DoCIEnjw9tCdKg7NWhD Vkj/CLVWQsO850wbafDSH80F0QxnMi8HqO5az+pdl1cvcbAFSpEdFukLEuDC/B3EFcL3 xPxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707154960; x=1707759760; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8HE4Ga8BSdBO6M6GmqVFa+HwwrrcZFVY3gTfyQML7uA=; b=fOoXBoOqQmzfz+jK8GbqGeGY9CJeOm2PTwTSgOMRpGetF+pBVLC4EZRrVhQZazxdGu ZoUxogonpVYaj+rkRhSv5Ka+d/jyYGNd/bJICrdtJ15oeVjyGV9LXf0Ra/CgI+C+Jg6a d13lrt62O4c3jOSdgLs7aY7FMpP78hrVZHc1//n8U1rRItdQiA4h5+GQFq91KC0ad505 8TCjvl4cAhGNN4xiQ/RicZRc63RXMW+a1QcmQaA51zxSZHatwMZNfghgejmvZUj5cCGV 5+yTcZW9mCT2UWsNUaWEY9MCL5okH6iRZsZdXi8igRHQqJ4A80swlGWLBt7L8/FjLe8D pYuA== X-Gm-Message-State: AOJu0YzbIwU3qNWcvTfpTNgt05gve51aVrXZOZv/J0d50uw2TJCVmBs5 PydF7jVY4kPyvaP1gaBClPIW+V1LVMc3zQx5B7xU5nefVnvol6/PVruMQsydNPY= X-Google-Smtp-Source: AGHT+IHU4KsrD3c0EP9g7y8hIFpyaAa4okXRRNN8qBm5KuQa3mySXXt7+dYoSw0Wr4a8G0dHjecPyQ== X-Received: by 2002:a05:622a:60c:b0:42a:9cd0:10d6 with SMTP id z12-20020a05622a060c00b0042a9cd010d6mr76578qta.34.1707154959703; Mon, 05 Feb 2024 09:42:39 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUJEoYPtNarb3IKpx+TlDaQWnlGCPYYwqRse61FZumUi17YhNjF1PV/xF22JLrvNXqFbkebE3MWbOPPsd6X9lkUYDUu8OcgOJYYw525y7FBiRK5b6k1ygrBSUjRHKinQ83+/bgI+lNtpfQIgMjD96anUt1JUN/A07pGruYvZXzU+IkWkDlqulHlYWtlENVhDo7EDFB9txF76qn+sovVDt+EhL7ut1S/hZswfg+VWs1j/g37rEX6ap6/4HYUNOLco77wrroLxU0HPFDsjm75+gckY78bO+VGj8ZjeEVOo3As9m48+X0kxJ0Q+DCxEkK31OMS3+rk9JgZYfc/rvBZ4DfknXCKEFbNUSYiTsiRn8LuNk6aoyUi8Yt0fkyR8E0kWWgCK3tQyQl37yuAXLLWMzwjLPdLL+LetoDhwayckEgT6hVcQTSAdr/64XA4acyBFh/KrJTmXe27EODPpZ7XhFeqlSK9tOkNG5gDuy05MIpWcUZfCBG5xHxACRXuyekJg91XbC9JxH2AkXUiOdSYdpHdIXrVkRrLeiPD0hPJPc/vsi8qojAJdzmxkzyfLzLhg19sg4M2WCCcphSLrPd+8rW3G5XCnkab07S61wEP7FLkMnOakYoKQoDGJkTOFQdDEWmESvGbq6OoQFA0uUJj+KlWqrYt9e4aDsPJbchaYnyqj5Tje8QC1runF4T0tAZjue/v+GvEzghpVJQWjy+T/0qjRPKiwuFVw3YGWf5PAbhZzViL Received: from ziepe.ca (hlfxns017vw-142-68-80-239.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.80.239]) by smtp.gmail.com with ESMTPSA id z5-20020ac86b85000000b0042c04cef1d6sm137895qts.66.2024.02.05.09.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Feb 2024 09:42:39 -0800 (PST) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rX2zO-000fQ2-NV; Mon, 05 Feb 2024 13:42:38 -0400 Date: Mon, 5 Feb 2024 13:42:38 -0400 From: Jason Gunthorpe To: James Gowans Cc: linux-kernel@vger.kernel.org, Eric Biederman , kexec@lists.infradead.org, Joerg Roedel , Will Deacon , iommu@lists.linux.dev, Alexander Viro , Christian Brauner , linux-fsdevel@vger.kernel.org, Paolo Bonzini , Sean Christopherson , kvm@vger.kernel.org, Andrew Morton , linux-mm@kvack.org, Alexander Graf , David Woodhouse , "Jan H . Schoenherr" , Usama Arif , Anthony Yznaga , Stanislav Kinsburskii , madvenka@linux.microsoft.com, steven.sistare@oracle.com, yuleixzhang@tencent.com Subject: Re: [RFC 00/18] Pkernfs: Support persistence for live update Message-ID: <20240205174238.GC31743@ziepe.ca> References: <20240205120203.60312-1-jgowans@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20240205120203.60312-1-jgowans@amazon.com> X-Rspamd-Queue-Id: 9C0C140026 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: p6h7bczt17c4yuy476ebte4ktwb9jbns X-HE-Tag: 1707154960-849536 X-HE-Meta: U2FsdGVkX18nQ2xrJVabBdph7kPu2DUi7jJ5biMfakDkn+IVyVp6cn3K0lp6vAESTcanJD8sl73nsJPsIXRK9RF64vEwo+w8RFYgcSQZRGuPRy3rB+VE5/WYKaWnA5kPyzLlr9JIC0lAIszmwZMm1sChD9nCI98gVTwYIH+SfKkWl/GKdGh153MwyaE4iykCtRC2ygOQN4mr+9qz15KtXfvye+xolLS7Tzl/OWSrmpYcuF1TCb3/Mz+Fodj+/DfoenMYOalyC6PRIN9dctC26RBP8jZEv3aKgrGVAbLN4m3esk8ooXECY2j8KG9s7NNTMhO9FfF5cJN3sRESmVqnsrdNrPy18vQGfgeJCnyHmeXlJSnJSfLTFmTZ2awn0JfvJ4cJwjKkVtHZha25/5VqVF0+w6Akc/FjnIE7AcAN3ItqIEPtuC6HkfpKOCEd4pEbu5IJE6OAhb8cv3/+Iv1HIzl/fwIy5TOyN1fE70ir+O3cax+UGdukKmtWLmxntRxEo4hwkhCjGcQM+UJwMXU+cU9wWeWYs+Zla4nsfXfh9Ko7ReFf1ytMEwdQyyZIfs5Vi/mZHOEinoRHDqdDX2nynWVTM6mHfp0hN257vi6Ntxak6EQNm0MRfiE/K+4927xbmkMsp+9j3l5bjwWdyX5B6Z58zPa7qdcpiyZuITIRGgwm4gZ7zjUf2pj653aTf/CoYVBhpyz01CscpjjOIdsSEMFTRyBDtjNk6oO7A0tj31qmscpD4o40HXqEJjUDG6MgLsdHqwFEVsRNpqgpufv1sJ11uIKFhlo7UkgZMYmbkxZWrIW41a2dO/U8tJYT7fbNwUHaJCo6O5zek7kWqEajQ1NbYbcbweiylct++e8n6Db4aHWOsPmBwMjT9kd3ll9dekrsk7MW4kiKT2Rj/BYN+wiBX5QXeuTnVhxGgU1Vka5Gh3Rudou/NAcVCUAlgxTM0VHJ1K83utT3+azKXed uWtZWG1T UX3+DmjNk2cpe6z5SWviWC53SzUH2VPp1TV/6V3qzvU5Ir4fC2zUgZU7Da2CWroSd/CgyevLIBJqGo6vHwzEJBxqFadIHxpHGWo5KUMkXxly2uSeAeuWR4UxjDMXauJ5zDb63QvAv5QxcoVOM202B/c/ymclpOb16OHcpbIoGMdwSvxh5QVxK5VFe493Qbb0cuiTy23T2Y9Tqg54++0uFlObW83x6BWkwNeYOVLfQDB76lcrPK4j0nbBZ2yFZ2LZ0YF88xrk1V3q+Aq24ehHs9EXmLmCMmfjbEZvJRQPeLWWLF9dAe1wu6f4JS6Xms4PZ6qkGF3R8+0EgqBnlThLwSqln51Dp+zi+LbWm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Feb 05, 2024 at 12:01:45PM +0000, James Gowans wrote: > The main aspect we’re looking for feedback/opinions on here is the concept of > putting all persistent state in a single filesystem: combining guest RAM and > IOMMU pgtables in one store. Also, the question of a hard separation between > persistent memory and ephemeral memory, compared to allowing arbitrary pages to > be persisted. Pkernfs does it via a hard separation defined at boot time, other > approaches could make the carving out of persistent pages dynamic. I think if you are going to attempt something like this then the end result must bring things back to having the same data structures fully restored. It is fine that the pkernfs holds some persistant memory that guarentees the IOMMU can remain programmed and the VM pages can become fixed across the kexec But once the VMM starts to restore it self we need to get back to the original configuration: - A mmap that points to the VM's physical pages - An iommufd IOAS that points to the above mmap - An iommufd HWPT that represents that same mapping - An iommu_domain programmed into HW that the HWPT Ie you can't just reboot and leave the IOMMU hanging out in some undefined land - especially in latest kernels! For vt-d you need to retain the entire root table and all the required context entries too, The restarting iommu needs to understand that it has to "restore" a temporary iommu_domain from the pkernfs. You can later reconstitute a proper iommu_domain from the VMM and atomic switch. So, I'm surprised to see this approach where things just live forever in the kernfs, I don't see how "restore" is going to work very well like this. I would think that a save/restore mentalitity would make more sense. For instance you could make a special iommu_domain that is fixed and lives in the pkernfs. The operation would be to copy from the live iommu_domain to the fixed one and then replace the iommu HW to the fixed one. In the post-kexec world the iommu would recreate that special domain and point the iommu at it. (copying the root and context descriptions out of the pkernfs). Then somehow that would get into iommufd and VFIO so that it could take over that special mapping during its startup. Then you'd build the normal operating ioas and hwpt (with all the right page refcounts/etc) then switch to it and free the pkernfs memory. It seems alot less invasive to me. The special case is clearly a special case and doesn't mess up the normal operation of the drivers. It becomes more like kdump where the iommu driver is running in a fairly normal mode, just with some stuff copied from the prior kernel. Your text spent alot of time talking about the design of how the pages persist, which is interesting, but it seems like only a small part of the problem. Actually using that mechanism in a sane way and cover all the functional issues in the HW drivers is going to be really challenging. > * Needing to drive and re-hydrate the IOMMU page tables by defining an IOMMU file. > Really we should move the abstraction one level up and make the whole VFIO > container persistent via a pkernfs file. That way you’d "just" re-open the VFIO > container file and all of the DMA mappings inside VFIO would already be set up. I doubt this.. It probably needs to be much finer grained actually, otherwise you are going to be serializing everything. Somehow I think you are better to serialize a minimum and try to reconstruct everything else in userspace. Like conserving iommufd IDs would be a huge PITA. There are also going to be lots of security questions here, like we can't just let userspace feed in any garbage and violate vfio and iommu invariants. Jason