From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 521C8C36005 for ; Mon, 28 Apr 2025 06:20:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E86F6B0005; Mon, 28 Apr 2025 02:20:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 695E36B0006; Mon, 28 Apr 2025 02:20:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5369B6B0007; Mon, 28 Apr 2025 02:20:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2DF826B0005 for ; Mon, 28 Apr 2025 02:20:38 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id F262EB8965 for ; Mon, 28 Apr 2025 06:20:38 +0000 (UTC) X-FDA: 83382453756.24.A0EF1B4 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf25.hostedemail.com (Postfix) with ESMTP id 53E33A000D for ; Mon, 28 Apr 2025 06:20:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Weg26xbq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of rientjes@google.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745821237; a=rsa-sha256; cv=none; b=kmCvVhzZuYZJg27hGOKG0xPN4kEwSuU8sPyifnN6OtQ909Ium033ofnb8hRrVFazd5uqTI P10fOpp4syFlbYL1y2NVr+BGZx/w1eHOV+o3tjj1TPKrgp3t2xwHj1Q6N6rpVO8D97MOn0 UAWXLaUYl+xcz2j5AjhChIg0ks5WHys= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Weg26xbq; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of rientjes@google.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745821237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=IXkZowAPn4kNB/SMK4aQRzRcIE+j1k5vZeIINfBypek=; b=vFIT2o7BVfv1Hyfs/yWBUtzpefcQcX2BY1wPeS2wfOqveHxPKpG2hbVjjWcyuH7F3009hS hMGuZnHAdO5cqythjs3oaLWZUXEY1UjG2RefGa0MRfmECHdM2/IcbXFhykGl44roKipARg KisYbNqnor1aH0bq+JvRRHAlhbn+CpU= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2242ac37caeso199815ad.1 for ; Sun, 27 Apr 2025 23:20:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1745821236; x=1746426036; darn=kvack.org; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=IXkZowAPn4kNB/SMK4aQRzRcIE+j1k5vZeIINfBypek=; b=Weg26xbqWV35kjEVA+ilNKrGxz//BBYUTDVnZ4ku4lTc3r5bPcDtgnakPm01PYEPwD YHVrgBLciliVfGbYDw8/jOsHPS//ftxtatUXT0nQYhxXxiPHiIuB+R9MYsSZO0MOrZ9u cCqRYZIMFF5H26uEDA0nG0alEIMbOVmfb3Cgljn6tixGWn7xtd1jrNW0nuAQ7ytQxQMl hFBc3sD+fCoNZ5Fi/SqJEvGpX/aWziPDKIIJWO8qwwXrMH/HxCpNwpioBvg5SkQN7ax6 9dkOIFVdLd5C0zBTDXGSmz7xNN78+QulJAv0NwCW/igKbC+NBXcbtAEv/TVjaGB6gkBq oJQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745821236; x=1746426036; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=IXkZowAPn4kNB/SMK4aQRzRcIE+j1k5vZeIINfBypek=; b=UsEAK4NA99hk5z2LsjR1Xc5a+NnN9PRk/o4s4aFCTg/zfVI7YRFopzC5EEUg60X7ib N6PlRZzbLvJhasLWnk5rnKFMZaghCMN8DqLUWzNdzMXNbRGZ4Uma9iGV1ZC0Il6Kdib0 EHMfeVWkH9aoMjgJQRBvCeU9fKcKaw6u2feh3yunD2eNUZICCqgwMsrgBxA7wU2lt7f0 RIHCDj3BC8BHeEdcwXp5pm2/WQYuFP/exQIZ/hlJ+vwqk6sxw2l2CXuJlxi0+xY66VgN 1R95WRTisK8UV7SAo5QTweeJxV2/5XFKXpmX90aRC04JiQPCSynsp48+Hkg9jLfBTnFQ h9QQ== X-Gm-Message-State: AOJu0Yy6WoCu3VD7WNNyb65vAeB4rNLm/jvege7idVbpUv4ESsTV0a4z rDJ8BkQmEEq/uNX2FEIRcDd2LzoCyCP1cbNg8DyTH/gSYlwQTsUcEnVj4B6NtA== X-Gm-Gg: ASbGncu66sRTO8g3Nr/g4egsZQsdFACzdm5GwxIvvJ0pnljKkngwY1UZYm2VZN8Nl6R vVcLi5gxmatrkCh+vtK8v2fjzvcHJzxG/brJ3J5jKI0xaH2e2ojMs2MjcmqupemF6oApEBw7VZm CVmuQ5iA+6lM+agXLs2Br+Ajz20LqiWyvfv+vjQk/z5T8xXWxk+328sOn5LGTThPlCroIo5XCmE /CmczjZzr88Sq6+VypNITgmCw/5iWhaO+FQXFHyuf7JXA0DPVIoIWIad79aPJihz/UndUPaoQAi NKoFw6xk5VehLSBCt7AAeGdriLW+ayr0gA+v9njSnbzZayCcABfH/rsrxKCY2yxxfADfMUIaUm5 r8QMSeP3cidIGFybKncLNB6RWkUx3haEj X-Google-Smtp-Source: AGHT+IEbvwRZ9JywV9BzJ11zLBZSswf6lRwuo8vaSFHDzKdK9dK/CYp5zUl9jXJST/Ww91LUZrwVUg== X-Received: by 2002:a17:902:d4c3:b0:216:21cb:2e06 with SMTP id d9443c01a7336-22dc90a4e72mr3004975ad.19.1745821235777; Sun, 27 Apr 2025 23:20:35 -0700 (PDT) Received: from [2a00:79e0:2eb0:8:7d97:75a5:a2d:e3f3] ([2a00:79e0:2eb0:8:7d97:75a5:a2d:e3f3]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-73e25ac87d5sm7166218b3a.157.2025.04.27.23.20.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Apr 2025 23:20:35 -0700 (PDT) Date: Sun, 27 Apr 2025 23:20:34 -0700 (PDT) From: David Rientjes To: Alexander Graf , Anthony Yznaga , Dave Hansen , David Hildenbrand , Frank van der Linden , James Gowans , Jason Gunthorpe , Junaid Shahid , Mike Rapoport , Pankaj Gupta , Pasha Tatashin , Pratyush Yadav , Vipin Sharma , Vishal Annapurve , "Woodhouse, David" cc: linux-mm@kvack.org, kexec@lists.infradead.org Subject: [Hypervisor Live Update] Notes from April 21, 2025 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 53E33A000D X-Stat-Signature: fykbu5xh5cpr343cim7nf1gemzj9zsqx X-HE-Tag: 1745821237-418211 X-HE-Meta: U2FsdGVkX1/t7iwkR5Rl9Vfl8rJHQse02u/ek18PKXRUDrGTqWJFqv/uF+OU2lsEmvS9LWqaXfa3+9BUcGbqGELdkMnEqXLa+Qc0TrK1JgFgdWJc4ae8/lF56Jt1o6tbITknEN7NdpqEdtkNHp6yunr+5kCOoYzbI3ZspdTVT/Kvw6jucbAtWR7OK+j6k3fHW1vevn9EPXuW+43OQhy346jLdPGWynUBkC2riNjlxVXisxhGD85jpuyxhwjx+7WSft3gmCL5deCQVFpx6Q8CJj4iD8gXF74cA9nIlcGd+uUW8r7vV3ctJBeZ39RyEJ8KHOecYPLshWGnp+TUmDvLt53oWW5ye+rQQQ80wtBjFctvm16/SpRNDxdbb3FXfsi5hoJ5V/O50bUg5PvCtJ/xppkXevwnOxs0ycsa9VjbzxMHtwcWjrRY4W3G8CaPzkEPEXjDktk+9Q6Sv7zQQDQPJYh2AAVRccE7e4W+F26wf5i2bXshi+cpcipOxeE4NJA1uteImdMATGjKfdkLJVCkwCnioZW90lsBHoHupHLzc6WB02CBxlnqBR87XGdDhIuRnrNXwSIsNooPdubiPo7R2rEo4obW4EBJgJ65llQVEnAiiVNJ29gwGUpuomYGp2wRnaaOpWaYud3v03jw/sBt5f5ItpNFLyhNzEzAw8SEqTBhSWjLdt+BHn9DC8oNTtGeO/DbLx4GVeKNaWyvxkXzpYyNWHIzyZdtMz5rWyrEMzk8PJbaCmXTPoPwQaEN8gztoky8bXdZgfdGESiLgDB530EnYQDTUpVooXjsKONxxcHkM8ujKQGZdh6ykuhU1ra7+IAJ96VJK1t2E1ua6/JQ+eTBW9pE3t9hxbXCnPwPXJZBJpUty44oNOHx7dyEAOV27d36pYzHWeZfIS/2PWO1+LAebEAd6i6pSFTt8In/SHsxy59frsTnwaJdhY1AYepcwltrpGzHCj81isRKf0p S3aWu6Jy 5PT4808/tx7SsYSobjanxr7ru5Ons4lBsYZALor4ZcvvKwgZxeDR5ZvXpOWScKTBSykxV34aX6+h3OgwUQxtD2Dw1ZtX11FHnXQo/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi everybody, Here are the notes from the last Hypervisor Live Update call that happened on Monday, April 21. Thanks to everybody who was involved! These notes are intended to bring people up to speed who could not attend the call as well as keep the conversation going in between meetings. ----->o----- KHO v6 is now staged in Andrew's mm-new tree. We discussed what it will take for this series to be pushed to Linus, specifically around Reviewed-by tags. There is not a ton of x86 specific code, but it would likely be useful to have Reviewed-bys from some x86 maintainers. Dave Hansen, would you be the right person to take a look at this from an x86 perspective? We definitely wanted to touch base with Jason Gunthorpe on this topic since he was on vacation at the time. Jason, do you have any feedback on KHO v6 that would be blocking for its eventual merge upstream? ----->o----- Pratyush looked at the LUO patches and noticed that it was largely doing what he would have done with fdbox. He expressed a concern about the global states, which was not part of the fdbox thinking. Pasha clarified that the finish step would be when the final cleanup would be done; we can still control what gets unfreezed and when. Pratyush asked if the prepare phase worked the same way for LUO. Pasha said that the list of participating subsystems is added before the prepare step, which does not freeze anything. Pratyush noted this is very similar to what he wanted to do with fdbox so suggested working on top of LUO. Pasha said that LUO v2 was happening right now and already began porting over the fdbox support. ----->o----- We discussed whether LUO largely replaces the need for guestmemfs as discussed in previous instances. It was noted that guestmemfs largely was aimed at preserving IOMMU and now we have aligned on iommufds, which was also supported by Jason previously. Since we didn't have James Gowans in this call this time, we wanted to follow-up on the upstream mailing list for this. James, do you agree on the convergence with LUO or is there still use cases where guestmemfs would be useful that isn't currently planned? ----->o----- We touched base very briefly about swiotlb in low memory, an issue that Pratyush ran into several weeks ago. Mike Rapoport noted that this was now supported with KHO v6 upstream which uses lowmem scratch support, so this should no longer be an issue. ----->o----- The u64 in the KHO FDT was discussed from the last sync, which KHO v6 implements. Each component has its own independent FDT that goes to the global FDT's physical address. We revisited the discussion from before when Alex had previously implemented it differently. The fixed maximum size was one of the biggest blockers. Now, with KHO v6, everybody gets their own subtree and can allocate from anywhere they want. I noted that one of concerns James had previously flagged was about dumping the state of the FDT for debugging purposes. This is noted as being solved in KHO v6. No additional concerns were flagged about the single u64 that was implemented. ----->o----- Mike noted that memblock is easy and useful for ftrace without a ton of additional complexity. ----->o----- We discussed the future of KSTATE[1]. Andrey noted that his vision allowed for building it on top of KHO as a protocol for describing state. This hasn't been done yet, but can be worked on. I asked if the KHO v6 pending in Andrew's tree would cause any issues for this extension. Andrey noted that this is a format for serialization and de-serialization data, the data itself could be an FDT blob. Pasha asked who would be benefiting from this serialization. Andrey noted this would be pretty much everything, including device drivers. At least with LUO v2, every component gets the u64 to store anything they want and this memory gets preserved by KHO. Pasha wondered if KSTATE could then plug into LUO v2, since it doesn't have any format dependency. Andrey said this could be done. It has similar flows compared to qemu. It was noted that KSTATE was going to be complementary, not overlapping, with KHO. Pasha suggested building KSTATE on top of LUO v2 and suggested Chris Li review the current KSTATE proposal. ----->o----- We discussed testing for LUO; Pasha noted that the plan is to add selftests and then asked a general question about existing kexec tests that exist, including with qemu emulator. We need to create a mechanism to do kexec testing with qemu that can be done directly with selftests. Andrey suggested using a nested VM and to test live update between two different kernel versions. I asked about how we could enumerate upstream the supported kernel versions that support live update, including their device drivers. I focused on how to describe the set of drivers that have been tested so that others can consume that information -- is this done through a code change that indicates that they can upgrade from a specific version. David Matlack suggested focusing on automation and frameworks that downstream consumers of the kernel can use in their own environments. Pasha suggested a zero-day testing infrastructure for reporting regressions and something like syzbot to track regressions. David Matlack also noted that he is working on VFIO selftests. ----->o----- Ashish Kalra asked about SEV-SNP support for live update. He noted "for SNP there is a VMSA page which is marked in-use/busy when the guest is running. Then the VMSA page for the currently running vCPU cannot be dumped by makedumpfile during vmcore generation as walking the guest memory and touching it will cause unrecoverable #NPF faults as VMSA is marked busy. So, this looks like a potential use case for preserving guest memory across kexec (kstate patches), so that the VMSA page be marked to be preserved and reserved." ----->o----- Next meeting will be on Monday, May 5 at 8am PDT (UTC-7), everybody is welcome: https://meet.google.com/rjn-dmzu-hgq Topics for the next meeting: - address any pending concerns from Jason as well as x86 maintainers that start looking at the KHO series + we specifically want to wait for Jason to discuss the decision on everything being preserved by fds vs recreating the state on the other side of kexec - determine next set of milestones for KHO v6 beyond what is already staged in Andrew's tree - determine next set of milestones for LUO v2 in its development and limitations on current support - confirm that LUO v2 subsumes guestmemfs or identify required support that is not yet present - update on KSTATE progress on top of LUO and what will be needed for PCIe devices - upstream patches for 1GB dev dax support - update on physical pool allocator that can be used to provide pages for hugetlb, guest_memfd, and memfds - SEV-SNP support for preserving guest memory and what foundational components AMD can depend on, building on top of KHO v6 or KSTATE - later: reducing blackout window during live update - later: testing methodology to allow downstream consumers to qualify that live update works from one version to another Please let me know if you'd like to propose additional topics for discussion, thank you! [1] https://github.com/aryabinin/linux/commits/kstate-v2.1/