From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF953EB64DA for ; Wed, 5 Jul 2023 12:19:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 188878D0002; Wed, 5 Jul 2023 08:19:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 137B48D0001; Wed, 5 Jul 2023 08:19:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F41148D0002; Wed, 5 Jul 2023 08:19:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E36EA8D0001 for ; Wed, 5 Jul 2023 08:19:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BBB9D1C8D1B for ; Wed, 5 Jul 2023 12:19:33 +0000 (UTC) X-FDA: 80977463826.04.FB1F376 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf24.hostedemail.com (Postfix) with ESMTP id B5ADC180012 for ; Wed, 5 Jul 2023 12:19:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TG2cXPGt; spf=none (imf24.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688559572; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZBre0g3T1mcgEtkinxsB/uoAXz8ZMrPHkJlWIM1M77A=; b=j6DnoLofGv5NtK2jpVhxXTastwR4+4eQ2WdhD3jJElDjQNSvgjmLu3VfnDo1xVdALz7H6E eFcqSPQ6j36n++nVyRksw7S0HZ/wLU/A1nw7YOplqheshyAhJo/s2Cr4GZgslfoc8dKFQq XyDaM8R2lA8ZNG9Ofx/Qrzl0sDlU/wk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688559572; a=rsa-sha256; cv=none; b=MXMvnVrJhQp1Xw/62fHuN3wdmF/QGPN71pU3qkVFg28GukWGL1yQS/eiEv8pgZOLSyyr3t ols+UuQlXNzdxFyDBRwwF4028V3OYhESonbK1ALdyL1O4NafwvAI/qKZI+WFbbmugExLaK d1OEereDSyuThk/y9PDLo5Hqxlml1l4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TG2cXPGt; spf=none (imf24.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=ZBre0g3T1mcgEtkinxsB/uoAXz8ZMrPHkJlWIM1M77A=; b=TG2cXPGtjDPuRohKGQ7FFcgcnz yy7Wboha5BtCIh5QyCVQVSLA+npV9WYK+6ISz4sxLcCPm3FjxHL5pQSL2MwjC+M7qfgfxjjZ3Wlj4 vAw8klij1kOJeqGb2eEsYx44AWGsS0u2vT5NiC09ZDaBcqpLiY0qxuvyZbFpp++oD92+J1yYxheCv G61n8lItcz3FLWozuuUSizvFPklOcxjS33w2Zwqp21PfKYBVMMySyh7MuwS5Ec3bGyRZ0Ux+PPuxi lAGXZKEmskubUXdJhdll17bgxfQLhSqAwx70+OrQU30mpcoBFgJ4RnkKY0xE6S7HznL8yluwQ2ghw iz6ft95g==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1qH1Te-00A3o9-Mw; Wed, 05 Jul 2023 12:19:22 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 4352A300023; Wed, 5 Jul 2023 14:19:21 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id 2CAC62022DED3; Wed, 5 Jul 2023 14:19:21 +0200 (CEST) Date: Wed, 5 Jul 2023 14:19:21 +0200 From: Peter Zijlstra To: "Huang, Kai" Cc: "kvm@vger.kernel.org" , "Raj, Ashok" , "Luck, Tony" , "david@redhat.com" , "bagasdotme@gmail.com" , "Hansen, Dave" , "ak@linux.intel.com" , "Wysocki, Rafael J" , "kirill.shutemov@linux.intel.com" , "Chatre, Reinette" , "Christopherson,, Sean" , "pbonzini@redhat.com" , "mingo@redhat.com" , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "nik.borisov@suse.com" , "hpa@zytor.com" , "Shahar, Sagi" , "imammedo@redhat.com" , "bp@alien8.de" , "Gao, Chao" , "Brown, Len" , "sathyanarayanan.kuppuswamy@linux.intel.com" , "Huang, Ying" , "Williams, Dan J" , "x86@kernel.org" Subject: Re: [PATCH v12 20/22] x86/virt/tdx: Allow SEAMCALL to handle #UD and #GP Message-ID: <20230705121921.GZ4253@hirez.programming.kicks-ass.net> References: <20230628152900.GI2438817@hirez.programming.kicks-ass.net> <20230628203823.GR38236@hirez.programming.kicks-ass.net> <42e13ccf7f27a68c0dd64640eed378c38ef40967.camel@intel.com> <20230630100659.GF2533791@hirez.programming.kicks-ass.net> <20230630102141.GA2534364@hirez.programming.kicks-ass.net> <20230630120650.GB2534364@hirez.programming.kicks-ass.net> <20230705102137.GX4253@hirez.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: B5ADC180012 X-Rspam-User: X-Stat-Signature: 1of78nhay811ucxqhf5iqbkb79ty5krs X-Rspamd-Server: rspam03 X-HE-Tag: 1688559571-526212 X-HE-Meta: U2FsdGVkX19wU373t3cprisDKhHMQOPaXnMzSVv9uk8Ir4ofbCa64+sgyVy2KIajr5xcbV3NpcLTh8Gk7M1nJVppl7dna3q641j2I9ZUZPQiRvnnOkpe7KUkbKKfhoZGGMBKqyjBBZlaPZNzizmo1iH8BCNctF/9QCjaKkpWgT3nOMaLFOdTNT9Qebc91lGpefGOvMu+Zktz2AkN22wOorQ+LrHpOvTwri/AqJPghftGP+GuiD7f1UPZOxeUrKaZREYINNo7JVyFws/tsVvmsMAn0jXC67+xDayJmo98r3aCRJVeQhTdKe4VnXbNa1pBSgLgBg5t6TeqQi+U/p0KHYIGvvV7bw3kTEvf7kqJ5eqTt6Mw9L8/7BI9ry2NcVnIWt2TKYs+cWf9KlYUaNSX0V62SI4WDjB7YvI++xXtMZ8QqbX3spkrurBpu65uemNPuqw0I2se1w3wxRGXKOIfhveW83pQ3KgEoCW/ENa7oxHROoKxpfB27ziBOvbxbIcEPeSJVhQyXTPZmTa1Z3TpwcZvpfyJG+ArFt+9M1jYV/hDtAsBLejr5GnBlfVsoPy8bGm2DLFWRFFTLhHWJhzAPlyIEWUo86CSiEhic1KLgh3vKHgTLKIAc8j8NuPRspB5Yq4GxX8VhBbbh7GDj1VktoXpQKW0v+AySDToW5NxXojv11nmIZmqdqgAvt869RodrsD9ldF+nqn4azQE2oOxk96OKHoy1flVFVqNl0++wfyWQO6hIlcly0d5mphqt5/aCP5/05YJRSNoZIAxcwWz1JRDbdFJguRRt05oCitegR83vRtffyYtynmd3e8B7oEYHPzPZO0xViH79fnH7vDPH2wIS1qKJt/O4nO+khb6gjIZOmqqWBcQsHfXYYTUoplzJPyKJLoKNfmO5JeLinxuXcMOtvlywhb1b8BM0fG31Fp8NV6xzr4rHXFEipjAjhFXfx2nUmzoasAFU+n76BQ 2puhsG6i qL6/CIWdNT5Ta/Ljoqvm2Z4Jjq5EdGHRPTZ0q/6oCEvL8smg2vz4okmJBPC5Yl2TXqWUvNJYQxM7M72KSw5ZQoVlVcz71QdZMrMKJsHOoV1cG2+lAX+BB7BlzwdtzOp6RKe0Wep5taTmPo59SUsBgSbxe0EVfM9dK+GhbWfBIpBEZaz95KwS+OOOmxA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000045, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jul 05, 2023 at 11:34:53AM +0000, Huang, Kai wrote: > Yeah I think from long-term's view, since SEAMCALLs to support live migration > pretty much uses all RCX/RDX/R8-R15 as input/output, it seems reasonable to > unify all of them, although I guess there might be some special handling to > VP.VMCALL and/or VP.ENTER, e.g., below: > > /* TDVMCALL leaf return code is in R10 */ > movq %r10, %rax > > So long-termly, I don't have objection to that. But my thinking is for the > first version of TDX host support, we don't have to support all SEAMCALLs but > only those involved in basic TDX support. Since those calls are out now, we should look at them now, there is no point in delaying the pain. That then gives us two options: - we accept them and their wonky calling convention and our code should be ready for it. - we reject them and send the TDX team a message to please try again but with a saner calling convention. Sticking our head in the sand and pretending like they don't exist isn't really a viable option at this point. > Also, the new SEAMCALLs to handle live migration all seem to have below > statement: > > AVX, AVX2 May be reset to the architectural INIT state > and > AVX512 > state > > Which means those SEAMCALLs need to preserve AVX* states too? Yes, we need to ensure the userspace 'FPU' state is saved before we call them. But I _think_ that KVM already does much of that. > And reading the spec, the VP.VMCALL and VP.ENTER also can use XMM0 - XMM15 as > input/output. Linux VP.VMCALL seems doesn't support using XMM0 - XMM15 as > input/output, but KVM can run other guest OSes too so I think KVM VP.ENTER needs > to handle XMM0-XMM15 as input/output too. Why would KVM accept VMCALLs it doesn't know about? Just trash the guest and call it a day. > That being said, I think although we can provide a common asm macro to cover > VP.ENTER, I suspect KVM still needs to do additional assembly around the macro > too. So I am not sure whether we should try to cover VP.ENTER. Not sure about asm, we have interfaces to save the XMM/AVX regs. kernel_fpu_begin() comes to mind, but I know there's more of that, including some for KVM specifically. > > I don't think they should be special, they're really just yet another > > leaf call. Yes, they have a shit calling convention, and yes VP.ENTER is > > terminally broken for unconditionally clobbering BP :-( > > > > That really *must* be fixed. > > Sure I don't have objection to this, and for VP.ENTER please see above. > > But I'd like to say that, generally speaking, from virtualization's point of > view, guest has its own BP and conceptually the hypervisor needs to restore > guest's BP before jumping to the guest. E.g., for normal VMX guest, KVM always > restores guest's BP before VMENTER (arch/x86/kvm/vmx/vmenter.S): > > SYM_FUNC_START(__vmx_vcpu_run) > push %_ASM_BP > mov %_ASM_SP, %_ASM_BP > > ... > mov VCPU_RBP(%_ASM_AX), %_ASM_BP > ... > vmenter/vmresume > ... > SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) > ..... > mov %_ASM_BP, VCPU_RBP(%_ASM_AX) > ... > pop %_ASM_BP > RET That's disgusting :/ So what happens if we get an NMI after VMENTER and before POP? Then it sees a garbage BP value. Why is all this stuff such utter crap?