From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28C17C3DA4A for ; Fri, 2 Aug 2024 22:40:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D9886B0085; Fri, 2 Aug 2024 18:40:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5624B6B0088; Fri, 2 Aug 2024 18:40:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DC416B0089; Fri, 2 Aug 2024 18:40:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1B6EC6B0085 for ; Fri, 2 Aug 2024 18:40:51 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AE2CCC1529 for ; Fri, 2 Aug 2024 22:40:50 +0000 (UTC) X-FDA: 82408776660.06.8F2573D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id 7788418000C for ; Fri, 2 Aug 2024 22:40:48 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C5hSFL3U; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722638390; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UIQ24+48TE0Ta9LNm8rRrMJXnuaT0qzUCWhnHkojtRo=; b=n0QlaAzMI236CtctK8s7S3reS8o6pkXQXdbjrDn2/kxuWGNRgKE3ya12dj/VRafVmffyPC JkslKBMZIq6t798XpoOWEj5iamflZjvsCs3cJGwSLBSHKUKm+A0iI4NTEv2hNxwtBoyFXk u+kKpoj3jLFwzD1YQBWgx4wuVUBMK6U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722638390; a=rsa-sha256; cv=none; b=wkTb4Z+xBb8vLb2AbIjc2bYEsNA/E+xvZcCYg2HnKHFS8oS3+FNRczwNzwT+JX3wX2B8AN NCB/oUYJWn31Sz8QDufPWrqyiVL6Mjpm59IbkrdYWMHC/UlGTklCgyvrNZZ9Qt5rhcGrKK QuVA00A9Y2Ldt1vHIk7yu+vJIxBznlc= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=C5hSFL3U; spf=pass (imf24.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722638447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=UIQ24+48TE0Ta9LNm8rRrMJXnuaT0qzUCWhnHkojtRo=; b=C5hSFL3UBYsGPDVqyTmba/8c/CqjFUOS4CmD/mWs5YP5mL2Qjcw+urMQmCL3jgFSdDGGKS So9nqwpc5MLZb7gXeufCdQLSZDIVa0aau6sEsUXbfHg6DOdgJWTelOLKS2ecmDw10TOxo2 Mt1F5EPFndv4uRzxV5O2RCS16o4EqSc= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-260-F7QkPTllPAuIaj9PHaC_ew-1; Fri, 02 Aug 2024 18:40:46 -0400 X-MC-Unique: F7QkPTllPAuIaj9PHaC_ew-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6bba0f9d3efso1938606d6.3 for ; Fri, 02 Aug 2024 15:40:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722638446; x=1723243246; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=UIQ24+48TE0Ta9LNm8rRrMJXnuaT0qzUCWhnHkojtRo=; b=f2lSqLUM3oGkMPjX0s5goXf7wGeLG4HTaEO7JMjeAgRHQq2gHRsvjxav3OgUeLpZVw diDDJjkw0fRoW/7I91ZCu/v1yOIGOnEWdl9c4PfvJfvpHXihZGIMPajrq50VD0JmVlh8 27cjxL4m6jkvwUkY2VbXpD+Io7g3HcY8GiruiiIJwo1RurXO7c9i4ZEfegCbGgtdFID5 TVSp1s6IXITsfOkYyMpgfGoVN0ePLceM10DDgNh7oxppb/DFy+QAhqpD4o/diMf0n/lr 0ebcpkUfyUwV/Q4CeG1bA+Hw1k3MHQi+NC8SmCfZ+IigZuvbAbO7CM3uM9JF9AvxJD4H QrzQ== X-Forwarded-Encrypted: i=1; AJvYcCUSFyBq0bhCMqMucwEMSBw3Fxvtif6Y0FR7X3I+GzURz4Awtus6e6XKDnPtjXng+7AN4l3CxzM70Q==@kvack.org X-Gm-Message-State: AOJu0YykhjKcJcLOPwoIMdT9GC2cGF86oafwRPuJPPHyau/Jtnj2uemK /OSHn2Jrabx8S7njKtpQ6nwpCAbwhXfbXtwm6P7LYpnC7wN1AFaSNo1XCvDZOolDsQmM8lTongS o3OURExHIyICCAOUlxhwksMTxt+NW+R7aKDpCXvdA6j6l3FuD X-Received: by 2002:a05:6214:3008:b0:6b7:885c:b750 with SMTP id 6a1803df08f44-6bb9832e268mr35451856d6.1.1722638445946; Fri, 02 Aug 2024 15:40:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGGuQ/TysjxPmr8e9IL8H+rnOjczzC8Pn9LtVkMqryMZ3GwIYnbZ1EzE5cR19PNfrempSFmMA== X-Received: by 2002:a05:6214:3008:b0:6b7:885c:b750 with SMTP id 6a1803df08f44-6bb9832e268mr35451646d6.1.1722638445539; Fri, 02 Aug 2024 15:40:45 -0700 (PDT) Received: from x1n (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6bb9c86baa8sm10847796d6.118.2024.08.02.15.40.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Aug 2024 15:40:45 -0700 (PDT) Date: Fri, 2 Aug 2024 18:40:42 -0400 From: Peter Xu To: David Woodhouse Cc: Carsten Stollmaier , Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , nh-open-source@amazon.com, Sebastian Biemueller , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Matthew Wilcox , Andrew Morton , "linux-mm@kvack.org" Subject: Re: [PATCH] KVM: x86: Use gfn_to_pfn_cache for steal_time Message-ID: References: <20240802114402.96669-1-stollmc@amazon.com> MIME-Version: 1.0 In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7788418000C X-Stat-Signature: zuhx7sdaiqbstihrypqcqj8mcpo834h3 X-HE-Tag: 1722638448-919898 X-HE-Meta: U2FsdGVkX1/1GvhSvvq4Q7QTzC/VL+hg3XcbMqeTaaviRjX7j86hflbXRcidIm8UHb4j6HhvJQrG0hRJi703H94enTRiAvM2hH47AUeZzNLwP/KarUg0P4IpxUEqJPvM3FXjSYynlKR1RDZMIHA6P9C7H+X+5KEyjFLmwmHPHn4I9O6+IHJM05R2aQh6RqpbZ2YyQjO3SNpNhDpBbbg07Ywq+grzyUlIEcVeZElCHzZ2x9e1m55pPJDsqTYiqneN1VTuukeSco76lTOynERi/AP6tXqyDvJi/oWUmahBV5Ntgb1RVI/wl5YDwfIzvOEFUICsJ3vmWdL39goQeFSzKplQcIoVtXXz4Q/Sqz9WPGR3bgZdu07HO77NJex/HeUwC6BulJ6xdkc3AdrgR38YeSBtMMChn3sntzQd+zExgrFo8dTK6iAAlVWaSPuB9xIuNCdhGTMANbarX/6ed+3DNqHlujIi5cvIy03obHB5Lr2stwTRBF9f20LoKiXwAaAWdqZ1odBCr5pc8qh3+t+lccT8kvOZsrw0vru9MKLarkkpXwHVsEmNPCQElYREXQtyTt1UFMQi9chjBFt79Gi7tAMmqV47U3Ettemrue0z3DD/l+RuKXJlpg7xrE8DOm955AJ87Yju5Jpz0PTrQtTBAmGswCTWv1yci2adfU9nYswDxlV/psqJYtxKcX529T01JSOivBhUnezf6pgcxV1sKtHF7ANzpD7JbuXPwez1UDmNJgniD1Cbpg6ZwNwKmtwzWZd3T2mRBlyMPC9NJVbIfjorsk8s/ExLKZDC4XSnaRuD1JnR8yLsFExwcbR/cKM50rMsPmI8gtMzYHyaRfdsYkZ1cr4uh2fJrRVXRDwd75b/6YRyelgJkctluLNkywH4T3Ux532/D+0ysQap+PKl4Wz3gs3F4tjDR80VNS6RhUpIayvt7ioKF/IodNvZUFuJwlVD30sEVyKqyjg/O2A /E8auQY2 XViojIjmwThWtYw8v4hrmIE6TxScrmW7esNWJ6VObXgwsEjrzXHXzRWvPWccEMIII+LJf29+h6QrtLbLfrPNT8pFGecIIaopkGuS7DH3/CnEx7nGzp13mLYRJXMGlMCRu6MDH+bxDzMXjaxEY0EVi7/FtwTpI2LTNno/ld7xczd5ppPhuxOyFd6QsDOXoc5kLEJZNUXv+f273YqcjbeWXWI0Ejrv/D78vaZQyQTwYXg3i1JcYNfqNxQzYp0MmC1Mw3XZXia/hEPmLuBoA6YgQ5S8XuyNLMGOV518+UGp6jPImrSyEc3ANR7nR5nN7PuOtaAvz4wLbnpQfG5k+XgpMLg/G5Cb8ZaUeBJbFJ9tSZHWiADhcgoYyznmPxLZRPR6PE6IjXySzHl6KsuCHNT2zsve4WXMb62rUQ3n0AspCg7cKkwwezgsYBq4Tlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Aug 02, 2024 at 01:03:16PM +0100, David Woodhouse wrote: > On Fri, 2024-08-02 at 11:44 +0000, Carsten Stollmaier wrote: > > On vcpu_run, before entering the guest, the update of the steal time > > information causes a page-fault if the page is not present. In our > > scenario, this gets handled by do_user_addr_fault and successively > > handle_userfault since we have the region registered to that. > > > > handle_userfault uses TASK_INTERRUPTIBLE, so it is interruptible by > > signals. do_user_addr_fault then busy-retries it if the pending signal > > is non-fatal. This leads to contention of the mmap_lock. > > The busy-loop causes so much contention on mmap_lock that post-copy > live migration fails to make progress, and is leading to failures. Yes? > > > This patch replaces the use of gfn_to_hva_cache with gfn_to_pfn_cache, > > as gfn_to_pfn_cache ensures page presence for the memory access, > > preventing the contention of the mmap_lock. > > > > Signed-off-by: Carsten Stollmaier > > Reviewed-by: David Woodhouse > > I think this makes sense on its own, as it addresses the specific case > where KVM is *likely* to be touching a userfaulted (guest) page. And it > allows us to ditch yet another explicit asm exception handler. > > We should note, though, that in terms of the original problem described > above, it's a bit of a workaround. It just means that by using > kvm_gpc_refresh() to obtain the user page, we end up in > handle_userfault() without the FAULT_FLAG_INTERRUPTIBLE flag. > > (Note to self: should kvm_gpc_refresh() take fault flags, to allow > interruptible and killable modes to be selected by its caller?) > > > An alternative workaround (which perhaps we should *also* consider) > looked like this (plus some suitable code comment, of course): > > --- a/arch/x86/mm/fault.c > +++ b/arch/x86/mm/fault.c > @@ -1304,6 +1304,8 @@ void do_user_addr_fault(struct pt_regs *regs, > */ > if (user_mode(regs)) > flags |= FAULT_FLAG_USER; > + else > + flags &= ~FAULT_FLAG_INTERRUPTIBLE; > > #ifdef CONFIG_X86_64 > /* > > > That would *also* handle arbitrary copy_to_user/copy_from_user() to > userfault pages, which could theoretically hit the same busy loop. > > I'm actually tempted to make user access *interruptible* though, and > either add copy_{from,to}_user_interruptible() or change the semantics > of the existing ones (which I believe are already killable). > > That would require each architecture implementing interruptible > exceptions, by doing an extable lookup before the retry. Not overly > complex, but needs to be done for all architectures (although not at > once; we could live with not-yet-done architectures just remaining > killable). > > Thoughts? Instead of "interruptible exception" or the original patch (which might still be worthwhile, though? I didn't follow much on kvm and the new gpc cache, but looks still nicer than get/put user from initial glance), above looks like the easier and complete solution to me. For "completeness", I mean I am not sure how many other copy_to/from_user() code in kvm can hit this, so looks like still possible to hit outside steal time page? I thought only the slow fault path was involved in INTERRUPTIBLE thing and that was the plan, but I guess I overlooked how the default value could affect copy to/from user invoked from KVM as well.. With above patch to drop FAULT_FLAG_INTERRUPTIBLE for !user, KVM can still opt-in INTERRUPTIBLE anywhere by leveraging hva_to_pfn[_slow]() API, which is "INTERRUPTIBLE"-ready with a boolean the caller can set. But the caller will need to be able to process KVM_PFN_ERR_SIGPENDING. Thanks, -- Peter Xu