From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4898CC001B0 for ; Wed, 16 Aug 2023 02:43:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA7FB8D001D; Tue, 15 Aug 2023 22:43:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D57188D0001; Tue, 15 Aug 2023 22:43:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C1F5E8D001D; Tue, 15 Aug 2023 22:43:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B2AA88D0001 for ; Tue, 15 Aug 2023 22:43:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 88F72A0E47 for ; Wed, 16 Aug 2023 02:43:53 +0000 (UTC) X-FDA: 81128422746.19.5DD5720 Received: from mail.loongson.cn (mail.loongson.cn [114.242.206.163]) by imf27.hostedemail.com (Postfix) with ESMTP id 7C4E040002 for ; Wed, 16 Aug 2023 02:43:50 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of maobibo@loongson.cn designates 114.242.206.163 as permitted sender) smtp.mailfrom=maobibo@loongson.cn ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692153831; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nvnZFXt5E2W8kVGLI0WyrHd1FsICbkq1fBKrQVeyTmM=; b=ZxFRLHQwfEd15fS87TUZJh02zlsRjC7RCe54iaEkNUOIR8d3wtnxmwHJu384DNwHIbZ+Ph EKon9L5PG3ALzROiQlDLrDki8gEmCJONlWY7Yasdv234jPxPsbK50t8y8AFzVeq37cvq6m XUwq2XClnBnuy7obvieQ/bW6zfWPUYg= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of maobibo@loongson.cn designates 114.242.206.163 as permitted sender) smtp.mailfrom=maobibo@loongson.cn ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692153831; a=rsa-sha256; cv=none; b=u347VfS1KJCmdIcoAKFX/fQ2D7hcowo3jd6OGqulrkupabuepVNaiGnd1SXCvc9eOgkeBQ q1VsMDF9WjL7OmmqQ1Rro+lr76zX//hzYAehhjv1/5cM2UFEAuSDhznxHhA33RCzl/JNEG TW9aNod2i2X7m/pgMF3IH6PeKcIp2Jk= Received: from loongson.cn (unknown [10.20.42.170]) by gateway (Coremail) with SMTP id _____8Cxc_DhN9xkL_sYAA--.51434S3; Wed, 16 Aug 2023 10:43:45 +0800 (CST) Received: from [10.20.42.170] (unknown [10.20.42.170]) by localhost.localdomain (Coremail) with SMTP id AQAAf8BxniPgN9xk0aJbAA--.53100S3; Wed, 16 Aug 2023 10:43:44 +0800 (CST) Message-ID: <107cdaaf-237f-16b9-ebe2-7eefd2b21f8f@loongson.cn> Date: Wed, 16 Aug 2023 10:43:44 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: [RFC PATCH v2 5/5] KVM: Unmap pages only when it's indeed protected for NUMA migration To: Sean Christopherson , Yan Zhao Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, pbonzini@redhat.com, mike.kravetz@oracle.com, apopple@nvidia.com, jgg@nvidia.com, rppt@kernel.org, akpm@linux-foundation.org, kevin.tian@intel.com, david@redhat.com References: <20230810085636.25914-1-yan.y.zhao@intel.com> <20230810090218.26244-1-yan.y.zhao@intel.com> <277ee023-dc94-6c23-20b2-7deba641f1b1@loongson.cn> Content-Language: en-US From: bibo mao In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-CM-TRANSID:AQAAf8BxniPgN9xk0aJbAA--.53100S3 X-CM-SenderInfo: xpdruxter6z05rqj20fqof0/ X-Coremail-Antispam: 1Uk129KBj93XoWxGF4kWFyUJF17ArW5ur15Awc_yoWrJF4fpF W5Ka18tF4DXrZ2kr97tw4xAFy2ga92gF18WryrK3sFyFn8tr92kw4xKrWa9FyfArn5Xr13 ta1jqFsxua4UZagCm3ZEXasCq-sJn29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7KY7ZEXa sCq-sGcSsGvfJ3Ic02F40EFcxC0VAKzVAqx4xG6I80ebIjqfuFe4nvWSU5nxnvy29KBjDU 0xBIdaVrnRJUUUv0b4IE77IF4wAFF20E14v26r1j6r4UM7CY07I20VC2zVCF04k26cxKx2 IYs7xG6rWj6s0DM7CIcVAFz4kK6r106r15M28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48v e4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI 0_Jr0_Gr1l84ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ Gr0_Gr1UM2AIxVAIcxkEcVAq07x20xvEncxIr21l57IF6xkI12xvs2x26I8E6xACxx1l5I 8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r106r15McIj6I8E87Iv67AK xVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IY64vIr41lc7I2V7IY0VAS07AlzV AYIcxG8wCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E 14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIx kGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAF wI0_Jr0_Gr1lIxAIcVCF04k26cxKx2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r 4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Jr0_GrUvcSsGvfC2KfnxnUUI43ZEXa7IU1EksDUU UUU== X-Rspam-User: X-Stat-Signature: 4dgzd8y7545izer3dh1urkn73yh7qcwt X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 7C4E040002 X-HE-Tag: 1692153830-125957 X-HE-Meta: U2FsdGVkX1/u3V/Jq/SDTMRth55o4dJvOGiaid1hMkCZ2ovrKlEG+ZJ2vQkpcyrZlrgAoIFLesr2HTjZ738Ifjw5bAZGvfkSGIcEisB/EK563JUcsWpAFXPE+ZmQeZQmAnK1MhmM1SyuV1tSs/VKgQ2JDWQcp05zpAWypry6RI+zyc/M4Vq6TVfA9hwKXZr5wGe5PSr07bqsyBx00PZy2yUfyTtTqNUmJvNhs+99Xrbhmqs57b1HsXAN09x+CqKY9omtqLMqkiVKsvPkU+WqCXU6ZVxK11AudoY49KHUvxCK2YKKVoUbyFDAbmjNgHD/GTsmaabH15pgSpUktBLm7WdF5YM5TY58TxDghZl6cihFbHLP7y6kDgL44uDXQSkzoE5TYcGCCyIy+PkQmEZOVHA/O1qRXDXEuLBYjteLMNvpdMkATntzQwomTIxIKsxlC7WPSHy0ROJF7DqTo0tr/XHoiAvGSMseUKxfKr6C9PcaJNyvnA6d1tuSKdaIjDl5mWrM1GBCmI9SKqkIElMnbiOYqDWRsmeh2CBKYx8fzokDcrR07YqfzG1UZ7pVNd94fGjlyRwo8oJbinDDv9bnrbzUKMqM9Z/wrxsG9ZsGBOszbFSWnan8mZtc+3xv+YJNhWhRyEbU4m2M8iOn7IJMgcSIcrI82c6UZKTAAKjjHuY+c7W19PkJ//7whLiuGRy/Pd42TaYv+8R5QTIJ3MrhP/3WL1LgDKhLzwR4iohzN0bnv30TlT7mEuKeS9fIvB6TXawdmUb9vADBzgt+/VySUyRFzGekBrSfHDnK7UQUOrWaq8tXqgX3AqyA0r0Z91csclhGe2G9je7JG6Z+qv/j6VlyrZxs9++PTpRj/yMYWwnx4E0Hi/8fJ1wBMu5L4JyKy+rW2EctCgTZMevikXfaAalFF3hbWgmRpHwuzU3DOsXQsP+jnknwAd73C5enyXY4WeWFjW/hDcO1YHZn3jN uRh0dh3w HRW7IqyuF3uJWwxvR5UgoLN2HWmzgrS4Sg7wGYOJAgw7Ir2MkDsJ+os0Krg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: 在 2023/8/15 22:50, Sean Christopherson 写道: > On Tue, Aug 15, 2023, Yan Zhao wrote: >> On Mon, Aug 14, 2023 at 09:40:44AM -0700, Sean Christopherson wrote: >>>>> Note, I'm assuming secondary MMUs aren't allowed to map swap entries... >>>>> >>>>> Compile tested only. >>>> >>>> I don't find a matching end to each >>>> mmu_notifier_invalidate_range_start_nonblock(). >>> >>> It pairs with existing call to mmu_notifier_invalidate_range_end() in change_pmd_range(): >>> >>> if (range.start) >>> mmu_notifier_invalidate_range_end(&range); >> No, It doesn't work for mmu_notifier_invalidate_range_start() sent in change_pte_range(), >> if we only want the range to include pages successfully set to PROT_NONE. > > Precise invalidation was a non-goal for my hack-a-patch. The intent was purely > to defer invalidation until it was actually needed, but still perform only a > single notification so as to batch the TLB flushes, e.g. the start() call still > used the original @end. > > The idea was to play nice with the scenario where nothing in a VMA could be migrated. > It was complete untested though, so it may not have actually done anything to reduce > the number of pointless invalidations. For numa-balance scenery, can original page still be used by application even if pte is changed with PROT_NONE? If it can be used, maybe we can zap shadow mmu and flush tlb in notification mmu_notifier_invalidate_range_end with precised range, the range can be cross-range between range mmu_gather and mmu_notifier_range. Regards Bibo Mao > >>> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c >>> index 9e4cd8b4a202..f29718a16211 100644 >>> --- a/arch/x86/kvm/mmu/mmu.c >>> +++ b/arch/x86/kvm/mmu/mmu.c >>> @@ -4345,6 +4345,9 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, >>> if (unlikely(!fault->slot)) >>> return kvm_handle_noslot_fault(vcpu, fault, access); >>> >>> + if (mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva)) >>> + return RET_PF_RETRY; >>> + >> This can effectively reduce the remote flush IPIs a lot! >> One Nit is that, maybe rmb() or READ_ONCE() is required for kvm->mmu_invalidate_range_start >> and kvm->mmu_invalidate_range_end. >> Otherwise, I'm somewhat worried about constant false positive and retry. > > If anything, this needs a READ_ONCE() on mmu_invalidate_in_progress. The ranges > aren't touched when when mmu_invalidate_in_progress goes to zero, so ensuring they > are reloaded wouldn't do anything. The key to making forward progress is seeing > that there is no in-progress invalidation. > > I did consider adding said READ_ONCE(), but practically speaking, constant false > positives are impossible. KVM will re-enter the guest when retrying, and there > is zero chance of the compiler avoiding reloads across VM-Enter+VM-Exit. > > I suppose in theory we might someday differentiate between "retry because a different > vCPU may have fixed the fault" and "retry because there's an in-progress invalidation", > and not bother re-entering the guest for the latter, e.g. have it try to yield > instead. > > All that said, READ_ONCE() on mmu_invalidate_in_progress should effectively be a > nop, so it wouldn't hurt to be paranoid in this case. > > Hmm, at that point, it probably makes sense to add a READ_ONCE() for mmu_invalidate_seq > too, e.g. so that a sufficiently clever compiler doesn't completely optimize away > the check. Losing the check wouldn't be problematic (false negatives are fine, > especially on that particular check), but the generated code would *look* buggy.