From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 781F3C433E3 for ; Mon, 27 Jul 2020 14:52:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 405DE20775 for ; Mon, 27 Jul 2020 14:52:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 405DE20775 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB6C06B0002; Mon, 27 Jul 2020 10:52:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3FB96B0003; Mon, 27 Jul 2020 10:52:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 907E26B0005; Mon, 27 Jul 2020 10:52:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id 775666B0002 for ; Mon, 27 Jul 2020 10:52:13 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 05C2E181AC9BF for ; Mon, 27 Jul 2020 14:52:13 +0000 (UTC) X-FDA: 77084146146.01.cook59_391031226f62 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id C11A910046460 for ; Mon, 27 Jul 2020 14:52:12 +0000 (UTC) X-HE-Tag: cook59_391031226f62 X-Filterd-Recvd-Size: 3492 Received: from huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Mon, 27 Jul 2020 14:52:11 +0000 (UTC) Received: from DGGEMS414-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 1CF99BE30548D6D8FEA0; Mon, 27 Jul 2020 22:51:52 +0800 (CST) Received: from [127.0.0.1] (10.174.186.173) by DGGEMS414-HUB.china.huawei.com (10.3.19.214) with Microsoft SMTP Server id 14.3.487.0; Mon, 27 Jul 2020 22:51:43 +0800 Subject: Re: [RESEND RFC PATCH v1] arm64: kvm: flush tlbs by range in unmap_stage2_range function To: Marc Zyngier CC: , , , , , , , , , , , , , , , References: <20200724134315.805-1-yezhenyu2@huawei.com> <5d54c860b3b4e7a98e4d53397e6424ae@kernel.org> From: Zhenyu Ye Message-ID: Date: Mon, 27 Jul 2020 22:51:41 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 In-Reply-To: <5d54c860b3b4e7a98e4d53397e6424ae@kernel.org> Content-Type: text/plain; charset="UTF-8" X-Originating-IP: [10.174.186.173] X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: C11A910046460 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Marc, On 2020/7/26 1:40, Marc Zyngier wrote: > On 2020-07-24 14:43, Zhenyu Ye wrote: >> Now in unmap_stage2_range(), we flush tlbs one by one just after the >> corresponding pages cleared.=C2=A0 However, this may cause some perfor= mance >> problems when the unmap range is very large (such as when the vm >> migration rollback, this may cause vm downtime too loog). >=20 > You keep resending this patch, but you don't give any numbers > that would back your assertion. I have tested the downtime of vm migration rollback on arm64, and found the downtime could even take up to 7s. Then I traced the cost of unmap_stage2_range() and found it could take a maximum of 1.2s. The vm configuration is as follows (with high memory pressure, the dirty rate is about 500MB/s): 192 48 After this patch applied, the cost of unmap_stage2_range() can reduce to 16ms, and VM downtime can be less than 1s. The following figure shows a clear comparison: | vm downtime | cost of unmap_stage2_range() --------------+--------------+---------------------------------- before change | 7s | 1200 ms after change | 1s | 16 ms --------------+--------------+---------------------------------- >> + >> +=C2=A0=C2=A0=C2=A0 if ((end - start) >=3D 512 << (PAGE_SHIFT - 12)) { >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __tlbi(vmalls12e1is); >=20 > And what is this magic value based on? You don't even mention in the > commit log that you are taking this shortcut. >=20 If the page num is bigger than 512, flush all tlbs of this vm to avoid soft lock-ups on large TLB flushing ranges. Just like what the flush_tlb_range() does. Thanks, Zhenyu