From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02A4EC2BD09 for ; Wed, 4 Dec 2019 16:00:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AEC552073C for ; Wed, 4 Dec 2019 16:00:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="gn5Q4wOC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AEC552073C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 839956B0B78; Wed, 4 Dec 2019 11:00:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 81A786B0B79; Wed, 4 Dec 2019 11:00:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7003C6B0B7A; Wed, 4 Dec 2019 11:00:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 59CD96B0B78 for ; Wed, 4 Dec 2019 11:00:18 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 0BB3C180AD811 for ; Wed, 4 Dec 2019 16:00:18 +0000 (UTC) X-FDA: 76227920916.03.brain99_1fa2781d08c17 X-HE-Tag: brain99_1fa2781d08c17 X-Filterd-Recvd-Size: 9299 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 4 Dec 2019 16:00:17 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id c124so400684qkg.0 for ; Wed, 04 Dec 2019 08:00:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+hrNsEc0GRu30jUNwwgijvfvJA3Gw0vbGFinPeI+s7A=; b=gn5Q4wOC/CWivBpaA0eyfip0aKM+qjW33B6QF5A/NHl5LG9ptLiIgrJx0VgtTY1YxR DtuNvXHvp/qSyUrI8MlkiGRI0p7IjuXYhZfma9Xp0MS9/kM9mBGgBx19r1zPiRO2Qsgh 1EubFBjJyJ79d/HDRqSHhulsugbf5yARPr1DF21T60p4zhlo5btmqa9oDvrM9dsQWayQ vZWr5mTUBYOIB5IZGQN8kRZeVY1OVboOKhkrvN9XAc97uj9L4ivZ+7eVjgQS1i18AhMG R0yI4e2vdElgCmvOc1VtDY52eRFf95njifE5hCv/JTuVpDtRQ0dvkeXoNLKIIhdIoqzr Hj6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+hrNsEc0GRu30jUNwwgijvfvJA3Gw0vbGFinPeI+s7A=; b=erDzqGVMqq2bhPm33gKmYDYkGSCa1mCoZP7/wOd2vA98GX6i6NaknB6FdqB9SHtvGO A8akCRVtv8gQzG8aoLJN3ZmjRZQ0PJMS4VBAiiba3pfGr61ENKjTDb112FBUEJ44JQgu AnHMsOM+LZdqewkNa/UmccAo6c1+R/QiXVFMQCa/51V3fmMPdRIepE1SxS5INjliEq/h nxEqXEFrEj2LpqO2684BAGPRQN6RWHsuzszIxBU05qvXoQvG/Z6KH/yBOOLCztFHwJn2 I3NkEZtbSuLObzHSiOUueIff8XMssielD/MqY+IIifB3eu2h0riJf3/H6Vb9rPVofVnd gs3g== X-Gm-Message-State: APjAAAVM1SlSfhh9G2vsP+DO0q1zNXBiJg5Wr2XyPLRbyQ2V9J3Nhh52 FUFsXP6157UYHUvou7jRTIEQkA== X-Google-Smtp-Source: APXvYqxGHmN/Bb5BgVqDSbpZGghOEE64QncuRmgF6H7L6Kzy+CxFQPsdvZXIYBmorhLxk5bGaaM22w== X-Received: by 2002:a37:a40d:: with SMTP id n13mr3811268qke.167.1575475216583; Wed, 04 Dec 2019 08:00:16 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id w21sm4177585qth.17.2019.12.04.08.00.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2019 08:00:15 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de Subject: [PATCH v8 24/25] arm64: kexec: enable MMU during kexec relocation Date: Wed, 4 Dec 2019 10:59:37 -0500 Message-Id: <20191204155938.2279686-25-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191204155938.2279686-1-pasha.tatashin@soleen.com> References: <20191204155938.2279686-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now, that we have transitional page tables configured, temporarily enable MMU to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramf= s is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/relocate_kernel.S | 135 +++++++++++++++++----------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relo= cate_kernel.S index 67efa42575a5..e623b02f72a3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (c) 2019, Microsoft Corporation. + * Pavel Tatashin */ =20 #include @@ -15,6 +17,48 @@ #include =20 GLOBAL(kexec_relocate_code_start) +/* Invalidae TLB */ +.macro tlb_invalidate + dsb sy + dsb ish + tlbi vmalle1 + dsb ish + isb +.endm + +/* Turn-off mmu at level specified by sctlr */ +.macro turn_off_mmu sctlr, tmp1, tmp2 + mrs \tmp1, \sctlr + ldr \tmp2, =3DSCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr \sctlr, \tmp1 + isb +.endm + +/* Turn-on mmu at level specified by sctlr */ +.macro turn_on_mmu sctlr, tmp1, tmp2 + mrs \tmp1, \sctlr + ldr \tmp2, =3DSCTLR_ELx_FLAGS + orr \tmp1, \tmp1, \tmp2 + msr \sctlr, \tmp1 + ic iallu + dsb nsh + isb +.endm + +/* + * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temp= orarily + * set zero_page table. Invalidate TLB after new tables are set. + */ +.macro set_ttbr arg, tmp + ldr \tmp, [\arg, #KEXEC_KRELOC_TRANS_TTBR0] + msr ttbr0_el1, \tmp + ldr \tmp, [\arg, #KEXEC_KRELOC_TRANS_TTBR1] + offset_ttbr1 \tmp + msr ttbr1_el1, \tmp + isb +.endm =20 /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot i= t. @@ -26,65 +70,52 @@ GLOBAL(kexec_relocate_code_start) * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. = The * machine_kexec() routine will copy arm64_relocate_new_kernel to the ke= xec * safe memory that has been set up to be preserved during the copy oper= ation. + * + * This function temporarily enables MMU if kernel relocation is needed. + * Also, if we enter this function at EL2 on non-VHE kernel, we temporar= ily go + * to EL1 to enable MMU, and escalate back to EL2 at the end to do the j= ump to + * the new kernel. This is determined by presence of el2_vector. */ ENTRY(arm64_relocate_new_kernel) - /* Clear the sctlr_el2 flags. */ - mrs x2, CurrentEL - cmp x2, #CurrentEL_EL2 + mrs x1, CurrentEL + cmp x1, #CurrentEL_EL2 b.ne 1f - mrs x2, sctlr_el2 - ldr x1, =3DSCTLR_ELx_FLAGS - bic x2, x2, x1 - pre_disable_mmu_workaround - msr sctlr_el2, x2 - isb -1: /* Check if the new image needs relocation. */ - ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 =3D kimage_head */ - tbnz x16, IND_DONE_BIT, .Ldone - raw_dcache_line_size x15, x1 /* x15 =3D dcache line size */ -.Lloop: - and x12, x16, PAGE_MASK /* x12 =3D addr */ - /* Test the entry flags. */ -.Ltest_source: - tbz x16, IND_SOURCE_BIT, .Ltest_indirection - - /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 - b .Lnext -.Ltest_indirection: - tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - mov x14, x12 /* ptr =3D addr */ - b .Lnext -.Ltest_destination: - tbz x16, IND_DESTINATION_BIT, .Lnext - mov x13, x12 /* dest =3D addr */ -.Lnext: - ldr x16, [x14], #8 /* entry =3D *ptr++ */ - tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: - /* wait for writes from copy_page to finish */ - dsb nsh - ic iallu - dsb nsh - isb - - /* Start new image. */ - ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 =3D kimage_start */ + turn_off_mmu sctlr_el2, x1, x2 /* Turn off MMU at EL2 */ +1: mov x20, xzr /* x20 will hold vector value */ + ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN] + cbz x11, 5f /* Check if need to relocate */ + ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR] + cbz x20, 2f /* need to reduce to EL1? */ + msr vbar_el2, x20 /* el2_vector present, means */ + adr x1, 2f /* we will do copy in el1 but */ + msr elr_el2, x1 /* do final jump from el2 */ + eret /* Reduce to EL1 */ +2: set_ttbr x0, x1 /* Set our page tables */ + tlb_invalidate + turn_on_mmu sctlr_el1, x1, x2 /* Turn MMU back on */ + ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; + ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; + mov x12, x1 /* x12 dst backup */ +3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 + sub x11, x11, #PAGE_SIZE + cbnz x11, 3b /* page copy loop */ + raw_dcache_line_size x2, x3 /* x2 =3D dcache line size */ + sub x3, x2, #1 /* x3 =3D dcache_size - 1 */ + bic x12, x12, x3 +4: dc cvau, x12 /* Flush D-cache */ + add x12, x12, x2 + cmp x12, x1 /* Compare to dst + len */ + b.ne 4b /* D-cache flush loop */ + turn_off_mmu sctlr_el1, x1, x2 /* Turn off MMU */ + tlb_invalidate /* Invalidate TLB */ +5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 =3D kimage_start */ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 =3D dtb address */ - br x4 + cbnz x20, 6f /* need to escalate to el2? */ + br x4 /* Jump to new world */ +6: hvc #0 /* enters kexec_el1_sync */ .ltorg END(arm64_relocate_new_kernel) =20 --=20 2.24.0