From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24311C4CED1 for ; Fri, 4 Oct 2019 18:53:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C538A2053B for ; Fri, 4 Oct 2019 18:53:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=soleen.com header.i=@soleen.com header.b="WvXNBYcG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C538A2053B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EAD958E0010; Fri, 4 Oct 2019 14:53:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E105B8E0008; Fri, 4 Oct 2019 14:53:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD8208E0010; Fri, 4 Oct 2019 14:53:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id A6BA78E0008 for ; Fri, 4 Oct 2019 14:53:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 2939A613E for ; Fri, 4 Oct 2019 18:53:01 +0000 (UTC) X-FDA: 76006999362.15.hen89_1a4bcddda0443 X-HE-Tag: hen89_1a4bcddda0443 X-Filterd-Recvd-Size: 13144 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Fri, 4 Oct 2019 18:53:00 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id u186so6771142qkc.5 for ; Fri, 04 Oct 2019 11:53:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=1RZkZY1cm3fyZ1mRNLzSw31HBeKwrW9anL0qGMNckK8=; b=WvXNBYcGxyrCweNSNRilE4CngzncSKrPfdcdz29smszgfVpe2vL25tsu8NPu+wd57D jNH/YbSOQdQT1Gn3sNw1oC/pUZTi621aLdZOTFeJElmFD6oDzt35317vqJkMQmRWbiNr 3HZFAzscthwvM37JdLIxLngCvf6oJW0MB1FOEo0BIwoDXfaFqjmZgWv/Ok97U3yf+40K fHmjpukrn0FiV4ROC4l0TMdeq0vCSsd2msM/z2gLG38fqgkORQ+wPZxtolke1/9hUaUC EpMcaAZY5LlFcu1A3+w8JV8nlMkFlgV0X3LW9rdw4lRGbEnNmZDBugSC+s8zCsLg4ezG 4hNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1RZkZY1cm3fyZ1mRNLzSw31HBeKwrW9anL0qGMNckK8=; b=VBVbcX77Jq06vVQOQVEiXTFeuevrsYhhDdgQ1nFdDMgG2nXY2k5LdMBiZeRijggi2x tkp9965HcRuEbWKeVTg3Pydz+w2XM9EQfFumwYBCRdLWOCt0v50oGiujNKPj69HryJ5H tGiGIG2veA+w7svDtgdr8vNLHruPxYj07C7uBRhe65jLuehy5x3lAiQ2q5dLnivqRqaI xSHE6QY3N50n7W5O25gYDdULzswE1cecqv8nlZAB7Iahjus6DIXe2CfRnVKaQPc4QKm3 LqeBn7QfhhPrcKWOi7FDtsZrHWb1iordgbHWs6mjqcI0hQvh+DIzgquUfouEalVJa3VD dkcw== X-Gm-Message-State: APjAAAVKdqlL320m05WKdkXjd7H3w7/HtQZ3OZeqVHXxljbobcyIByRf 4FyHOB9zahN3DT4xWupbYuLo3A== X-Google-Smtp-Source: APXvYqxUuU67FEPU7NjrrVlytpALHcbwBn51HoFK5KIVhE8fOtIcGXFGGXEzcDfaAzB0IiGcIDlM1g== X-Received: by 2002:a37:6813:: with SMTP id d19mr11649474qkc.101.1570215179923; Fri, 04 Oct 2019 11:52:59 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id p77sm4042514qke.6.2019.10.04.11.52.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Oct 2019 11:52:59 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com, linux-mm@kvack.org, mark.rutland@arm.com Subject: [PATCH v6 16/17] arm64: kexec: configure trans_pgd page table for kexec Date: Fri, 4 Oct 2019 14:52:33 -0400 Message-Id: <20191004185234.31471-17-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191004185234.31471-1-pasha.tatashin@soleen.com> References: <20191004185234.31471-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Configure a page table located in kexec-safe memory that has the following mappings: 1. identity mapping for text of relocation function with executable permission. 2. identity mapping for argument for relocation function. 3. linear mappings for all source ranges 4. linear mappings for all destination ranges. Also, configure el2_vector, that is used to jump to new kernel from EL2 o= n non-VHE kernels. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 32 +++++++ arch/arm64/kernel/asm-offsets.c | 6 ++ arch/arm64/kernel/machine_kexec.c | 125 ++++++++++++++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 16 +++- 4 files changed, 170 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexe= c.h index d5b79d4c7fae..450d8440f597 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,23 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif =20 +#if defined(CONFIG_KEXEC_CORE) +/* Global variables for the arm64_relocate_new_kernel routine. */ +extern const unsigned char arm64_relocate_new_kernel[]; +extern const unsigned long arm64_relocate_new_kernel_size; + +/* Body of the vector for escalating to EL2 from relocation routine */ +extern const unsigned char kexec_el1_sync[]; +extern const unsigned long kexec_el1_sync_size; + +#define KEXEC_EL2_VECTOR_TABLE_SIZE 2048 +#define KEXEC_EL2_SYNC_OFFSET (KEXEC_EL2_VECTOR_TABLE_SIZE / 2) + +#endif + +#define KEXEC_SRC_START PAGE_OFFSET +#define KEXEC_DST_START (PAGE_OFFSET + \ + ((UL(0xffffffffffffffff) - PAGE_OFFSET) >> 1) + 1) /* * kern_reloc_arg is passed to kernel relocation function as an argument= . * head kimage->head, allows to traverse through relocation segments. @@ -97,6 +114,15 @@ static inline void crash_post_resume(void) {} * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other * arguments are currently unused, and must be set to 0 + * trans_ttbr0 idmap for relocation function and its argument + * trans_ttbr1 linear map for source/destination addresses. + * el2_vector If present means that relocation routine will go to EL1 + * from EL2 to do the copy, and then back to EL2 to do the jump + * to new world. This vector contains only the final jump + * instruction at KEXEC_EL2_SYNC_OFFSET. + * src_addr linear map for source pages. + * dst_addr linear map for destination pages. + * copy_len Number of bytes that need to be copied */ struct kern_reloc_arg { unsigned long head; @@ -105,6 +131,12 @@ struct kern_reloc_arg { unsigned long kern_arg1; unsigned long kern_arg2; unsigned long kern_arg3; + unsigned long trans_ttbr0; + unsigned long trans_ttbr1; + unsigned long el2_vector; + unsigned long src_addr; + unsigned long dst_addr; + unsigned long copy_len; }; =20 #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offs= ets.c index 900394907fd8..7c2ba09a8ceb 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -135,6 +135,12 @@ int main(void) DEFINE(KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); DEFINE(KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); + DEFINE(KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0= )); + DEFINE(KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1= )); + DEFINE(KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector))= ; + DEFINE(KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr)); + DEFINE(KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr)); + DEFINE(KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machin= e_kexec.c index fb6138a1c9ff..71479013dd24 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,13 +20,10 @@ #include #include #include +#include =20 #include "cpu-reset.h" =20 -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -74,15 +71,124 @@ static void *kexec_page_alloc(void *arg) return page_address(page); } =20 +/* + * Map source segments starting from KEXEC_SRC_START, and map destinatio= n + * segments starting from KEXEC_DST_START, and return size of copy in + * *copy_len argument. + * Relocation function essentially needs to do: + * memcpy(KEXEC_DST_START, KEXEC_SRC_START, copy_len); + */ +static int map_segments(struct kimage *kimage, pgd_t *pgdp, + struct trans_pgd_info *info, + unsigned long *copy_len) +{ + unsigned long *ptr =3D 0; + unsigned long dest =3D 0; + unsigned long src_va =3D KEXEC_SRC_START; + unsigned long dst_va =3D KEXEC_DST_START; + unsigned long len =3D 0; + unsigned long entry, addr; + int rc; + + for (entry =3D kimage->head; !(entry & IND_DONE); entry =3D *ptr++) { + addr =3D entry & PAGE_MASK; + + switch (entry & IND_FLAGS) { + case IND_DESTINATION: + dest =3D addr; + break; + case IND_INDIRECTION: + ptr =3D __va(addr); + if (rc) + return rc; + break; + case IND_SOURCE: + rc =3D trans_pgd_map_page(info, pgdp, __va(addr), + src_va, PAGE_KERNEL); + if (rc) + return rc; + rc =3D trans_pgd_map_page(info, pgdp, __va(dest), + dst_va, PAGE_KERNEL); + if (rc) + return rc; + dest +=3D PAGE_SIZE; + src_va +=3D PAGE_SIZE; + dst_va +=3D PAGE_SIZE; + len +=3D PAGE_SIZE; + } + } + *copy_len =3D len; + + return 0; +} + +static int mmu_relocate_setup(struct kimage *kimage, unsigned long kern_= reloc, + struct kern_reloc_arg *kern_reloc_arg) +{ + struct trans_pgd_info info =3D { + .trans_alloc_page =3D kexec_page_alloc, + .trans_alloc_arg =3D kimage, + }; + + pgd_t *trans_ttbr0 =3D kexec_page_alloc(kimage); + pgd_t *trans_ttbr1 =3D kexec_page_alloc(kimage); + int rc; + + if (!trans_ttbr0 || !trans_ttbr1) + return -ENOMEM; + + rc =3D map_segments(kimage, trans_ttbr1, &info, + &kern_reloc_arg->copy_len); + if (rc) + return rc; + + /* Map relocation function va =3D=3D pa */ + rc =3D trans_pgd_map_page(&info, trans_ttbr0, __va(kern_reloc), + kern_reloc, PAGE_KERNEL_EXEC); + if (rc) + return rc; + + /* Map relocation function argument va =3D=3D pa */ + rc =3D trans_pgd_map_page(&info, trans_ttbr0, kern_reloc_arg, + __pa(kern_reloc_arg), PAGE_KERNEL); + if (rc) + return rc; + + kern_reloc_arg->trans_ttbr0 =3D phys_to_ttbr(__pa(trans_ttbr0)); + kern_reloc_arg->trans_ttbr1 =3D phys_to_ttbr(__pa(trans_ttbr1)); + kern_reloc_arg->src_addr =3D KEXEC_SRC_START; + kern_reloc_arg->dst_addr =3D KEXEC_DST_START; + + return 0; +} + int machine_kexec_post_load(struct kimage *kimage) { + unsigned long el2_vector =3D 0; unsigned long kern_reloc; struct kern_reloc_arg *kern_reloc_arg; + int rc =3D 0; + + /* + * Sanity check that relocation function + el2_vector fit into one + * page. + */ + if (arm64_relocate_new_kernel_size > KEXEC_EL2_VECTOR_TABLE_SIZE) { + pr_err("can't fit relocation function and el2_vector in one page"); + return -ENOMEM; + } =20 kern_reloc =3D page_to_phys(kimage->control_code_page); memcpy(__va(kern_reloc), arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); =20 + /* Setup vector table only when EL2 is available, but no VHE */ + if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) { + el2_vector =3D kern_reloc + KEXEC_EL2_VECTOR_TABLE_SIZE; + memcpy(__va(el2_vector + KEXEC_EL2_SYNC_OFFSET), kexec_el1_sync, + kexec_el1_sync_size); + } + kern_reloc_arg =3D kexec_page_alloc(kimage); if (!kern_reloc_arg) return -ENOMEM; @@ -92,10 +198,19 @@ int machine_kexec_post_load(struct kimage *kimage) =20 kern_reloc_arg->head =3D kimage->head; kern_reloc_arg->entry_addr =3D kimage->start; + kern_reloc_arg->el2_vector =3D el2_vector; kern_reloc_arg->kern_arg0 =3D kimage->arch.dtb_mem; =20 + /* + * If relocation is not needed, we do not need to enable MMU in + * relocation routine, therefore do not create page tables for + * scenarios such as crash kernel + */ + if (!(kimage->head & IND_DONE)) + rc =3D mmu_relocate_setup(kimage, kern_reloc, kern_reloc_arg); + kexec_image_info(kimage); - return 0; + return rc; } =20 /** diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relo= cate_kernel.S index d352faf7cbe6..14243a678277 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -83,17 +83,25 @@ ENTRY(arm64_relocate_new_kernel) ldr x1, [x0, #KRELOC_KERN_ARG1] ldr x0, [x0, #KRELOC_KERN_ARG0] /* x0 =3D dtb address */ br x4 +.ltorg +.Larm64_relocate_new_kernel_end: END(arm64_relocate_new_kernel) =20 -.ltorg +ENTRY(kexec_el1_sync) + br x4 /* Jump to new world from el2 */ +.Lkexec_el1_sync_end: +END(kexec_el1_sync) + .align 3 /* To keep the 64-bit values below naturally aligned. */ -.Lcopy_end: .org KEXEC_CONTROL_PAGE_SIZE - /* * arm64_relocate_new_kernel_size - Number of bytes to copy to the * control_code_page. */ .globl arm64_relocate_new_kernel_size arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel + .quad .Larm64_relocate_new_kernel_end - arm64_relocate_new_kernel + +.globl kexec_el1_sync_size +kexec_el1_sync_size: + .quad .Lkexec_el1_sync_end - kexec_el1_sync --=20 2.23.0