From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8DFFC433E0 for ; Mon, 25 Jan 2021 19:19:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4ECE420E65 for ; Mon, 25 Jan 2021 19:19:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4ECE420E65 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 19CBD8D0024; Mon, 25 Jan 2021 14:19:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 106078D0023; Mon, 25 Jan 2021 14:19:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E64B88D0024; Mon, 25 Jan 2021 14:19:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id CCA108D0001 for ; Mon, 25 Jan 2021 14:19:41 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 831851EE6 for ; Mon, 25 Jan 2021 19:19:41 +0000 (UTC) X-FDA: 77745261762.17.bead40_13161ee27588 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 5DC84180D0184 for ; Mon, 25 Jan 2021 19:19:41 +0000 (UTC) X-HE-Tag: bead40_13161ee27588 X-Filterd-Recvd-Size: 10928 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) by imf20.hostedemail.com (Postfix) with ESMTP for ; Mon, 25 Jan 2021 19:19:39 +0000 (UTC) Received: by mail-qk1-f180.google.com with SMTP id r77so13432946qka.12 for ; Mon, 25 Jan 2021 11:19:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=yxqxdjcYeXzHRYJR2uL7fg5YfMG49Nqr7zKyoy7ZCso=; b=B+qQeHZmm1nCpaer5jGR+2XL6vSG8oc6ubxBiW55/ZEpjT2ip14JUJefspWFfBQ6rt iBqlLQCI3KgXOUvLw0zhTVAQK5uxX/UjiKR/MkioK7T2r12lNsCoQBZbQn+9NVQ2oNuT YK+uko972boltBWppWe8whrFOtZ2Mhtqv/iP9xoIMVV3NqgpyuNg2c2fwqeyz4qeD/o3 ktZpoUs8f9jp3HUc1a5pQ/Utk2SPkmBTyIeWR5osMkCLaaoOXqGVKy3KDgzs4me7BHgM zboESC1tS7LywP8+X1Y0VNi5CfUjDoofhhyY29oDe1fIrxlNrkHZWRfJADOUB+qVFZGG qrAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yxqxdjcYeXzHRYJR2uL7fg5YfMG49Nqr7zKyoy7ZCso=; b=HvmvhPTB0rGRmfEendy4bgqBFkrArmwySouoWhySv3SfNvdTE/aSruB7YvKAxWGaGu 3sgOMfMJ5s+f5itUTg31uTn8Z7WN5ruNRjhSEs9B0/sBMrS20zXAJHrDDEKPoshr5wXZ pdb1gB2+praogwlLjv+XD9ncJDdf097Q9bTCfBzFLylajYZOeioqmaUSpPCpTnWlSfVD ENaMOoehrjyaLMakfEuTJ3WjCOtSvrPGsxV3k+pRGLXePjY8zgfmk3rR5f/EHfaWwMtW l7JXZSsDmFHFxpRwZSOkaQvIRiOCQYXZQVIPKCEheHsLfMCFxdQTKXPdpBtk/mLYLJeO P9hQ== X-Gm-Message-State: AOAM533roXtzu5O5mdEbWmpiZh/BHnjWUpOicBluouCdFL1tlhCSJKmi 19XK9gkjbSnNETb3YIlMzdHXkQ== X-Google-Smtp-Source: ABdhPJzSy2uQ+VCegZg1vz7vlssjTu7VocHSaS3LdXt5GoYDFqUz9exkpacYbL1y/NfjKgTESmOAjQ== X-Received: by 2002:a05:620a:1406:: with SMTP id d6mr2217884qkj.312.1611602377878; Mon, 25 Jan 2021 11:19:37 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id s6sm9047638qtx.63.2021.01.25.11.19.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 11:19:37 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v10 08/18] arm64: trans_pgd: hibernate: idmap the single page that holds the copy page routines Date: Mon, 25 Jan 2021 14:19:13 -0500 Message-Id: <20210125191923.1060122-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210125191923.1060122-1-pasha.tatashin@soleen.com> References: <20210125191923.1060122-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: James Morse To resume from hibernate, the contents of memory are restored from the swap image. This may overwrite any page, including the running kernel and its page tables. Hibernate copies the code it uses to do the restore into a single page that it knows won't be overwritten, and maps it with page tables built from pages that won't be overwritten. Today the address it uses for this mapping is arbitrary, but to allow kexec to reuse this code, it needs to be idmapped. To idmap the page we must avoid the kernel helpers that have VA_BITS baked in. Convert create_single_mapping() to take a single PA, and idmap it. The page tables are built in the reverse order to normal using pfn_pte() to stir in any bits between 52:48. T0SZ is always increased to cover 48bits, or 52 if the copy code has bits 52:48 in its PA. Signed-off-by: James Morse [Adopted the original patch from James to trans_pgd interface, so it can = be commonly used by both Kexec and Hibernate. Some minor clean-ups.] Signed-off-by: Pavel Tatashin Link: https://lore.kernel.org/linux-arm-kernel/20200115143322.214247-4-ja= mes.morse@arm.com/ --- arch/arm64/include/asm/trans_pgd.h | 3 ++ arch/arm64/kernel/hibernate.c | 32 +++++++------------ arch/arm64/mm/trans_pgd.c | 49 ++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 21 deletions(-) diff --git a/arch/arm64/include/asm/trans_pgd.h b/arch/arm64/include/asm/= trans_pgd.h index 7fbf6a3ccff7..5d08e5adf3d5 100644 --- a/arch/arm64/include/asm/trans_pgd.h +++ b/arch/arm64/include/asm/trans_pgd.h @@ -33,4 +33,7 @@ int trans_pgd_create_copy(struct trans_pgd_info *info, = pgd_t **trans_pgd, int trans_pgd_map_page(struct trans_pgd_info *info, pgd_t *trans_pgd, void *page, unsigned long dst_addr, pgprot_t pgprot); =20 +int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans= _ttbr0, + unsigned long *t0sz, void *page); + #endif /* _ASM_TRANS_TABLE_H */ diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.= c index 94fc275cdd21..9df32ba0d574 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -194,7 +194,6 @@ static void *hibernate_page_alloc(void *arg) * page system. */ static int create_safe_exec_page(void *src_start, size_t length, - unsigned long dst_addr, phys_addr_t *phys_dst_addr) { struct trans_pgd_info trans_info =3D { @@ -203,7 +202,8 @@ static int create_safe_exec_page(void *src_start, siz= e_t length, }; =20 void *page =3D (void *)get_safe_page(GFP_ATOMIC); - pgd_t *trans_pgd; + phys_addr_t trans_ttbr0; + unsigned long t0sz; int rc; =20 if (!page) @@ -211,13 +211,7 @@ static int create_safe_exec_page(void *src_start, si= ze_t length, =20 memcpy(page, src_start, length); __flush_icache_range((unsigned long)page, (unsigned long)page + length)= ; - - trans_pgd =3D (void *)get_safe_page(GFP_ATOMIC); - if (!trans_pgd) - return -ENOMEM; - - rc =3D trans_pgd_map_page(&trans_info, trans_pgd, page, dst_addr, - PAGE_KERNEL_EXEC); + rc =3D trans_pgd_idmap_page(&trans_info, &trans_ttbr0, &t0sz, page); if (rc) return rc; =20 @@ -230,12 +224,15 @@ static int create_safe_exec_page(void *src_start, s= ize_t length, * page, but TLBs may contain stale ASID-tagged entries (e.g. for EFI * runtime services), while for a userspace-driven test_resume cycle it * points to userspace page tables (and we must point it at a zero page - * ourselves). Elsewhere we only (un)install the idmap with preemption - * disabled, so T0SZ should be as required regardless. + * ourselves). + * + * We change T0SZ as part of installing the idmap. This is undone by + * cpu_uninstall_idmap() in __cpu_suspend_exit(). */ cpu_set_reserved_ttbr0(); local_flush_tlb_all(); - write_sysreg(phys_to_ttbr(virt_to_phys(trans_pgd)), ttbr0_el1); + __cpu_set_tcr_t0sz(t0sz); + write_sysreg(trans_ttbr0, ttbr0_el1); isb(); =20 *phys_dst_addr =3D virt_to_phys(page); @@ -434,7 +431,6 @@ int swsusp_arch_resume(void) void *zero_page; size_t exit_size; pgd_t *tmp_pg_dir; - phys_addr_t phys_hibernate_exit; void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *, void *, phys_addr_t, phys_addr_t); struct trans_pgd_info trans_info =3D { @@ -462,19 +458,13 @@ int swsusp_arch_resume(void) return -ENOMEM; } =20 - /* - * Locate the exit code in the bottom-but-one page, so that *NULL - * still has disastrous affects. - */ - hibernate_exit =3D (void *)PAGE_SIZE; exit_size =3D __hibernate_exit_text_end - __hibernate_exit_text_start; /* * Copy swsusp_arch_suspend_exit() to a safe page. This will generate * a new set of ttbr0 page tables and load them. */ rc =3D create_safe_exec_page(__hibernate_exit_text_start, exit_size, - (unsigned long)hibernate_exit, - &phys_hibernate_exit); + (phys_addr_t *)&hibernate_exit); if (rc) { pr_err("Failed to create safe executable page for hibernate_exit code.= \n"); return rc; @@ -493,7 +483,7 @@ int swsusp_arch_resume(void) * We can skip this step if we booted at EL1, or are running with VHE. */ if (el2_reset_needed()) { - phys_addr_t el2_vectors =3D phys_hibernate_exit; /* base */ + phys_addr_t el2_vectors =3D (phys_addr_t)hibernate_exit; el2_vectors +=3D hibernate_el2_vectors - __hibernate_exit_text_start; /* offset */ =20 diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index ded8e2ba0308..527f0a39c3da 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -273,3 +273,52 @@ int trans_pgd_map_page(struct trans_pgd_info *info, = pgd_t *trans_pgd, =20 return 0; } + +/* + * The page we want to idmap may be outside the range covered by VA_BITS= that + * can be built using the kernel's p?d_populate() helpers. As a one off,= for a + * single page, we build these page tables bottom up and just assume tha= t will + * need the maximum T0SZ. + * + * Returns 0 on success, and -ENOMEM on failure. + * On success trans_ttbr0 contains page table with idmapped page, t0sz i= s set to + * maximum T0SZ for this page. + */ +int trans_pgd_idmap_page(struct trans_pgd_info *info, phys_addr_t *trans= _ttbr0, + unsigned long *t0sz, void *page) +{ + phys_addr_t dst_addr =3D virt_to_phys(page); + unsigned long pfn =3D __phys_to_pfn(dst_addr); + int max_msb =3D (dst_addr & GENMASK(52, 48)) ? 51 : 47; + int bits_mapped =3D PAGE_SHIFT - 4; + unsigned long level_mask, prev_level_entry, *levels[4]; + int this_level, index, level_lsb, level_msb; + + dst_addr &=3D PAGE_MASK; + prev_level_entry =3D pte_val(pfn_pte(pfn, PAGE_KERNEL_EXEC)); + + for (this_level =3D 3; this_level >=3D 0; this_level--) { + levels[this_level] =3D trans_alloc(info); + if (!levels[this_level]) + return -ENOMEM; + + level_lsb =3D ARM64_HW_PGTABLE_LEVEL_SHIFT(this_level); + level_msb =3D min(level_lsb + bits_mapped, max_msb); + level_mask =3D GENMASK_ULL(level_msb, level_lsb); + + index =3D (dst_addr & level_mask) >> level_lsb; + *(levels[this_level] + index) =3D prev_level_entry; + + pfn =3D virt_to_pfn(levels[this_level]); + prev_level_entry =3D pte_val(pfn_pte(pfn, + __pgprot(PMD_TYPE_TABLE))); + + if (level_msb =3D=3D max_msb) + break; + } + + *trans_ttbr0 =3D phys_to_ttbr(__pfn_to_phys(pfn)); + *t0sz =3D TCR_T0SZ(max_msb + 1); + + return 0; +} --=20 2.25.1