From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 028B3C282DE for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 93BC6280016; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8781E280010; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C847280016; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4D5BE280010 for ; Thu, 13 Mar 2025 14:11:58 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C87B080432 for ; Thu, 13 Mar 2025 18:11:59 +0000 (UTC) X-FDA: 83217321558.01.ABE87AF Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf06.hostedemail.com (Postfix) with ESMTP id 48B91180007 for ; Thu, 13 Mar 2025 18:11:52 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IzJSOY8p; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889513; a=rsa-sha256; cv=none; b=PWtIVyS/+IYYvYd+YP19F6yBt3jVxuZld8jCIEWJfIcszw/Y4Qy0LsDVwZO9OflK/A0Ju6 BUAFp+6bMKiCguOUbb1FLJRX+d2alyKq3+BXP3ZVdIry9gXP9iQMV3VLZEsxVMWu1YKMiV tSx1M+gvVwyZ46oprRSI0z6sZV0KLJw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=IzJSOY8p; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=35h_TZwgKCLcgXZhjXkYdlldib.Zljifkru-jjhsXZh.lod@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889513; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=BQ9qRzrPEN1GcbrK0z/y6HJjUqoMRcgLmlmWfOqlnG0znq3yhd9OwDEWGAxyqP5kPSZTh4 Dyw8f2d8xumpOgs96KIsJSKa0wzbUsbRocUZRXcFE5fkKbtGEsQ2PIWrwArDCJMfITAbWi OmDfTqN7mkTzxjDJARHy4Jmoh9MZ/KY= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-438e4e9a53fso9453255e9.1 for ; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889511; x=1742494311; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=IzJSOY8pX1aTHn4OOp/egkqnRIKxQEOWzcaS7TaQOCpFBiL4XhRUj+hVo8mrimUC/G G4WFmCL2ugr+Gre7c4bfu8bjWJUSwDx18U2EDGnNgoQpbb/Z/Kvde42h04CTIdhpJqrE YEhY8wyY0H/12yb5IX9rOrukdrI/n7QcloZdi+LUYhGekLEqi/DI+koZ4iyAv5J3lmvC 2pBn4siEQ4H6lDEZnTxJRzFQMGg5YZGzkREqXtBM5Al++V0MX+8RWNqGvPZFdeuM6Q2y s04PTbYZAtR+SQCYh5I+GOfnbsiw7/OQfv6+lStlP+tUt7M+7sBI741baoJzKYUYm4iL zBTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889511; x=1742494311; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=qReEEc666AMqIDQRUym2Wwf4KbAvZHrrIxFKZ4H3hE56dztf07jKJUu/wcFjM7tJSW 4S0ILKEhiq886uAEj/dK6+SODETiUmrHWAgoyO05kgi2fuRs1gIwbx4nzJHIjCqSLEjI bqIyojOE5hTjkjKbAnCjbZBvgWffJCu1wl58g+EF8oVgn6boxFlHtR0bW5cPqciA6fDv vXJtXb1fPBWJoTJtf5z0LjaaT2tOcUg7mfjBdT99livD7hq1F6rCy0Ck2njjTFFo/jz1 B9Jo427U0SIsdHT0nQ2MvZzRtjnV8qxXTP8Df/XhJIt5OwKnuh/vqwAXLCs7pBlYCy/a 5HfA== X-Forwarded-Encrypted: i=1; AJvYcCWgfst0qZCHBdTcikRvqmP457kwYdH09xDy6/86bNQdjSjOzsCkmNNLOrlJnhADD0NGRiM3K0e0/w==@kvack.org X-Gm-Message-State: AOJu0YzsIS1O2FeC4IozuPo8zks7qPwaUd3zPUnjy6qQvqV2cxOQe9w7 l9E4ttp3zUb1qZtSsWhxSdqlCscKdde1mq8XzdrjyiGo2iKCDdv+ruh0VAcLon86qMzLWRawddp w1W4wuFLQqQ== X-Google-Smtp-Source: AGHT+IF292MH6JqAkiIN8jtyln5hVYHbTRuF9S/Wm78jwx44s0FdKYuBZ0MucR5gttxIvSprRDkRwKMLpsnBdw== X-Received: from wmbfj6.prod.google.com ([2002:a05:600c:c86:b0:43c:f256:f4b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d12:b0:43c:eec7:eab7 with SMTP id 5b1f17b1804b1-43d01bdd224mr152891025e9.11.1741889510788; Thu, 13 Mar 2025 11:11:50 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:24 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-5-04972e046cea@google.com> Subject: [PATCH RFC HACKS 05/11] Add asi_map() and asi_unmap() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" X-Rspam-User: X-Rspamd-Queue-Id: 48B91180007 X-Rspamd-Server: rspam05 X-Stat-Signature: cwdnmpmz7j4j1pzka83csr7d9g5rmksb X-HE-Tag: 1741889512-775319 X-HE-Meta: U2FsdGVkX18kc3++nFYNev/K0E7QwPXqFE7Y6cN99xsmgU4oZVWGEYGglOF3i65SjB3zbsH8c/fjttyglzLx7kfhS/PBNtsa/AqXo9XXB8lO9MREsK1L7HBHprL1TUpX5Uo8nuLVkNvXfOuBnBfWNQ9j0ThtReJGLKDRwrBlkxUs5gHLDY6tUiqaDvGsMJEcHIzqP3C4XH/zltMj/MZZgfyxJXaJGZih4KPoxFRee2xiNB0CRrcRhHBi4/25jXT5yKK2ygXHDDTuO3vJloVeyNmIBoAfZfwgE4G62IV3YoI0Jt1QWy806N45iw48126pDHd7RpmUSP1qRIHMaVwsaRQ57Dw6Jm5zXUt/25Cv7gnaXCg3yWamgN5Y6817U8UtL7yg2idhVrp6gPEoLfq7FwWUgYn6YTRpatJi3pk9z1HdM7rNQ3IancwhszS2fSesSjybuxGBOWRCt935ACOQjkm3F8+beRHAfEdHW8C/GsfmjEG74CTJ7hD3b4LSfgThGMaJSb/ptUg+9p8bGyIYITsE9wv1xeyuBoP+9J5xg8m7FOFiOGWMyZuZLtwGqGB9qQ8PJh4030JeeNhfx12QCuR873QAAy3bVRwcZDB7SkPcLyyOfF2QYbSsDdfVtPzNfzkbdrxHPalW/S+mFxzCfyYcal34/yrIEhVrvyG4e/NUqF/FI468H9y2ztXThUdi0Q0FMobcP2X62TyEXKEyXaKMpAyQyUJYmllDUIU50VqXzxP6JYX+hnEgMfeUavyQbngI9cE6CZnn+UOiSxuric6v5zgITB+LjzS5By1MhiNJ3Ic1UpaGy0mY0kfNp3yxthFoYovd2Vm7rBUlHsLxI7sAzw2o6G4pVjRAjm/lubFnrp+XFaI222HVFHGTjL0t39pKQCDjVJIYfwoJczNfRklQy++yQfchowENlkbBiGFJagY1KkcRcrTiJSW/n59fZxTO34tberzGrKVNtYH Yd43Kugq h/NMBixFYIe2HS0p9FRSchO2LEW/tPAyIQxITXnEP0Mrk0rmuMqZ2OnZPTiYWkitWmvqsa+vR9JSXvQZqIwmSgsj9O3/duTtFNdhysAeHpr7AlecpNbiMshkJJVPO9rtMs+CYkW6js9RkjEhpHQsIuedDNMXip2d66ERJxP+JRt+O4iRKe9LuLAZ+uIATINrNyaa+R3O80MrzqcjUbjeFHsyR4qOWl5pHKFu8eyJss5p1xhd3EVQ3M7JMTG2ZcBRgerVEzORVf5+VAPR1bSKjyOFL41m4JKsb7vFnP+k0D9aoW+Pw0ZHrOWsjhuyBTK0eNpxKcFFteAxoXOBY6gdhOK54Kw+Qh7ymmLwnwo+NhKC8haXycTu7dag6tBTp2+LqwTW9iFoioLLemYl6S13lFJxEZOKoodIUUPMVPfRBoiOyTbPeieAaVOvXjoUIMm6u1xkjo7AgzpUhUTbREtU4SAAC1O6oyNQ3cdvAtU+COgMu4HwFlBCU9V51ndt9PzSpm6ChEZeKkhEgFVkpbrV/7XtFxbn4xoGBwYPnO4qCwFf9XIl0xIa6oEqsC1eQcsDt2Ym3Kvfd2JNYEuZuRgTmelyaCyJS6v5UgGt5R+oRTCu5oh4vOl9b6cK9XaJEDUL/9Mb5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do not worry too much about the implementation of this. I am trying to implement this more neatly using the x86 PAT logic but haven't managed to get it working in time for the RFC. To enable testing & review, I'm very hastily throwing something together that basically works, based on a simplified version of what was used for the latest RFC [0]. [0] https://lore.kernel.org/linux-mm/20250110-asi-rfc-v2-v2-0-8419288bc805@google.com/ Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 3 ++ arch/x86/mm/asi.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++ include/linux/asi.h | 2 ++ include/linux/vmalloc.h | 4 +++ mm/vmalloc.c | 32 +++++++++++-------- 5 files changed, 105 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index b8f604df6a36508acbc10710f821d5f95e8cdceb..cf8be544de8b108190b765e3eb337089866207a2 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -23,6 +23,9 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } +void asi_map(struct page *page, int numpages); +void asi_unmap(struct page *page, int numpages); + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index e5a981a7b3192655cd981633514fbf945b92c9b6..570233224789631352891f47ac2f0453a7adc06e 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,8 +1,85 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include + #include +#include static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD]; struct asi __asi_global_nonsensitive = { .pgd = asi_global_nonsensitive_pgd, }; + +/* + * Map the given pages into the ASI nonsensitive physmap. The source of the + * mapping is the regular unrestricted page tables. Only supports mapping at + * pageblock granularity. Does no synchronization. + */ +void asi_map(struct page *page, int numpages) +{ + unsigned long virt; + unsigned long start = (size_t)(page_to_virt(page)); + unsigned long end = start + PAGE_SIZE * numpages; + unsigned long page_size; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + for (virt = start; virt < end; virt = ALIGN(virt + 1, page_size)) { + pte_t *pte, *pte_asi; + int level, level_asi; + pgd_t *pgd = pgd_offset_pgd(asi_global_nonsensitive_pgd, virt); + + pte_asi = lookup_pgtable_in_pgd(pgd, virt, &level_asi); + page_size = page_level_size(level_asi); + + pte = lookup_address(virt, &level); + if (!pte || pte_none(*pte)) + continue; + + /* + * Physmap should already be setup by PAT code, with no pages + * smaller than 2M. This function should only be called at + * pageblock granularity. Thus it should never be required to + * break up pages here. + */ + if (WARN_ON_ONCE(!pte_asi) || + WARN_ON_ONCE(ALIGN_DOWN(virt, page_size) < virt) || + ALIGN(virt, page_size) > end) + continue; + + /* + * Existing mappings should already match the structure of the + * unrestricted physmap. + */ + if (WARN_ON_ONCE(level != level_asi)) + continue; + + set_pte(pte_asi, *pte); + } +} + +/* + * Unmap pages previously mapped via asi_map(). + * + * Interrupts must be enabled as this does a TLB shootdown. + */ +void asi_unmap(struct page *page, int numpages) +{ + size_t start = (size_t)page_to_virt(page); + size_t end = start + (PAGE_SIZE * numpages); + pgtbl_mod_mask mask = 0; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + vunmap_pgd_range(asi_pgd(ASI_GLOBAL_NONSENSITIVE), start, end, &mask); + + flush_tlb_kernel_range(start, end - 1); +} diff --git a/include/linux/asi.h b/include/linux/asi.h index 2d3049d5fe423e139dcce8f3d68cdffcc0ec0bfe..ee9811f04a417556cf2e930644eaf05f3c9bfee3 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -13,6 +13,8 @@ struct asi {}; static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_map(struct page *page, int numpages) { } +static inline void asi_unmap(struct page *page, int numpages) { } #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 31e9ffd936e39334ddaff910222d4751c18da5e7..c498ba127b4a511b5a6f10afa2aae535509fc153 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -11,6 +11,7 @@ #include /* pgprot_t */ #include #include +#include #include @@ -324,4 +325,7 @@ bool vmalloc_dump_obj(void *object); static inline bool vmalloc_dump_obj(void *object) { return false; } #endif +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask); + #endif /* _LINUX_VMALLOC_H */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 61981ee1c9d2f769d4a06ab542fc84334c1b0cbd..ffeb823398809388c0599f51929a7f3506ed035f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -427,6 +427,24 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, } while (p4d++, addr = next, addr != end); } +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long end, + pgtbl_mod_mask *mask) +{ + unsigned long next; + pgd_t *pgd = pgd_offset_pgd(pgd_table, addr); + + BUG_ON(addr >= end); + + do { + next = pgd_addr_end(addr, end); + if (pgd_bad(*pgd)) + *mask |= PGTBL_PGD_MODIFIED; + if (pgd_none_or_clear_bad(pgd)) + continue; + vunmap_p4d_range(pgd, addr, next, mask); + } while (pgd++, addr = next, addr != end); +} + /* * vunmap_range_noflush is similar to vunmap_range, but does not * flush caches or TLBs. @@ -441,21 +459,9 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, */ void __vunmap_range_noflush(unsigned long start, unsigned long end) { - unsigned long next; - pgd_t *pgd; - unsigned long addr = start; pgtbl_mod_mask mask = 0; - BUG_ON(addr >= end); - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - if (pgd_bad(*pgd)) - mask |= PGTBL_PGD_MODIFIED; - if (pgd_none_or_clear_bad(pgd)) - continue; - vunmap_p4d_range(pgd, addr, next, &mask); - } while (pgd++, addr = next, addr != end); + vunmap_pgd_range(init_mm.pgd, start, end, &mask); if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); -- 2.49.0.rc1.451.g8f38331e32-goog