From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC5DFC28B28 for ; Thu, 13 Mar 2025 18:12:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12241280019; Thu, 13 Mar 2025 14:12:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00C6B280010; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF080280019; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BCBEE280010 for ; Thu, 13 Mar 2025 14:12:04 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D0D11140430 for ; Thu, 13 Mar 2025 18:12:05 +0000 (UTC) X-FDA: 83217321810.12.85FC9B0 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf29.hostedemail.com (Postfix) with ESMTP id 77DF8120014 for ; Thu, 13 Mar 2025 18:12:03 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aLvy1TEZ; spf=pass (imf29.hostedemail.com: domain of 38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741889523; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=3iLeUMbB+d/7Ly/Sz4eOrmSImKiCtxD8VosRkpzGJ4UBwcdcnmYaoSCDx828mKk/KFFkmr yHtFowjX06uJhblZt8ZamSv+6Qx9qJ5NzVm9fmBrFmaZHSIRclk0tPHgsGXQGreXWyBkEM 1dggzb2fHEtTTOUoOs1bJ6igwmmFJ6s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741889523; a=rsa-sha256; cv=none; b=AgH87D9MlVL1P+FLtR0PUPcjAmNtKQowiZikqANfO8tUREH0W7oMQr/8i8svyO53PkK24M DoTLh0KmxeqDcXb46Q7aqxytE6U9beSt/JGrl1WYn1yUADdqMZXXdLawBdpWy8e7gZ2Xz0 Rf8kir9jAdcmnxHb//zMQs1XsKnxcMo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=aLvy1TEZ; spf=pass (imf29.hostedemail.com: domain of 38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com designates 209.85.221.74 as permitted sender) smtp.mailfrom=38R_TZwgKCMIriksuivjowwotm.kwutqv25-uus3iks.wzo@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-39143311936so606414f8f.0 for ; Thu, 13 Mar 2025 11:12:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889521; x=1742494321; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=aLvy1TEZh6bdIYx9kXpG1JtkZsCMjqTZbqYpkfhn+a8C6masvcnqBAzC3i4wcGhjLj gpN8xXHRFVF9xIuIVMQAwU67LBmj4ObTwPUl9Qc5nPdSQU/jRv9VcMPjxM5DZ+CdIzIY 8jIpk45sUNiYnYWI4mbL/kIrOwN7FhgVBFSV77F+N3wRZsxmREqnp/TSmrwAH1jaL0Xs IRxHHQ+qxtAO5gKuTidZyyvsNjNYc5g8DRikowmqz0bsZJNEU4vYL3wxTqL/7I1/kxXn JoIrgZHJ9zdekqK0xdfymNNBlBtvl8Pn6KJsEKtqYplnR2+awvBaw38ByFNhIOM3LbTz XAzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889521; x=1742494321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=nNPodanKVIEpv24U854BzYha63o7Wb8Ib4o2m3R7NoJtttjPIK48lAA4gGuek2Rv/w OgVcVrWN9jWCdFjd9beEAMbQiSnk22Sb8Mmq53FaDZekTyWtMj95L5ZRTNmIy7zLFSvd mYB2VwlQgh9HZjV6wFp4Au5JV0SxKdGXvpfV5Ph0woi+Lt4wOiMo2H7dQGoTkr4rR7hE a05Z7GJH/I424tYxazFiEcEruRtGP/4bz9CHPgMEVXODUB2r5uTt4jW71DH6R4iEgbxV CItewwgtjlK3z1VNee5OL+2HFgVjgvSj/PgWZ1lvbCpruqvcTSFcYWng0Nas30cerDJJ +eVg== X-Forwarded-Encrypted: i=1; AJvYcCWSGLa9aFFG+COj8P8KxhoRm64wqzdA82gRb9zc5zrtsxD6O6BB6+/sx+KZ5KHFsKgxjMIpsXbXPg==@kvack.org X-Gm-Message-State: AOJu0YyXNe+0gbP6iYUT3ce5epRZ2s7X7QKVhTEwsdZKaKrAV7pweNL1 RdeuNdGGBg/5Gouj9atxc19K4xWNNuyEufDt4u/9eFz0TALVsq10k8xGt3X5zn6VccSXe09b84V YvnJSGSxjoQ== X-Google-Smtp-Source: AGHT+IGTDPbDngPkmEoWDX7m7yWVpzg47wDDczIbjQCcv6h8SnX4qWQyopmuiT4sdFB+hEFVGiIYdXMgmMnnIQ== X-Received: from wro9.prod.google.com ([2002:a05:6000:41c9:b0:391:2f9a:9a8b]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1f83:b0:390:f9d0:5e3 with SMTP id ffacd0b85a97d-39263affd7amr11502380f8f.1.1741889521421; Thu, 13 Mar 2025 11:12:01 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:29 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-10-04972e046cea@google.com> Subject: [PATCH RFC 10/11] mm/page_alloc: Add support for nonsensitive allocations From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 77DF8120014 X-Stat-Signature: puobnx8e17oyu77ecjzgxpsin7j735tf X-HE-Tag: 1741889523-576836 X-HE-Meta: U2FsdGVkX1/gmNBpA6cdqXtdmXnLtI969O/PWVYZELev47jE/W3Q/1DG20iye5t8WqcMHlCdA/ogPPMg8QhlI46wyEwMvWIoEA3ywx6AWMVGYjmuUMPBXoQ/FjDYyXLXoJCt8YC5EH0mForgVpqgtYaWf4Jr/WJ8n4jK5Rw/el2nDoaujgRTYBzmBsDGcNIv39y9z1BzvUXhBRgasB0m4kxiw962nmYHXgXtk68vtgHFFVvCLMSeFpyXeaSecPnD1blZdcb6psyZiO8qZlOjRATUlYnC8iy9CA38b+zNADHFy4lOgv5qECc/idVLN4K6h0FsNAiLSs+ve9mnapCSjrF/987WRgdfqKYXQwmG5L5gjJYxfBXLKUwqq9fCIeuQWe2ye/F/NlKB7ucWgQ39PBrjrQtlnnu5FgQRtiWoe4QqacjUq+v7OJzPPxQfyX8w8zUYUf8I4gVL83bRC5DslsTT4rSo2M+MbWZWFgPHt2mBKZqRCM4T5Ao+JpoUp0GoynjGZVGy0mfLda2nv/xf0MpMxeyLE+ze8EbTa988AAmK7frYSPjGn5EF0ofkFsHy1yoddgkcS65O4PQp+bc+ZiJ+c30EdD/EICBQYgsr4syFojTx3mRVbvdB5w2PRhif29BY0O2m4jUgQaQXUX2jUDKFqcl33OndxsyAvgQUFdZyomzM9P+tPMGOVueQW5j4Gof+0sJMX1GwZrxEOL3fAPs15HdBF+nhanoXR7E19y6lS2bCkJsdG534OGqR/SRAoIJPJsC7HJjazQSOmteebr5vhOWvi6sLrNbgNkn9IRQk5XTLzvSu3RCVrmbf5pxhOD5GiaE97KLIefHQCpbsJLEVQODRHr54qXwY7v+pjR5jFmG/ii+525kbTMMZP/nq14AlI8/kwBslrZO80lQ5WW+iPNvqK8y1D2PPWJm1ibrwT2z2tkSCS+8F9mNtA9kIHlobkaQRaSittkcHAda YpVzL8fR z1Isx+Js22tRNxQPsuoHXJMIsIfAb8Skk3CYdlB6u+X0+3Zw7ysWvP8bJmOgCglN1LVfN93Up0al26Zux7dgxnB4tkgTuz+pCBXCCCEBcPRjbq3VkmqZ1mRegSIAjy3vjLRiRilAh37ogukmpQiN9u7K+0jiKEkNYAQuEIUSNfcS0luIQFTp8iCr+1on5PZo9KpUc6238qGqHXFmXbzJo9l69CwIzLdTIS4B8evXwDL0lsed3/YKdzZYq4Ct35gjNFuzOEr4K/Dgs9L3YdsFUqwCvxVHAuxE4qJUAbbJsntrCGPXGtEu9Gv9+9kP0s6SKhfP1XRjycAb0eA2qs2hhwfdlUrRRLx6dTavXfBPQRUtdOcZN7P24MGffiKO2ZbgHW+4J7IubcwVEuFCaOQiFaD1fQ4pIESUdP2IC9zMbEejz1TeFzR5aHzEqtF6soCnwzXFW8B6l5MGnuiNmHmNSPlQhvHdCXRnjxAJg4fbU5YqDEWdP72WLVD+0us2sl1D+kdqDEnEv7YOjJdpXFBxHuPd9PdpGqJeGpgrFczaRThM3DiH2scrnqLpFJNYMKsNYbo77ADWBhdUTdWtk9uK6CGIbBhJhDtiCeyXSpyR0+Z4M570= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Creating nonsensitive pages from sensitive ones is pretty easy: Just call asi_map(). Since this doesn't require any allocation or synchronization (at least, the synchronization is implied by the zone lock) just do it immediately in the place where the pageblock's migratetype is updated. Implement a migratetype fallback mechanism to let this happen, restricted to requiring a whole pageblock. Now that it's possible to create nonsensitive pages, the allocator supports non-__GFP_SENSITIVE allocations, so remove the temporary hack setting this flag unconditionally in the entry point. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 5 ++++ mm/page_alloc.c | 67 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 57 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index cf8be544de8b108190b765e3eb337089866207a2..4208b60852ace25fc9716c276bd86bf920ab3340 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -2,7 +2,12 @@ #ifndef _ASM_X86_ASI_H #define _ASM_X86_ASI_H +#include +#include +#include + #include +#include #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae711025da15823e97cc8b63e0277fc41b5ca0f8..0d8bbad8675c99282f308c4a4122d5d9c4b14dae 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -15,6 +15,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ +#include #include #include #include @@ -422,6 +423,18 @@ void set_pageblock_migratetype(struct page *page, int migratetype) migratetype != MIGRATE_UNMOVABLE_NONSENSITIVE)) migratetype = MIGRATE_UNMOVABLE_SENSITIVE; + /* + * TODO: doing this here makes this code pretty confusing, since in the + * asi_unmap() case it's up to the caller. Need to find a way to express + * this with more symmetry. + * + * TODO: Also need to ensure the pages have been zeroed since they were + * last in userspace. Need to figure out how to do that without too much + * cost. + */ + if (migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE) + asi_map(page, pageblock_nr_pages); + set_pfnblock_flags_mask(page, (unsigned long)migratetype, page_to_pfn(page), MIGRATETYPE_MASK); } @@ -1606,19 +1619,35 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, /* - * This array describes the order lists are fallen back to when - * the free lists for the desirable migrate type are depleted + * This array describes the order lists are fallen back to when the free lists + * for the desirable migrate type are depleted. When ASI is enabled, different + * migratetypes have different numbers of fallbacks available, in that case the + * arrays are terminated early by -1. * * The other migratetypes do not have fallbacks. + * + * Note there are no fallbacks from sensitive to nonsensitive migratetypes. */ -static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { - [MIGRATE_UNMOVABLE_SENSITIVE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE }, #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION - /* TODO: Cannot fallback from nonsensitive */ - [MIGRATE_UNMOVABLE_NONSENSITIVE] = { -1 }, +#define TERMINATE_FALLBACK -1 +#else +#define TERMINATE_FALLBACK #endif - [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE }, - [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE }, +static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] = { + [MIGRATE_UNMOVABLE_SENSITIVE] = { + MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + [MIGRATE_UNMOVABLE_NONSENSITIVE] = { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE + }, +#endif + [MIGRATE_MOVABLE] = { + MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE, TERMINATE_FALLBACK + }, + [MIGRATE_RECLAIMABLE] = { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, }; #ifdef CONFIG_CMA @@ -1931,6 +1960,21 @@ int find_suitable_fallback(struct free_area *area, unsigned int order, if (area->nr_free == 0) return -1; + /* + * It's not possible to change the sensitivity of individual pages, + * only an entire pageblock. Thus, we can only fallback to a migratetype + * of a different sensitivity when the entire block is free. + * + * This cross-sensitivity fallback occurs exactly when the start + * migratetype is MIGRATE_UNMOVABLE_NONSENSITIVE. This is because it's + * the only nonsensitive migratetype (so if it's the start type, a + * fallback will always differ in sensitivity) and it doesn't appear in + * the 'fallbacks' arrays (i.e. we never fall back to it, so if it isn't + * the start type, the fallback will never differ in sensitivity). + */ + if (migratetype == MIGRATE_UNMOVABLE_NONSENSITIVE && order < pageblock_order) + return -1; + *claim_block = false; for (i = 0; i < MIGRATE_PCPTYPES - 1 ; i++) { fallback_mt = fallbacks[migratetype][i]; @@ -4694,13 +4738,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac = { }; - /* - * Temporary hack: Allocation of nonsensitive pages is not possible yet, - * allocate everything sensitive. The restricted address space is never - * actually entered yet so this is fine. - */ - gfp |= __GFP_SENSITIVE; - /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. -- 2.49.0.rc1.451.g8f38331e32-goog