From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40E7DC021B2 for ; Tue, 25 Feb 2025 12:56:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 932F96B007B; Tue, 25 Feb 2025 07:56:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E2F96B0082; Tue, 25 Feb 2025 07:56:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A9FB6B0085; Tue, 25 Feb 2025 07:56:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5C9F56B007B for ; Tue, 25 Feb 2025 07:56:41 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CDAF2C1501 for ; Tue, 25 Feb 2025 12:56:40 +0000 (UTC) X-FDA: 83158466160.09.02412DD Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) by imf30.hostedemail.com (Postfix) with ESMTP id DA97480004 for ; Tue, 25 Feb 2025 12:56:38 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fEdzSdWS; spf=pass (imf30.hostedemail.com: domain of jackmanb@google.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=jackmanb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740488199; a=rsa-sha256; cv=none; b=OMwnnxaAhWGDzykBa/6A5SxCob1AqscEBJ+oefvH+O+jgSuo9PpKF1X/dLV39n5FFtwD9A iytgbKO4eZWiebYCZzijWH7mlqaptRLljZkrQKJlnL+pXF2Not6u8RbJ445q8XvjoSiSCh pwM9IDEeIxFtxLDTqQpQzF8gJ7kGWz0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fEdzSdWS; spf=pass (imf30.hostedemail.com: domain of jackmanb@google.com designates 209.85.128.53 as permitted sender) smtp.mailfrom=jackmanb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740488199; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MC9KPLvVqWKMt0reEhfmjJu692XzkFN80RWzCGUDFJk=; b=bNQTCyOFtX1opsBN1uUiyQX9peq0/vqGvP20iDpX/Cbr9fatEBbtVjnO/qdVu4a6+UlWxG q17AgklCGO7ID59/+8VSU7nOrj0cneLz6DsPFrp6EuPYfYclYsPIt7tcgY/TsSnfM1dyEu 2L6vO4KNOz11SegfeygHulYjOV8lJt4= Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-439849103c8so48445e9.0 for ; Tue, 25 Feb 2025 04:56:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740488197; x=1741092997; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MC9KPLvVqWKMt0reEhfmjJu692XzkFN80RWzCGUDFJk=; b=fEdzSdWS4ktPmfIv4/MA347CQsxrC0FJJpiF1hQW/So3/JlQg5bwvxojL9yAJPRbbF LJOm3vslsgfseMuQXtuIi/r77+jvZHIRfaHKMAxWdnP9bXPxs1T50yXNMApogEfod9rq BLQudqASt3dYh0s8EX9BWeEq3OWBziqzb6bjVXkHHOwycC21ZJZc1ghixnTRT2VDVPz3 LyUe1wnsQkUKhcnDYq9DVFlBDkP4z81qzVfHUu3RAkATuOYoj56t6QWek62ndIoctRXx BVO25wWpbfurkA5kaGjJMAEdALDwBxC6iqf+NfdCWHktV2PvCvD9NpiPb4lobhGo9OUo x6ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740488197; x=1741092997; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MC9KPLvVqWKMt0reEhfmjJu692XzkFN80RWzCGUDFJk=; b=VGz2AqrVeWG0z8khoVIVv/aMVByKnB8jz0kRckwZdWkBcxPYF5BBpCs42ry0U8TT4a gVomkBFz9QmFQTgjOzVNWENtPktAdoMIr1AGFuHqRWtnK9Nu3x8j7h4PYrtRyFFIf2an Qu8h1dQP1oI0WLdj2vVzpkTEDE6PiqkdTuHsuJyOyMF4/j9l69uqgYzUKpum0dQn3d6O k8qpIsQA5ybin/rCxaGpzy2M2Hxwg5B0DpDl4Ql0fhS7Ye7bnLZmy8nPlrhENYnZGVjE D8T+ZWa2qdH8HRgAhZXD0Q8cjgj7Ix9gVvND5xlZEhvjhbYDm57NN4sFMJItdOM30aIs ei5Q== X-Forwarded-Encrypted: i=1; AJvYcCVftKe72NvaMnSv5h7y2aRwt2fEnERUloYMeHvWdhPSReGBh/ibSR4omnAQgfSmhEO/o7tyWlZZZQ==@kvack.org X-Gm-Message-State: AOJu0YyciY32duybMve4xfGX+LtKIQRR+XWJu7+taqnnSeFnkllq4QMn tPXXUA1Z/MLOPcq/Yo+gymBgvITQDGlZh2+l5Ganm75MtzMlcvZZ+88hp8LOdQ== X-Gm-Gg: ASbGncsLkYrEdhmoSu90ZTvqck3ZTQWCFROXauS+VLK63cKDkaRxiociKr7XIet476N MlW8CMiXlQF2BQ735OReP77lwhSxJlOJSGSUzf/EZil6HYJk7uYhHGq1ugsvJNpSZGHj5GPqf0T VCT1BoUct0oEuIOsTw1O0jTrzeTWzJv9VQqZ7E1Hq2qjcaZElJwWu4Ml9DMj9K0NSk648NhdxDA mgGqdXGRFMOYAqV6AjlSsjalZQ2uLBGCeSjvgqebN1ZbcaDqBkFVswuCyJJVunbNbkD1iiPjo2R hwQ5oah8oJHgsT7cizrv0NDx9reLbMbwE1vPotss+IMcOOaK1UxbkbqkWBZ+mw== X-Google-Smtp-Source: AGHT+IHHx1tnED5DbqgakpRRi9n7AbytxVfYjoT3lvOaag4Ekssq2e8VL3NZ4DRLo24ldsb4E07AvA== X-Received: by 2002:a05:600c:590c:b0:439:8f59:2c56 with SMTP id 5b1f17b1804b1-43ab0fa8bd4mr1345945e9.2.1740488197133; Tue, 25 Feb 2025 04:56:37 -0800 (PST) Received: from google.com (44.232.78.34.bc.googleusercontent.com. [34.78.232.44]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43ab156a183sm24997255e9.40.2025.02.25.04.56.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Feb 2025 04:56:36 -0800 (PST) Date: Tue, 25 Feb 2025 12:56:32 +0000 From: Brendan Jackman To: David Hildenbrand Cc: Brendan Higgins , David Gow , Rae Moar , Andrew Morton , Oscar Salvador , Lorenzo Stoakes , Vlastimil Babka , Michal Hocko , linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yosry Ahmed Subject: Re: [PATCH RFC 0/4] mm: KUnit tests for the page allocator Message-ID: References: <20250224-page-alloc-kunit-v1-0-d337bb440889@google.com> <0449ff75-0a6b-4c1e-bf12-ff052aad5287@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <0449ff75-0a6b-4c1e-bf12-ff052aad5287@redhat.com> X-Rspamd-Queue-Id: DA97480004 X-Stat-Signature: qedapfkmt5cnceat4bj6jcgt71gohpez X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740488198-669190 X-HE-Meta: U2FsdGVkX1/dJTt6EqrV0USIb7r3fHe0wUH8rQztT5ODbWvryEnjc/zJ9161SpeHPRqp7/F8bSaFqmkoxVuQYw5CLc8S56vTRMIdeBn39QnxUaCKNqjGtfdHJW/CwhYMPMWXeTbSwF2M+Z7ZS0ik4CH4M4ZKCzCyBD1EwtkqVcxeiQqolkucLZBBd16pdeue8i5cooMbXyoGYUcdPq3KQoeLF49C3REYDXAsROP+nvXWubu/ypSUpln/ZQ3hO8AwFMraK0yPUUWTG+x3gYGZA436EJBHBDfjMAtKk4d2eXn/V1sNgRhb2LuQEq7ZRGT3/phlwF5S58K27ddN+R6z7JKMflYO6EaxHM99jqigAXMI9p1/nHs54NevRvzZDyHie8T7aaDsunoD1ceDZe4dJcX2GGgxNUPIpXc5YfRGgtbhHfZJAO50sz1by1DAEHJOTjYfjJULMmnTG0lva46mgYveCapOLB2y3DCVuosq4/0o9eS7ovDc3V6dTEl7jN3J8d+2N8ohybfJwq/4jZzN/fU/Kww2avallzqHdANndCzapM2OqFoxkDqYUSzMyKBPOmok1sDX4hzoFep0niBgu4CBGi2LR8fd/8UqSPC4pv4y7u1aVrh83LfAijy08ZZZ8aYhA09C+DONu3Lseoir0PBpPpFPHwrvIrwv/gQ5I0/jWeNPruN0eE1buGJKy01eMWK5xqN/kQqC+7opgKC/EkfzCkTkMuV8DJyp6GvfKvpxBaWgUd9K+WD8d5AERXCAxBC2giXbUVxxLi1VzOTNa4EsOgcXfXRl3eGSx/iAiJJydDAUdmGIjsfbwwV7kdWTFXMOd1ifQRXuNSuCongyNPcRRlHYKN0L2DyMsV/1tk3K14UllZL+dUd4tUxi7CTR2TTn8Co2huJVDsYK/Oeh7bmBO2CuEmBA65VqRBvGglK61oWZmVxBVhlaX3wgjorWbWCZzkc9rcgeKMWog+u CGopvA1L d0Lu3xfjzDZDftvlXZGDnwTFyDD27yYn26jQLMU/1LkJX4PQyng8q4ajcKD3296mFPUeGoskNw7Vf//KT8QXalrHkpntZVkWlxStuXeZMxtHuVQKfLivSFIY/t9X7QL+d/GalYclf8Kn9Kc6cOp4yOZxNM2kPBhgJhjjsG09GToG7mmwvwn9z7zB4xAzjroPAPsCCjBl6ouPeu2Yu9R+wUgxcv6wD1rhd47mw3CKu83ZOq3WrjVhmGMZGWKy6wygVVnPZoOxqUO6pZjiDQZtVTJzP9iNztDrCqkjeryuX8HmDj5UyCayGSSwY1Jgjml1TNiZ9020eoyY3pp+fFqKeCURnYhSsFFQ72ThElQHnt+Ot+xA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.003686, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Feb 25, 2025 at 11:01:47AM +0100, David Hildenbrand wrote: > > This is an RFC and not a PATCH because: > > > > 1. I have not taken much care to ensure the isolation is complete. > > There are probably sources of flakiness and nondeterminism in here. > > > > 2. I suspect the the basic idea might be over-complicated: do we really > > need memory hotplug here? Do we even need the instance of the > > allocator we're testing to actual memory behind the pages it's > > allocating, or could we just hallucinate a new region of vmemmap > > without any of that awkwardness? > > > > One significant downside of relying on memory hotplug is that the > > test won't run if we can't hotplug anything out. That means you have > > to fiddle with the platform to even run the tests - see the > > --kernel_args and --qemu_args I had to add to my kunit.py command > > above. > > > > So yeah, other suggestions welcome. > > > > 2b. I'm not very confident I'm using the hotplug API properly. > > Me neither ;) > > Dynamically adding memory to that "fake" node is certainly interesting, but > which guarantees do we have that some other features (page migration, memory > offlining, page reporting ...) don't interact in weird ways with this "fake" > node? Adding special-casing all over the place for that feels wrong. I > assume this is point 1. you note above. Yeah, basically this is the big downside. Changing the system we're trying to test in order to make it testable can't be avoided entirely, but I am also pretty unhappy with sprinkling "if (node_isolated(node))" everywhere. I guess the ideal approach is one where, instead of having to modify the meaning of node_data, we just support replacing it completely, e.g: struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int preferred_nid, nodemask_t *nodemask, struct pagelist_data *node_data) { struct alloc_context ac = { .node_data = node_data }; // ... } Ideally this could be done in such a way that it disappears completely outside of KUnit builds, the interface should be private and we'd wanna get rid of any unnecessary pointer chasing with stuff like: #ifdef CONFIG_PAGE_ALLOC_KUNIT_TESTS static inline struct ac_node_data(struct alloc_context *ac, int node) { return ac->node_data[node]; } #else #define ac_node_data(ac, node) (NODE_DATA(node)) #endif I initially rejected this approach because it felt "too intrusive", but now that I've actually written this RFC I think it could be less intrusive than the node_isolated() thing I've proposed here. The most obvious challenges I can see there are: - There might be places that page_alloc.c calls out to that care about node_data but where we really don't want to plumb the alloc_context through (maybe vmscan.c is already such a place)? - I dunno how many more such helpers we'd need beyond ac_node_data(), like do we need ac_nodes_possible_mask() etc etc etc? But maybe worth a try - can you see any obvious reason this idea is stupid? > So I don't quite love the idea on first sight ... but I haven't grasped all > details of the full picture yet I'm afraid. Do you have any thoughts about "making up" memory instead of hot-unplugging real memory for test usage? That might simplify things a bit? It seems possible that very little mm code cares if the memory we're managing actually exists. (For ASI code we did briefly experiment with tracking information about free pages in the page itself, but it's pretty sketchy and the presence of debug_pagealloc makes me think nobody does it today). There might be arch-specific issues there, but for unit tests it seems OK if they don't work on every ISA.