From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7A3CC05027 for ; Mon, 6 Feb 2023 22:54:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20BA36B0071; Mon, 6 Feb 2023 17:54:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BBB56B0073; Mon, 6 Feb 2023 17:54:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 034E76B0074; Mon, 6 Feb 2023 17:54:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E87226B0071 for ; Mon, 6 Feb 2023 17:54:24 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B107C8053B for ; Mon, 6 Feb 2023 22:54:24 +0000 (UTC) X-FDA: 80438372448.27.9BDC757 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2062.outbound.protection.outlook.com [40.107.94.62]) by imf06.hostedemail.com (Postfix) with ESMTP id C701418000B for ; Mon, 6 Feb 2023 22:54:20 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=LsbsolV1; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=none (imf06.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.94.62) smtp.mailfrom=gregory.price@memverge.com; dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675724061; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4bb8jCOTM2gb8DMT6ila7Mgf5ik9dm/bOrt1zl8RwRQ=; b=JoAP/M59hqe1Ef96UIshruT/QhCYcu3lekaU2maoBJJtU+3BqN0QQGC5B3mKJaK0du+T6y Uk5IphH45Uol+Ni1ZweJqQkXJmT7tgscVuzpKBAOXgvTV+sckA3Rr4+1QmVcXUxWJJPSor 9AY93juazSMmuVhVumf52vhr62cy3hc= ARC-Authentication-Results: i=2; imf06.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=LsbsolV1; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=none (imf06.hostedemail.com: domain of gregory.price@memverge.com has no SPF policy when checking 40.107.94.62) smtp.mailfrom=gregory.price@memverge.com; dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1675724061; a=rsa-sha256; cv=pass; b=nffhm0fMQKjHOzN9St4wcUVRJFLdSZcQHCB0JUtl5NyptSSOdjxuJs00DebBU3DujzgAy3 msI8mw9bXEYIakHu0zOjtJCJyCifT/O/SM5fpr2uSqUwCEiReV1Kx/txC6bxteOJWrdecB C3ThYx4/e+1m4P38zjwzzB7PonrWoTM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DvPNaZaybCICAO7WyI2yzSegZMbRTk6OPpv/6+MWGHRQGff+lEag6BgCWXZvLoUpVXzdMyw20gnHWTwucgs9q1wKtlIjJc8AhupGcvOWlOjbFiV31pkvgGrvhRHUmMnrmttLO97t22lOma94xEIzXhYZUKV3B6VMvkvFML1ZcVaSmOuXbhQZ5UjPCSi3AeXToSGP2iZaU7vCrbA9Ka1Z0Sw4akJN61GeDdkekFd3VElu0JmC8aoZYkANrNjp9RKxgnh9z/MaQVQueG9JKebUdM0edFmVRN6jr4bjTR+Qa8h4l0SPjCYNlCxDhCQxPlZw9TrXWUnrcQ16hHq2nIWszw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4bb8jCOTM2gb8DMT6ila7Mgf5ik9dm/bOrt1zl8RwRQ=; b=TAFoMdBY00nbUSMX6gx+T76Ie2OtLxjojPbbH8kIJZd5GJu8ra9Anm415GzhJKmdCmgZ37RhI42dGcPzibAibaFPRcXdv00NyZC6cVO6wDpQcY6O4iOw8hw81zAOWdGM7G5oRsRE6BnCpQBxN1R7qc1g+tZx0QZtuqKPLCQKIZz3XxC8veKigDV6enjGM70MQ/kC1FdDR5RDaYtQ0wowHqxsAWKrXtrwTkSrHeuyAlphGQq1m0IxO6FXlozXmyLavuN2DzpdqPXnjalDNFo38Umno2+HC9edS1hTLSwY5ym28LpL3DlyJawi9j7GzWQamWOaGmNCK8vaBvyOAthD4Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4bb8jCOTM2gb8DMT6ila7Mgf5ik9dm/bOrt1zl8RwRQ=; b=LsbsolV1F0h68sZw5QCHlNDJSXvDBB+SMJFzd5x/q12alMNLQqCVHf6X9JkkHp9N2hXBxxxtt0LO9XaOfdA3lobumtzzRQLoZa+ts7oKooGHq9RIas4cJlnXUUbdKXjXFzE29zsd5kN10+QiIR0vcXFdt+Y5ysB3nAxfhoBWfW0= Received: from BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) by MN2PR17MB3999.namprd17.prod.outlook.com (2603:10b6:208:202::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 22:54:17 +0000 Received: from BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::d253:1eb3:9347:c660]) by BN6PR17MB3121.namprd17.prod.outlook.com ([fe80::d253:1eb3:9347:c660%4]) with mapi id 15.20.6064.034; Mon, 6 Feb 2023 22:54:17 +0000 Date: Mon, 6 Feb 2023 14:05:45 -0500 From: Gregory Price To: Dan Williams Cc: linux-cxl@vger.kernel.org, Michal Hocko , David Hildenbrand , Dave Hansen , linux-mm@kvack.org, linux-acpi@vger.kernel.org Subject: Re: [PATCH 17/18] dax: Assign RAM regions to memory-hotplug by default Message-ID: References: <167564534874.847146.5222419648551436750.stgit@dwillia2-xfh.jf.intel.com> <167564544513.847146.4645646177864365755.stgit@dwillia2-xfh.jf.intel.com> <63e17c1c4f8a3_e3dae29457@dwillia2-xfh.jf.intel.com.notmuch> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <63e17c1c4f8a3_e3dae29457@dwillia2-xfh.jf.intel.com.notmuch> X-ClientProxiedBy: SJ0PR03CA0115.namprd03.prod.outlook.com (2603:10b6:a03:333::30) To BN6PR17MB3121.namprd17.prod.outlook.com (2603:10b6:405:7c::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN6PR17MB3121:EE_|MN2PR17MB3999:EE_ X-MS-Office365-Filtering-Correlation-Id: fb850aa3-ce16-4d12-120a-08db08951362 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2V6GaeMloasgGh/J2OMfjDOSwHvTyi9ppmoRe0+4/vnCiy+PdxU/j95zJzEM0Fu6BB8v7AOuueyMs1p3Co5fULUb1kvOH5fMIzXPIdhunP8ox67TCzRmUuVESt0k1bSnPIWKp1qkdWKa56dHb4BuI3dXOfa8cE8YH44gBZIzUK7rqk20/ajks5/oyodTp5NAYoO5mCaFwUXKHr4fb/OpdnpNbLnoxZWSC0MEI/KGus9IVoJFmaM71MdjylUy0OgAqQjBp9lEtSTqGTIDlFhN10+3FguJijgJ08rGAv6SEWcW2nXqPYjntklm3tiTeco3AOnjvdKLYQ5pz9rXmwbPQxg1mm9r0gqCxAHFapXFJg18hW4I7qBrMfdJ/L7hl0N/uF6nRn89cc1dJa7xcZyeSiyNpvqmEd0SLP3OCko8xGP/yAF8txpYeh5IE5qwtbx6DXu1U0Ovjaq8KpIl0CwMLuI+27MStQKphavbIfMna6GzRyggNI5CsUb4+CNrYONJqRLXewsiRiJ+EFW4gkbLlKNcru3mRdlDZVRgg69TQKMTuOX0my5D+r8wzddZNZdav2Q7s2Z1UhWshr5YA6pye1nPHWWiepNwCUTTBizfc0po2K7Lfie/Rk855K5w0uVf9FppkiM2pE6/pOHih3jmD9oNeL7TGUBDps5c4KUZLfxS5s18t9XgZ7nJs+TaM60P X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BN6PR17MB3121.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(346002)(396003)(136003)(39850400004)(366004)(376002)(451199018)(6486002)(54906003)(316002)(478600001)(36756003)(2906002)(38100700002)(86362001)(2616005)(83380400001)(6916009)(66946007)(66476007)(4326008)(5660300002)(8676002)(6666004)(8936002)(41300700001)(44832011)(66556008)(6506007)(6512007)(186003)(26005)(67856001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?20mqZJirYXDuevE6RTd8e634mkp5DvdyPMC+Y4Xehlf69ONhrt38AuFYYCid?= =?us-ascii?Q?iaJVKXEhQ3nWTKMwDlX7Y+jRAXQ2DTF4dqBcozNyVCsDCkKIzj/UwuTk0HYj?= =?us-ascii?Q?KEXMCYyfPuVjsuxN1CN3h2Zen8StHnwBZtxdze/7qtCBD1eNQDkr3Fzy4QNr?= =?us-ascii?Q?NsZncIloX5qYZgsLGPkcplh15cPYOEkl/AL11Vy+qVuoISI6JWIoKbukCtiP?= =?us-ascii?Q?O4DdNNLjIA0xU7/V+9YPnxVEH7qumItvHVekBn9hChUOL0tfMd2oYp9FswWt?= =?us-ascii?Q?e0PpOAwyTfrPKil0NpbZo5caJHi8AkvkiIC9Wp+D517swxEWKoVw2vSspAJe?= =?us-ascii?Q?HU7rqsZ/lilrboxFOYhToC2fCcMkH/lRq4LBM+BIMXJF2TP65LpEE7UZ2MMY?= =?us-ascii?Q?DAxpYCwVB/dymkkP8df4p8RF9yjgVCl2aGBfgwETsXiampacXk9ot2zGI3Ip?= =?us-ascii?Q?mgzxsbBQZevJOnvQTsfaU5qZP3BP8NlBV4emuwAuOsfOc/C8LOeEvQNA4Kpi?= =?us-ascii?Q?nH1TqfidLpI9Czqa77PM6O+Qec8UTELAsoB/OmE5k0ASc+2xy5gNd1OXb1lq?= =?us-ascii?Q?PyiroRzx1+9bPcBpKPu1gIB4O8Aw6+c05kgQ+VdDNAnTJ0fI7tP8hUzda3h7?= =?us-ascii?Q?wIHiVUGg2jWLvnKm9hmkyxxfoOZLez4vdvZFO9+JbjpBUDu/vJnR2zpnv3sO?= =?us-ascii?Q?rT3canFRLVdgnLqWSvpAoN/k9ghm99ryHJSf18+OxINoO/VPsMmJ2cl0ff0J?= =?us-ascii?Q?UKyc/zwY9+OfbR1cKUazFGy7IRnk1v5D7QOOBJodzvpn/RX4a6D6/icr5cjl?= =?us-ascii?Q?u2JN/iSebvYkpuEpw4y/isjsmP7ZXqI7YJieKUMC+mHi51j4NCTTIqHJKsGm?= =?us-ascii?Q?FUhd6M5m8hum4ko2pnb38dvZ0KeYVvF9fRQD9l0CnYHHtoio1IsIV01NK+1k?= =?us-ascii?Q?3y3ok8CjGJzFSTtvISJkPU0UXT3YlZIPio//OwwDiSPhwvKiXOAYefm1U/+o?= =?us-ascii?Q?13MAVsXSlFyJy5k1rBc+sshHfJer58/UbT2AR77hr86Ydgcjb+9KSxOGF6Su?= =?us-ascii?Q?z6vSfaSN3KoJHBlesac1u5oC2n+n1AJnRuw60/Lu3adNj47vDdYA/FfmzMKp?= =?us-ascii?Q?jx/v/Pk4ya0a2No9iktRZB9tlUBHppTzFcgYYrU4pM6lL80HfV9VP/o4VrR0?= =?us-ascii?Q?XhrarpvDoNDcYp2YU4aSzmdWClhvK8vMALaFSBPRur/JYxKD+k1gBP7LKEBj?= =?us-ascii?Q?oVkJeds1NPtk70CoIrLiwT0hkVvG/QAGh66YJ/Hz83kbPt8vGtZZDDvR5606?= =?us-ascii?Q?GsXzvjnyWGm5xo8lJC7c0zXvzNonDq7oq75o+P+KbY+Fo8g/Ifee1ouKUR23?= =?us-ascii?Q?2VqGIm4KJEjYoraEFGMZmUieLgsJW/v2PALMVqAkSvmUxA8GJiQ1azkkHOTx?= =?us-ascii?Q?ovp3nXc4LfPt2FgOn5DJpsISFJdPvzd0tCCMmoMhAo4OgZnoMR2jbr6u6wDe?= =?us-ascii?Q?d2ocfZ6QVkUmcV0PYEaIRJQ23Owh1Fd7Ku7DZcjH+6DLY+dSBCk+KjJvAI+V?= =?us-ascii?Q?ykn7uINXnytojYHzJ2gDXXKvGibvfTTUQN2IsIXgEMl9VZY8yPG7ZJMmL4LV?= =?us-ascii?Q?Bo4qXAzrrOlrZbULMaY6134=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb850aa3-ce16-4d12-120a-08db08951362 X-MS-Exchange-CrossTenant-AuthSource: BN6PR17MB3121.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 22:54:17.3744 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nAZ0iEipxYMVPBnYcT1gPfUc35FPl801ZkNglyyHRBy1KE7co8zT84Bn/Sc0K8TqluJCNVuNLs018CJMB/sLYIyCjdDVCtTUWSZ0WeLeF0k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR17MB3999 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C701418000B X-Rspam-User: X-Stat-Signature: aq9haaxkqzftnstax1bds6ogfxpnaubx X-HE-Tag: 1675724060-892586 X-HE-Meta: U2FsdGVkX19nRjp3fSaBXxDgQaiQSjfw7CqBFzKZf+/MKZY/8zJ1lO7sUxq8PS2jQ4tZyM00HmDn8RkdPyCx/G5K89lY8+ngKsVQWM2+9eJt7TWmIZ1vTbGxdkuU4jHW3lsE/eBwd8Xdb3f2OCWZ70SksCWve7E8r75YM0irYStL975faoZefOhB4m46sCl2Q3SSIUNNWRji4fYK8GcA0stCzRJigz3fWiN6uSzm0I9leTkRsJbPUQZhyPe33YXJJKNiBaL1JaLgjBA3EdTsLaXSm6TqBoK1qQ05TW9CsSGMU0e6EzvNm7oL9jGWdfRkG+8fAU7YdRV2ZpPYyCcRpXdXjET/4tJM1mRLQERQ/HY8jrTo+Go7sJUqo822YQTVXeXfwq3BZTyH9Yxrnihv9/yl0O+EdZnCSa0rWfFa5cS+kuNOevq4azoIPNMeIkZeyDa8YBsUwH9HSRLNMjQZTmkb6YBcEm74NwB6qzJhzxW0a0VWeJ9dSkAGOpCQWCVdLFpYLXDgIWDWjjYkB3LdTa0Q47kyg6KzLlH8itI913/Gv4eIZ8KEFkaTBUdGxA2Mg2woqpUuAdQUgokzADhRvng06+hJe8mHk+qIxbrhq66jmz+VjDwh72O998KKrroFWIkhDAqP4htjBYe3WV7STeapJiL33oQU46K2qBP9B0MBhEZ9Rnpbpg+DpkWgwAsFQwFwNf9NnKIgMZdaMHKjNmbAYEYH5Kls63HRcyLqk3j5Eg26jDOQ8QM138nCoSN2H6hmHNgw5mojO8RJsK7DRFnrtpjmLFzxrhFxRi82lbV7ELyPkLqvSIjzRq5yX/xJRydkz/NF8oZYanN7HGr6qvUxfHx3hmKhCP32iMwWyYGk3vk8s3hJpAxK+yg/k5wGftws7vnMdLQXktGGpDXk0bTij9M4Z4jUkBj5a7XTL11itRxflwZ+PzzUD9EG0xeGwiNgdz5e2hwTwozTxnw r5hSzF/L aCS3aYOT/dItj6X518VDJOoEtQfGhVTvhJdyBCAka9cWruX5Pg3UyCBpCNPgUVDWgWLgv43e62576CL4uqFM4xzNt7vPPy4owgDcQH2lWURAnj1XNk/+OfxyHxfdiFjZwL0AbMphNXziBVa9CngXoXwUNXeDycJp9DU/OYxbjl/Z3SQdBFYuJHDFKnGUymPKqSfuNzSx851FUQGRiuNrq7/EeEqQNr2oOyPtNVKfA58KLjZfN+B/09S2W/hixOihg0lBqSGILSWNO37XHHvVfXqaB0Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Feb 06, 2023 at 02:15:56PM -0800, Dan Williams wrote: > Gregory Price wrote: > > On Sun, Feb 05, 2023 at 05:04:05PM -0800, Dan Williams wrote: > [..] > > > > Stupid question: when defaulting to online, do these devices get placed > > into Zone Normal? Is there a way for us, at a minimum, to online this > > as Zone Moveable in an effort to assist the "hope the kernel has not > > pinned a page" problem (and to try to keep kernel resources out of this > > zone in general). > > > > If this is covered by a different patch or already set up this way, > > ignore me :] > > Have a look in Documentation/admin-guide/mm/memory-hotplug.rst, the > 'daxctl recconfigure-device' man page and the the policy options of how > hot-added memory is brought online. > > The routing can be anything from fully offline device-dax, to fully > online ZONE_NORMAL, or even a mix of device-dax subdivision, > ZONE_NORMAL, and ZONE_MOVABLE memblocks all within one CXL ram region. Hm. I'm just thinking, for early expander devices the assumption that the default behavior should be auto-online is good. Just kinda working through this in my head for multi-headed devices and early pools, and I suppose the kernel paremeter covers that. Most of those will be used in specialty, vertically integrated systems, so we're good to go. In the future when we get fully featured DCD's, I imagine we will have a memory region of size X, with N memory blocks, but only some of those blocks are online. I suppose it could also be done such that DCD capacity add events drive the creation of new blocks under the region, as opposed to having the region pre-create the blocks. Either way I think the kernel parameter probably covers this case as well, I'm just wondering if at some point the default will yet again wish to be false, but i think that's a "CXL has become so successful all memory is now CXL DCDs and computers look radically different" scenario. Anyway, this seems good. Reviewed-by: Gregory Price