From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFB93C4332F for ; Mon, 30 Oct 2023 04:24:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E4A66B017D; Mon, 30 Oct 2023 00:24:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 295216B017F; Mon, 30 Oct 2023 00:24:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15C326B0181; Mon, 30 Oct 2023 00:24:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0740C6B017D for ; Mon, 30 Oct 2023 00:24:44 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C8C1C1205C6 for ; Mon, 30 Oct 2023 04:24:43 +0000 (UTC) X-FDA: 81400836846.22.9878832 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2084.outbound.protection.outlook.com [40.107.237.84]) by imf13.hostedemail.com (Postfix) with ESMTP id D58C72000D for ; Mon, 30 Oct 2023 04:24:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=KaysYbuh; spf=pass (imf13.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.237.84 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698639881; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WJ9LnQ+uY3eBG7M4nyDlMzv2l2lTUtyumg2o6d1ZBVM=; b=MCBqnFiuLmQYvCGplkSpx02O9aKlvBIRbEej6OaLEoah3nk1kyLvKHE+eIqSbSr+oyTMMw jOygKP2Iw0P8vdf9DdQv4Ay4SMrBCiWzRfipuqj1zaLbgjnCTGDp6/2u9xfEHVQE9cU0Ao 7g2Fxk+cKdVyH4okhRJCG/58Az0e7BQ= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=memverge.com header.s=selector2 header.b=KaysYbuh; spf=pass (imf13.hostedemail.com: domain of gregory.price@memverge.com designates 40.107.237.84 as permitted sender) smtp.mailfrom=gregory.price@memverge.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=none) header.from=memverge.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1698639881; a=rsa-sha256; cv=pass; b=eQdAJiWyp2BisOc5xkc6f4iWShILrwNSmypHYuVvTRulpGchCYaYUs9qCWMceACpd9CM0i WDR/DNYgdMVsle3NHPZPH7azAMZQVmHb5oYidAtdW5Rhe8+T2JaI251ms9q/eGuEVf2I8L WPkQlWcYqbjeRjquek8q/cVwZVrFUx4= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bCKFkpnbEKwHwotqYvw8jPIj8t4KsGb5jLxnw/xY9O7cmrRRCDjJkHNCw2rDfmyTeNAs8ob2wdQAjOjZVTcgtHTuul5busCDjTwYZufCZ1l/NoPWKvak4agupWzVgRQOvEgsQAO1WCghoTrGblBTLmG/rHgD2Dx5JyF+vjF99bRLvMZxcfrsDWPDNqanf0RC4tosl2UNJethhPtDC2w2lQeyaquKqH7dkzHE9ThpKRIQB8NDEn5UTDCe9tbJr+m0wCRNL+oSwpFftm3LLjp2nv+DGz/R1EBvc3MoUV8OJRTK+UsOAPIsq/g0eKIs7uHMm/LH6NJg4RP3nD02YX3YBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WJ9LnQ+uY3eBG7M4nyDlMzv2l2lTUtyumg2o6d1ZBVM=; b=KebdP7VANGDVyGraj6ybdIcFFCGFzzOCwdszwGKzjhwcP4wGksHji16A4mBoU/AoDTH5JpOYvsWSPmf9wElKAH9yXgamWm3eIginP9rz4+mx5RphpqbxI1YdBD2Ru/zLSldEg7NBW2/L55fr8XIg9lV6Ua4IqqLpvMIxxiT43DvDZUym7jV5dypknROziB2rQvsJdJvCv4Ku6eXse8z6dbQAgBpj4aOJ7TIOIrr6ayWVfpHa+EtxM/qOSHzW//5r7j+89s/iMT6uyDvfmh8/ialYJoU9OuaqiU8xbBd5yyOkose//X2allhx+Cynfnxtvku6Lw59wbHHuVs+j5yXhg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=memverge.com; dmarc=pass action=none header.from=memverge.com; dkim=pass header.d=memverge.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=memverge.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WJ9LnQ+uY3eBG7M4nyDlMzv2l2lTUtyumg2o6d1ZBVM=; b=KaysYbuhXkHBDUULv4YuF7u9h7HYmM5J5WDi+pEvJow44P+cj+qdjLrZn73mOa1j3rvOJss66N+hrC6dXoYZ6ahzvfsE5klXaL9MPDOdgZD02h1PJAmWNeQ7OWeDFFh5sicDxTBxlvRAZAXmDk99CjJpv2/Pg0wNkk0lZiMyeAs= Received: from SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) by MW4PR17MB4683.namprd17.prod.outlook.com (2603:10b6:303:104::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.15; Mon, 30 Oct 2023 04:24:36 +0000 Received: from SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::381c:7f11:1028:15f4]) by SJ0PR17MB5512.namprd17.prod.outlook.com ([fe80::381c:7f11:1028:15f4%4]) with mapi id 15.20.6954.015; Mon, 30 Oct 2023 04:24:35 +0000 Date: Mon, 30 Oct 2023 00:19:39 -0400 From: Gregory Price To: "Huang, Ying" Cc: Gregory Price , linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, akpm@linux-foundation.org, sthanneeru@micron.com, "Aneesh Kumar K.V" , Wei Xu , Alistair Popple , Dan Williams , Dave Hansen , Johannes Weiner , Jonathan Cameron , Michal Hocko , Tim Chen , Yang Shi Subject: Re: [RFC PATCH v2 0/3] mm: mempolicy: Multi-tier weighted interleaving Message-ID: References: <87edhrunvp.fsf@yhuang6-desk2.ccr.corp.intel.com> <87fs25g6w3.fsf@yhuang6-desk2.ccr.corp.intel.com> <87ttqidr7v.fsf@yhuang6-desk2.ccr.corp.intel.com> <87lebrec82.fsf@yhuang6-desk2.ccr.corp.intel.com> <87a5s0df6p.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87a5s0df6p.fsf@yhuang6-desk2.ccr.corp.intel.com> X-ClientProxiedBy: SJ0PR03CA0064.namprd03.prod.outlook.com (2603:10b6:a03:331::9) To SJ0PR17MB5512.namprd17.prod.outlook.com (2603:10b6:a03:394::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ0PR17MB5512:EE_|MW4PR17MB4683:EE_ X-MS-Office365-Filtering-Correlation-Id: 46c2affa-9940-4f34-2548-08dbd9001f70 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: znZH7VAs9KZd9eYJGIFOoLDT7D1uipMOF+SyINFrahTWpNzXVBlfmo97IT3DPAM/wsv1G8VKO5r2azDGj887hwWvb2LgbX3niL2bymsuAbFq5xy4eAU1i4NTF8EckFYPIAv5zmhAROdIcgQlpUJ7EIBY9GkS4RDfV0TIG/xlavw7f4Sw6GA7Bj3cg6rcY4j9EmonrL39ko4MflaNOHpzrE/Gzq86/OemB2109a1e4dUBFnUeKbeyonBt0W/hJB/1aNdfnBlexRHhDE+khvJQsrHehbXXQhGm/NOJAjbfikzkDYPHh0lppK57wPC6tR5Y08pMZYxDJ6FpM8lObCbfPKxz0U+6+195zGK8GYipvVkmO8yna3TwrjDQyO0gQrvCNOsG8Ru35tohfgJ9w3CxKnFAQD/9BiasbftGwn7sq3xM/JdL9KlArCiL9y02lFf2nyc+iO52wl3H6Nt1gSab5ZJUMDscxyWhVeKJWIiLPK/u7BJlthBU4iYHxhFZbfW3zWNn40AGUgbfFLOo5pMtXYCqSo7G15Hprah/UlQTrOQ9kgUgtDIjbooAV4/9bxv2wX7NWYNF42hXn+lqtqZiOMcb79zbS+OMx6XnVMM2Icw= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR17MB5512.namprd17.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(136003)(346002)(376002)(39830400003)(396003)(230922051799003)(186009)(451199024)(1800799009)(64100799003)(5660300002)(41300700001)(7416002)(2906002)(54906003)(66476007)(66946007)(66556008)(8936002)(966005)(6486002)(8676002)(4326008)(44832011)(316002)(6916009)(478600001)(66899024)(38100700002)(83380400001)(86362001)(36756003)(6512007)(6506007)(26005)(2616005)(16393002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?IYlhymq8cTF3qWV7CZ4qZEtW8/qv/SF8vOIsneI5FQxfwnfZt6QS1WaP/9qz?= =?us-ascii?Q?HnYobMy3FUe/uo+0KSnYD5UZivpbmPH4ADvyHRaZiDtEOidYrW+8DZCN6OsM?= =?us-ascii?Q?DAA5zKsHTSnE5stD+1vT4SPWSNBGC5rCsTnf+g2LaYFCyM84Yo7lQ/hlML2/?= =?us-ascii?Q?Y6vhjlfqIj0j31DZIDU6rp6uOIHVxGJV37l7bo6pSzi9ZQmd5wH69YuGUcQy?= =?us-ascii?Q?fi8Qv9pL6Heg59ORVb88wo518mhJGFHE66tRiDofbQp3f9QandY0YlHNGd+6?= =?us-ascii?Q?TvrH+ruqhFaytSsBScj2rQeDZYz5/lMnQqyoGrguMsfZ/oowBzJpfwkoHqd3?= =?us-ascii?Q?VXL9HL4o1xDaxhBGMJCIbXLf5wE8w7znxMFjkad8mT8qn1HbNetTKkqPgijw?= =?us-ascii?Q?D+uHau3DDrnO5ey0OOpsOfq/eycD5Hp7IA9uR4CibyCmsnJKk0WAN85Ja9lM?= =?us-ascii?Q?+CI6VGawq5S+PEZco9V0IkV1bNGfkMQ5uxjrq+9t6Qab/3y/SPSjMHj/I8VX?= =?us-ascii?Q?/CJ0PyfPEs2OCFEHJ9M9FKmjArPa5jh3Iiu6tkcuvBhrOj1ZTaIJ/2jnLVcM?= =?us-ascii?Q?L3K0XZ53yj/ZZUX0gAvZ3uGxBfxaEtc6s9bpTqtXhKOtBp+CfgVEiQ4kDdmn?= =?us-ascii?Q?mBCJ4ZRVu2VtlcGAcoXgRyRRa+7srTXYQZ+lV2bWBWCgG1NkDhiV8g4F/x05?= =?us-ascii?Q?X5Q0ORk0XKdiRaMECsiD7vaDfFplrvu331oZsDXomW9l4p4ShFsJwJuEJiwx?= =?us-ascii?Q?kqffIdbspOKXLxMqhdPOEukCVS3wWI6/v/W+8v9x8iMXnOAN4RVVK4wtzPGW?= =?us-ascii?Q?kUZojeCUpv90sgquWB579QzeZ3JzSC2HPijsiRxQPetTcwxP+MBGD6ZyrF0T?= =?us-ascii?Q?up6wiWuUh8WGrNLa2OW+yftx4o3OZY7ezm0xjmewOAzfc5VxIXaq29Eh/AXT?= =?us-ascii?Q?moTfPy+XCiRVY0aQ9q0QYa3AGcuwDL13yQ8Pd90mkbNcGOSBOWV5VxBqY1tv?= =?us-ascii?Q?C36J3dICASZR562jZ/LD4ZOSIYizN/R/UnESfM8xwrX4s7x3YSFPkZ2+9pah?= =?us-ascii?Q?lpRTfzBvRqyDN7eGGqRQmhv0XfFv/MZ3mTjbRV0pVtjmx04z7kJVFNQgi2Py?= =?us-ascii?Q?YTj6h7ROqc1yYstCn01hNLL8NZgY4wO2Uzqp8n+eFAWC2aVul4gZEZ0tdYEZ?= =?us-ascii?Q?pa469X1e5z1ZCnL7OEAYE9Eh/E73t2DR9uPVp9DbOpGICIVhZIMqzo+BqYsx?= =?us-ascii?Q?9B+Zay9jBk6LysmaOE/4sRojXDc6Dd7+/76dIXeX8DTdTt6N2LRHOp9Aat1U?= =?us-ascii?Q?z/wb2lKQ+uNry8Qko9WtZmh/cNQtkwHcMxgruex6HH7BlT6oK+tZYE7CQ7/Z?= =?us-ascii?Q?vBcu8LSym3T60PDgw8KbMJ1zMrA2zujvOWusMrq1rH/gKZoxi6xVCh2LhVqD?= =?us-ascii?Q?O/GbTz2xmJvk7qAPNfFxogOsye4IEwrlFegblGHYHG9IzS/tBqPLAlhQ86TF?= =?us-ascii?Q?1M7HR5uQEVwnBALe42pOI2QSCjhp3MGPsi+OQU6x4KpX2WVUbk6CWpW3kwKq?= =?us-ascii?Q?DPXtd68jlBAWOKAqNMMjiHCJkbW+yVh0+n2OSnjLecGzlQR1AqdUDa51PYbn?= =?us-ascii?Q?UA=3D=3D?= X-OriginatorOrg: memverge.com X-MS-Exchange-CrossTenant-Network-Message-Id: 46c2affa-9940-4f34-2548-08dbd9001f70 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Oct 2023 04:24:35.5726 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Wegk/qus82kCUDx0QDQzVXXKSIjHdHbl3uTs9vGun6eTx4HL3qGzyEgdfnn/Y7Cn0Mgce4qbjLZ3aaiC5BECFWINs04z7mZayOoZ1VfD138= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR17MB4683 X-Rspamd-Queue-Id: D58C72000D X-Rspam-User: X-Stat-Signature: pps9ytkundux9c5f6xuuzj794f1r7f58 X-Rspamd-Server: rspam01 X-HE-Tag: 1698639880-635846 X-HE-Meta: U2FsdGVkX1/JOm3AQcRxrNDOIO5MCE5M7VSplV30B64+oIPCk9gESu5S5u0Nq+mTeFm+j4a4KcDfKP3DQfJ1bbE/3VYqzUvhxZ3KjjSuHjwVe5G0cAIZYKWWmYboOYt1lrmNgYhzNUmbP5umA0j9bfDAdmAR+JGyNKS95UYthmX0hNnOC/mAcgZlAdoHeD+rwu5JYqZ/th9q9U2BlQoqMhp2+t0HAQkFxVPNHgZsuiaUSt7JN/6UgDBYwz61xuClzD0u4CkN6yp8MWH3wv03Prlr0Sfbj6+uwyKkPzPFg9xKzMoT0+Bof2qe8pRNvPh/VpfSutANddfMFc5wY2368bO/KlHPT0DvKWafnj8gVkMwSJLssb9GJjy9xV9HRF3pMX6v4BQPZAW7y4Qf+thI2njeKPqcdX0o3EBQWRkOpF8gjd1gh+YmdsdI/+qR0gqQ2E/Zwx7JgHxG+IfL6+onl67WWESJ2mbDbA9mwqv4q0KLvhbu/tsKarV0tlaO2nvg3gmhrQnn+aRGY9qSFiJDHlFJlKb/vpcKvnkgwb2s9mII9jKljqnH/QlD5Pj2Dp85cwD1fpk26K85sboP/jYMEPN4A1QrDd/WhkZcyJHpe97Q9CdKhXvZaKNGdFIxO0C2ws//YPm4tNgWzyxpvcKZkUnXe9XdRckpaTJahoHfhC9VgTtMDeXjWH0qdp9vdsyvRtmdjg+EgvHYxHXztasBHu+lDVenpcCZjxdJQpwKdJaf5NKtMqzwHnAtDA0+k5jZEQPZX677xgb+NuJqck6T5+ZC0lsHyTQBoKlon+EPwimahupi0BN9Fmo4ok/2/49o0r0338NpBpRnoefP7yfiWTx1iu2Esi0LtNE+yAeSMkWsxW6G6PsxLjKLluFlljtvCAbpt5771yv3+X09hU+sFTmla+FdqaO6NYveDvtZVtxf6I+FLMndF2FB15nhSGcjt45p9+cQs/JD5BYLb9m B+w7l34g 7o9vAKN8slnGPM4udHnJakx5APcfHiXCH3LeTFw3f4rdL0ezNaj5jr9RWDVIVk8hoxKZgxmocuWSEujL5DiAncGiTjL7JwuyaQ8sGfxAorxWeHgHKo1EXELYvWM5jY1+W4SUjULE7YWNLSbJb9aVSKSQJdGKDuX8si6uA4vPaW7d7NqLfX93xtd4PUSv1pqCHBEXvXTNCnf6CXmDT5OtZ9cYmZQ1108Zxw35V6U0lKSS+tkoE7QLBX/ph85I1LrTIhBucHnO3QY+27M2nwvOVGQYojsd184XxAUxzMzkIe7KQvBze4lqnEspk4EEyAuToPckOGE5wu5JM7bL+mnhNzXcjOM9Pn70ehPPHy+rCdBxpwQTTm7DTN+gvmnpIUaJKfZ8YqvE9VbMP/rg+WhfnehAJNGF60wINhvMvtN3Y4M6jiyFWEEkHGoajU59DKVCbQOsiv7E+fC9K7q5OWiogCUXViQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Oct 30, 2023 at 10:20:14AM +0800, Huang, Ying wrote: > Gregory Price writes: > > The extending adds complexity to the kernel code and changes the kernel > ABI. So, IMHO, we need some real life use case to prove the added > complexity is necessary. > > For example, in [1], Johannes showed the use case to support to add > per-memory-tier interleave weight. > > [1] https://lore.kernel.org/all/20220607171949.85796-1-hannes@cmpxchg.org/ > > -- > Best Regards, > Huang, Ying Sorry, I misunderstood your question. The use case is the same as the N:M interleave strategy between tiers, and in fact the proposal for weights was directly inspired by the patch you posted. We're searching for the best way to implement weights. We've discussed placing these weights in: 1) mempolicy : https://lore.kernel.org/linux-cxl/20230914235457.482710-1-gregory.price@memverge.com/ 2) tiers https://lore.kernel.org/linux-cxl/20231009204259.875232-1-gregory.price@memverge.com/ and now 3) the nodes themselves RFC not posted yet The use case is the exact same as the patch you posted, which is to enable optimal distribution of memory to maximize memory bandwidth usage. The use case is straight forward - Consider a machine with the following numa nodes: 1) Socket 0 - DRAM - ~400GB/s bandwidth local, less cross-socket 2) Socket 1 - DRAM - ~400GB/s bandwidth local, less cross socket 3) CXL Memory Attached to Socket 0 with ~64GB/s per link. 4) CXL Memory Attached to Socket 1 with ~64GB/s per link. The goal is to enable mempolicy to implement weighted interleave such that a thread running on socket 0 can effectively spread its memory across each numa node (or some subset there-of) such that it maximizes its bandwidth usage across the various devices. For example, lets consider a system with only 1 & 2 (2 sockets w/ DRAM). On an Intel System with UPI, the "effective" bandwidth available for a task on Socket 0 is not 800GB/s, it's about 450-500GB/s split about 300/200 between the sockets (you never get the full amount, and UPI limits cross-socket bandwidth). Today `numactl --interleave` will split your memory 50:50 between sockets, which is just blatantly suboptimal. In this case you would prefer a 3:2 distribution (literally weights of 3 and 2 respectively). The extension to CXL becomes obvious then, as each individual node, respective to its CPU placement, has a different optimal weight. Of course the question becomes "what if a task uses more threads than a single socket has to offer", and the answer there is essentially the same as the answer today: Then that process must become "numa-aware" to make the best use of the available resources. However, for software capable of exhausting bandwidth with from a single socket (which on intel takes about 16-20 threads with certain access patterns), then a weighted-interleave system provided via some interface like `numactl --weighted-interleave` with weights either set in numa nodes or mempolicy is sufficient. ~Gregory