From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 908C4C10DCE for ; Wed, 6 Dec 2023 00:52:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 267976B0085; Tue, 5 Dec 2023 19:52:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F1336B0087; Tue, 5 Dec 2023 19:52:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0929C6B0088; Tue, 5 Dec 2023 19:52:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EB56B6B0085 for ; Tue, 5 Dec 2023 19:52:31 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BF4E740337 for ; Wed, 6 Dec 2023 00:52:31 +0000 (UTC) X-FDA: 81534567702.04.B4E8CAB Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.11]) by imf21.hostedemail.com (Postfix) with ESMTP id 5EFE91C0009 for ; Wed, 6 Dec 2023 00:52:29 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dnNwjZrv; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.11 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701823950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GZMZMHdiiiXNdlhfUahIdhzUq2D0dk/AvilXXch3Dok=; b=evEQqLlBwBbfZd1tQlkaauxj90PGtcOtlFATBytmL5v4b6mkJGnroIHuGXgRBS7mzjCdVj 18BdIOX40il4J/256IxW5BY2lraUA0Ftrj7iW9xtEm/62qDeX2GcjdqpIwz4kSDYWONo3Q eeMO7ML2gsKoyGP5EFHn5ualvtDAloY= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dnNwjZrv; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf21.hostedemail.com: domain of ying.huang@intel.com designates 192.198.163.11 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701823950; a=rsa-sha256; cv=none; b=mJVYx5eVY+D7bkt/gpYfBq7TyhyTIVt9hClt4XAgs93k8HMwewr6Ih6MgS+NeprFr5XSTE oMghuofhgBQ5+GIOLNdzdk3LlP7yWnc8lY3LqBAL46IIddLMaa1AzznpGR0Zt6aR+Tzwlx ifmXi5Vf8x9f/+cy3Pvjpfmh/uMHWu8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1701823949; x=1733359949; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=80ctbQu0ehO8Eb46q4E7qyIosvMkmzHirGugGzD0fGM=; b=dnNwjZrvEgSYI5ZGB1YRAxN1p4+VNwcwF0ZyKFUefjhlOGnrPZ09NKng YtHH8+FCx9AK8mnYwPYvDRiov65tR1h2tm41R3pGSFBJrs4Aqu47QDNtu nuoE7iyoKOPLT5yca4bNv5LGjMXCR2BV2lU9+Alw4nuIiRNvNFEBCTwGi h6OU5u4h8lYtyppJ0FjjIiwFK5xr/q3CsJiz3VYIxsi+pB0q8FNu3i5iB lvfDiHlvGuEuBl3QG0+Xf8iz5h/Go6eRW/YG1XUjF1inzyDg5rHbXgInw xAzj58bCyqDFXG/nnS8B+GQYH1z9RGhl7ZASApb2pBsftFb9y6sHk3MMh g==; X-IronPort-AV: E=McAfee;i="6600,9927,10915"; a="821340" X-IronPort-AV: E=Sophos;i="6.04,253,1695711600"; d="scan'208";a="821340" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmvoesa105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Dec 2023 16:52:28 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10915"; a="861937995" X-IronPort-AV: E=Sophos;i="6.04,253,1695711600"; d="scan'208";a="861937995" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Dec 2023 16:52:23 -0800 From: "Huang, Ying" To: Gregory Price Cc: Michal Hocko , "tj@kernel.org" , "John Groves" , Gregory Price , "linux-kernel@vger.kernel.org" , "linux-cxl@vger.kernel.org" , "linux-mm@kvack.org" , "cgroups@vger.kernel.org" , "linux-doc@vger.kernel.org" , "akpm@linux-foundation.org" , "lizefan.x@bytedance.com" , "hannes@cmpxchg.org" , "corbet@lwn.net" , "roman.gushchin@linux.dev" , "shakeelb@google.com" , "muchun.song@linux.dev" , "jgroves@micron.com" Subject: Re: [RFC PATCH v4 0/3] memcg weighted interleave mempolicy control In-Reply-To: (Gregory Price's message of "Tue, 5 Dec 2023 09:47:51 -0500") References: <87o7fveeze.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf4i2xe1.fsf@yhuang6-desk2.ccr.corp.intel.com> <87fs0h2fb4.fsf@yhuang6-desk2.ccr.corp.intel.com> Date: Wed, 06 Dec 2023 08:50:23 +0800 Message-ID: <875y1c2lyo.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5EFE91C0009 X-Stat-Signature: 946w35884afm1n3tmmhysf1dcmjnicem X-HE-Tag: 1701823949-994193 X-HE-Meta: U2FsdGVkX1/Hl6SMOp4jfRk/ipSeACGMRpMvHJCzWzDUzZLtjV7WWOBMTz+uPcfmOUqWzjQ7WCAu87ugoZXeEtumanUbL77CYbmIJ4j61m8tXwWcfWucJoR5moCiG0vNgOgT5rURx0Gib0oaO9jb3TgIyo+3MQ3yC5Ko4+8kwuvICoDKUsDqjPVAmjj3tIc10HvUDvdWuI8AOIXnGrjAlP9HZc/mFUSA/UgV4TUmo+MMkN3iW9C8JxESIpP6FFrMlKYlW7760YZcj7HcvjRkRtYJG6HLNQqUvvdjYsLw8JQJkVhGQ9yRG/TpRBJpH/6T6IhIvBIL+iidUR9x4qzrjEhzNUbnctCYixiKK4seiCfUXq4lsf5qP1o4zPpOM9jhEA2BzExvG49NeYjsoI5wQppPrBtLKZHRcNVdWhDCcno8NXwHOrP0j0Ghqv/y/uppQpA9g9gpCqeDOsuWOe4K2VukkJjzHOV4bWpWMHtbKgE7pXjE1bq0KctJMtdGYTvcxVOXNnLWhSnCySBOsq2EIWzW1aei7ZoPN3UcsQ+XZZFf6xz2jvl5+UD3BLsBsFJdSN280vngzjk0oYuWG9IkIG3Ddy8cOinz/WRyJJwfbqg7lxlDrBwuHGt6OT1E6gjNmtZX4I9ih8jxWXnSGmERZxSFN2Hr7mau4HX8dinyDG5xKbye2t2tK58Hz91WMxDfSifW505O3piclq+h/epUQqXgMcZ3SPKNwAXIoUFsXxp/ChjiQaWlbyRxrt6Xf0VZIzopt2oj12QlReIHt1DtxMZ3bqWcMrZ3VsZZvlj+tSnd9mTe57wydaIIz1fGvjgOFyK/lZkUXhymTDhUqfnRe0v00YiOmspsBmJDe1n1GSCvrXFOsd8Ign/hre/EFM8mbGX/PEDvu4i4Fymlzm9p7CGKcq06nT/dYYKkb3U1H1Z3P2eEYS5GS/4PkySISqXMxjAF+sVVN7TUCq841UH wiQtItM1 dFIXbYzsxT+Q3Il4MqCERDADmlEgI5mR+/w+wKLAreaF5kz5XNYt5wpCb54spJoGbqbMloxIe3siTJsRAqa9B8UstDQtR4EHRvS6VtXXKSBAL+mjqmtP3jYveAjLqQoXSCDKniaYQxU6x+mN97tNwXKp8dffXi4Ax7yUBospyECGv7l8NmPPSO0jFeuMOG2eoc+9ZPn+SoM2dlNItWj83/7w45GWHA5sWABsdP58vzDXAwTqI0lL0SZ4od7Ve836JFUPZwR5CICwgd5o/7xYhsTJiFgEzgo6S7XOtzZw1C4ximyfld7b7c35UHfq+qCcXhXYJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Gregory Price writes: > On Tue, Dec 05, 2023 at 05:01:51PM +0800, Huang, Ying wrote: >> Gregory Price writes: >> >> > On Mon, Dec 04, 2023 at 04:19:02PM +0800, Huang, Ying wrote: >> >> Gregory Price writes: >> >> >> >> > If the structure is built as a matrix of (cpu_node,mem_nodes), >> >> > the you can also optimize based on the node the task is running on. >> >> >> >> The matrix stuff makes the situation complex. If people do need >> >> something like that, they can just use set_memorypolicy2() with user >> >> specified weights. I still believe that "make simple stuff simple, and >> >> complex stuff possible". >> >> >> > >> > I don't think it's particularly complex, since we already have a >> > distance matrix for numa nodes: >> > >> > available: 2 nodes (0-1) >> > ... snip ... >> > node distances: >> > node 0 1 >> > 0: 10 21 >> > 1: 21 10 >> > >> > This would follow the same thing, just adjustable for bandwidth. >> >> We add complexity for requirement. Not there's something similar >> already. >> >> > I personally find the (src,dst) matrix very important for flexibility. >> >> With set_memorypolicy2(), I think we have the needed flexibility for >> users needs the complexity. >> >> > But if there is particular pushback against it, having a one dimensional >> > array is better than not having it, so I will take what I can get. >> >> TBH, I don't think that we really need that. Especially given we will >> have set_memorypolicy2(). >> > > From a complexity standpoint, it is exactly as complex as the hardware > configuration itself: each socket has a different view of the memory > topology. If you have a non-homogeneous memory configuration (e.g. a > different number of CXL expanders on one socket thant he other), a flat > array of weights has no way of capturing this hardware configuration. One important task of the software is to hide the complexity of hardware from the users. At least it should provide the option. It only add complexity based on real requirements. > That makes the feature significantly less useful. In fact, it makes the > feature equivalent to set_mempolicy2 - except that weights could be > changed at runtime from outside a process. > > > A matrix resolves one very specific use case: task migration > > > set_mempolicy2 is not sufficient to solve this. There is presently no > way for an external task to change the mempolicy of an existing task. > That means a task must become "migration aware" to use weighting in the > context of containers where migrations are likely. > > Two things to consider: A task... > a) has no way of knowing a migration occured > b) may not have visibility of numa nodes outside its cpusets prior to > a migration - making it unlikely/not possible for them to set > weights correctly in the event a migration occurs. > > If a server with 2 sockets is set up non-homogeneously (different amount > of CXL memory expanders on each socket), then the effective bandwidth > distribution between sockets will be different. > > If a container is migrated between sockets in this situation, then tasks > with manually set weights, or if global weights are a single array, will > have poor memory distributions in relation to the new view of the system. > > Requiring the global settings to be an array basically requires global > weights to be sub-optimal for any use cases that is not explicitly a > single workload that consumes all the cores on the system. > > If the system provides a matrix, then the global settings can be optimal > and re-weighting in response to migration happens cleanly and transparently. For these complex requirements, we will have process_set_mempolicy2(). I think that it's even more flexible than the global matrix. -- Best Regards, Huang, Ying