From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3F80C197A0 for ; Fri, 17 Nov 2023 16:05:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C3B366B04EA; Fri, 17 Nov 2023 11:05:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF3656B04EB; Fri, 17 Nov 2023 11:05:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD9576B04EC; Fri, 17 Nov 2023 11:05:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9E3FB6B04EA for ; Fri, 17 Nov 2023 11:05:00 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 74E24C0DFB for ; Fri, 17 Nov 2023 16:05:00 +0000 (UTC) X-FDA: 81467919960.19.E2B3422 Received: from outbound-smtp28.blacknight.com (outbound-smtp28.blacknight.com [81.17.249.11]) by imf16.hostedemail.com (Postfix) with ESMTP id 2E6AD18002A for ; Fri, 17 Nov 2023 16:04:57 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.11 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700237098; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sv1QJO/5ojlFj/n8OL5LnbwhF0zbol0fcV3u+zrIHJM=; b=YhEErLJiTnB+G/981D0pn4+mHmktD+uSpk7LTh7Khgwyc91oni6gKjizar46+7e0bGWBtg 6+BnlrW7b6A5kvs/2NWwGmiRmcIx9B3cD7xAFyChP8AM+Oe6muCWnPfqjm2KmmQt+Cs/Gz vzhDMCw2XAmnI6CTS4JODvpkCgPbDr0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700237098; a=rsa-sha256; cv=none; b=GtK1O7IEyH/p6J/RYc6E8mc0NXofaFMc5wwW4wsBn2yzdy0sNZaVfScM89bE2MYyMRgEx+ mb3224KqiA29QJo4ijVwD0Faz5LzxKZVR7bEzZHe+qXAXVjeadGC9AqCkwwOmWqnCSV/0Q gbmR13cZhAjpxpVL3H5hSatTjVenLgs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.11 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp28.blacknight.com (Postfix) with ESMTPS id D58F8CCE5B for ; Fri, 17 Nov 2023 16:04:55 +0000 (GMT) Received: (qmail 17770 invoked from network); 17 Nov 2023 16:04:55 -0000 Received: from unknown (HELO mail.blacknight.com) (mgorman@techsingularity.net@[81.17.254.21]) by 81.17.254.26 with ESMTPA; 17 Nov 2023 16:04:55 -0000 Date: Fri, 17 Nov 2023 16:04:53 +0000 From: Mel Gorman To: Peter Zijlstra Cc: "Huang, Ying" , Baolin Wang , David Hildenbrand , akpm@linux-foundation.org, wangkefeng.wang@huawei.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, John Hubbard Subject: Re: [RFC PATCH] mm: support large folio numa balancing Message-ID: <20231117160453.dkbpwub7aq3jxksf@techsingularity.net> References: <606d2d7a-d937-4ffe-a6f2-dfe3ae5a0c91@redhat.com> <871qctf89m.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf57en8n.fsf@yhuang6-desk2.ccr.corp.intel.com> <20231117100745.fnpijbk4xgmals3k@techsingularity.net> <20231117101343.GH3818@noisy.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20231117101343.GH3818@noisy.programming.kicks-ass.net> X-Rspamd-Queue-Id: 2E6AD18002A X-Rspam-User: X-Stat-Signature: dduro9mra517yunb6yasccj5ar61frn8 X-Rspamd-Server: rspam03 X-HE-Tag: 1700237097-704830 X-HE-Meta: U2FsdGVkX1/DKEmzZWuVDZdqA7A0MluhH6yru/mMxQqNCzpzB+1qtTRj8Hph/ScIwhu0DJiSU1All4J4JViz/uIuoc8hnX58ho1rjfJSTBehUHFSUUxbdAOh8RvnjK2ctvpTLaoH3wf6EXwKpYQHYbQaJJP9IT/B9W6GlaTSvVNPtkz4qtoRFNmixhpVgQuaGmvEj53Up+4f2X9Gn79XahUTcKRCOGQqj2Z3O8czMkksN5bNNT1S+Ti4HVCOuXMLILenMVRI4s8pyJH2KWNvXoPkDcTXh1R3yIxs5wBzcG4+9WGRNcrXCTkGlPUSzIBuCg+lKggLrD9uUTfQZ4GwoftKnrRiBoR0kIgfAe9J8R92eqVnuQ5k1U0A0+IChLNIs6tF+vLIrFit9skzUQb9tjUt4Fsy0gX0TikjJB32n9P2GW/ovd3bYm9YQVFN7Hioqnz4Yd2cKK6IhWzixOmZP/z4pnJJj9RnLQtv/bJ2BNLrXXHUpRbsbSWU1PZkF8dDq4gKomxSNfcy7wAHod/q8FWypbeWi0HUWzqt0zQebNWTemHeKzEvK5Y0Fy6EclrrVzYwHjM9Bhq9PAC6l4ZggmSAOyOL+N7fp5GIZjZWII4jJjTQ+4gfmUztjOsAdzosNTi3kCFnVGIKS7IB+XkVfgkYxex6ZglJWERRzldIjPuS9ZfVJsgeI/aTvcEJVGv+ahgv7BafoV/LKvU+B2WL93SDGr4zm+5VSop7/bYEmqfuwpx6/obFHd2B6C2o6mOirt0BlqUN5+FphQj4miH5BJoLddh8LlYQg0rirC+Z3i6LAV0bsV8vMD7SvmvWDSZvrGP7aL04+zwoECF97UFIsaRiFQrOYHBPDLEYXVyo/2Yz/6cMOOyli+Bb5uOmvPZ1sCs4EP/z3zQ70vHfUoyBwVWUwd4GQNfuyxhfaUgRsKr+qqmz7qXsdyLA2kusjAvPHLe/LnYhKhTpoQ9o5dN Z0H3WYpd 3eXgSIfnCKOpiDTmM0ImhbQdUEjcohMSTmjv192P0RLIW3zSMAgW7W31oIE9h+tdclxbCccg2QuVt1UrkhWj/U71NJ0zAMW3kfC176RSLHYpXBGpiBBSuv1bPJG3YAzqT8Kr7Ez1dunDPOoFULqkIC8Q2cZDMXfvkUt8yBIzMFdduibM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Nov 17, 2023 at 11:13:43AM +0100, Peter Zijlstra wrote: > On Fri, Nov 17, 2023 at 10:07:45AM +0000, Mel Gorman wrote: > > > This leads into a generic problem with large anything with NUMA > > balancing -- false sharing. As it stands, THP can be false shared by > > threads if thread-local data is split within a THP range. In this case, > > the ideal would be the THP is migrated to the hottest node but such > > support doesn't exist. The same applies for folios. If not handled > > properly, a large folio of any type can ping-pong between nodes so just > > migrating because we can is not necessarily a good idea. The patch > > should cover a realistic case why this matters, why splitting the folio > > is not better and supporting data. > > Would it make sense to have THP merging conditional on all (most?) pages > having the same node? Potentially yes, maybe with something similar to max_ptes_none, but it has corner cases of it's own. THP can be allocated up-front so we don't get the per-base-page hints unless the page is first split. I experimented with this once upon a time but cost post-splitting was not offset by the smarter NUMA placement. While we could always allocate small pages and promote later (originally known as the promotion threshold), that was known to have significant penalties of it's own so we still eagerly allocate THP. Part of that is that KVM was the main load to benefit from THP and always preferred eager promotion. Even if we always started with base pages, sparse addressing within the THP range may mean the threshold for collapsing can never be reached. Both THP and folios have the same false sharing problem but at least we knew about the false sharing problem for THP and NUMA balancing. It was found initially that THP false sharing is mostly an edge-case issue mitigated by the fact that large anonymous buffers tended to be either 2M aligned or only affected the boundaries. Later glibc and ABI changes made it even more likely that private buffers were THP-aligned. The same is not true of folios and it is a new problem so I'm uncomfortable with a patch that essentially says "migrate folios because it's possible" without considering any of the corner cases or measuring them. -- Mel Gorman SUSE Labs