From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2FBFC197A0 for ; Mon, 20 Nov 2023 08:01:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6393E6B03FC; Mon, 20 Nov 2023 03:01:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E9F36B0402; Mon, 20 Nov 2023 03:01:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B1066B0403; Mon, 20 Nov 2023 03:01:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 362676B03FC for ; Mon, 20 Nov 2023 03:01:38 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0CF78A0443 for ; Mon, 20 Nov 2023 08:01:38 +0000 (UTC) X-FDA: 81477588276.17.56D11B3 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) by imf16.hostedemail.com (Postfix) with ESMTP id E117018001C for ; Mon, 20 Nov 2023 08:01:34 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf16.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700467296; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZLLHSILZJb/nXk9ReBM9607kNQrxxO6Y5ks+GANV3g=; b=ssnbqTYpCha7lpWGYLEm71rdWejY7Yq5YcPHBjJuhRStMfj9xTAcU9+aax4YyY5tMxYsXS q4FuwdvtClSdFwEFcqiPGdcpJ7JIUy6UWhlvPr/t8EO8ZlUsTaMhLiIdE5zsChZZ+e8QNf SBjR68dX4TXGblKDLWVwtwp5diXxnd8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf16.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.112 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700467296; a=rsa-sha256; cv=none; b=TPDh8hzuz+XjAcRPzYSygp4VQnDT9hlmVQLfEvisiKm5EvurXND3BTaGyjQ7jYV7WO+P+z p5wN83LPxBqz2dmw9am3OwcN/VZL0YfJfiGI08Unt8U5daW1hoAiVlMR4ZD3MwYCniJANv oUa5r2wWb15a5liHzyIIGxOjGubgG00= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=10;SR=0;TI=SMTPD_---0VwkI1Fs_1700467289; Received: from 30.97.48.46(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VwkI1Fs_1700467289) by smtp.aliyun-inc.com; Mon, 20 Nov 2023 16:01:30 +0800 Message-ID: <55ca937e-92ba-4d01-a8f1-3a2a66054451@linux.alibaba.com> Date: Mon, 20 Nov 2023 16:01:43 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] mm: support large folio numa balancing To: Mel Gorman , "Huang, Ying" Cc: David Hildenbrand , akpm@linux-foundation.org, wangkefeng.wang@huawei.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, John Hubbard , Peter Zijlstra References: <606d2d7a-d937-4ffe-a6f2-dfe3ae5a0c91@redhat.com> <871qctf89m.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf57en8n.fsf@yhuang6-desk2.ccr.corp.intel.com> <20231117100745.fnpijbk4xgmals3k@techsingularity.net> From: Baolin Wang In-Reply-To: <20231117100745.fnpijbk4xgmals3k@techsingularity.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E117018001C X-Stat-Signature: 4zbj8fcit19z95cqaerp61tyhooi4qsh X-HE-Tag: 1700467294-560956 X-HE-Meta: U2FsdGVkX1+OkIyDzKzw977WyTg/GZ65cuYAng5mRJ+U5YkwvcU/vz0OV57IwFe0r6zcwC5fdQt2iyC2CxiTkbpRU62EaQJaLrFZ0SFkxL2/B/cfrEXINYGqkRBoxui0MzXaPcmMFLU1CRgOaHsvHcAjAqKlpK9ixdJmnNp9pQke7MG8nuqvdQ9tgmi2e5WDgPcsykgZBdL/hMbQ2QnGWRwwpt73oHANHSR/uvQe/7c28m04vWs5CLe69HC8yZqovWp9PKeUNZEUYFZWDBT7Gq3U0Z+yMgQ20xn+oTwlvaNmbglRnK56+EZKGV01H+Lm7XDFu84E3XkYglnb2b5lX3TBA+3cYPfG6dbOCP999vE6ALCn7Ro+FAyCNipEOGSRsRRq+E0HoAln5RsxiNtuIjMgUMktAUfqTQfMt9vBQbHhAljlssBvfcMZrXpWmyeU3MSaqSPJ1yL56j33HFoZrVuoQkIhzNNeLWincPTShr308E6+xAqjUReZlLdUBZRT/3RoLnqrHcdVQrfxkcWDYO+HdwCChELFb5poUERAIyxH0cxmMURX+F0UuIyngsztJLHv4RdONublbwFrdrZk9w3G58+AS4ObBpPKXsDzPpQxJaUHEx11oh603HDqLLtuHf8S5CvMDyqrMrJ+pIXFV/KDibpPZsSWgYcDF3JmviKjcUH21Dfvub70vAaC89AosZ+TIrJASSoBii207ObDVyxyOf1UGfHfk3Xxbbk/8Ud+T+z16GAvH3VtS2cTDw2VbLR0G6YVngHfFcmvZ2dTtHY1ALqLDzcrqZQ00fCr6siXtwDkI/Sg7Jr0UPtqHRi3jruUI7pvhqkvFN9x8yh/mDWwy/pMl3tId9UYyuiKVtgg2Ub2EUjR22Bx4zLbW+XtzqOwy2qkn3rKdnTFl5vvqhxhpE2MWSeiWcdC46L6EbutzmpDVEwEUsS7thiB8e2pSuSUT6ijU+kxYWt+z4X RPANfcmC dZbtMLiW6PY9kiceQuwbjku6vnR43swt15yOgxmjtA7zHo6gYRv+XOyKb4mJd4LtgB6KO4YSJL/5jik6eFLYmCzixG/HvRKKe0AXOWf9PvCwByBYJcZBlQ//L3e9VB3ZI1y5qui8sD4m4gwXopF2GRJJrcupmLhAgDY985/wYnPT+lM/Nft4mTPit2bds4kDvOpbkltyxOmUn9WY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 11/17/2023 6:07 PM, Mel Gorman wrote: > On Wed, Nov 15, 2023 at 10:58:32AM +0800, Huang, Ying wrote: >> Baolin Wang writes: >> >>> On 11/14/2023 9:12 AM, Huang, Ying wrote: >>>> David Hildenbrand writes: >>>> >>>>> On 13.11.23 11:45, Baolin Wang wrote: >>>>>> Currently, the file pages already support large folio, and supporting for >>>>>> anonymous pages is also under discussion[1]. Moreover, the numa balancing >>>>>> code are converted to use a folio by previous thread[2], and the migrate_pages >>>>>> function also already supports the large folio migration. >>>>>> So now I did not see any reason to continue restricting NUMA >>>>>> balancing for >>>>>> large folio. >>>>> >>>>> I recall John wanted to look into that. CCing him. >>>>> >>>>> I'll note that the "head page mapcount" heuristic to detect sharers will >>>>> now strike on the PTE path and make us believe that a large folios is >>>>> exclusive, although it isn't. >>>> Even 4k folio may be shared by multiple processes/threads. So, numa >>>> balancing uses a multi-stage node selection algorithm (mostly >>>> implemented in should_numa_migrate_memory()) to identify shared folios. >>>> I think that the algorithm needs to be adjusted for PTE mapped large >>>> folio for shared folios. >>> >>> Not sure I get you here. In should_numa_migrate_memory(), it will use >>> last CPU id, last PID and group numa faults to determine if this page >>> can be migrated to the target node. So for large folio, a precise >>> folio sharers check can make the numa faults of a group more accurate, >>> which is enough for should_numa_migrate_memory() to make a decision? >> >> A large folio that is mapped by multiple process may be accessed by one >> remote NUMA node, so we still want to migrate it. A large folio that is >> mapped by one process but accessed by multiple threads on multiple NUMA >> node may be not migrated. >> > > This leads into a generic problem with large anything with NUMA > balancing -- false sharing. As it stands, THP can be false shared by > threads if thread-local data is split within a THP range. In this case, > the ideal would be the THP is migrated to the hottest node but such > support doesn't exist. The same applies for folios. If not handled So below check in should_numa_migrate_memory() can not avoid the false sharing of large folio you mentioned? Please correct me if I missed anything. /* * Destination node is much more heavily used than the source * node? Allow migration. */ if (group_faults_cpu(ng, dst_nid) > group_faults_cpu(ng, src_nid) * ACTIVE_NODE_FRACTION) return true; /* * Distribute memory according to CPU & memory use on each node, * with 3/4 hysteresis to avoid unnecessary memory migrations: * * faults_cpu(dst) 3 faults_cpu(src) * --------------- * - > --------------- * faults_mem(dst) 4 faults_mem(src) */ return group_faults_cpu(ng, dst_nid) * group_faults(p, src_nid) * 3 > group_faults_cpu(ng, src_nid) * group_faults(p, dst_nid) * 4; > properly, a large folio of any type can ping-pong between nodes so just > migrating because we can is not necessarily a good idea. The patch > should cover a realistic case why this matters, why splitting the folio > is not better and supporting data. Sure. For a private mapping, we should always migrate the large folio. The tricky part is the shared mapping as you and Ying said, which can have different scenarios, and I'm thinking about how to validate it. Do you have any suggestion? Thanks.