From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3CF0C48286 for ; Mon, 5 Feb 2024 02:50:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 218A46B0071; Sun, 4 Feb 2024 21:50:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C8E66B0072; Sun, 4 Feb 2024 21:50:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0912B6B0074; Sun, 4 Feb 2024 21:50:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EDB4D6B0071 for ; Sun, 4 Feb 2024 21:50:24 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8E2A9A014E for ; Mon, 5 Feb 2024 02:50:24 +0000 (UTC) X-FDA: 81756221568.28.F6A228C Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) by imf15.hostedemail.com (Postfix) with ESMTP id 59B62A0008 for ; Mon, 5 Feb 2024 02:50:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=jt+T+wjw; spf=pass (imf15.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.99 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707101422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t0gk3xBIxgK8STj8RToaRUAGDh+0bUS8n9Avz3FOyak=; b=bHHRgU5FV7RMLR+WWayBYrB6jmh2GsAgebR17b+qLUeKX7exr9BFHsjGWyHy506/jXxCFs kSomNv77VCAlL5YOh2pVOHkK+WAzUfsibFuYyypLEFLFT6l/YEALA1Lw91fFbbBsQn+BvF nD72/V4d3+0ZkMfwNIxCPKc2o66Qtqg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707101422; a=rsa-sha256; cv=none; b=fZ4h9Ms5ffZnaDVUVJMw8WD4xBboGp4EsyW2+y+LrdH3KS1KqcTJ1ORJoHw0VyIYI6D3cH Vp+K8ZcjP3IUDchYWzxuDWpZ/v0QSMjUUW/wGh2Q/cQe1UcJDb8LrLduJnOmSlZ7AqZEd3 xhkQPk+6xjWZGmN3kX3prI5SD1OwXX0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=jt+T+wjw; spf=pass (imf15.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.99 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1707101418; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=t0gk3xBIxgK8STj8RToaRUAGDh+0bUS8n9Avz3FOyak=; b=jt+T+wjwnAsHod4w78hjPtemo/SM4xuCyDPPxKW0FM0iamlRTHeAb0EwfLgLSKU9dQWeWv/IMg4JNvDg7VkQCWXCtNGyNhST8qQL7gL3Zt1gEhORwdkOaE02hkfsRl6c1ltMSG78ZvMJFH0YGrU1Ww0c7ikMIep8esoTALdRzrc= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=7;SR=0;TI=SMTPD_---0W02DprY_1707101415; Received: from 30.97.56.40(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W02DprY_1707101415) by smtp.aliyun-inc.com; Mon, 05 Feb 2024 10:50:16 +0800 Message-ID: <909cee7d-0201-4429-b85d-7d2662516e45@linux.alibaba.com> Date: Mon, 5 Feb 2024 10:50:32 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH] mm: hugetlb: remove __GFP_THISNODE flag when dissolving the old hugetlb To: Michal Hocko Cc: akpm@linux-foundation.org, muchun.song@linux.dev, osalvador@suse.de, david@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <6f26ce22d2fcd523418a085f2c588fe0776d46e7.1706794035.git.baolin.wang@linux.alibaba.com> <3f31cd89-f349-4f9e-bc29-35f29f489633@linux.alibaba.com> From: Baolin Wang In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Stat-Signature: 47orrg7x4phtkdikp1pmgd1ay3adu4m1 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 59B62A0008 X-Rspam-User: X-HE-Tag: 1707101420-495653 X-HE-Meta: U2FsdGVkX1+JwnQXK5vSgbWMG4VC0rethLIaCW4xjcww5TeuCW+Lb72ytPGpKe+vjy9ULFS2XA6GjA492HVFm7mnIgWT2uC+4EgU2y7Xj6+zgr/+6kR4vAHOBlMcsObrsbXUg1TqB7A76hgk2ajc2O1ZhcD/+fnW8KE3gD9DlZP1/dZW7tSr3O7SSCegKGE0ISnSWyLTFLiUHxY4eESBotqcamzL2VsJxAKz9+T64K/P6fBks2oj+cQkA5ZWc+Ctvht99FpSDBpV9996GvGGiGMviD6kp5bZd49tOYxAsco974YGhykZ7gvh1V0+vqSchJKVJTHTnjrJjvnb+F9uQIl4JAngpDMurAu2+lpCZ1hU54nvPpnjhFJDf5o+rTXL85fzLP/raKppwdbpfKpMzaiqZF+1sUoGcYvrb2SzlvFOwWXBJ3M1W7tyg2ojmdnOewmrNWhPHeghbHb+6+zKfUtD99srkCNXFUZlkTZG1/6czdvcz6d1WyIlbYAuhKrjj55KG9rqceLcc7+3abiBh4nXUdHeTIz0Ei/LneTOqnBMwK6kaDgDy/S54CmUck211h1MtzpiVMJgD/NNOGf9uvgBX5q+FcBw79Hq/fT/YJIfHoW9dqsvMgi8p/+3JRSVHCfjKiNVLnXyzWhyhPiB+eH8Kh+5sviKHd/rEn75vNXOxZ8DxfHFj9RjGBU6WOVza9zCmk9EeS9JvfXBb+1cHb82nRql63bEmbp9TEdk4+rBjnJMKBR3DAoWf8ocNz4/K1tAN2rofZt0RAAyV2pn+CoBtH7YXo2P6PCKv9WWOUJcTPlzqLvOFOlfTF42Zp/Yv1UyOdNaRH4AVCU2/5hwYJgT7Oge/m/QMVMXhxXz9Gqa0bbSNzoIoXe5h1XdB0c5S+2owxp7HfobK6Cnw/TpxeaQG8ZakFaJY9ZBFu9JIk4RqpNsW+RFOcf1rvVOcbDl1WxEhVyOFeR5P3gz4IT Bv7TWnFm fAWv95dHkc6eWH1twPrY+G8ykPDjrvsb35c7ULxiSpxIB0Jt/KuBdvIdXcBHpmBdRS79JzLT1y3RP+KESFmrilENbbTvnkqmHKgK055mhalpXQ9pMZzjlJnFBgQhnHQMv+dhP7A4QIzIG+kOO+9HHKh1XMJN+yXr/HHgnaNUnTZi1ItwdsW9YYfDWRDJsA8Xcsyzqfsx4iz5YeIFv4vKn0cv3hK59ObgdvoN/N/U7nmoKgPkEUK5CjzD3Y9yeH/+CDcJ6Uzw77tEKSEyYlxBmmFhYwrnW1gdfM/IgHYhujwf8urH5GIEL/7yzgWE/h3OUZMz6mVboGn+3794= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/2/2024 5:55 PM, Michal Hocko wrote: > On Fri 02-02-24 17:29:02, Baolin Wang wrote: >> On 2/2/2024 4:17 PM, Michal Hocko wrote: > [...] >>>> Agree. So how about below changing? >>>> (1) disallow fallbacking to other nodes when handing in-use hugetlb, which >>>> can ensure consistent behavior in handling hugetlb. >>> >>> I can see two cases here. alloc_contig_range which is an internal kernel >>> user and then we have memory offlining. The former shouldn't break the >>> per-node hugetlb pool reservations, the latter might not have any other >>> choice (the whole node could get offline and that resembles breaking cpu >>> affininty if the cpu is gone). >> >> IMO, not always true for memory offlining, when handling a free hugetlb, it >> disallows fallbacking, which is inconsistent. > > It's been some time I've looked into that code so I am not 100% sure how > the free pool is currently handled. The above is the way I _think_ it > should work from the usability POV. Please see alloc_and_dissolve_hugetlb_folio(). >> Not only memory offlining, but also the longterm pinning (in >> migrate_longterm_unpinnable_pages()) and memory failure (in >> soft_offline_in_use_page()) can also break the per-node hugetlb pool >> reservations. > > Bad > >>> Now I can see how a hugetlb page sitting inside a CMA region breaks CMA >>> users expectations but hugetlb migration already tries hard to allocate >>> a replacement hugetlb so the system must be under a heavy memory >>> pressure if that fails, right? Is it possible that the hugetlb >>> reservation is just overshooted here? Maybe the memory is just terribly >>> fragmented though? >>> >>> Could you be more specific about numbers in your failure case? >> >> Sure. Our customer's machine contains serveral numa nodes, and the system >> reserves a large number of CMA memory occupied 50% of the total memory which >> is used for the virtual machine, meanwhile it also reserves lots of hugetlb >> which can occupy 50% of the CMA. So before starting the virtual machine, the >> hugetlb can use 50% of the CMA, but when starting the virtual machine, the >> CMA will be used by the virtual machine and the hugetlb should be migrated >> from CMA. > > Would it make more sense for hugetlb pages to _not_ use CMA in this > case? I mean would be better off overall if the hugetlb pool was > preallocated before the CMA is reserved? I do realize this is just > working around the current limitations but it could be better than > nothing. In this case, the CMA area is large and occupies 50% of the total memory. The purpose is that, if no virtual machines are launched, then CMA memory can be used by hugetlb as much as possible. Once the virtual machines need to be launched, it is necessary to allocate CMA memory as much as possible, such as migrating hugetlb from CMA memory. After more thinking, I think we should still drop the __GFP_THISNODE flag in alloc_and_dissolve_hugetlb_folio(). Firstly, not only it potentially cause CMA allocation to fail, but it might also cause memory offline to fail like I said in the commit message. Secondly, there have been no user reports complaining about breaking the per-node hugetlb pool, although longterm pinning, memory failure, and memory offline can potentially break the per-node hugetlb pool. >> Due to several nodes in the system, one node's memory can be exhausted, >> which will fail the hugetlb migration with __GFP_THISNODE flag. > > Is the workload NUMA aware? I.e. do you bind virtual machines to > specific nodes? Yes, the VM can bind nodes.