From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBAA3C433FE for ; Thu, 20 Oct 2022 09:33:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 349988E0001; Thu, 20 Oct 2022 05:33:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F9906B0073; Thu, 20 Oct 2022 05:33:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1E7E88E0001; Thu, 20 Oct 2022 05:33:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0F2776B0071 for ; Thu, 20 Oct 2022 05:33:24 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D5DCCAB9E3 for ; Thu, 20 Oct 2022 09:33:23 +0000 (UTC) X-FDA: 80040814686.07.FFCD4B2 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf21.hostedemail.com (Postfix) with ESMTP id 9DEC51C000A for ; Thu, 20 Oct 2022 09:33:22 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0VSeaYB9_1666258397; Received: from 30.97.48.62(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VSeaYB9_1666258397) by smtp.aliyun-inc.com; Thu, 20 Oct 2022 17:33:19 +0800 Message-ID: Date: Thu, 20 Oct 2022 17:33:17 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.0 Subject: Re: [PATCH 2/2] mm: migrate: Try again if THP split is failed due to page refcnt To: "Huang, Ying" Cc: akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, shy828301@gmail.com, jingshan@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <87mt9qnbrf.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Baolin Wang In-Reply-To: <87mt9qnbrf.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666258403; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nw6owlg/FthdwvRfB9wruxtCc4RFKpMCUZWx2KpGdFg=; b=Pk1VmqxIsyf6xF1muMiJaFiQPWAbbC0C1NweJ8SZ3yy+3sPsgyBvKAzCAVofIIHBriknPT 9/9RutRfPszuVDKnPHwAmhB5KepvZS5NC5QvdxbD6vbhA28tN/QPiCKLKx4nMN0koltcHZ A+7GPbwGUlZWLELuZxv9oiEspR5DRdo= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666258403; a=rsa-sha256; cv=none; b=dTrRoW2bS65rEXsxZqX86Syj0VzElNBJrbaZu1aMw2frG/UBmtyGsV1/90cJ1G2UC8Jbov AXKSwXuWMKmvAAlTCda0ndpiLVjbCflKKobLZMhM/9GbpwkWv1gIaRVNaxayPlKKaHvviu VDJlLBvBykf12m6Si1BiUHgEkh0S0KE= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9DEC51C000A X-Rspam-User: Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Stat-Signature: m43sk7eqmp3os4zxngf9w9999xxr5of6 X-HE-Tag: 1666258402-325797 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/20/2022 4:24 PM, Huang, Ying wrote: > Baolin Wang writes: > >> When creating a virtual machine, we will use memfd_create() to get >> a file descriptor which can be used to create share memory mappings >> using the mmap function, meanwhile the mmap() will set the MAP_POPULATE >> flag to allocate physical pages for the virtual machine. >> >> When allocating physical pages for the guest, the host can fallback to >> allocate some CMA pages for the guest when over half of the zone's free >> memory is in the CMA area. >> >> In guest os, when the application wants to do some data transaction with >> DMA, our QEMU will call VFIO_IOMMU_MAP_DMA ioctl to do longterm-pin and >> create IOMMU mappings for the DMA pages. However, when calling >> VFIO_IOMMU_MAP_DMA ioctl to pin the physical pages, we found it will be >> failed to longterm-pin sometimes. >> >> After some invetigation, we found the pages used to do DMA mapping can >> contain some CMA pages, and these CMA pages will cause a possible >> failure of the longterm-pin, due to failed to migrate the CMA pages. >> The reason of migration failure may be temporary reference count or >> memory allocation failure. So that will cause the VFIO_IOMMU_MAP_DMA >> ioctl returns error, which makes the application failed to start. >> >> I observed one migration failure case (which is not easy to reproduce) is >> that, the 'thp_migration_fail' count is 1 and the 'thp_split_page_failed' >> count is also 1. >> >> That means when migrating a THP which is in CMA area, but can not allocate >> a new THP due to memory fragmentation, so it will split the THP. However >> THP split is also failed, probably the reason is temporary reference count >> of this THP. And the temporary reference count can be caused by dropping >> page caches (I observed the drop caches operation in the system), but we >> can not drop the shmem page caches due to they are already dirty at that time. >> >> Especially for THP split failure, which is caused by temporary reference >> count, we can try again to mitigate the failure of migration in this case >> according to previous discussion [1]. > > Does the patch solved your problem? The problem is not easy to reproduce and I will test this patch on our products. However I think this is a likely case to fail the migration, which need to be addressed to mitigate the failure. >> [1] https://lore.kernel.org/all/470dc638-a300-f261-94b4-e27250e42f96@redhat.com/ >> Signed-off-by: Baolin Wang >> --- >> mm/huge_memory.c | 4 ++-- >> mm/migrate.c | 18 +++++++++++++++--- >> 2 files changed, 17 insertions(+), 5 deletions(-) >> >> diff --git a/mm/huge_memory.c b/mm/huge_memory.c >> index ad17c8d..a79f03b 100644 >> --- a/mm/huge_memory.c >> +++ b/mm/huge_memory.c >> @@ -2666,7 +2666,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) >> * split PMDs >> */ >> if (!can_split_folio(folio, &extra_pins)) { >> - ret = -EBUSY; >> + ret = -EAGAIN; >> goto out_unlock; >> } >> >> @@ -2716,7 +2716,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) >> xas_unlock(&xas); >> local_irq_enable(); >> remap_page(folio, folio_nr_pages(folio)); >> - ret = -EBUSY; >> + ret = -EAGAIN; >> } >> >> out_unlock: >> diff --git a/mm/migrate.c b/mm/migrate.c >> index 8e5eb6e..55c7855 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -1506,9 +1506,21 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, >> if (is_thp) { >> nr_thp_failed++; >> /* THP NUMA faulting doesn't split THP to retry. */ >> - if (!nosplit && !try_split_thp(page, &thp_split_pages)) { >> - nr_thp_split++; >> - break; >> + if (!nosplit) { >> + rc = try_split_thp(page, &thp_split_pages); >> + if (!rc) { >> + nr_thp_split++; >> + break; >> + } else if (reason == MR_LONGTERM_PIN && >> + rc == -EAGAIN) { > > In case reason != MR_LONGTERM_PIN, you change the return value of > migrate_pages(). So you need to use another variable for return value. Good catch, will fix in next version. Thanks for your comments.