From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7758DC433FE for ; Thu, 20 Oct 2022 19:21:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDB038E0002; Thu, 20 Oct 2022 15:21:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C64398E0001; Thu, 20 Oct 2022 15:21:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B047C8E0002; Thu, 20 Oct 2022 15:21:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9E1878E0001 for ; Thu, 20 Oct 2022 15:21:40 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 74B0A1602A5 for ; Thu, 20 Oct 2022 19:21:40 +0000 (UTC) X-FDA: 80042297160.22.E39966F Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf17.hostedemail.com (Postfix) with ESMTP id EC57540039 for ; Thu, 20 Oct 2022 19:21:39 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id g28so453275pfk.8 for ; Thu, 20 Oct 2022 12:21:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MGWd55MQb3vBkEb2MC5NHIx5pr8IGRG3XR9906nE3CU=; b=TTJYvP8Sy39CMvqQXExyqYeiJNpgGzBuDkgLgRstXGV+u9l2gj387BjJvdJLf+fWSv 4ZGD3e9nC0Al4r2RCDU6B0nUhjOjRk18sxI7pk23EAFhfUH19TfvJOT+fA0e0nOMYD+F IzjquSgkJXdSc+tOjXwF78644ZG8/ygfRc84cV1khJ9mOanhrv1TsQPtlWAFMCRJ5v6j CLbO/llDW6moxiJXLyuFfp/zATSiHfozQ0MJ5jsQFBAntZaPTYh6hhtZC/LCyeauaORh Tat2v1A4LbYx1XhhKO0McVOcxJ0yGzgcaLQKs2aEPFMMy8yTBDPDwThfCF1wmvGNQMgs FGmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MGWd55MQb3vBkEb2MC5NHIx5pr8IGRG3XR9906nE3CU=; b=JhTegzBEoHw2Smmc2IY5P1pjBf6GsVhjY9m+9y2YTNPdLAyT10HxOL0UDxqlOYJ8LN fLfa30aEF4Ju/GhG3V+CDqY/oXFujxR5I2Kir3eVdTAPp5cdYXrAGM88A9soiaGwpFQA NyHW6EkIMFfzGeWQWY0REXff1u/kA9XzrLq5wzx8XurzlbdusF1IJjslCg8NjCU3/72n hVqVODsitCqNmLulcVvr6ZjNbgA6rdEhiCkC8oRpBBZ+HRcGybO5qxYpRPRJlgIjXhlu GvTDdF3gPFEbvRaPT8yTY6/Cz+QErCfNlvzTvBAjPu754j4mu5ObuT8Jf7AMQGtDpR04 boyQ== X-Gm-Message-State: ACrzQf3jk2GBBo+IS9Q6DyvYyr28wxyhnYq8Sa/veE8Ig6GNVo40bYd5 CWwDBf+hNhL8OuaWe5knLUMqHmRqJUlHFsvlrWU= X-Google-Smtp-Source: AMsMyM5wM+eGGSaToRccX+wHcPfdv3XLev25laMftlhjjePiHav3A6iPAptZaMvnGH8fQB5fGfLwBG+JS5BnQMQsBeg= X-Received: by 2002:aa7:9dcb:0:b0:565:89a8:c708 with SMTP id g11-20020aa79dcb000000b0056589a8c708mr15721779pfq.4.1666293698956; Thu, 20 Oct 2022 12:21:38 -0700 (PDT) MIME-Version: 1.0 References: <87mt9qnbrf.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: From: Yang Shi Date: Thu, 20 Oct 2022 12:21:26 -0700 Message-ID: Subject: Re: [PATCH 2/2] mm: migrate: Try again if THP split is failed due to page refcnt To: Baolin Wang Cc: "Huang, Ying" , akpm@linux-foundation.org, david@redhat.com, ziy@nvidia.com, jingshan@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666293700; a=rsa-sha256; cv=none; b=ufgYsuPxpPRQobC4KWTrTtB3rUEKQ6jg1w9SZXa2x/SQfDBWyxGnh0/cBGihIGtXeaGrjl ZBywQ5qmdujUca6IYIDy3sgEn2tZBg4luIHP8OHXGzSYb0ZRD0AjKzVTb57q8fcXOC9NDu Eg1cOycmBwrBpK1Zv47T8qqBwi8NQdM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TTJYvP8S; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=shy828301@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666293700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MGWd55MQb3vBkEb2MC5NHIx5pr8IGRG3XR9906nE3CU=; b=U9jRGGM+UBR03iZRr4VMXHIbQj0ce/Kkb1wEiIF+Q8jymL/NrSXHTvtDv4dm9KwadxAZ9k jgW+u1R14fPbpOSdD9NA0Iu+mm5K5v2rkVfP2ldTJxLlP7M7FVAtq9XtkwdhIR6PkZ/RtS YwEFNJypv/0QkJOmQlGeKK6s+MBHdeM= X-Rspamd-Queue-Id: EC57540039 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TTJYvP8S; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of shy828301@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Stat-Signature: e9mxbq8cskqcgns7p8ai17zmrdrgws8i X-HE-Tag: 1666293699-219160 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 20, 2022 at 2:33 AM Baolin Wang wrote: > > > > On 10/20/2022 4:24 PM, Huang, Ying wrote: > > Baolin Wang writes: > > > >> When creating a virtual machine, we will use memfd_create() to get > >> a file descriptor which can be used to create share memory mappings > >> using the mmap function, meanwhile the mmap() will set the MAP_POPULATE > >> flag to allocate physical pages for the virtual machine. > >> > >> When allocating physical pages for the guest, the host can fallback to > >> allocate some CMA pages for the guest when over half of the zone's free > >> memory is in the CMA area. > >> > >> In guest os, when the application wants to do some data transaction with > >> DMA, our QEMU will call VFIO_IOMMU_MAP_DMA ioctl to do longterm-pin and > >> create IOMMU mappings for the DMA pages. However, when calling > >> VFIO_IOMMU_MAP_DMA ioctl to pin the physical pages, we found it will be > >> failed to longterm-pin sometimes. > >> > >> After some invetigation, we found the pages used to do DMA mapping can > >> contain some CMA pages, and these CMA pages will cause a possible > >> failure of the longterm-pin, due to failed to migrate the CMA pages. > >> The reason of migration failure may be temporary reference count or > >> memory allocation failure. So that will cause the VFIO_IOMMU_MAP_DMA > >> ioctl returns error, which makes the application failed to start. > >> > >> I observed one migration failure case (which is not easy to reproduce) is > >> that, the 'thp_migration_fail' count is 1 and the 'thp_split_page_failed' > >> count is also 1. > >> > >> That means when migrating a THP which is in CMA area, but can not allocate > >> a new THP due to memory fragmentation, so it will split the THP. However > >> THP split is also failed, probably the reason is temporary reference count > >> of this THP. And the temporary reference count can be caused by dropping > >> page caches (I observed the drop caches operation in the system), but we > >> can not drop the shmem page caches due to they are already dirty at that time. > >> > >> Especially for THP split failure, which is caused by temporary reference > >> count, we can try again to mitigate the failure of migration in this case > >> according to previous discussion [1]. > > > > Does the patch solved your problem? > > The problem is not easy to reproduce and I will test this patch on our > products. However I think this is a likely case to fail the migration, > which need to be addressed to mitigate the failure. You may try to trace all migrations across your fleet (or just pick some sample machines, this should make data analysis easier) and filter the migration by reasons, for example, MR_LONGTERM_PIN, then compare the migration success rate before and after the patch. It should be a good justification. But it may need some work on data aggregation, process and analysis, not sure how feasible it is. > > >> [1] https://lore.kernel.org/all/470dc638-a300-f261-94b4-e27250e42f96@redhat.com/ > >> Signed-off-by: Baolin Wang > >> --- > >> mm/huge_memory.c | 4 ++-- > >> mm/migrate.c | 18 +++++++++++++++--- > >> 2 files changed, 17 insertions(+), 5 deletions(-) > >> > >> diff --git a/mm/huge_memory.c b/mm/huge_memory.c > >> index ad17c8d..a79f03b 100644 > >> --- a/mm/huge_memory.c > >> +++ b/mm/huge_memory.c > >> @@ -2666,7 +2666,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > >> * split PMDs > >> */ > >> if (!can_split_folio(folio, &extra_pins)) { > >> - ret = -EBUSY; > >> + ret = -EAGAIN; > >> goto out_unlock; > >> } > >> > >> @@ -2716,7 +2716,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > >> xas_unlock(&xas); > >> local_irq_enable(); > >> remap_page(folio, folio_nr_pages(folio)); > >> - ret = -EBUSY; > >> + ret = -EAGAIN; > >> } > >> > >> out_unlock: > >> diff --git a/mm/migrate.c b/mm/migrate.c > >> index 8e5eb6e..55c7855 100644 > >> --- a/mm/migrate.c > >> +++ b/mm/migrate.c > >> @@ -1506,9 +1506,21 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > >> if (is_thp) { > >> nr_thp_failed++; > >> /* THP NUMA faulting doesn't split THP to retry. */ > >> - if (!nosplit && !try_split_thp(page, &thp_split_pages)) { > >> - nr_thp_split++; > >> - break; > >> + if (!nosplit) { > >> + rc = try_split_thp(page, &thp_split_pages); > >> + if (!rc) { > >> + nr_thp_split++; > >> + break; > >> + } else if (reason == MR_LONGTERM_PIN && > >> + rc == -EAGAIN) { > > > > In case reason != MR_LONGTERM_PIN, you change the return value of > > migrate_pages(). So you need to use another variable for return value. > > Good catch, will fix in next version. Thanks for your comments.