From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2222C433FE for ; Mon, 24 Oct 2022 01:42:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ADB7940008; Sun, 23 Oct 2022 21:42:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15DD4940007; Sun, 23 Oct 2022 21:42:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 04D27940008; Sun, 23 Oct 2022 21:42:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EA01F940007 for ; Sun, 23 Oct 2022 21:42:02 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BBCE280B04 for ; Mon, 24 Oct 2022 01:42:02 +0000 (UTC) X-FDA: 80054142084.19.21C1A8F Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf02.hostedemail.com (Postfix) with ESMTP id 387C58000E for ; Mon, 24 Oct 2022 01:42:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666575721; x=1698111721; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=zP+QvtF6/dvJCPBCVPE+qDCgefA5nlCatvsEozgumH4=; b=OI4PZxvVnS3JrNxxkEZEC2rikOZu5DG0MFlLk9Ex9hWqVMc3tV45QlQL nUXrgI/53Aeqm0m7qXPJgIuUEis+uZl0MkMZVSpw/HJAHysRHcgOuihyH F3wl5IGDlkilRGZz8wcmEyfEsoidS0Rc5vXOqDOjCqOlq2dXqplN7k4Sq btdYZk2M98qR3H2nkb51GzgAzqHzrD3Dk6L5hPdnEx7cu2W8eSbSZNuJI dOD9YkZHnNsDEC5ZLE4snzlBpsO0/Z1vIw8+4N+k3aQgg8eZk9AWB3k7k 4pX2yHB2OkgJ+62tfVxn+WX02khd3QeMML0dxIyeBjZ34FkoLx/KngVIc g==; X-IronPort-AV: E=McAfee;i="6500,9779,10509"; a="306050337" X-IronPort-AV: E=Sophos;i="5.95,207,1661842800"; d="scan'208";a="306050337" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Oct 2022 18:42:00 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10509"; a="773664338" X-IronPort-AV: E=Sophos;i="5.95,207,1661842800"; d="scan'208";a="773664338" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Oct 2022 18:41:57 -0700 From: "Huang, Ying" To: Baolin Wang Cc: , , , , , , , Subject: Re: [PATCH v2 2/2] mm: migrate: Try again if THP split is failed due to page refcnt References: <88831f1764c8fbd5b5fdad27cd5ae3d2ca796e44.1666335603.git.baolin.wang@linux.alibaba.com> Date: Mon, 24 Oct 2022 09:41:12 +0800 In-Reply-To: <88831f1764c8fbd5b5fdad27cd5ae3d2ca796e44.1666335603.git.baolin.wang@linux.alibaba.com> (Baolin Wang's message of "Fri, 21 Oct 2022 18:16:24 +0800") Message-ID: <875ygam213.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666575722; a=rsa-sha256; cv=none; b=uGWtzWWaQRu5Wx9O700MLoWAFMx3Zlgn/uEXz7XdusTriyAXigQWOzGEBjjgm0Gv74WIIM 8oxPfi4OJPwmmZ03Tddj13f6wmi9ueWUeLOu1LgCOy5tRpmOEJOw0cjThY4CPaDA1lY0Bu yJ/x064AEqBCnFLQh3MfbxqX+EZfYC8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=OI4PZxvV; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666575722; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EjU8pUPoT6HzKfxDX7q0YwMKPg6CgaBd+3s+Q+qdtK0=; b=e//7F42PEJEB92/riRFzJJ7l9QnFgy7iJt/MtHIOvxbYm4/zoSyq4x5HJRg7bFPZ5vxvhA Ur4AWNtVfFKXoQDJ3/an2maqctLXmohJWIShpOotUjBjs2CW0bvha0qFOwsbllJAwxVNHZ aUAjE2iLNm/Ky0QyMduWT8FPHbFJHDU= X-Stat-Signature: fhpkkns4swcfjm5raug3eid916h5i86h X-Rspamd-Queue-Id: 387C58000E Authentication-Results: imf02.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=OI4PZxvV; spf=pass (imf02.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1666575720-233490 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Baolin Wang writes: > When creating a virtual machine, we will use memfd_create() to get > a file descriptor which can be used to create share memory mappings > using the mmap function, meanwhile the mmap() will set the MAP_POPULATE > flag to allocate physical pages for the virtual machine. > > When allocating physical pages for the guest, the host can fallback to > allocate some CMA pages for the guest when over half of the zone's free > memory is in the CMA area. > > In guest os, when the application wants to do some data transaction with > DMA, our QEMU will call VFIO_IOMMU_MAP_DMA ioctl to do longterm-pin and > create IOMMU mappings for the DMA pages. However, when calling > VFIO_IOMMU_MAP_DMA ioctl to pin the physical pages, we found it will be > failed to longterm-pin sometimes. > > After some invetigation, we found the pages used to do DMA mapping can > contain some CMA pages, and these CMA pages will cause a possible > failure of the longterm-pin, due to failed to migrate the CMA pages. > The reason of migration failure may be temporary reference count or > memory allocation failure. So that will cause the VFIO_IOMMU_MAP_DMA > ioctl returns error, which makes the application failed to start. > > I observed one migration failure case (which is not easy to reproduce) is > that, the 'thp_migration_fail' count is 1 and the 'thp_split_page_failed' > count is also 1. > > That means when migrating a THP which is in CMA area, but can not allocate > a new THP due to memory fragmentation, so it will split the THP. However > THP split is also failed, probably the reason is temporary reference count > of this THP. And the temporary reference count can be caused by dropping > page caches (I observed the drop caches operation in the system), but we > can not drop the shmem page caches due to they are already dirty at that time. > > Especially for THP split failure, which is caused by temporary reference > count, we can try again to mitigate the failure of migration in this case > according to previous discussion [1]. > > [1] https://lore.kernel.org/all/470dc638-a300-f261-94b4-e27250e42f96@redhat.com/ > Signed-off-by: Baolin Wang Thanks! Reviewed-by: "Huang, Ying" Best Regards, Huang, Ying > --- > Changes from v1: > - Use another variable to save the return value of THP split. > --- > mm/huge_memory.c | 4 ++-- > mm/migrate.c | 19 ++++++++++++++++--- > 2 files changed, 18 insertions(+), 5 deletions(-) > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index ad17c8d..a79f03b 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -2666,7 +2666,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > * split PMDs > */ > if (!can_split_folio(folio, &extra_pins)) { > - ret = -EBUSY; > + ret = -EAGAIN; > goto out_unlock; > } > > @@ -2716,7 +2716,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > xas_unlock(&xas); > local_irq_enable(); > remap_page(folio, folio_nr_pages(folio)); > - ret = -EBUSY; > + ret = -EAGAIN; > } > > out_unlock: > diff --git a/mm/migrate.c b/mm/migrate.c > index 1da0dbc..6d49a3e 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1506,9 +1506,22 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > if (is_thp) { > nr_thp_failed++; > /* THP NUMA faulting doesn't split THP to retry. */ > - if (!nosplit && !try_split_thp(page, &thp_split_pages)) { > - nr_thp_split++; > - break; > + if (!nosplit) { > + int ret = try_split_thp(page, &thp_split_pages); > + > + if (!ret) { > + nr_thp_split++; > + break; > + } else if (reason == MR_LONGTERM_PIN && > + ret == -EAGAIN) { > + /* > + * Try again to split THP to mitigate > + * the failure of longterm pinning. > + */ > + thp_retry++; > + nr_retry_pages += nr_subpages; > + break; > + } > } > } else if (!no_subpage_counting) { > nr_failed++;