From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4615FC433FE for ; Thu, 20 Oct 2022 08:25:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACFC56B0071; Thu, 20 Oct 2022 04:25:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7F746B0073; Thu, 20 Oct 2022 04:25:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 96EBD8E0001; Thu, 20 Oct 2022 04:25:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 86F5B6B0071 for ; Thu, 20 Oct 2022 04:25:04 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 572E1160EC9 for ; Thu, 20 Oct 2022 08:25:04 +0000 (UTC) X-FDA: 80040642528.08.BD31F29 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf11.hostedemail.com (Postfix) with ESMTP id CE8CF40015 for ; Thu, 20 Oct 2022 08:25:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666254303; x=1697790303; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=N/IkiQgsQXyxu6iYoSAwhE2auxZ8xnMdjB2X/PD31M0=; b=F7xSDkciq4rFgCS4yTtruRAqORyIMlcgde2lka4rfh4Y816VR0NjR9If 7K3KvJd7FG55S6tPLlWSoJuJR0++IhivMtWL7E+lBN9PUOgo/pTWSjCw0 MabfbtG/vA6PZYoSPrt0EdDI7A5IoFy7NRlRhtdN8zIWdCZAGDiJtrqJb u2uh0HCOALtxR9V6zD344Yv4vlMtLhR9LDx1uWYGXvT0dC58Sac32r0/6 aG2hij06pbVB82hXtDcNekLe6ikHEhoajeSOhBtApsIwEU1fArkKlE50L M8+V3G3NNPsldmnNDb8+kKrV7OlZ6Sc6n43/9BabJngxaqwMpr/xFGh8m g==; X-IronPort-AV: E=McAfee;i="6500,9779,10505"; a="368694412" X-IronPort-AV: E=Sophos;i="5.95,198,1661842800"; d="scan'208";a="368694412" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2022 01:25:02 -0700 X-IronPort-AV: E=McAfee;i="6500,9779,10505"; a="958787937" X-IronPort-AV: E=Sophos;i="5.95,198,1661842800"; d="scan'208";a="958787937" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga005-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2022 01:25:00 -0700 From: "Huang, Ying" To: Baolin Wang Cc: , , , , , , Subject: Re: [PATCH 2/2] mm: migrate: Try again if THP split is failed due to page refcnt References: Date: Thu, 20 Oct 2022 16:24:20 +0800 In-Reply-To: (Baolin Wang's message of "Thu, 20 Oct 2022 15:49:01 +0800") Message-ID: <87mt9qnbrf.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666254304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jWsYbFkon1uUPtk5RgG9Rv092qIQMrbbnL9yNlnWIR0=; b=8LeEqMMDsQQvgK3wRz2yhCZc85fFtk2roquvozePT0RNS/A83KKW9jA5x2xFc4rZRk2fjM OMLuPjKOuIts7nd8WYjN0LiGPsiYi/8mCwNgrBr5Fw0Xq5JMYASDOkUvmcVz0TvhhODuAo /7JcUgWyLSo2LAydKeNBPYpGbW5FjZo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=F7xSDkci; spf=pass (imf11.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666254304; a=rsa-sha256; cv=none; b=Y5hv2R/qjQqrAHdqnDIuVV5AOmtKHcnkmKyrvjrz9l5e53ZLWr5SWKo0H744ieRutICIzP dNHnrACaVLucKuJvH/Ty9+flckMCv0xC9Yat8ye6FsGUttWiSLLJmttJvKUd1Ei32r0JHb AGTmRud7QsrCVXGSe138KSPh4FwzZKk= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: CE8CF40015 X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=F7xSDkci; spf=pass (imf11.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: 5u4xrjhdbtrxet51toedzhhs3asmc1i1 X-HE-Tag: 1666254303-412417 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Baolin Wang writes: > When creating a virtual machine, we will use memfd_create() to get > a file descriptor which can be used to create share memory mappings > using the mmap function, meanwhile the mmap() will set the MAP_POPULATE > flag to allocate physical pages for the virtual machine. > > When allocating physical pages for the guest, the host can fallback to > allocate some CMA pages for the guest when over half of the zone's free > memory is in the CMA area. > > In guest os, when the application wants to do some data transaction with > DMA, our QEMU will call VFIO_IOMMU_MAP_DMA ioctl to do longterm-pin and > create IOMMU mappings for the DMA pages. However, when calling > VFIO_IOMMU_MAP_DMA ioctl to pin the physical pages, we found it will be > failed to longterm-pin sometimes. > > After some invetigation, we found the pages used to do DMA mapping can > contain some CMA pages, and these CMA pages will cause a possible > failure of the longterm-pin, due to failed to migrate the CMA pages. > The reason of migration failure may be temporary reference count or > memory allocation failure. So that will cause the VFIO_IOMMU_MAP_DMA > ioctl returns error, which makes the application failed to start. > > I observed one migration failure case (which is not easy to reproduce) is > that, the 'thp_migration_fail' count is 1 and the 'thp_split_page_failed' > count is also 1. > > That means when migrating a THP which is in CMA area, but can not allocate > a new THP due to memory fragmentation, so it will split the THP. However > THP split is also failed, probably the reason is temporary reference count > of this THP. And the temporary reference count can be caused by dropping > page caches (I observed the drop caches operation in the system), but we > can not drop the shmem page caches due to they are already dirty at that time. > > Especially for THP split failure, which is caused by temporary reference > count, we can try again to mitigate the failure of migration in this case > according to previous discussion [1]. Does the patch solved your problem? > [1] https://lore.kernel.org/all/470dc638-a300-f261-94b4-e27250e42f96@redhat.com/ > Signed-off-by: Baolin Wang > --- > mm/huge_memory.c | 4 ++-- > mm/migrate.c | 18 +++++++++++++++--- > 2 files changed, 17 insertions(+), 5 deletions(-) > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index ad17c8d..a79f03b 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -2666,7 +2666,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > * split PMDs > */ > if (!can_split_folio(folio, &extra_pins)) { > - ret = -EBUSY; > + ret = -EAGAIN; > goto out_unlock; > } > > @@ -2716,7 +2716,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) > xas_unlock(&xas); > local_irq_enable(); > remap_page(folio, folio_nr_pages(folio)); > - ret = -EBUSY; > + ret = -EAGAIN; > } > > out_unlock: > diff --git a/mm/migrate.c b/mm/migrate.c > index 8e5eb6e..55c7855 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1506,9 +1506,21 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, > if (is_thp) { > nr_thp_failed++; > /* THP NUMA faulting doesn't split THP to retry. */ > - if (!nosplit && !try_split_thp(page, &thp_split_pages)) { > - nr_thp_split++; > - break; > + if (!nosplit) { > + rc = try_split_thp(page, &thp_split_pages); > + if (!rc) { > + nr_thp_split++; > + break; > + } else if (reason == MR_LONGTERM_PIN && > + rc == -EAGAIN) { In case reason != MR_LONGTERM_PIN, you change the return value of migrate_pages(). So you need to use another variable for return value. > + /* > + * Try again to split THP to mitigate > + * the failure of longterm pinning. > + */ > + thp_retry++; > + nr_retry_pages += nr_subpages; > + break; > + } > } > } else if (!no_subpage_counting) { > nr_failed++; Best Regards, Huang, Ying