From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E96FC61DA4 for ; Mon, 13 Mar 2023 12:44:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FD546B0071; Mon, 13 Mar 2023 08:44:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B8446B0075; Mon, 13 Mar 2023 08:44:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8752A6B007B; Mon, 13 Mar 2023 08:44:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 752516B0071 for ; Mon, 13 Mar 2023 08:44:58 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 43410C08C7 for ; Mon, 13 Mar 2023 12:44:58 +0000 (UTC) X-FDA: 80563844676.01.1C28D43 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf30.hostedemail.com (Postfix) with ESMTP id 45CFB80007 for ; Mon, 13 Mar 2023 12:44:55 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ipGk3030; spf=pass (imf30.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678711495; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VvuEsOqB0rMIX/ofACTDOngn/bkpGjpWRTNnWf8PO/s=; b=ndtLJXDLUMTedVwMESXyAm8vhYQg7PxXLQjABgtsFzGyQgZeyWiPs9d4V3bdPw1DN00hlU JZ+LzyH/GLT/Gm7dHFRHBQnEqw6+ePuBTWnxRhASKd81ZoykHzrkhTy4jNuvkyPvleyZQE 0VeS3Ti20MseZeXyldcIvf5tV9bmhy0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ipGk3030; spf=pass (imf30.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678711495; a=rsa-sha256; cv=none; b=c5Tr/Lbcf9GvierePhn3vE2ahJi/yFAmXG2DMVxVnFgFpObZgV69dqR9cqCU+Sz42oFMsQ rqIBtqwNh4QGK9aVUvIyiBYMW/pbciMWFUxw/QTUX75DKuMxZEZcicQLpsaCHVOhC/Tz/Q gz4ck4HhJj4NgimMjcNaZEfClxPEezQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678711495; x=1710247495; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=L0nog3Zuea8dweSuw9vCb41izmv85iPbZrSBDZeUARA=; b=ipGk30300fyBykJnNGvsH5GBUb3Ee3nTSCNnmzo9FXkn+qzypRlhjFSg KID+VD2a0/wZYlgNh3TODds4c9NZVPSkUrcbJMiNPMbw8TsnfGUF7j+Xh +ZW2sSWPnxyPLBNm0uY2GiKhszmj+3LJ9SlpKlfhkmIIy/lP2svFanZ+0 peuSgnXKm28oDphfVY2FM5aiRmj6rZ9EmYLLjsFnuNv361Fcmh91iDroe Xxklrdf7/0KwK8b7Y24+EhXWVHKjrrIkM9HvGNXQYSn39ziaMbY/Fq1e1 kwvqls5pvwKiK/exgkbPVBBdmTapLHBmVV2eWdGCNa+OfPp6xunKuM0e/ g==; X-IronPort-AV: E=McAfee;i="6500,9779,10647"; a="364797293" X-IronPort-AV: E=Sophos;i="5.98,256,1673942400"; d="scan'208";a="364797293" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Mar 2023 05:44:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10647"; a="655962641" X-IronPort-AV: E=Sophos;i="5.98,256,1673942400"; d="scan'208";a="655962641" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga006.jf.intel.com with ESMTP; 13 Mar 2023 05:44:51 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, mike.kravetz@oracle.com, sidhartha.kumar@oracle.com, naoya.horiguchi@nec.com, jane.chu@oracle.com, david@redhat.com Cc: fengwei.yin@intel.com Subject: [PATCH v4 4/5] rmap:addd folio_remove_rmap_range() Date: Mon, 13 Mar 2023 20:45:25 +0800 Message-Id: <20230313124526.1207490-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230313124526.1207490-1-fengwei.yin@intel.com> References: <20230313124526.1207490-1-fengwei.yin@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 45CFB80007 X-Stat-Signature: m6yc87cmj3b4jfnpi5rrdjdzoq6kd4ar X-HE-Tag: 1678711495-328796 X-HE-Meta: U2FsdGVkX1+YERHk9Ck7ekoamggAwd3Zq6NTFGHAhaevLjCTKEuSagClNQFkfDw/S+xeX3LN4GPIyk8zn25CL84AFUJYUS2/sjO9JuMIpWIngi+4oJzFz7CNdsLnzJWuekOzwK4ImLQ7wNyEdLBOdJGI6iL1WOR2kFnn3yTwLaynXX4Pt4SFH2o1xXx2btPd2rvEwrWJqfhbKMN2GuyKKLyO4rZAIjcYaoq9uCiGkqXaC2BCJgz0lmZ5UAxNaUMoJy5m8Dl+RiYRbk+ozXeDlbubI5UIeGeFoUvrpCDTEICXksbkxJkTS3/syEwqerYrQIWa4vxLskrYc0K5oKizu43YwaCSPNf0MXrbCc9I28DTv86+IiRGD4OXFN583Uj5c4H8CJ34ukUZvBZLZt0Pmsb6bH3UYHxvLMuIWWp+yelfO1HyEmD+GDpwPlrUL4T4ZlBIW2OyE0hL5PrmcanKERcyv6MDj+fZZgY7kpScAxszMP1NUmRJqFh/1ztl66aXRjqJOiErlBuLShpLfwsOAsjzcZp8NyUh1n1QGPCAGeNoVej0AZEfmuo/7PnfdvQKD6YB4gl7qF7v/BaRroF1/SqLSO92dA4TivthJyRZqXcCzqSpX9fQu3ddg/+91mywIzxi/PZCb9JIftzpPHZ64zkrPwvnEuborf770RlFOsw/JHgMwNz6CQA33gZYOZsT6Ggc6E/SbGWFzzkaAt0+pe8CJDkySZVC8KUOvuyagMJx0vgxaa8aQkMcGViCqkaxlrITvbDMoAxyvMgrJsyoRnIIJQUKSkhWoFGH0gaGWwu+Q+1oiW4v0yktDZj3smf38kmygfdrNROCq46j8DUT2ri2M4ySKuCgbZQ7GyBFgXTx8BakWkcJlX7yjO8HgwRjg/3vzjiz63o8DMYE/+s1pS+DeubCgOxN4FIVjOy2jf14nJyiDF3blCvAuqrRbwr/yE4vMig/WF0NA7H5THl lH1edye0 CdlDC/wV570Gbs3mqdYOgN9Gepc6p0K0oxBU2URnjvT/Fkjjp1wuRCQAVp6k/3aSTT1e9tvcorHiRViN/C3HYpL+lilAVh0ezBYZDA5IJfZtt5LAWqnIhBgVR8VL9pBNILR2+VV561M1t1QUMF0AaQ29brQtoJ/kFJRXqP9AXyxVUIM7Ut9aRM7C9+EPe/8Ht1wxYexTHmuVMOILVn9yazDUhL86OiGx5iRRqFMxeWU33aNx0kC3WtyDH8/TQmCHPGzWC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_remove_rmap_range() allows to take down the pte mapping to a specific range of folio. Comparing to page_remove_rmap(), it batched updates __lruvec_stat for large folio. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 4 +++ mm/rmap.c | 58 +++++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b87d01660412..d2569b42e21a 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -200,6 +200,10 @@ void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_range(struct folio *, struct page *, + unsigned int nr_pages, struct vm_area_struct *, + bool compound); + void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 72fc8c559cd9..bd5331dc9d44 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1355,23 +1355,25 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, } /** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from + * folio_remove_rmap_range - take down pte mapping from a range of pages + * @folio: folio to remove mapping from + * @page: The first page to take down pte mapping + * @nr_pages: The number of pages which will be take down pte mapping * @vma: the vm area from which the mapping is removed * @compound: uncharge the page as compound or small page * * The caller needs to hold the pte lock. */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) +void folio_remove_rmap_range(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + bool compound) { - struct folio *folio = page_folio(page); atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int nr = 0, nr_pmdmapped = 0, last; enum node_stat_item idx; - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + VM_BUG_ON_FOLIO(compound && (nr_pages != folio_nr_pages(folio)), folio); + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); /* Hugetlb pages are not counted in NR_*MAPPED */ if (unlikely(folio_test_hugetlb(folio))) { @@ -1382,12 +1384,16 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + + if (last) + nr++; + } while (page++, --nr_pages > 0); } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1441,6 +1447,30 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma, compound); } +/** + * page_remove_rmap - take down pte mapping from a page + * @page: page to remove mapping from + * @vma: the vm area from which the mapping is removed + * @compound: uncharge the page as compound or small page + * + * The caller needs to hold the pte lock. + */ +void page_remove_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) +{ + struct folio *folio = page_folio(page); + unsigned int nr_pages; + + VM_BUG_ON_FOLIO(compound && (page != &folio->page), folio); + + if (likely(!compound)) + nr_pages = 1; + else + nr_pages = folio_nr_pages(folio); + + folio_remove_rmap_range(folio, page, nr_pages, vma, compound); +} + static bool try_to_unmap_one_hugetlb(struct folio *folio, struct vm_area_struct *vma, struct mmu_notifier_range range, struct page_vma_mapped_walk pvmw, unsigned long address, -- 2.30.2