From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BB5BC54798 for ; Tue, 5 Mar 2024 10:22:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 310576B0071; Tue, 5 Mar 2024 05:22:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BFA86B00B6; Tue, 5 Mar 2024 05:22:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 143886B00A1; Tue, 5 Mar 2024 05:22:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 002B26B00B6 for ; Tue, 5 Mar 2024 05:22:34 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C59A21C0960 for ; Tue, 5 Mar 2024 10:22:34 +0000 (UTC) X-FDA: 81862596228.22.694EB78 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf22.hostedemail.com (Postfix) with ESMTP id 4D3B7C0012 for ; Tue, 5 Mar 2024 10:22:33 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=sJX1Tg5F; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709634153; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VGKYLFPfUGJHtT00zVJnn7onAYX1D5CI2UtbLSnte14=; b=u30pC5So7iE99tBSWjQDjQ2XbMuqdhUvRPVbgg0kKYcldwrBnDwH7+wG3dRV/DUDkrf3wV fTKYOUew/kmK/1iXfSOAv5tsMf1VNNM/mE2p4ErtZ8JNXMunE+PiFhT5fNRaNmRD/QjsC0 n5gQ2XhmjgwUjwWCXX2dwWUv4iySW5s= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=sJX1Tg5F; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf22.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709634153; a=rsa-sha256; cv=none; b=Abd7cDaB7fuTH4oOvaHLIUXvhq0nQHUzVotd2BX9RKr0ggoNtAc73YYI9XPWUcJb7BceRy pQbuaJK1LxlnEoujziz6l4Hhz3P8QdklHwQHunDSQOZ40IF+xDz3JrWt94m/PmTBRGMfG5 eAR6SQ05deg3z7eXvqEJ7T0zyqPFz2o= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 6B2206145F; Tue, 5 Mar 2024 10:22:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37E9EC433F1; Tue, 5 Mar 2024 10:22:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709634152; bh=EE9BRQu4WZxxi1OZrE2CKyiBDx8STpwb4qcA00ZbW2s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=sJX1Tg5FKs34en7E/G+kuK/7egbMx0GJIJuNOqXoVHmxPxsk7TFDIK234lueOqawk DfVy2OhjuiuyjZbXTM6acAkLylbMo160eDC1sJnVeoQgpcULEW38upnzWL1Lqr/7Rp S+7/VEHx42Ezy9GnCvmGq1Ejj2VhvO/SSG68quM57xXQ+O9/+eROVha2xpE5zyY3cw c+b8EhkxOF9+IM77zAXowcECpDsgxXJ/bZkKfOvKYzluPnpTdFWR0s9lad7fKI63HZ R+ya0BlQN3UT9rbMc+IDO37qwGt3goNxjii9a+6PVBp0+Q1oZsJEax4jeolprYJrU7 8QlQEfR74JKGw== From: Leon Romanovsky To: Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Jason Gunthorpe , Chaitanya Kulkarni Cc: Leon Romanovsky , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: [RFC 01/16] mm/hmm: let users to tag specific PFNs Date: Tue, 5 Mar 2024 12:22:02 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 4D3B7C0012 X-Stat-Signature: abcaoxqrxenqcnudzp9w6qj7pi9pjwmx X-HE-Tag: 1709634153-575851 X-HE-Meta: U2FsdGVkX18SuCOeoiwG8mIIAXJaOb8/KYEW6BVAH3C0eOaRRadqnJ4NyAVJHaW1+R1q++KYwjAMo3gCYZwMwY13ZSjYhmtOkJFphtSSzFwkujQ61kesJAarIyhKupTvlQLJsvwoj7yQr2B4ShvfXBjvcHezqCpT4GpDl8JUjqzKeewJR6Z3PBdPR2UBUN7uwSCIv97C8L25oAfdNxLW/CqcuomQ0EED66QpgihPcC1MV0mzmrb3HEZ4AiJIXAuLM5C9NegFSxndRVQS7aoI2rWJPlo/3Os397lsADOpzSJBXackMToELv7WoFQjL0tdYyJsp5TQNpSXKjXg2HPDjk4bwwRSDSStETHeYdYhLvQUm90n7AWTlPDDQ+lTmKXJGdqnRVvlkcH7JLYyuiFf6ywzGvVi36O+uuI/YQzazWybWDgT1xZO/VcqQn1uog/47SLiU0upEUrnhwhBsqIMMFsA+aWRp1KkrH5ZGQEjDCqzJv+tRiH4U+Jxe4NZ5R8E8HF52EwFQvaDAKdVNEgL+ZTyx8+Au2tvmSsbM64sOEG2DiDWF9xwCf1fL5JYqnVe0weoUW/Otx8Zq6Bm2kY1Q/ugxxsF/JF1Fzt55+z7kowR0+REm9Y6wE5qe/eWV4oWmkPrrLCKm37IkYsLHoJSCZXQ1sTVoH3vZZORJ4GVMfk4/nWMiLDMeDg3CpPzjf6SwEUDJrCWUEKNbC7y2ArTIHcet4UY7mUdV8UeAC8RNvj1AEFvRfM6kbGl6frtnnVxPl+gzVhEoORG3ZTqZxTXsTCOmaz2QK7VXG1GkbuE8yCXkG4rvMWddsARmXMjfbxNuWivhBKQSjbdOrhvvI1Qftb15e9+0iFoBN20yo0WRxbcaxb7iGBvZX1Q9V52qPD/tKU3o0Og3BkIJaCiKiWbGv97fMmIu4WVZy0h6cooQwN/3tCUZXnNfrT7GjfNcjKW5xtHy31ZVSrxPlzJ54U FfT9wRxo P6MKd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Message-ID: <20240305102202.q7de1SMFDRfVPj-GGbKTZ6qb9f1bKms-qCTBnko1ryo@z> From: Leon Romanovsky Introduce new sticky flag, which isn't overwritten by HMM range fault. Such flag allows users to tag specific PFNs with extra data in addition to already filled by HMM. Signed-off-by: Leon Romanovsky --- include/linux/hmm.h | 3 +++ mm/hmm.c | 34 +++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..b90902baa593 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,7 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_STICKY - Flag preserved on input-to-output transformation * * On input: * 0 - Return the current state of the page, do not fault it. @@ -36,6 +37,8 @@ enum hmm_pfn_flags { HMM_PFN_VALID = 1UL << (BITS_PER_LONG - 1), HMM_PFN_WRITE = 1UL << (BITS_PER_LONG - 2), HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), + /* Sticky lag, carried from Input to Output */ + HMM_PFN_STICKY = 1UL << (BITS_PER_LONG - 7), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), /* Input flags */ diff --git a/mm/hmm.c b/mm/hmm.c index 277ddcab4947..9645a72beec0 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -44,8 +44,10 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, { unsigned long i = (addr - range->start) >> PAGE_SHIFT; - for (; addr < end; addr += PAGE_SIZE, i++) - range->hmm_pfns[i] = cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= cpu_flags; + } return 0; } @@ -202,8 +204,10 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } return 0; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -236,7 +240,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) goto fault; - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -253,14 +257,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = swp_offset_pfn(entry) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | swp_offset_pfn(entry) | cpu_flags; return 0; } required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -304,11 +308,11 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_unmap(ptep); return -EFAULT; } - *hmm_pfn = HMM_PFN_ERROR; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | HMM_PFN_ERROR; return 0; } - *hmm_pfn = pte_pfn(pte) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | pte_pfn(pte) | cpu_flags; return 0; fault: @@ -453,8 +457,10 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i = 0; i < npages; ++i, ++pfn) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; i < npages; ++i, ++pfn) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } goto out_unlock; } @@ -512,8 +518,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); - for (; addr < end; addr += PAGE_SIZE, i++, pfn++) - range->hmm_pfns[i] = pfn | cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++, pfn++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= pfn | cpu_flags; + } spin_unlock(ptl); return 0; -- 2.44.0