From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E08B5C19F32 for ; Wed, 5 Mar 2025 18:35:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DBAD280014; Wed, 5 Mar 2025 13:35:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18B06280008; Wed, 5 Mar 2025 13:35:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A49D280014; Wed, 5 Mar 2025 13:35:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DCB54280008 for ; Wed, 5 Mar 2025 13:35:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 76D0A55233 for ; Wed, 5 Mar 2025 18:16:36 +0000 (UTC) X-FDA: 83188302792.25.CFC8B7D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf03.hostedemail.com (Postfix) with ESMTP id C8E8520017 for ; Wed, 5 Mar 2025 18:16:34 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dfAZgM5N; spf=pass (imf03.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741198594; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YKv+lR41FJBiDAXX7yUI5s168DUMpR7MPmdi8m1JtXw=; b=Y/Uj3P6NeL4GmHFJXBhQS9550lqlx40dDVTr+LoS7anHurp9VDXhtL2o9mw/OBu5nAzTAS rlfrEyFxkAZoyQZ8J/LitlVxS6mCFsiDN17id0HL2HwOZCNYUkF1Zal/gsnbqLheGWgyvg Y1VKkEez5r+lA+l89aSC3yWcm/OCrQ4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=dfAZgM5N; spf=pass (imf03.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741198594; a=rsa-sha256; cv=none; b=WLYNLqomUGcxSwgr4sYkfSDNSn4cPTkhK4r/78xvftPvKgKpn1V6fiqQkGfDOXzBOz8Bx5 RVvy9r+cV7IDwxMMaaQdeT65kCuOKmiiFQSG1q76uDwU/UoKsibPRGWtq4gCmLgg0fhe7a okVSVqEu97niDFeuFTjJaA6TDsRdbWw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 409F15C6BBE; Wed, 5 Mar 2025 18:14:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8DEBAC4CED1; Wed, 5 Mar 2025 18:16:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741198593; bh=jzL4THh0ZOpM9TD9jtctj6vlHRlDDc1QOi0WL+ouKWc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dfAZgM5NRq4d1FSeYirYc97Aa37ub9mtL5+bYLp6su191iwtezpNxNXtZ9QKnYlfG ZbU4WvCz8p8tYxGgqYlbUlcF7P0PADFKnlM/EF+o3gGKkkEDqFvW6xeVSkOCtyYwx3 8m2pMPtJp+qwemce3LMOXfT2qVT5VDOjvkpFLvUpflQ0olmQBk+jZngEtXZcUxs4iA oMGEqY6WB+JVziG+jlXgXL6tNNF+rh3EucCJmQC/exsYPZNpJ3IGf1YyfvdjOY3m1K 1cP0WLchBvEthG2w3kt4Kx3pa369OEKOQ0igOLvv2ef71WBsy4u/HRBVQwnfXD0QEf fDAJimD9a+P1A== From: SeongJae Park To: Cc: SeongJae Park , "Liam R. Howlett" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Vlastimil Babka , kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 12/16] mm/madvise: batch tlb flushes for process_madvise(MADV_DONTNEED[_LOCKED]) Date: Wed, 5 Mar 2025 10:16:07 -0800 Message-Id: <20250305181611.54484-13-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250305181611.54484-1-sj@kernel.org> References: <20250305181611.54484-1-sj@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: puk6id64joh6qnfqizn637dmim7eo394 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: C8E8520017 X-Rspam-User: X-HE-Tag: 1741198594-162105 X-HE-Meta: U2FsdGVkX19AnnlMEeBBv3cRnyfA8i8Y/rqTOqp3oMq83vwW6N+6CCy+VNpdLKtb2/5PA5n7Dy/hGxjyKfOCApYp+1frAQtPYJu9qSz+NjvcCh5OrO5KXOuo10aYPkoHg8yPVmUBOa0YxixBhVPopPudBX72Vf8a8q/LajelW215qXORNOABFm2VRfMLx+JF0oRNRh2Jv51+1bx6rfVEGvkkc7WxkiPDt+Ngbj61zka7VPBK5KNkfCedVRaOgiiJ9kz5zN7AGJXEsU6K2fxghT+xTZb3M0Q8Hs5uWoj9Y/k6rg7UKUNUynuhBLg81mfRZwBvhTAhBGbMEcBlPXJarLdXS4JcvpF8bEvCxu3u/NaRaJf/eEhAlmj6Y7KbakQ4k/47lKfNxHJpwKq6MVia/jp+hvxlpUkpniM7uUSKVZ9KNwctNoJacq8fz5lv5qek6UkEFJd9mj6KEMoaBoqtO4+/UgnUieA0Vq74CRwdUPwjAqQlUhyGMTGULmerJ0J+g0ul7LFYTlCaWEh2RZo9DPgZ/pjOzMlp6QjscXS5jdG/uz8u4v6Dlq6KjzjCCeqpHeypfvlsFhQU/+raFEUCmDCCXHVdGCwJiueqSeXe21deXAsJNIPl1cDSr7UAAT4bIVl9526j3BKfyW+OVnmdh3I78sbKJyJDQ9T2KrBlRKNJhXDZJ14W4xFdPY0FW9slhd4QrhsCnQ16PS5eI96E7ZaeVnvosRIevBZUrDT1/kVAX9mgS5hveu05Bgu+mAAMQksGcFMD7dBlfqDz87MMmMdRUvTNI6LV4xnLMntJzYICcZ6MCTr5eaZ6aSsKwZ+TLPHQtevN5NJbMHn2XyQUnEZccyB0KtO+6IOJmAjZ3axXhHJkhYToggh+Mnnv9/JfcIvv8BPtMoSus/31BqghISvZHYp7J4ORgr+J2aasLm2nl5t3LOx90hR5jETFT9H6oV4WEZoGunjucAuV0Rp Sy93bc1g Ul6Qmp72Mwib1PCwIAHHzYMDfwbJeMFAh3j600c05i9r0c5Qlq+byfc0XLXcoK0yS6Jc3higJQ8h0jCJXQONeD4XadK96ICnfGPQyXIZ6E0adCxv6ZjROt1KZMrR+acADTskT7e4L5HrJEogBc3oHe2bkYosrhch3gorZ+XUQ1QNqzepchv8r5ihi44S6QlgGO6zVZsFzazirFb+hr8s3GzfkFA/jaPJ3JP3IyPonzyFCs30s6pTUH/s37KZvFG4ECL098CshIKfqocFsLvNedfrvcA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: MADV_DONTNEED[_LOCKED] internal logic can be invoked with batched tlb flushes. Update vector_madvise(), which is called for process_madvise(), to use that in the way, by passing an mmu_gather object that it initializes before starting the internal works, and flushing the gathered tlb entries at once after all the internal works are done. Signed-off-by: SeongJae Park --- mm/madvise.c | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 767c5d21ee75..efa4184d6cf5 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -905,6 +905,7 @@ static bool madvise_dontneed_free_valid_vma(struct vm_area_struct *vma, struct madvise_behavior { int behavior; + struct mmu_gather *tlb; }; static long madvise_dontneed_free(struct vm_area_struct *vma, @@ -964,7 +965,8 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, } if (behavior == MADV_DONTNEED || behavior == MADV_DONTNEED_LOCKED) - return madvise_dontneed_single_vma(NULL, vma, start, end); + return madvise_dontneed_single_vma( + madv_behavior->tlb, vma, start, end); else if (behavior == MADV_FREE) return madvise_free_single_vma(NULL, vma, start, end); else @@ -1802,19 +1804,50 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior) return do_madvise(current->mm, start, len_in, behavior); } +static bool vector_madvise_batch_tlb_flush(int behavior) +{ + switch (behavior) { + case MADV_DONTNEED: + case MADV_DONTNEED_LOCKED: + return true; + default: + return false; + } +} + +static void vector_madvise_init_tlb(struct madvise_behavior *madv_behavior, + struct mm_struct *mm) +{ + if (!vector_madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_gather_mmu(madv_behavior->tlb, mm); +} + +static void vector_madvise_finish_tlb(struct madvise_behavior *madv_behavior) +{ + if (!vector_madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_finish_mmu(madv_behavior->tlb); +} + /* Perform an madvise operation over a vector of addresses and lengths. */ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, int behavior) { ssize_t ret = 0; size_t total_len; - struct madvise_behavior madv_behavior = {.behavior = behavior}; + struct mmu_gather tlb; + struct madvise_behavior madv_behavior = { + .behavior = behavior, + .tlb = &tlb, + }; total_len = iov_iter_count(iter); ret = madvise_lock(mm, behavior); if (ret) return ret; + vector_madvise_init_tlb(&madv_behavior, mm); while (iov_iter_count(iter)) { unsigned long start = (unsigned long)iter_iov_addr(iter); @@ -1843,14 +1876,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, } /* Drop and reacquire lock to unwind race. */ + vector_madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); madvise_lock(mm, behavior); + vector_madvise_init_tlb(&madv_behavior, mm); continue; } if (ret < 0) break; iov_iter_advance(iter, iter_iov_len(iter)); } + vector_madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); ret = (total_len - iov_iter_count(iter)) ? : ret; -- 2.39.5