From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A5D2C282DE for ; Wed, 5 Mar 2025 18:34:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F236280011; Wed, 5 Mar 2025 13:34:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 77BB4280008; Wed, 5 Mar 2025 13:34:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61FA3280011; Wed, 5 Mar 2025 13:34:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 41802280008 for ; Wed, 5 Mar 2025 13:34:35 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3387380964 for ; Wed, 5 Mar 2025 18:16:39 +0000 (UTC) X-FDA: 83188302918.06.36780F7 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id 305F9A002B for ; Wed, 5 Mar 2025 18:16:37 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EHqFAg4A; spf=pass (imf25.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741198597; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yclikyou6ZqEyjnb1AJ2zxIYKQUJqLR2+o0ldRAlVQg=; b=M1+yW9RGiLpe79hAv2kvJgF0COl+BDD2Kg8yMla1nBsDfTebQ7N3m+ARJGeqHe+0SfHb0S YS39fI1H25SbGsZQW04jBPpv6N4JfR+mGD80z+0AQ2QpxKjtInUAFODS+/GV5mXBPi044z S/OQrC3F3b53QQCx9Jr7eLP6ktxyZyc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741198597; a=rsa-sha256; cv=none; b=i/A2I15A/YYdUxiyNTtDaoEaDMyu1ZzNnc0zZnu89wZi6qqAAdAYTNm/d+ZexUWrNTEKeJ H0dwbaKkfDZGRFN6kcLK5ojKNkTqpqgNmmqSgLTXJc9bLgjfEefdhfB37MOFYChNmGBv2x b8eiBNCBac1WIADCnNBCLZE6zJZw8io= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EHqFAg4A; spf=pass (imf25.hostedemail.com: domain of sj@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=sj@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 91C055C6BBE; Wed, 5 Mar 2025 18:14:19 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DDADAC4CED1; Wed, 5 Mar 2025 18:16:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1741198596; bh=1LPOXs5PBkvrUa9S0qnB8qMN3Uc/uSVSQOTjrIyqAEE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EHqFAg4AOa7S72mrOT/H3kNLtZLpur42QTtIRfYbMO+8bjFaIMS1gEdG7yG47ywRy 4rqL7AOqSLUArKtqalvcDEx3ixdn8dlfbcGDTM1pqiALKHLYqhtQxCLfVmbiUAhrTX 2RcH9u9I5jIJsy6XNsl18EuyAwWPEEvd/joVHWmWx0elQQUeWx3GfIyDmIE/G9M6Re XFyFhxghJiYElbYzzl1lLS52XUGPvwNvv+54MiCrwgqJI55xsVpJFhza5Ax/bx2POQ n1qO5tDPVQXxLPtGBrry7hNgEENXcWxlm8HRSwn0cqoTuKx/tfYYqSPbb7fu1+nRVF LQwENlJY4X9sw== From: SeongJae Park To: Cc: SeongJae Park , "Liam R. Howlett" , Andrew Morton , David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Vlastimil Babka , kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH 14/16] mm/madvise: batch tlb flushes for madvise(MADV_{DONTNEED[_LOCKED],FREE} Date: Wed, 5 Mar 2025 10:16:09 -0800 Message-Id: <20250305181611.54484-15-sj@kernel.org> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250305181611.54484-1-sj@kernel.org> References: <20250305181611.54484-1-sj@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: kyqz435unjtp9ugrhsjodgst8jk47jtk X-Rspamd-Queue-Id: 305F9A002B X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1741198597-721530 X-HE-Meta: U2FsdGVkX18OrtDktJvC/FusEykoIicfomqHUumRlsSJ7FhqbMtWElExOkDiF2he79xIi/obsrOzXOLeWBdeYIZ8JnKEiU1y3TV2taLIMVBKClL5lBrsQHFtkPlkR37lvIFFG/mUUb07ZbNUvyCfCTcdZy3g+YtJNO9sVZhRkOTvLaMwRojhV554jxsY8SJelovmP2CEpSxPELaLvhNG86x4/wjWpn4BVMJBiaWJKE4synRFGs3+/LKcT+Gpzc8xM8W7dJh/W57oOK2c256vzobn8Yu2GDd/gg0PeTliM6y4MF4rVvgf06X7XaTOIFRgHniJks3QNprUGLFiu4ZgRXzJoh4XU6SZCVCV1JiayQWtAFfPdHoh56HeVzvXGBsSQRY/UKTrnrRh0am+J4yNPHMa2CQ7qqWI9rw7aVQqQTkt4/s6QS1pi2psHNn9mhe+dmp5PZYSdPL8/b/mRE+8up3XgPUxERs+YvABkdiO7qzSDNiDU64FgbpTVvsLXmT/qU+z4HGS7Z+jaMN9InAgb7ku3DinBejxH1+GpKP7NvoaZvddAhH9kLxIulT/oYB0+FN1fUJovuI1TcAf547ybSiIdgz9jLt2a6aLOamsELeNSoxH7B8ExSLKvvlVXw/McDTZg2OzZs3Ie4AzT3V42kIV0bs+fNfR/z8QFfJJdY3ySmm+Qcim3Pm/8Augc9nhDewUFfF/v5tOhL43ycIeDBormud2V60sUOVKhyHFmAhy8xvpnsAL7z8r9QAYO0EH/4iojUX8S+5/6jqgJZikJ2WRl7WpkatpTJRjths6CULic1Go20b1bEBwC36AcnpuK7HoZRNEj6mOMiiOzxbNyF4yXdnI6KlYz53N1MpG0HV0YbVIk6Ni/6GskRWPCGCLrCU88hVzdrZGP9u3Epu01XjxKgTGCXI2IBDM1Y1/zTXb2PnsEA+I95QRzCa0xc8i6UBm9U2Fx+TlglA1FT5 2WlSusz9 SFMjR6G1ZGuYzTgVofaULm1dRr0uq+nllCZw0v6ZIsHAul9JTNZ9zCyOUirfr5q76c2WrI5oWMAnFDRcMiQzYFWbdkVapiCeh/EmCppLqX4x7uRp87/F9RQjIkgESvglqss+IuMnyV2qpvIkI+L225rdVE3BRpwYULLtvXDM36SDlSk2Ehinan6nNcHNvLZC6d/Uiyd14NgDrzRpALRtL+JxDQnwkkHovIgbXm7n695MPIFa+hOCyHKQqepPWu2Veqs2g8XzzpJ7V6EhVUbZhz0/XUw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: MADV_DONTNEED[_LOCKED] and MADV_FREE internal logics can be invoked with batched tlb flushes. madvise() is called for single address range, but if there are multiple vmas for the range, tlb flush will happen multiple times. Update do_madvise(), which is called for madvise(), to use that in the efficient way, by passing an mmu_gather object that it initializes before starting the work, and flushing the tlb entries at once after all the internal works are done. Signed-off-by: SeongJae Park --- mm/madvise.c | 68 ++++++++++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index f1beadb6176a..0d292b8e1a0e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1635,6 +1635,32 @@ static void madvise_unlock(struct mm_struct *mm, int behavior) mmap_read_unlock(mm); } +static bool madvise_batch_tlb_flush(int behavior) +{ + switch (behavior) { + case MADV_DONTNEED: + case MADV_DONTNEED_LOCKED: + return true; + default: + return false; + } +} + +static void madvise_init_tlb(struct madvise_behavior *madv_behavior, + struct mm_struct *mm) +{ + if (!madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_gather_mmu(madv_behavior->tlb, mm); +} + +static void madvise_finish_tlb(struct madvise_behavior *madv_behavior) +{ + if (!madvise_batch_tlb_flush(madv_behavior->behavior)) + return; + tlb_finish_mmu(madv_behavior->tlb); +} + static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior) { size_t len; @@ -1787,14 +1813,20 @@ static int madvise_do_behavior(struct mm_struct *mm, int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int behavior) { int error; - struct madvise_behavior madv_behavior = {.behavior = behavior}; + struct mmu_gather tlb; + struct madvise_behavior madv_behavior = { + .behavior = behavior, + .tlb = &tlb, + }; if (madvise_should_skip(start, len_in, behavior, &error)) return error; error = madvise_lock(mm, behavior); if (error) return error; + madvise_init_tlb(&madv_behavior, mm); error = madvise_do_behavior(mm, start, len_in, &madv_behavior); + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); return error; @@ -1805,32 +1837,6 @@ SYSCALL_DEFINE3(madvise, unsigned long, start, size_t, len_in, int, behavior) return do_madvise(current->mm, start, len_in, behavior); } -static bool vector_madvise_batch_tlb_flush(int behavior) -{ - switch (behavior) { - case MADV_DONTNEED: - case MADV_DONTNEED_LOCKED: - return true; - default: - return false; - } -} - -static void vector_madvise_init_tlb(struct madvise_behavior *madv_behavior, - struct mm_struct *mm) -{ - if (!vector_madvise_batch_tlb_flush(madv_behavior->behavior)) - return; - tlb_gather_mmu(madv_behavior->tlb, mm); -} - -static void vector_madvise_finish_tlb(struct madvise_behavior *madv_behavior) -{ - if (!vector_madvise_batch_tlb_flush(madv_behavior->behavior)) - return; - tlb_finish_mmu(madv_behavior->tlb); -} - /* Perform an madvise operation over a vector of addresses and lengths. */ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, int behavior) @@ -1848,7 +1854,7 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, ret = madvise_lock(mm, behavior); if (ret) return ret; - vector_madvise_init_tlb(&madv_behavior, mm); + madvise_init_tlb(&madv_behavior, mm); while (iov_iter_count(iter)) { unsigned long start = (unsigned long)iter_iov_addr(iter); @@ -1877,17 +1883,17 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter, } /* Drop and reacquire lock to unwind race. */ - vector_madvise_finish_tlb(&madv_behavior); + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); madvise_lock(mm, behavior); - vector_madvise_init_tlb(&madv_behavior, mm); + madvise_init_tlb(&madv_behavior, mm); continue; } if (ret < 0) break; iov_iter_advance(iter, iter_iov_len(iter)); } - vector_madvise_finish_tlb(&madv_behavior); + madvise_finish_tlb(&madv_behavior); madvise_unlock(mm, behavior); ret = (total_len - iov_iter_count(iter)) ? : ret; -- 2.39.5