From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD0BCC25B74 for ; Fri, 31 May 2024 01:47:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C9A16B009B; Thu, 30 May 2024 21:47:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 279866B009D; Thu, 30 May 2024 21:47:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11A286B009E; Thu, 30 May 2024 21:47:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E72A66B009B for ; Thu, 30 May 2024 21:47:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7BAA080F1B for ; Fri, 31 May 2024 01:47:35 +0000 (UTC) X-FDA: 82177004070.13.0129E00 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by imf23.hostedemail.com (Postfix) with ESMTP id B1DBF140006 for ; Fri, 31 May 2024 01:47:31 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cPRmKjrC; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf23.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717120053; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PJDANj4+0n7pfXaR6YkRJnkCicgMDSkDK6L13cU1/V4=; b=wSekUJatOfsYxxeoObLEfCtXRpPvRN71h4DqCfPynZgknz4kUzopuTlIJIvzOPZk9Ywh6O aaJZyfa0HIBWxtgGhPTc0tPSR3sMXxRPg9FmvohKMPefZ1yuq+tcuuk5PNcNt8PMBVosrL 8yUKH4xt792ZTUMTlMevQJQeCAaDLSA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717120053; a=rsa-sha256; cv=none; b=LpqYgkOkZvsKakYpIOHfCL68OMaFE1sS6yduei123BWHMtLLvGXQDc+imOTMlVogCFumfC NzmW44VBvsGSjcmkJA7XEQpoVPmhcWsFyXiQL6Uho/aqWih63k9KG8mI8DvZNedbPfxv6n 2UEg4qVwoDpBjpsPHp1TKKow88R/QNI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=cPRmKjrC; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf23.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.10 as permitted sender) smtp.mailfrom=ying.huang@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1717120052; x=1748656052; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=bRLXaD0t9oBTss9DDyeGZGopPy1q6rqAwm3aGhuXiBE=; b=cPRmKjrC771k734HTv8XcCSZGTDvQ/+a9Ll/Jssuza0TNvV4KJqzatVN u4610XtHtgTNFVu0zdE9DJFME9mQmD2wP1C77hFisaTKdQg3vp4pBrkr4 qowxrepCLKZ4tpNcJXnY0rloS6HdHSZXIzrboOZgpEpNIZcRZDeWLpZia p4K6y2PquLHSyewOo9HU0VgNd6ZHu3LUDrpIgiaurR3i7xQDhJSQae8LB 0/F+5vS1OLOG+5GMy1wiu5D3G0V1Ksn1I1OmqmeL257dtU4+DZZ2KRbAk VG0plW+4Cd65uJ83N9K9Teo1GqDIX4v6PcfDw9b1k4h6j5VtJDXqxCDJh Q==; X-CSE-ConnectionGUID: FCjvIgVaRly+wdKpPt4aVQ== X-CSE-MsgGUID: 13HsXLICQmyaNyJWM15ZaQ== X-IronPort-AV: E=McAfee;i="6600,9927,11088"; a="31142763" X-IronPort-AV: E=Sophos;i="6.08,202,1712646000"; d="scan'208";a="31142763" Received: from orviesa008.jf.intel.com ([10.64.159.148]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 May 2024 18:47:30 -0700 X-CSE-ConnectionGUID: 8bXyluNQQ5iUZIhU/5+CYw== X-CSE-MsgGUID: mdxGSkWESN2KkwjT+/CaNA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,202,1712646000"; d="scan'208";a="36579659" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orviesa008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 May 2024 18:47:25 -0700 From: "Huang, Ying" To: Byungchul Park Cc: Dave Hansen , , , , , , , , , , , , , , , , Subject: Re: [PATCH v10 00/12] LUF(Lazy Unmap Flush) reducing tlb numbers over 90% In-Reply-To: <20240530093306.GA35610@system.software.com> (Byungchul Park's message of "Thu, 30 May 2024 18:33:07 +0900") References: <982317c0-7faa-45f0-82a1-29978c3c9f4d@intel.com> <20240527015732.GA61604@system.software.com> <8734q46jc8.fsf@yhuang6-desk2.ccr.corp.intel.com> <44e4f2fd-e76e-445d-b618-17a6ec692812@intel.com> <20240529050046.GB20307@system.software.com> <961f9533-1e0c-416c-b6b0-d46b97127de2@intel.com> <20240530005026.GA47476@system.software.com> <87a5k814tq.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240530071847.GA15344@system.software.com> <871q5j1zdf.fsf@yhuang6-desk2.ccr.corp.intel.com> <20240530093306.GA35610@system.software.com> Date: Fri, 31 May 2024 09:45:33 +0800 Message-ID: <87wmnazrcy.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Stat-Signature: crdrhubxte9optqtguij3zfq3wg51xyt X-Rspamd-Queue-Id: B1DBF140006 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717120051-699367 X-HE-Meta: U2FsdGVkX18PnCHBusD7CT7VqbVsrCc17NgizxHhsCQDeMRfYESAOcdmqfbIoBeKV5bLe8QwQINVgQYqDJCsSUGDDiABt/cA5c9PmArecPCEMjipzKyFkE+xxJdpDwuqmx+C9NceKc5TwHVSxNCvD0p+xnzg1EUrqMEdeAECU95bhX4Q+V4EcU6w+2mNjtsFq6qHpgP/H9iMtKvR4KK60JmICJWww4CSwMjPpxmk4drdlfqSJP4jM41rxZT43mzmJzTyF/EegocR6I9vEuUnganAJVSjWsQquI3tU+OJrDgl0If6ro2V0Fh1oxEGTdd9mYDyCpkMi+n7sjBhgJH/ei4xd82xqATux0Dwb0leqoFzwul+k53aBoyZbN2GqbiKKrWE29z4c7OfQF++1o6TS2RcnhbSFw0VokBr4V7txEUYDWWeDUaO+fheWiGJwe0OTJu+XJyKHvBCQAkeG6BURHCgp3xdMNtzFuGUAZbYPt5sZiSBeqo0v0F9osKEGeaGMx1Mjaq6sEDKSeO4vnnWsZDhT5wvk2O2WnZxfXhwiGpUAZHT7a1kAS61xjC6snsMAyQOYFfWlfNpdSuaMMFscuSS6S18w2sFXF0Aaq7SdQZo/q+TtMSsskT/dTZX7tWJE76A48D5hSiWuqFGHrhwITMaND3/N4iGYc/nvf7pRBRTRBBgjFsbRAVZBykgl4T4kP09zf3c5ANtXxXOToNfTQuD4LyBFEgGkMJn8PAifwggUtcAAqBeS38UGAf0Ha/axpzS2UUhAyQZubA9Y2wNFu31oj9AmEa/vZR4c3TqeqSttiBZycZe9ao5jNBcmhABef9X6q+vA7i802xf05GK3q374NAJZib+xhdcmkmddRdx5TfSP2+tYB0fAN6veeaHLhCsG9xInq9/cKUDmqPCNM/bBuNwZGs0flQVeoGTnHpF2JcpMK+gmzZ+sWJ1D9Ca1MrN9G6jxOl05kLbnjk jNctPTBp /tiXY3vgAiojAkMiZK9qhkaILHnFfsPHt+Y0xJCDCB45uHYrSKT55BWnOm/ucZoyKQ+V0zK1OgPyLC6/8Hlvm9XnMyH/uoZRUqC+bO+z15EJUrdPP5QZDiQ/lbnYjrUGhyEbUKgD5Re0DHNZl4u1npQGhSond/NyhIwRsRWbUAP2R4ZqYeIWz0xaqaTjfQM6lRDXzMq9/ST1cvlao/U3/PjPKBZY2yd+xeJFF4T9CK38ZFTmzchzS4Qw1WHWLhZqs7dD1wZ3lPgygxaljcNGwHL4dWyV0peVa51tcIRZdRuvbX6+RC8+VYFtswvHMTFii/IW8g1vctmTGUo0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Byungchul Park writes: > On Thu, May 30, 2024 at 04:24:12PM +0800, Huang, Ying wrote: >> Byungchul Park writes: >> >> > On Thu, May 30, 2024 at 09:11:45AM +0800, Huang, Ying wrote: >> >> Byungchul Park writes: >> >> >> >> > On Wed, May 29, 2024 at 09:41:22AM -0700, Dave Hansen wrote: >> >> >> On 5/28/24 22:00, Byungchul Park wrote: >> >> >> > All the code updating ptes already performs TLB flush needed in a safe >> >> >> > way if it's inevitable e.g. munmap. LUF which controls when to flush in >> >> >> > a higer level than arch code, just leaves stale ro tlb entries that are >> >> >> > currently supposed to be in use. Could you give a scenario that you are >> >> >> > concering? >> >> >> >> >> >> Let's go back this scenario: >> >> >> >> >> >> fd = open("/some/file", O_RDONLY); >> >> >> ptr1 = mmap(-1, size, PROT_READ, ..., fd, ...); >> >> >> foo1 = *ptr1; >> >> >> >> >> >> There's a read-only PTE at 'ptr1'. Right? The page being pointed to is >> >> >> eligible for LUF via the try_to_unmap() paths. In other words, the page >> >> >> might be reclaimed at any time. If it is reclaimed, the PTE will be >> >> >> cleared. >> >> >> >> >> >> Then, the user might do: >> >> >> >> >> >> munmap(ptr1, PAGE_SIZE); >> >> >> >> >> >> Which will _eventually_ wind up in the zap_pte_range() loop. But that >> >> >> loop will only see pte_none(). It doesn't do _anything_ to the 'struct >> >> >> mmu_gather'. >> >> >> >> >> >> The munmap() then lands in tlb_flush_mmu_tlbonly() where it looks at the >> >> >> 'struct mmu_gather': >> >> >> >> >> >> if (!(tlb->freed_tables || tlb->cleared_ptes || >> >> >> tlb->cleared_pmds || tlb->cleared_puds || >> >> >> tlb->cleared_p4ds)) >> >> >> return; >> >> >> >> >> >> But since there were no cleared PTEs (or anything else) during the >> >> >> unmap, this just returns and doesn't flush the TLB. >> >> >> >> >> >> We now have an address space with a stale TLB entry at 'ptr1' and not >> >> >> even a VMA there. There's nothing to stop a new VMA from going in, >> >> >> installing a *new* PTE, but getting data from the stale TLB entry that >> >> >> still hasn't been flushed. >> >> > >> >> > Thank you for the explanation. I got you. I think I could handle the >> >> > case through a new flag in vma or something indicating LUF has deferred >> >> > necessary TLB flush for it during unmapping so that mmu_gather mechanism >> >> > can be aware of it. Of course, the performance change should be checked >> >> > again. Thoughts? >> >> >> >> I suggest you to start with the simple case. That is, only support page >> >> reclaiming and migration. A TLB flushing can be enforced during unmap >> >> with something similar as flush_tlb_batched_pending(). >> > >> > While reading flush_tlb_batched_pending(mm), I found it already performs >> > TLB flush for the target mm, if set_tlb_ubc_flush_pending(mm) has been >> > hit at least once since the last flush_tlb_batched_pending(mm). >> > >> > Since LUF also relies on set_tlb_ubc_flush_pending(mm), it's going to >> > perform TLB flush required, in flush_tlb_batched_pending(mm) during >> > munmap(). So it looks safe to me with regard to munmap() already. >> > >> > Is there something that I'm missing? >> > >> > JFYI, regarding to mmap(), I have reworked on fault handler to give up >> > luf when needed in a better way. >> >> If TLB flush is always enforced during munmap(), then your solution can >> only avoid TLB flushing for page reclaiming and migration, not unmap. > > I'm not sure if I understand what you meant. Could you explain it in > more detail? > > LUF works for only *unmapping* that happens during page reclaiming and > migration. Other unmappings than page reclaiming and migration are not > what LUF works for. That's why I thought flush_tlb_batched_pending() > could handle the pending tlb flushes in the case. > > It'd be appreciated if you explain what you meant more. > In the following email, you have claimed that LUF can avoid TLB flushing for munmap()/mmap(). https://lore.kernel.org/linux-mm/20240527015732.GA61604@system.software.com/ Now, you said it can only avoid TLB flushing for page reclaiming and migration. So, to avoid confusion, I suggest you to send out a new series and make it explicit that it can only optimize page reclaiming and migration, but not munmap(). And it would be good too to add some words about how it interact with other TLB flushing mechanisms. -- Best Regards, Huang, Ying