From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF4D3C369BA for ; Tue, 15 Apr 2025 17:28:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F8FC6B016E; Tue, 15 Apr 2025 13:28:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A85D6B016F; Tue, 15 Apr 2025 13:28:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 398186B0170; Tue, 15 Apr 2025 13:28:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1BDD16B016E for ; Tue, 15 Apr 2025 13:28:20 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BA7F5817DE for ; Tue, 15 Apr 2025 17:28:21 +0000 (UTC) X-FDA: 83336962002.21.79E2BD1 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 9D874180003 for ; Tue, 15 Apr 2025 17:28:19 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1744738100; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mj1oX4uH4nJRL8toIJyqfYxrcDiF5lKUR/vX6O7p1rM=; b=QjUHZZ11O6m4Kl08xp5ITyGUzIzHKptwzx8sQ2FF98S/WUWh6j5GPsoKS0aGHvGP4ShEA7 My7lkitql7X13w7CaEVHlMPh2ynDDAmHyOY+yHrZYGNh47iKx2jzWblmjRAC+BaxllqVqm BdYOSwVEqKao2i0UJNi0/SYMgMQoBAw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1744738100; a=rsa-sha256; cv=none; b=2RbqgHneon7yiofHkgiwjy50tScrPgUIkDwltqg6kK9WkTPG7cQVFOUO1t72fj4wpY97PV 8QK+hAEEN8JZLnJgyN+j2VZLu3SKmvjHS5l4EAio8Dfjh7h+7bjjFOmbYtNCu6pC3/xPg1 A07ckcIwyLwdPxpHQ+fYVoQX2iSk9z4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E85C115A1; Tue, 15 Apr 2025 10:28:16 -0700 (PDT) Received: from [10.57.86.225] (unknown [10.57.86.225]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D5E83F59E; Tue, 15 Apr 2025 10:28:16 -0700 (PDT) Message-ID: <19d2b1c6-ef45-49f5-b11b-a57adc522852@arm.com> Date: Tue, 15 Apr 2025 18:28:14 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 11/11] arm64/mm: Batch barriers when updating kernel mappings Content-Language: en-GB To: Catalin Marinas Cc: Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <20250304150444.3788920-1-ryan.roberts@arm.com> <20250304150444.3788920-12-ryan.roberts@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 9D874180003 X-Rspam-User: X-Stat-Signature: txbb9jks1awofbjmtjrp9baa8719hgb3 X-HE-Tag: 1744738099-365987 X-HE-Meta: U2FsdGVkX1+AS0ix4A9c5oh6Fbo3s7VWs0j/5lkm+S4shD5FzlfmWJ9n6834yvVYgXOF5IeiA/d18pXkOjL9j/B6DJVRcw3+bdhQLN5R6owTVSxs3+pm3m0sZ7PZFRrROni9E3LpupfX9zSRYocaNIoAEbGKAqaMCA/OG5XgYYik/nbD3J81ZzcMPjZniKYMCNQFZ8otsqIuDrIhACpnqz/lEZb7PLoAmoR5QBQzTGTSUupQwnB8sFc3M+HRDeYPf63i7o8kDtD7TmdbB3dI6PsiLNeK1IS+v/pcbXN4Sjq7S6ycD4NWd41rXolVFZK9D49SECcAXiThuCVre6KGP5ChNmHjcZaNiEsIORmveB34BNHaZYpACaOaASVMZUqzIYH8aVw4hqc5w7BJJQsJpL0yDFl7cRqX8dxDhmYKRQ3Hv1I8CuXkWccUkqX3EWclMV0tIyzcZlNQh2lBDxmwJ+Z+aUnpWscuAV4OUTtQeHVRls5wLxlLhkGjY2G/eTnqz4Q5++Oe4mzTY9Pn7i7CAZHSro7QKmG1E5f2fFkQple9wG+Dg/wLduhF8KAd39IDokY//e+IU9Bqiz7N1QoxaMxe2RZh7yxTV9y3T6MQ8C46rPriJguugDaKqGcZuPzbY10eBuoIPVO+ml+uqjYDt8gweeOIhg5tEokEZCdgDPSiNS0q/jNtIilUVxop9Z9wiaqUVW25iKDThmegmBrT0eXju6t24RKqJVkPLk9FkRbwQfPuyB8UK+sHX8894ls13DQfzAlEdSALBAcotJGHuG01tDFyNOdjnFwVyeBbB+FzKgMwJtYvkdsmtTSTgTdvtOD1Tj/7+fFNP/HtZsT5HRe09iv+PZDZf4HG+nRAMCwrBF8+q6rLlA836wOR4HM+KuXOwk+Ob6Xw4MLkeRWI7sEUi7FFdaeiCl0GaHWu+hfZ+czYI6HINveWgrw8px0dnP1fb+Up1yBWiYTOCOj RHLT8JDT kB8JA/WbmegOYHwAghvYs9SC9gWCfFvEBaq7dqrc/29aAE1kRLYBgJv4IJWYC1WFuR+EXu15t6Ow13pdSyPv+k7GGBIxiNAxbhoEXTPwl33EXGZBdiOYOoptxmHVWhkFaSD2640+IbINrYrceKThoaxvVaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 15/04/2025 11:51, Catalin Marinas wrote: > On Mon, Apr 14, 2025 at 07:28:46PM +0100, Ryan Roberts wrote: >> On 14/04/2025 18:38, Catalin Marinas wrote: >>> On Tue, Mar 04, 2025 at 03:04:41PM +0000, Ryan Roberts wrote: >>>> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h >>>> index 1898c3069c43..149df945c1ab 100644 >>>> --- a/arch/arm64/include/asm/pgtable.h >>>> +++ b/arch/arm64/include/asm/pgtable.h >>>> @@ -40,6 +40,55 @@ >>>> #include >>>> #include >>>> >>>> +static inline void emit_pte_barriers(void) >>>> +{ >>>> + /* >>>> + * These barriers are emitted under certain conditions after a pte entry >>>> + * was modified (see e.g. __set_pte_complete()). The dsb makes the store >>>> + * visible to the table walker. The isb ensures that any previous >>>> + * speculative "invalid translation" marker that is in the CPU's >>>> + * pipeline gets cleared, so that any access to that address after >>>> + * setting the pte to valid won't cause a spurious fault. If the thread >>>> + * gets preempted after storing to the pgtable but before emitting these >>>> + * barriers, __switch_to() emits a dsb which ensure the walker gets to >>>> + * see the store. There is no guarrantee of an isb being issued though. >>>> + * This is safe because it will still get issued (albeit on a >>>> + * potentially different CPU) when the thread starts running again, >>>> + * before any access to the address. >>>> + */ >>>> + dsb(ishst); >>>> + isb(); >>>> +} >>>> + >>>> +static inline void queue_pte_barriers(void) >>>> +{ >>>> + if (test_thread_flag(TIF_LAZY_MMU)) >>>> + set_thread_flag(TIF_LAZY_MMU_PENDING); >>> >>> As we can have lots of calls here, it might be slightly cheaper to test >>> TIF_LAZY_MMU_PENDING and avoid setting it unnecessarily. >> >> Yes, good point. >> >>> I haven't checked - does the compiler generate multiple mrs from sp_el0 >>> for subsequent test_thread_flag()? >> >> It emits a single mrs but it loads from the pointer twice. > > It's not that bad if only do the set_thread_flag() once. > >> I think v3 is the version we want? >> >> >> void TEST_queue_pte_barriers_v1(void) >> { >> if (test_thread_flag(TIF_LAZY_MMU)) >> set_thread_flag(TIF_LAZY_MMU_PENDING); >> else >> emit_pte_barriers(); >> } >> >> void TEST_queue_pte_barriers_v2(void) >> { >> if (test_thread_flag(TIF_LAZY_MMU) && >> !test_thread_flag(TIF_LAZY_MMU_PENDING)) >> set_thread_flag(TIF_LAZY_MMU_PENDING); >> else >> emit_pte_barriers(); >> } >> >> void TEST_queue_pte_barriers_v3(void) >> { >> unsigned long flags = read_thread_flags(); >> >> if ((flags & (_TIF_LAZY_MMU | _TIF_LAZY_MMU_PENDING)) == _TIF_LAZY_MMU) >> set_thread_flag(TIF_LAZY_MMU_PENDING); >> else >> emit_pte_barriers(); >> } > > Doesn't v3 emit barriers once _TIF_LAZY_MMU_PENDING has been set? We > need something like: > > if (flags & _TIF_LAZY_MMU) { > if (!(flags & _TIF_LAZY_MMU_PENDING)) > set_thread_flag(TIF_LAZY_MMU_PENDING); > } else { > emit_pte_barriers(); > } Gah, yeah sorry, going to quickly. v2 is also logicially incorrect. Fixed versions: void TEST_queue_pte_barriers_v2(void) { if (test_thread_flag(TIF_LAZY_MMU)) { if (!test_thread_flag(TIF_LAZY_MMU_PENDING)) set_thread_flag(TIF_LAZY_MMU_PENDING); } else { emit_pte_barriers(); } } void TEST_queue_pte_barriers_v3(void) { unsigned long flags = read_thread_flags(); if (flags & BIT(TIF_LAZY_MMU)) { if (!(flags & BIT(TIF_LAZY_MMU_PENDING))) set_thread_flag(TIF_LAZY_MMU_PENDING); } else { emit_pte_barriers(); } } 000000000000105c : 105c: d5384100 mrs x0, sp_el0 1060: f9400001 ldr x1, [x0] 1064: 37f80081 tbnz w1, #31, 1074 1068: d5033a9f dsb ishst 106c: d5033fdf isb 1070: d65f03c0 ret 1074: f9400001 ldr x1, [x0] 1078: b707ffc1 tbnz x1, #32, 1070 107c: 14000004 b 108c 1080: d2c00021 mov x1, #0x100000000 // #4294967296 1084: f821301f stset x1, [x0] 1088: d65f03c0 ret 108c: f9800011 prfm pstl1strm, [x0] 1090: c85f7c01 ldxr x1, [x0] 1094: b2600021 orr x1, x1, #0x100000000 1098: c8027c01 stxr w2, x1, [x0] 109c: 35ffffa2 cbnz w2, 1090 10a0: d65f03c0 ret 00000000000010a4 : 10a4: d5384101 mrs x1, sp_el0 10a8: f9400020 ldr x0, [x1] 10ac: 36f80060 tbz w0, #31, 10b8 10b0: b60000a0 tbz x0, #32, 10c4 10b4: d65f03c0 ret 10b8: d5033a9f dsb ishst 10bc: d5033fdf isb 10c0: d65f03c0 ret 10c4: 14000004 b 10d4 10c8: d2c00020 mov x0, #0x100000000 // #4294967296 10cc: f820303f stset x0, [x1] 10d0: d65f03c0 ret 10d4: f9800031 prfm pstl1strm, [x1] 10d8: c85f7c20 ldxr x0, [x1] 10dc: b2600000 orr x0, x0, #0x100000000 10e0: c8027c20 stxr w2, x0, [x1] 10e4: 35ffffa2 cbnz w2, 10d8 10e8: d65f03c0 ret So v3 is the way to go, I think; it's a single mrs and a single ldr. I'll get this fixed up and posted early next week. Thanks, Ryan