From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53935E77188 for ; Tue, 7 Jan 2025 03:26:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCDEF6B0083; Mon, 6 Jan 2025 22:26:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C7DE56B0089; Mon, 6 Jan 2025 22:26:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6C356B008C; Mon, 6 Jan 2025 22:26:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 99C626B0083 for ; Mon, 6 Jan 2025 22:26:47 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 284BA1A0666 for ; Tue, 7 Jan 2025 03:26:47 +0000 (UTC) X-FDA: 82979218854.03.12F8877 Received: from shelob.surriel.com (shelob.surriel.com [96.67.55.147]) by imf01.hostedemail.com (Postfix) with ESMTP id 68B7840010 for ; Tue, 7 Jan 2025 03:26:45 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736220405; a=rsa-sha256; cv=none; b=c3pDAtcvh95ExGjgfpbcQJz6CqDoJ5c+DBU0RyVD2e6IWe3wk6a2oVgHB0r81eAzCow/kO QlsL0PJOhwYFKL4d+wXTswcvHRem/X+Hbz6BayxdOMHLe4Rn64GZ9QNYat/h2vqmArOaKr qEV9xzrQmaLmmhcPcw9kZNK1GHvyVs8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of riel@shelob.surriel.com designates 96.67.55.147 as permitted sender) smtp.mailfrom=riel@shelob.surriel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736220405; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8zPfBj62QNhZU7OTTJpBg4FM9n0kiBXK6Rgl+/aoGaA=; b=Dc06lSHtsDHJxbKnr8Ttj1ldIFcRPiBVyZ6ua8S9CjjYTJxgAB0KvLd1FDqsF+Ct8vlAPK SYcAZ/4uctpu+0pFzCfB5PME1lIg2NSdVZGLgIuWga4LHIcskx07pw8IYcU8P8orgjC7Tm 8jMyG1/ph37cKguPTfbJu9phfD/wxU4= Received: from fangorn.home.surriel.com ([10.0.13.7]) by shelob.surriel.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.97.1) (envelope-from ) id 1tV0Dr-0000000085m-2QqM; Mon, 06 Jan 2025 22:25:39 -0500 Message-ID: <95a7349f887e538b5e63f77da6b2a1d7efc9a43f.camel@surriel.com> Subject: Re: [PATCH v3 00/12] AMD broadcast TLB invalidation From: Rik van Riel To: Yosry Ahmed Cc: x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, akpm@linux-foundation.org, nadav.amit@gmail.com, zhengqi.arch@bytedance.com, linux-mm@kvack.org, Reiji Watanabe , Brendan Jackman Date: Mon, 06 Jan 2025 22:25:39 -0500 In-Reply-To: References: <20241230175550.4046587-1-riel@surriel.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.54.1 (3.54.1-1.fc41) MIME-Version: 1.0 X-Stat-Signature: z5o5kpxac599wa43fq8nmb3rh9wb5ykj X-Rspam-User: X-Rspamd-Queue-Id: 68B7840010 X-Rspamd-Server: rspam08 X-HE-Tag: 1736220405-909118 X-HE-Meta: U2FsdGVkX1+W7NyXcjhRyC/hITqtoGiubWrNClMDbw13hYvm4c7CpT0NxWX8IR6xi/nMycWpv/1/qN5AsH8xGV2xodjB+Jm1w6Qiy3+5lj503OIZd07HX3FGdI3jibqpAkenrOuz5j/eusMsxYUlH3gMaCk8UIKL0MpJJ28CeqvN2zb6nUy8G660nDgeEiaXCV/egwqQWdy2zIWAzv0hkxZN3Z/jEvc6i8fKqumiFE+L0LSHWETb2MWJTNudfMGQPN44wgZWqMs1l0diLj6uOQkWHHrLZCV36UO9SEcQTDy0zl/DCdWmaGm11UaOzC1OdCkj2lVSYba4eJ1XUFnbnZxaJZtmPqERPrjP/TaRXNHXSi2jLckopGReGA8QQcnFWiNSz9Wqx9b/OPULQOgeyC3wAM9L7xRCD73KmSz5RFwaR8cCXz92K9RSlbhecHPtjBDBeafyesoaHAFgcrp8VFmqy9izeJnnSo922LqLGRBePdpv+jhpWbCJDMlMH2sHmBe3tGFSFB5eAnbmMtkUXAv9FLjoVQ56lvcKrWkYPxodm8z9f3EPLWFeW8lYpPhBP0a3ijjU9qkKxl48Pe2z3Bym74KW8Ly9pSAiBBIhDNvYSTdOs/SmW2Y0j7WsHpMFo02AUwv2xTtP94eyTJ+hpZWKlwTQQPYJxaWU/EDKKVrXWotm6hz6QXCpg4TcpFibQho77dW8uSD02F2jmt4yrnAagsJ8rF0irCS6eoNeSf+EtAUnDEDPdQJ8uIekpv/HDNLoZ62r9+EdelTOyQel5gDZCt9DNk4Vhl+W79zqHFLnawXlIf604KCZ8Uak1JZNEyBecbCiWzs2nQTLoSIZ85xSmEaAhhrNC/OBDssHDDJvQvtKfAGCxKXLD8DtF8nDT+/rr9EA05Te4RY+5nDNqDf+dhoL2VHqzENpHxoGoKjmNyerPnzdo0Aj/P80feIO2f7wBiFeF7QXgeQ5GXN pF1BarZF iOIUEACmCLFdKrrHSDOFI5d1TKjqJh0P5AGws9kly+nadyMUZgJUen0rRRAZPO6TPt1W7vYnXQGZ6MKpYBkLC98eJ47uf+nevdJ3ZbkS2/ND5fRipDbFFXEgPhHrAyDGm9yeSkSpBHIh6qldm1Jm9CgnG7wIfAbUy1oSGhU2yz8twRZiwNBKFbezp7ZxJdC30eZwxLLVqUOFkVUkh15ZhR6+4VMgCB9D5S3jVIvwy4+5aWR0bDU9jOIMt1J7j4geQaFiVtltS5/Y37aw1LFpa5lgrHmCMdq+9geR7YIKzI71EfGE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 2025-01-06 at 14:49 -0800, Yosry Ahmed wrote: >=20 > We briefly looked at using INVLPGB/TLBSYNC as part of the ASI work to > optimize away the async freeing logic which sends TLB flush IPIs. >=20 > I have a high-level question about INVLPGB/TLBSYNC that I could not > immediately find the answer to in the AMD manual. Sorry if I missed > the answer or if I missed something obvious. >=20 > Do we know what the underlying mechanism for delivering the TLB > flushes is? If a CPU has interrupts disabled, does it still receive > the broadcast TLB flush request and handle it? I assume TLB invalidation is probably handled similarly to how cache coherency is handled between CPUs. However, it probably does not need to be quite as fast, since cache coherency traffic is probably 2-6 orders of magnitude more common than TLB invalidation traffic. >=20 > My main concern is that TLBSYNC is a single instruction that seems > like it will wait for an arbitrary amount of time, and IIUC > interrupts > (and NMIs) will not be delivered to the running CPU until after the > instruction completes execution (only at an instruction boundary). >=20 > Are there any guarantees about other CPUs handling the broadcast TLB > flush in a timely manner, or an explanation of how CPUs handle the > incoming requests in general? The performance numbers I got with the tlb_flush2_threads microbenchmark strongly suggest that INVLPGB flushes are handled by the receiving CPUs even while interrupts are disabled. CPU time spent in flush_tlb_mm_range goes down with INVLPGB, compared with IPI based TLB flushing, even when the IPIs only go to a subset of CPUs. I have no idea whether the invalidation is handled by something like microcode in the CPU, by the (more external?) logic that handles cache coherency, or something else entirely. I suspect AMD wouldn't tell us exactly ;) --=20 All Rights Reversed.