From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89673C02198 for ; Mon, 10 Feb 2025 15:27:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0A606B0083; Mon, 10 Feb 2025 10:27:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DBA3F6B0088; Mon, 10 Feb 2025 10:27:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C811D280001; Mon, 10 Feb 2025 10:27:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A8B2C6B0083 for ; Mon, 10 Feb 2025 10:27:21 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 273491210B7 for ; Mon, 10 Feb 2025 15:27:21 +0000 (UTC) X-FDA: 83104413882.22.27B232E Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf03.hostedemail.com (Postfix) with ESMTP id 46CA920013 for ; Mon, 10 Feb 2025 15:27:19 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rL1GQDI9; spf=pass (imf03.hostedemail.com: domain of jackmanb@google.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=jackmanb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739201239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yZ316OTEKAnaWAw3ip6jFLMYmi3bAEAv/lpBYPcdN08=; b=LGeMZuIgba0nOAwakuzWI0cZccHlzl9irhtl2eqe4YIRhbKSyUYz/mQIg+xQliWDFItCp1 J+NeIMyq5Fh61daZxzzANKoqrGkI+tReAQ8YYlc0MMBh3LsmehzXRA/d43BHMX+0GLFu2B VwF1RsXHerhl6yiImuXQfh0pThvRJOE= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rL1GQDI9; spf=pass (imf03.hostedemail.com: domain of jackmanb@google.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=jackmanb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739201239; a=rsa-sha256; cv=none; b=cFKJ2SWcIq+ZIcsmgoRA6Pxii5OfwR7njHCuH/+VTagqG4xPoIYg0bq4cC8Vijni1LgoMo 04clJO/2dMnMHsekwA0DVMrsQhRWKTz49UX0aZ7IoI7zZWGJhddbywMjRsBnMBooJ0O/hU d24c2FBbIt7vOpajK+RAhbUyBTQhZw8= Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-47180c199ebso392971cf.0 for ; Mon, 10 Feb 2025 07:27:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739201238; x=1739806038; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=yZ316OTEKAnaWAw3ip6jFLMYmi3bAEAv/lpBYPcdN08=; b=rL1GQDI9VKm4vFWeJdH7zlY38eIkkO81oYtfzYH2VtKtfQCuKhzoc7h6OZlhIu3EiJ ysds0/MAiwbbc/VznBs1bEiUrr1YWN/kUarTHsbzJXutI4y46ST9zJX3Pi3L5xDt9opm 8WmITwdnrf6/42ueu3I29nMcD9nQlhkaIdlUBkON6V5qrIu3JupEbyYXOAhxCUdCX/Zy PqFwDRzYyUc4nysjpdzkCIoCVnIMyztJriO86UWH9Gs3H1CEzvhYsnhG5RXXIVE3Ypf2 iegqcDdvAyBWnZ3nGu1H1RfGeIOIbb6wLA6Apf+GzrdiS4YDXD3Th1qK8OE6IOGABXSQ no+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739201238; x=1739806038; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=yZ316OTEKAnaWAw3ip6jFLMYmi3bAEAv/lpBYPcdN08=; b=Bcq1Snd7zRLTdA8lXeGsnpIAjcxVQGl2Z84GHzZ5toaRKEgKOr8BgV6IIELH0tYJbF 6PI14guVpHYLSTB1DT84LDrGQTX33aUH+zOVX5oyO+f8WCsd3Y89S+kjCc4KuLj2had4 NEvbJerIG3+KY/fRW0gQ2KCF9eKJKnmmC0ovcSLM086zdOCr/waK9c6yNeuO6+n0WWrp q+UsOitGLVgGHYWKoGx8SHkIKrGG6S6XKBfEdWXIDa3Sk+GIa3fbPq2AafwICxWcI/5g YwMW18faE4IADm9AErRmyaPGaeYh0h79wNK8/E2LHDkWa0nuA0uHUvHLEO+OkOvjIzA8 FNcg== X-Forwarded-Encrypted: i=1; AJvYcCW3l6YbCNNvdAhLgK1dwQwnYR0PWUSGGU6AEjTzFCnnDiddrddBERLfbvOvmXlXk0ZuXy3HXldlzw==@kvack.org X-Gm-Message-State: AOJu0YyeNo1sgsEFIA49hY7q69vdcoWlAKlOUaVNKmVtji7z9A9So+dx xmgZoez1f0BBfZkcT1DkpZzLGmdI+lUWZlbZRUaTaZq1U7El5UlS46EanN4BBUEYs1wvlBV00yV gMVOfuw5ZODfTBHZYemR6VPgTgtKXfWIyv/iy X-Gm-Gg: ASbGncuo3UlI5xb7bfjxaPz/qEJb90DGlAjOdrCHvoPVwu8KQ0On2P7M86W4y1purOJ G5fuWjfLpwZiOxBHOcpkWl8aLiPGnXl8KCj12+blMTc+sJsV+Cvy1ZbR841Gy7Dg3Lld98oNfjc KnPqZkuQ9cgf10qDWaZVrlEL1BTKc= X-Google-Smtp-Source: AGHT+IGMC9rpyokyquYS20i+q+ma7qGGS+XM/pl59dYsC1sgzfA9pzyKGEeQgh9YV3MTbqORXSHO8DmjEdKJ7ZFsWzg= X-Received: by 2002:a05:622a:1829:b0:466:8c7c:3663 with SMTP id d75a77b69052e-47177e0c91fmr6467531cf.5.1739201238266; Mon, 10 Feb 2025 07:27:18 -0800 (PST) MIME-Version: 1.0 References: <20250206044346.3810242-1-riel@surriel.com> <20250206044346.3810242-11-riel@surriel.com> In-Reply-To: <20250206044346.3810242-11-riel@surriel.com> From: Brendan Jackman Date: Mon, 10 Feb 2025 16:27:06 +0100 X-Gm-Features: AWEUYZnA61IuaO7K52ByypDfx9F-3cd0JBhr6z4LbJwawKvTYqUz-OtEqws_R8s Message-ID: Subject: Re: [PATCH v9 10/12] x86/mm: do targeted broadcast flushing from tlbbatch code To: Rik van Riel Cc: x86@kernel.org, linux-kernel@vger.kernel.org, bp@alien8.de, peterz@infradead.org, dave.hansen@linux.intel.com, zhengqi.arch@bytedance.com, nadav.amit@gmail.com, thomas.lendacky@amd.com, kernel-team@meta.com, linux-mm@kvack.org, akpm@linux-foundation.org, jannh@google.com, mhklinux@outlook.com, andrew.cooper3@citrix.com, Manali Shukla Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 46CA920013 X-Stat-Signature: 4t65oxckmjybb8pztfd6sssjme817ji7 X-HE-Tag: 1739201239-94414 X-HE-Meta: U2FsdGVkX1/UiuUaWATYNRFW+ZJODbTAHL+Rw3sCDUoZu5+GQ1N8mxkA0nfCg1mhS9PCxArLhg7h/qvHMma4qf6Ot25FdI0TKewH+ogtU/OvEjYV0QJtlqT3N/yLuOyvgxstlRVoB1A1wK9A1/VqoEwGF0XSMKaZy5Ioz4yoA75j3nz5o21d4LGl3U5MGMymSjZKcxa1RoEpaLIg12TzZ8+XzMjfMjm/K+wW1dLPfP/f/bm/+WIbUnfMuxC51syQF8F1HzvkAwGeZ88ZeiEWvAzzDblhM0QasA3OPQ/homSZvrv+UL3nGYGnHvZ3rKMKSHMUVvgW2o//Vd32CyHQSn21srGZqPlH4r/+ON/TSPRw7CnAFP72WjtMr9T/NrrQgJnB+TU/d67p+3lcA3tzdUzlSDCHyX+Pupt4GmnhnDp/dXMIE3GxOy6LquRlbLTFQMkkptaEWgMtghtPXJKAAdBO7VWNZW4IY+Ildyk9xqiGpgJsQeE84iBcZ22PWqk5NYgVHAo9URbpllf+ZrtEWnyMZyAAM2XG89Axta3MCxhT0P7L1uQIr/VTNeelAjURBXAcX7+xneuLySOa39Hy9/kpwjUd564lZ391q4T0tJosoIuaFziAnWRN6MDO61g6OtQ5syVfvD75Myg5K6vq/RrHnqIu2qyvSB0FXrg6rWxzRLRcYwc90VDQ8PYs6B4C/vPeL5pXhuXKW38m1whi+jjWlUWJ6Qqny+9vR/XtoroiiRSGpeAzu0+Ch6PdgBfMYEmf9piDLVLvr8F447tcqjifUGz8La9UePkKAcegRWcqSeLREu8B/VFLj5vRBAOwPzkkrgqbH7x7KnNLSmvf9/XOC0fSukeV8TjonMoTgDFjqfqwThbM1ps6tmS75Z16NXmdqfXkgf0wP4DysCvnUaBDjv7xvGqKmYtG15YNN1Vn1Z05WpdB2C1DXzW0sitAl7odJTmVHcK6261D1Qw 81dMRTho uAqn2vXBmIuX+dbupxe56w+X6LUyXMTmRbre9grZZeMaLRpfRpbB2t77kV5HVZjn/b7iMp4AFwIinaSeblAN0xcv3Hm6SGBEv9x6YhpYgOmznGORs2hzm4pRBgwrhhYM1/xgXPEMdXa6qqWr7tCv0o5qI9LNsdW31raE7KLUm2mkSUfpFzecm1A2ECV55c1ihRg0MXo/603hoofvPMOhf9BJLN/BsN3P22kysgrz/tmqPIdSKt+n1cKcPth8ZYxrZOLpMFHuGrDD2lzc6Zo+treRFAJmOSrSLKvkzQ6lHWGaGaTM0eEqMW3rdl08JY+1q2Q70j3KIHeXpDwHadK591PBVmnyYWc92uLFwmmzXocMZpfU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 6 Feb 2025 at 05:46, Rik van Riel wrote: > /* Wait for INVLPGB originated by this CPU to complete. */ > -static inline void tlbsync(void) > +static inline void __tlbsync(void) > { > - cant_migrate(); Why does this have to go away? > diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h > index 234277a5ef89..bf167e215e8e 100644 > --- a/arch/x86/include/asm/tlbflush.h > +++ b/arch/x86/include/asm/tlbflush.h > @@ -106,6 +106,7 @@ struct tlb_state { > * need to be invalidated. > */ > bool invalidate_other; > + bool need_tlbsync; The ifdeffery is missing here. > +static inline void invlpgb_flush_user_nr_nosync(unsigned long pcid, > + unsigned long addr, > + u16 nr, bool pmd_stride) > +{ > + __invlpgb_flush_user_nr_nosync(pcid, addr, nr, pmd_stride); > + if (!this_cpu_read(cpu_tlbstate.need_tlbsync)) > + this_cpu_write(cpu_tlbstate.need_tlbsync, true); Why do we need the conditional here? I always thought code like this was about avoiding cacheline contention, but given this is a percpu and it's really only of interest to the present CPU, is this worthwhile? I guess it might be sharing a cacheline with some other percpu that is more shared? Anyway, not a big deal, I'm mostly asking for my own education. > @@ -794,6 +825,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, > if (IS_ENABLED(CONFIG_PROVE_LOCKING)) > WARN_ON_ONCE(!irqs_disabled()); > > + tlbsync(); > + > /* > * Verify that CR3 is what we think it is. This will catch > * hypothetical buggy code that directly switches to swapper_pg_dir > @@ -973,6 +1006,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, > */ > void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) > { > + tlbsync(); > + I have a feeling I'll look stupid for asking this, but why do we need this and the one in switch_mm_irqs_off()? > @@ -1661,12 +1694,53 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) > local_irq_enable(); > } > > + /* > + * If we issued (asynchronous) INVLPGB flushes, wait for them here. > + * The cpumask above contains only CPUs that were running tasks > + * not using broadcast TLB flushing. > + */ > + if (cpu_feature_enabled(X86_FEATURE_INVLPGB)) It feels wrong that we check the cpufeature here but not in e.g. enter_lazy_tlb(). > + tlbsync(); > + > cpumask_clear(&batch->cpumask); > > put_flush_tlb_info(); > put_cpu(); > } > > +void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, > + struct mm_struct *mm, > + unsigned long uaddr) > +{ > + u16 asid = mm_global_asid(mm); > + > + if (asid) { > + invlpgb_flush_user_nr_nosync(kern_pcid(asid), uaddr, 1, false); > + /* Do any CPUs supporting INVLPGB need PTI? */ Not today, but 1. I don't think avoiding static_cpu_has() is worth the cost of making the reader take that logical step. 2. AFAIK a new CPU bug never led to enabling KPTI on a CPU that didn't have it before, and I think that would be a pretty dystopian future (and hopefully by then we'll have ASI instead...). But we can't really rule it out.