From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCA4EC433EF for ; Fri, 8 Jul 2022 03:27:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 350B5900002; Thu, 7 Jul 2022 23:27:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2FEF86B0073; Thu, 7 Jul 2022 23:27:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EF07900002; Thu, 7 Jul 2022 23:27:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 102D96B0071 for ; Thu, 7 Jul 2022 23:27:44 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CDCF9205C8 for ; Fri, 8 Jul 2022 03:27:43 +0000 (UTC) X-FDA: 79662498006.16.E6F95AA Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) by imf07.hostedemail.com (Postfix) with ESMTP id 62EDF4001B for ; Fri, 8 Jul 2022 03:27:43 +0000 (UTC) Received: by mail-qt1-f170.google.com with SMTP id e7so133920qts.1 for ; Thu, 07 Jul 2022 20:27:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=Fkdge/Nw+hp2LyWDD1RraaWizvaUbNrgq9CNzWE36oI=; b=Z0O4AZvZL310mr4YeJPsjbOoNh8onqFaE/pWH//yE7WVUTEaK2bQ/qUbKFRev6wiWM 7B+3pNjELKQtQ35iL5Ti25V89NS2O/qguKW/B2u0aHlBBJGXWzyl6e5EVnQmQMaKIcfy iDOgUQl+XiigijRCbeSJUA0JQVppMzui5IVJtWiZWooAD7TJAti/V4oKeTF3Q8Z1oBMw LFrc0y1HEyE1yA3GJ0/v3LfEJqvbsGi1c/E646hU2xlWGfeKY7U84PypmiHtZ8oUo9R9 EWzLzVhm8TasMdtEQIi+FBfKdfM02XmmD5gZcASQ05L929yb79/b+eQ0Kw8+y6PA01gR lpyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=Fkdge/Nw+hp2LyWDD1RraaWizvaUbNrgq9CNzWE36oI=; b=E/I0XXkzRc/TDCSEU3FPSvVKPkFaYjc1gHiF7KIcSf0u1sdUgWkimR82nR4DX+cFMn A8DKUaV+PIiOJB976AgNoYlfofggO0U4r9B9p2Ol2uPhIzzrx0qdUMnsH/QpREh535Nl aEpaXCuKR9MEorI2Z/G76L9fIXGCY5IwzT50lRDJcYCNCEDZOEwnDYc0x7bdEAtqajp1 pkLFGT9U81TtoUAEwWHjpC9PMyrBi701hf7A2b838vticUGyT7uNojGCohCVBzQiBjAY fASTq3niakNhCvMteOQZ20/rND5lGPsycdX0r2u22a6i3/mjahu6DwBlhwtNHgFlEDrE Cb9A== X-Gm-Message-State: AJIora9UiSgszxL7LPGVh5rwtphAnHDF0JECriGZE1VCRUYTYMH7kB/s +HCjJrO3ivWFuXWANm7MIzw71g== X-Google-Smtp-Source: AGRyM1vXypjVqTpH1CyAa1HHU80BKivIBbnRrcSR4/b8WpNNVUA/LaHgEMeY/tpwdkEHNhe/iWeXYg== X-Received: by 2002:a05:622a:178e:b0:31d:49dd:c35 with SMTP id s14-20020a05622a178e00b0031d49dd0c35mr1169035qtk.555.1657250862606; Thu, 07 Jul 2022 20:27:42 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id c13-20020a05620a164d00b006a6a4b43c01sm31604577qko.38.2022.07.07.20.27.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 20:27:42 -0700 (PDT) Date: Thu, 7 Jul 2022 20:27:27 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Nadav Amit cc: Andrew Morton , Dave Hansen , LKML , Nadav Amit , Peter Zijlstra , Ingo Molnar , Andy Lutomirski , Thomas Gleixner , x86@kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v2] x86/mm/tlb: avoid reading mm_tlb_gen when possible In-Reply-To: <20220606180123.2485171-1-namit@vmware.com> Message-ID: References: <20220606180123.2485171-1-namit@vmware.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657250863; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Fkdge/Nw+hp2LyWDD1RraaWizvaUbNrgq9CNzWE36oI=; b=08JTSiHLFU+JiG1fHifqEfTTE1vg/utwS0et0cLnvhYOUZauETDPmDJYiz/f1lt1nfjEVP XOFrQBaoXtoTlg680s4baL2icoCKOErPsuGeW8iQH0By4HN/g/8E7SCWxidxH+4MV8NDWL kSPE9meKQPsjtgumcDQL8Pd9NexguM4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657250863; a=rsa-sha256; cv=none; b=sahcVb3bsQB2cyr0x08J1aIvPZ/9PUUkeqf1ZwvfYHBPsjfGgabKUKdaw7ACTAGp/L0Tuu bRXnL0CMM60rmeiRZBwMDC5hPv7vQNRRSXHIzJIy709463+nvyYFhsLIFyrR0wW4Aw9Tyv K6rEyjqHYWSlMiF6IOGIKLuv+zZxt18= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Z0O4AZvZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.160.170 as permitted sender) smtp.mailfrom=hughd@google.com Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Z0O4AZvZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.160.170 as permitted sender) smtp.mailfrom=hughd@google.com X-Rspamd-Server: rspam02 X-Stat-Signature: n75zqwpjixruoer81m78adtkhem746ha X-Rspamd-Queue-Id: 62EDF4001B X-Rspam-User: X-HE-Tag: 1657250863-718452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, 6 Jun 2022, Nadav Amit wrote: > From: Nadav Amit > > On extreme TLB shootdown storms, the mm's tlb_gen cacheline is highly > contended and reading it should (arguably) be avoided as much as > possible. > > Currently, flush_tlb_func() reads the mm's tlb_gen unconditionally, > even when it is not necessary (e.g., the mm was already switched). > This is wasteful. > > Moreover, one of the existing optimizations is to read mm's tlb_gen to > see if there are additional in-flight TLB invalidations and flush the > entire TLB in such a case. However, if the request's tlb_gen was already > flushed, the benefit of checking the mm's tlb_gen is likely to be offset > by the overhead of the check itself. > > Running will-it-scale with tlb_flush1_threads show a considerable > benefit on 56-core Skylake (up to +24%): > > threads Baseline (v5.17+) +Patch > 1 159960 160202 > 5 310808 308378 (-0.7%) > 10 479110 490728 > 15 526771 562528 > 20 534495 587316 > 25 547462 628296 > 30 579616 666313 > 35 594134 701814 > 40 612288 732967 > 45 617517 749727 > 50 637476 735497 > 55 614363 778913 (+24%) > > Acked-by: Peter Zijlstra (Intel) > Cc: Dave Hansen > Cc: Ingo Molnar > Cc: Andy Lutomirski > Cc: Thomas Gleixner > Cc: x86@kernel.org > Signed-off-by: Nadav Amit > > -- > > Note: The benchmarked kernels include Dave's revert of commit > 6035152d8eeb ("x86/mm/tlb: Open-code on_each_cpu_cond_mask() for > tlb_is_not_lazy() > --- > arch/x86/mm/tlb.c | 18 +++++++++++++++++- > 1 file changed, 17 insertions(+), 1 deletion(-) > > diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c > index d400b6d9d246..d9314cc8b81f 100644 > --- a/arch/x86/mm/tlb.c > +++ b/arch/x86/mm/tlb.c > @@ -734,10 +734,10 @@ static void flush_tlb_func(void *info) > const struct flush_tlb_info *f = info; > struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm); > u32 loaded_mm_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); > - u64 mm_tlb_gen = atomic64_read(&loaded_mm->context.tlb_gen); > u64 local_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen); > bool local = smp_processor_id() == f->initiating_cpu; > unsigned long nr_invalidate = 0; > + u64 mm_tlb_gen; > > /* This code cannot presently handle being reentered. */ > VM_WARN_ON(!irqs_disabled()); > @@ -771,6 +771,22 @@ static void flush_tlb_func(void *info) > return; > } > > + if (f->new_tlb_gen <= local_tlb_gen) { > + /* > + * The TLB is already up to date in respect to f->new_tlb_gen. > + * While the core might be still behind mm_tlb_gen, checking > + * mm_tlb_gen unnecessarily would have negative caching effects > + * so avoid it. > + */ > + return; > + } > + > + /* > + * Defer mm_tlb_gen reading as long as possible to avoid cache > + * contention. > + */ > + mm_tlb_gen = atomic64_read(&loaded_mm->context.tlb_gen); > + > if (unlikely(local_tlb_gen == mm_tlb_gen)) { > /* > * There's nothing to do: we're already up to date. This can > -- > 2.25.1 I'm sorry, but bisection and reversion show that this commit, aa44284960d550eb4d8614afdffebc68a432a9b4 in current linux-next, is responsible for the "internal compiler error: Segmentation fault"s I get when running kernel builds on tmpfs in 1G memory, lots of swapping. That tmpfs is using huge pages as much as it can, so splitting and collapsing, compaction and page migration entailed, in case that's relevant (maybe this commit is perfect, but there's a TLB flushing bug over there in mm which this commit just exposes). Whether those segfaults happen without the huge page element, I have not done enough testing to tell - there are other bugs with swapping in current linux-next, indeed, I wouldn't even have found this one, if I hadn't already been on a bisection for another bug, and got thrown off course by these segfaults. I hope that you can work out what might be wrong with this, but meantime I think it needs to be reverted. Thanks, Hugh