From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 450B8C83F1A for ; Thu, 10 Jul 2025 19:17:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDA9D8D0002; Thu, 10 Jul 2025 15:17:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB23F8D0001; Thu, 10 Jul 2025 15:17:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA1148D0002; Thu, 10 Jul 2025 15:17:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B39078D0001 for ; Thu, 10 Jul 2025 15:17:17 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 546B25D77D for ; Thu, 10 Jul 2025 19:17:17 +0000 (UTC) X-FDA: 83649313314.12.D1D268E Received: from mail.zytor.com (terminus.zytor.com [198.137.202.136]) by imf13.hostedemail.com (Postfix) with ESMTP id 508B020011 for ; Thu, 10 Jul 2025 19:17:15 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=zytor.com header.s=2025062101 header.b=H+2YFCTn; dmarc=pass (policy=none) header.from=zytor.com; spf=pass (imf13.hostedemail.com: domain of hpa@zytor.com designates 198.137.202.136 as permitted sender) smtp.mailfrom=hpa@zytor.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752175035; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dK3+vVfqOJwgcvbSOXlTrXE17g//6OeCs7hq/OKo4hU=; b=fLCJxL6kZD1XJfhx+yYikszkz+jzO46MpL+4QFpttCbruhj/xZCKKhTycwGYzfgjHUedr2 SnpTyiVlFG8F9Jui8pT2pzooNHSpyoO22BhNy2GJCG+uGZDWDIjRoujNhyQ1R6YPkO3gBq ZjfcdwSE2sk8l7FL1AMD1FHuWA+aIhY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752175035; a=rsa-sha256; cv=none; b=Ix0oE+zZ94yiCrhnQfiIsQY606quUvEOOWbsH4DZtrNVaKsKaoUm2zqaGTwC7nOuly+JOP fjDIVMRZ0SJO38JK2bvVRmx2iNMEjK+0sDhmDOKPxWrBwL4ahbGQF9JZoV9FbUceL7BzFQ 6u3Srdbtz+rCRjxvF3gbl+5vtGs8DL0= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=zytor.com header.s=2025062101 header.b=H+2YFCTn; dmarc=pass (policy=none) header.from=zytor.com; spf=pass (imf13.hostedemail.com: domain of hpa@zytor.com designates 198.137.202.136 as permitted sender) smtp.mailfrom=hpa@zytor.com Received: from [127.0.0.1] (c-76-133-66-138.hsd1.ca.comcast.net [76.133.66.138]) (authenticated bits=0) by mail.zytor.com (8.18.1/8.17.1) with ESMTPSA id 56AJGXME778639 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 10 Jul 2025 12:16:34 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 mail.zytor.com 56AJGXME778639 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2025062101; t=1752174995; bh=dK3+vVfqOJwgcvbSOXlTrXE17g//6OeCs7hq/OKo4hU=; h=Date:From:To:CC:Subject:In-Reply-To:References:From; b=H+2YFCTnw5z6ZLmuZ9B1BVWch1D/9ZhhVQGNSyZbDzQrCCeVVzHocaCB1sQi4QU9r 0C7ngFQOxCuhCK+pcoHRI+6BB6C8lJ+vTL5qIPXuMNidq+Mqe4sCI6TX74RFBOwba2 vtVKDGjVg5M0FIXvySLLNsmT+JOjBN45GKe97AOAYfPpsWgNXzUPSepEbr2jmaRovM ck1NS7DVSImoOUSNby0QoW+XP/8Z1TSEFUdvPKEwZdEV6dfXhdXJMi717yROJRIbEJ d1PaNAZeAHojU682AVXBZ2cfMBYha32WzVsXtbTSuT7T46yVhu/ug35jtxbCOVOA2K +zI/CfaUj+IPg== Date: Thu, 10 Jul 2025 12:16:33 -0700 From: "H. Peter Anvin" To: dan.j.williams@intel.com, Peter Zijlstra CC: Jonathan Cameron , Catalin Marinas , james.morse@arm.com, linux-cxl@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, gregkh@linuxfoundation.org, Will Deacon , Davidlohr Bueso , Yicong Yang , linuxarm@huawei.com, Yushan Wang , Lorenzo Pieralisi , Mark Rutland , Dave Hansen , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, Andy Lutomirski Subject: Re: [PATCH v2 0/8] Cache coherency management subsystem User-Agent: K-9 Mail for Android In-Reply-To: <68701051ec185_1d3d1001d@dwillia2-xfh.jf.intel.com.notmuch> References: <20250624154805.66985-1-Jonathan.Cameron@huawei.com> <20250625085204.GC1613200@noisy.programming.kicks-ass.net> <20250625093152.GZ1613376@noisy.programming.kicks-ass.net> <686f4e20c57cd_1d3d100b7@dwillia2-xfh.jf.intel.com.notmuch> <20250710105622.GA542000@noisy.programming.kicks-ass.net> <68700a5428a2f_1d3d1008b@dwillia2-xfh.jf.intel.com.notmuch> <575B5DF2-AE1D-43E9-9A4B-09FB78EFFC43@zytor.com> <68701051ec185_1d3d1001d@dwillia2-xfh.jf.intel.com.notmuch> Message-ID: <4CA4EB2A-8F45-4596-ADB9-B0C2307316B1@zytor.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 508B020011 X-Rspamd-Server: rspam03 X-Stat-Signature: 6bdkk8kt1jie3i18tbmjdke4gi6mbput X-HE-Tag: 1752175035-314089 X-HE-Meta: U2FsdGVkX19faLi+yBktLzCe9kicSOlGcMVM2Z8TrWZxz6uwMHNlzdC0o2F5emivNwJ8e03n7COoAjUNUU6uxcSwBlx+WWrrS9VWkC+A7X9pnd7DAW/8J2owT+QYQ3v+aSZMlpUi1yyztSsgPw49qiAO8oTCPRFs7WS8jM6t/AV7ZnXuk4vlEqCx7fYyRNYHLS4EVzmDhZ8dn1miCun0gWLOVjHek7DBp7/CeAWmDcS+9n2SO0OjgNBwDG6FzCb/vxCajNeAw7qdjqeVEiT4wtBxxMEqHRIlzPWTxUSwm+uj2aCCZH7Tu1Ni7plPjkDR8wZ44N7Y9YLjRVPuvXBQ+SPDAKmvXqyRSjTTf8MJiF4Sqc/5ZVTwcXArU6ehUBt7wWkT0EogDcA4RQqhiRUR8pmFfm60j0nbegy4hZIovcx/Aqt+Gm8Z6+2BaG7FGYGCLd+YsN3f7z6VydsyZqH6/Mj0itJfCvo2YT3sEEArE4yLmgbx/O7DqIHnsCN3xoENO/DqKBF46C+kihr73cdi7EbTL3uwrIOOO6Y1EWRCId9K5MrTjNZyZiNL0hjOWoJDnC4Op47lWnVi13Ixa75SCx1MMDFTBjlMgZ4kCefE3TO5V8amlmnQml6tnt7lNq74QLAhEJE78mrQaP0aQ1/WKIbnKircTe7SgMA9YMC+mnz0idXjhHqvOfkP601+bPUI54uVXPsZZoLJRnJCFxupO/rgX9KiCqdwNDo/FNQZCjbdwwQ2A/wJofHHZ8ePXCL+snL5b5yKz9hnQmbhBwYSWlDFj+iUCmIy5y57h+uSFdOPXuyXZRZcFxaIsT57cDsaymlkyiwHte80XZtkstecTLaiGLPBRsu6l88R5qOVeYpwyqF6ESpPYFHMKirhDVr5/w6kWrbwKtTKjZm7G1aDB3e9t5B43KVD6BphvkQ2WSZvBncH1fzBQ4TfeTTwMAk/rh9HtmsxPv5Xbao2eGn oNtRXQCV 3IzlDjE9DMti0/ukVBZZCd14IZzpemh3VWfLxPwYZ+I1qBPB0AFfK/FeHaGYK+O7UA/NIunNNVUNgp22IC2p0uSou8NL81yH1IgREyDg/teZY+DS2R3EPTZ2crEnhUGeA1DvYeEs19ys0SzFpvuA9W5L+etzdddvQBgNCBYjq+wmTw8IAkR7c+oOVkZ4oUDqhhYXdbZbaSAECWOm4GffIyzuXeqYkUefS0GidfUH6VkIlPHtaZcoTHWQLGxIZ7fEqg7UISuoy1Ej8fi99ehd6ZWzYK9oQOJzfQahaklTpviWYrSNd2n4Iqvnu6Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On July 10, 2025 12:11:13 PM PDT, dan=2Ej=2Ewilliams@intel=2Ecom wrote: >H=2E Peter Anvin wrote: >[=2E=2E] >> >> > In the near term though, current CXL platforms that do not support >> >> > device-initiated-invalidate still need coarse cache management for= that >> >> > original infrequent provisioning events=2E Folks that want to go f= urther >> >> > and attempt frequent DCD events with WBINVD get to keep all the pi= eces=2E >> >>=20 >> >> I would strongly prefer those pieces to include WARNs and or worse= =2E >> > >> >That is fair=2E It is not productive for the CXL subsystem to sit back= and >> >hope that people notice the destructive side-effects of wbinvd and hop= e >> >that leads to device changes=2E >> > >> >This discussion has me reconsidering that yes, it would indeed be bett= er >> >to clflushopt loop over potentially terabytes on all CPUs=2E That shou= ld >> >only be suffered rarely for the provisioning case, and for the DCD cas= e >> >the potential add/remove events should be more manageable=2E >> > >> >drm already has drm_clflush_pages() for bulk cache management, CXL >> >should just align on that approach=2E >>=20 >> Let's not be flippant; looping over terabytes could take *hours*=2E But= those are hours during which the system is alive, and only one CPU needs t= o be looping=2E > >Do not all CPUs need to perform the invalidation for L1 copies of the >line? > >Not trying to be flippant, but if wbinvd is only a one-shot per Peter's >proposed policy and the system experiences another CXL reconfiguration >event, then looping is the only option or fail the memory plug event=2E > >> The other question is: what happens if memory is unplugged and then a >> cache line evicted? I'm guessing that existing memory hotplug >> solutions simply drop the writeback, since the OS knows there is no >> valid memory there, and so any cached data is inherently worthless=2E > >Right, the expectation is that unplug is always coordinated and that >surprise unplug is unsupported / might lead to system instability=2E > > CLFLUSH goes through the cache coherency protocol and is therefore system = wide, which WBINVD is not=2E