From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8435C83F1D for ; Thu, 10 Jul 2025 10:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 578786B0089; Thu, 10 Jul 2025 06:56:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 529546B008A; Thu, 10 Jul 2025 06:56:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43F2D6B008C; Thu, 10 Jul 2025 06:56:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 33E7E6B0089 for ; Thu, 10 Jul 2025 06:56:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DA23B58505 for ; Thu, 10 Jul 2025 10:56:46 +0000 (UTC) X-FDA: 83648052012.09.44EDF72 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 206C612000F for ; Thu, 10 Jul 2025 10:56:42 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Q53Dg/bF" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752145005; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7pEA1uNTqPvv6+XR/zuY/aMB8/9LJc6Rsl4Iqenx7gQ=; b=oRqFVqkKkHbbRFpybVpbLjPacKv7n1dX54926Pr+FUngro8Y8VwZXUWWYRZzclTHaDHH89 reXGEpWtkVczD94SSG0kvnzqiBFK62GFjSBFW/gTF+JZee3ZaZaGLOyteSn2heQbRMCWjJ DoV8530W4ol/eWxMndPFeu9ZBorVNyo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Q53Dg/bF"; spf=none (imf29.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752145005; a=rsa-sha256; cv=none; b=S+Q0NWgHSj9xLXfxs+ddSUBBbSI7gvXTVaEs1MGnrVChnnFDRR6W7YwIf+08v6qyy4m0W+ I30zKi7GY17gjfPX7QS9fuK3531AJIkiPJEO2FjMnWwy+SDFuefT11J9QboTI4zoIVpNep 18r0dIHkYM//+oZpAaD/lUvvk1AA0y4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=7pEA1uNTqPvv6+XR/zuY/aMB8/9LJc6Rsl4Iqenx7gQ=; b=Q53Dg/bFa0IOiKKDOS+53JMqLd rWu6gVsz2DbYQS7/ZS4GsN7sNLBcFe5gtoDTA70AVg1E6+YN+gLVdwXFrHrG7RJBHNdiLIHL2Hqig eJ+2QJCO3siJAdamLStruKmbDre9sLI7jMYWnF5BCA5rSs1oL0ob/BF6Fe8Z63IeS5ED3rgQepKHw Rp9hPkGAAcNYMOaycRM3nSDZknPNn/2oJ9rQv0kHpzKQaU4FkLsK9SaAIbtD4844S1AGIiqdNQ/1B I/5ta9es7JbfQwyn1KjbLHwNTOA5vk9C05tctaYwRZOGHrIASCwI7FNGr7RW4hxkUIp3aRH6mBZtm iTMs9ePg==; Received: from 77-249-17-252.cable.dynamic.v4.ziggo.nl ([77.249.17.252] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.98.2 #2 (Red Hat Linux)) id 1uZowy-00000008aeW-27gb; Thu, 10 Jul 2025 10:56:24 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 1000) id 0B932300125; Thu, 10 Jul 2025 12:56:23 +0200 (CEST) Date: Thu, 10 Jul 2025 12:56:22 +0200 From: Peter Zijlstra To: dan.j.williams@intel.com Cc: "H. Peter Anvin" , Jonathan Cameron , Catalin Marinas , james.morse@arm.com, linux-cxl@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, gregkh@linuxfoundation.org, Will Deacon , Davidlohr Bueso , Yicong Yang , linuxarm@huawei.com, Yushan Wang , Lorenzo Pieralisi , Mark Rutland , Dave Hansen , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, Andy Lutomirski Subject: Re: [PATCH v2 0/8] Cache coherency management subsystem Message-ID: <20250710105622.GA542000@noisy.programming.kicks-ass.net> References: <20250624154805.66985-1-Jonathan.Cameron@huawei.com> <20250625085204.GC1613200@noisy.programming.kicks-ass.net> <20250625093152.GZ1613376@noisy.programming.kicks-ass.net> <686f4e20c57cd_1d3d100b7@dwillia2-xfh.jf.intel.com.notmuch> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <686f4e20c57cd_1d3d100b7@dwillia2-xfh.jf.intel.com.notmuch> X-Rspam-User: X-Rspamd-Queue-Id: 206C612000F X-Rspamd-Server: rspam09 X-Stat-Signature: hkqyghc98kancn388npon36jeengjfcj X-HE-Tag: 1752145002-507541 X-HE-Meta: U2FsdGVkX1+zaWsZGQnpd2ChWWEOWXy0ryL3AvVaiMrxaPoWIfA8TUNs1LTpOrTXLCbeXshdcV1gsV2+eQJtfH5PGCZxNu31rvQGggHT2KA1xUDZoY7zT/KFqLVQyNlPrH2xE20EaH4VWYGtZ+YMCMTAJMB2Y2Q530TxHMC94lAW+A7b16600ooCjI7FUSuHfbRx9dBagz3kEBbLkOe5tO36gTcyV4dJo9Z5Cqr3EbQadLkt+c0WA+MdHmoplKXGRRCO2MC1tLx7pLzmHq1YhQlOuZza4jZQ5YyYF6eOjYW8Ia01kldndHtMrDyQfRD5A6BtQfziu0nNHj2jSpmvrQ4NvKuVefhTEl6jpMZ0fQ/MpkjVJIIywBf2u1WJbVKDMN5laUw7IrzZcA3nfTsbvGKhljAgrtdVx93uRi6Hyz8u1M2iknNuKLOD87KhEUth8l8yDzXbUiHJVa90Uz2P7X/K84jy/wDTPnbGJ8RCDgVQGFZzUYmVEshMH/f22lJoSnSGwalPdJc59AmzrK7HIUg1YrQAgrFNcvwktz1DxM7yWZBkfAyzodRWeXcH8ckiPUq2JMYJ6raUH5enB9z4ykAs8Fzq0HoQ5E9GbBN9goTAc8C6ps3WJAcNmqshd7qtRCWrL9bE9UyzH/B4CgDSOQ6ElxNhaQWFshIxLmOfJgEvVA4424g78Omf9sYI9pll5ONoq4s6JZvQgUteyAR6c92ICp+pikzJN5ve1NlFO3zSAMN6fxWwgk5emkUTz7D55y4paJIejQJmuefmE6VwW+qohgUs1PmQHVgZNGpQy5kUmNY+D9mHvpp9ljSwXh8QXCJA80zt1B5q7MqHGulpVkTbZshJLLFd3RGw5m+t2buyu7fOOv2for0CTXyq2FAB6BQVx7K2Q+2vdLXM2CRDOuLb6918vgO8BuovYapPZjKYLpPpSPdYsyp4thpytdp4mBI5MQ2+9Cv1mwlRYmL yaOxndOM SneAR8WWIeVUCOfHgyV8Czw7Gnkp/4W9trc48SOxMExm/Rj+nS8stKXFDW6pmcwClr+8lPYkpFl6xfvHlT+PJklCqTP/BDQQBMn/R/No4eoKSypBLpcVb8BZtM0SNi48ZsFvtYbS+em0hgsDFyUzkCUalSWmtWvis/BOI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jul 09, 2025 at 10:22:40PM -0700, dan.j.williams@intel.com wrote: > "Regular?", no. Something is wrong if you are doing this regularly. In > current CXL systems the expectation is to suffer a WBINVD event once per > server provisioning event. Ok, so how about we strictly track this once, and when it happens more than this once, we error out hard? > Now, there is a nascent capability called "Dynamic Capacity Devices" > (DCD) where the CXL configuration is able to change at runtime with > multiple hosts sharing a pool of memory. Each time the physical memory > capacity changes, cache management is needed. > > For DCD, I think the negative effects of WBINVD are a *useful* stick to > move device vendors to stop relying on software to solve this problem. > They can implement an existing CXL protocol where the device tells CPUs > and other CXL.cache agents to invalidate the physical address ranges > that the device owns. > > In other words, if WBINVD makes DCD inviable that is a useful outcome > because it motivates unburdening Linux long term with this problem. Per the above, I suggest we not support this feature *AT*ALL* until an alternative to WBINVD is provided. > In the near term though, current CXL platforms that do not support > device-initiated-invalidate still need coarse cache management for that > original infrequent provisioning events. Folks that want to go further > and attempt frequent DCD events with WBINVD get to keep all the pieces. I would strongly prefer those pieces to include WARNs and or worse.