From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67C8CC47258 for ; Wed, 31 Jan 2024 10:29:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE60A6B0089; Wed, 31 Jan 2024 05:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D96586B008C; Wed, 31 Jan 2024 05:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE7B86B0092; Wed, 31 Jan 2024 05:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AA6C76B0089 for ; Wed, 31 Jan 2024 05:29:23 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 74A18C0C82 for ; Wed, 31 Jan 2024 10:29:23 +0000 (UTC) X-FDA: 81739234206.18.4B39686 Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) by imf18.hostedemail.com (Postfix) with ESMTP id 550D11C0018 for ; Wed, 31 Jan 2024 10:29:21 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=arndb.de header.s=fm2 header.b=JLTxhWG0; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="D 8RPznj"; spf=pass (imf18.hostedemail.com: domain of arnd@arndb.de designates 66.111.4.221 as permitted sender) smtp.mailfrom=arnd@arndb.de; dmarc=pass (policy=none) header.from=arndb.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706696961; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iKsNv795W6E/GbLdxQpU+VkSSklR3RfYg2NP17KrEuU=; b=FbNc164Snxhx/st0mMk+rC5S/BJDiXTEpl4V8EDLf7JaSNcQViczdlOD6zch2MONGlYMbK SU8uutUg37r8GiKbswMyYOedOlRRVc2k93oWfY43Wt342/UNvhUW5QVSRFbr9DafS3nkrp tQxCvHI2+b4Fdfl0+z4xRQLoUn9wgJA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706696961; a=rsa-sha256; cv=none; b=A4eN5Bv3Ag9mfL5dXlSFQYc+pz3frVgcDeXewjFCItgSH6HcV55LZgGEhpR7hMLzBP9OsI GZ03JLqMCfLFRJ5Y+JEDeRih4M/xWDbNX6uYkCONSdgZuiuqfqVawrvUq3Kne3cHBBmb/w OIvBWbQ3GzKXbdecLgsrE0DD0pF4GwY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=arndb.de header.s=fm2 header.b=JLTxhWG0; dkim=pass header.d=messagingengine.com header.s=fm3 header.b="D 8RPznj"; spf=pass (imf18.hostedemail.com: domain of arnd@arndb.de designates 66.111.4.221 as permitted sender) smtp.mailfrom=arnd@arndb.de; dmarc=pass (policy=none) header.from=arndb.de Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailnew.nyi.internal (Postfix) with ESMTP id 51C26584FB1; Wed, 31 Jan 2024 05:29:20 -0500 (EST) Received: from imap51 ([10.202.2.101]) by compute5.internal (MEProxy); Wed, 31 Jan 2024 05:29:20 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=arndb.de; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1706696960; x=1706704160; bh=iKsNv795W6E/GbLdxQpU+VkSSklR3RfYg2NP17KrEuU=; b= JLTxhWG0Z9hIr/3XbhRbYsJwWiCHv+FVO+pTjbIITWydr9qM3b8HsQs0pIReCjDi 3RiGFjmNoJ4CdhwLrLca9sEDEZE29naSHJ6RTHBe3HwBLUx///9zcq1z2WhncZPF ESHVAQIXyxAUSt6KVv8HxBryiw7q/ljwZFyJYl6GZi2XqgQd7dG9DT0NbSMUqxXr Bg3Slm3BWjhSUR2Lmep4wMtmW5juwp4ZKxFaRYOHXpDm2LbwZK7o7GzfAfFGSOss lFq55FfrU3YdFbb/M83Sm/37KhWhRpFn7PjS53XwKTk3qFq031VU283dv84K7S15 gTuCfcjWtIbfHcRGYXN25Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1706696960; x= 1706704160; bh=iKsNv795W6E/GbLdxQpU+VkSSklR3RfYg2NP17KrEuU=; b=D 8RPznj1JPIm3sJJASdG9GOieBFePMt861KHPTegWEprstweYS8VJo0HJY67r1r3x Xi5+flrjO1ap7pGFwfqnlcxY/2EmBjQfY3X+24KgqrujuyBeUKaaQDfArDLcQ621 9X6ioXHVW0KuphFt2vC5H68+KMDPXsT68FD/Ie015N+moT9oFoSNqSvb1aWyjckp XtbSxYibx21a4qkFvEohLzOf9MT9HmCwkhuEcAngg4ElhXSKC/JRXWd8ZnoRu1YX rYwfSTaqo0SSdigwpZB3m4Nmj5XoRUGR86jm/41CZiqz0B05GRD//1fPlerSTpXx qLDPIYx+dZCDG76xo/LQg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrfedtledgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtgfesthhqredtreerjeenucfhrhhomhepfdet rhhnugcuuegvrhhgmhgrnhhnfdcuoegrrhhnugesrghrnhgusgdruggvqeenucggtffrrg htthgvrhhnpeegfeejhedvledvffeijeeijeeivddvhfeliedvleevheejleetgedukedt gfejveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe grrhhnugesrghrnhgusgdruggv X-ME-Proxy: Feedback-ID: i56a14606:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 44453B6008D; Wed, 31 Jan 2024 05:29:18 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.11.0-alpha0-144-ge5821d614e-fm-20240125.002-ge5821d61 MIME-Version: 1.0 Message-Id: <1036060c-25dd-439b-b081-893d0cb000f6@app.fastmail.com> In-Reply-To: <269edff0-d989-4ac8-b0c3-bce31283806b@kalrayinc.com> References: <20230120141002.2442-1-ysionneau@kalray.eu> <20230120141002.2442-32-ysionneau@kalray.eu> <995eb624-3efe-10fc-a6ed-883d52d591bb@linaro.org> <269edff0-d989-4ac8-b0c3-bce31283806b@kalrayinc.com> Date: Wed, 31 Jan 2024 11:28:57 +0100 From: "Arnd Bergmann" To: "Yann Sionneau" , "Krzysztof Kozlowski" , "Yann Sionneau" , "Jonathan Corbet" , "Thomas Gleixner" , "Marc Zyngier" , "Rob Herring" , "Krzysztof Kozlowski" , "Will Deacon" , "Peter Zijlstra" , "Boqun Feng" , "Mark Rutland" , "Eric W. Biederman" , "Kees Cook" , "Oleg Nesterov" , "Ingo Molnar" , "Waiman Long" , "Aneesh Kumar" , "Andrew Morton" , "Nicholas Piggin" , "Paul Moore" , "Eric Paris" , "Christian Brauner" , "Paul Walmsley" , "Palmer Dabbelt" , "Albert Ou" , "Jules Maselbas" , "Guillaume Thouvenin" , "Clement Leger" , "Vincent Chardon" , =?UTF-8?Q?Marc_Poulhi=C3=A8s?= , "Julian Vetter" , "Samuel Jones" , "Ashley Lesdalons" , "Thomas Costis" , "Marius Gligor" , "Jonathan Borne" , "Julien Villette" , "Luc Michel" , "Louis Morhet" , "Julien Hascoet" , "Jean-Christophe Pince" , "Guillaume Missonnier" , "Alex Michon" , "Huacai Chen" , "WANG Xuerui" , "Shaokun Zhang" , "John Garry" , "Guangbin Huang" , "Bharat Bhushan" , "Bibo Mao" , "Atish Patra" , "Jason A . Donenfeld" , "Qi Liu" , "Jiaxun Yang" , "Catalin Marinas" , "Mark Brown" , "Janosch Frank" , "Alexey Dobriyan" , "Julian Vetter" , jmaselbas@zdiv.net Cc: "Benjamin Mugnier" , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, devicetree@vger.kernel.org, linux-mm@kvack.org, Linux-Arch , linux-audit@redhat.com, linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: Re: [RFC PATCH v2 31/31] kvx: Add IPI driver Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 550D11C0018 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 17t9sep7it7g3ca9trhcsc3q4u65nj43 X-HE-Tag: 1706696961-941232 X-HE-Meta: U2FsdGVkX1/6FVQpSdvXrTgY3ZHlS4JaUHUsE+PLg1X7wfLjHg6PnvhZeiwph5xg/u0fB5rnSXIEJ9MJ0T8nN1Pbo131wrEXYwLoJqAyIIT2SvVBN4IAn6bsQpr0ZAtW4E2K12fnqPMvVEhNMc3gG6IF2gUug55AGZwsPMM0blpp3rglMWSXmogGSR4Hf+JLw8wyJduI/O9YpbLuFpf51GsLMYYeQK4kUk+Q8pNQ5f1Ji7kFrJoclJkGuNJ1M26jQ1Ox6klRfXkt5rOr7K5iVOlE1W29shaUNsXxtVFRljuw65a9eBgCWSTiCZgjYmY2dLuSsdnOapdxEOLilCUvdLsgGvkGF5uCfJ2WRdTk1iGlDDkhGWp3g9EHG0VLTR05noK8MJDAHO6RHwg7C/4Pd8u5QCKc25WclKSEPcJCZ0f8LkEeekMWjRDcemTVAdIjg6o8JiEF/g3bSWO+NtEnIQMeSg3me20Nj3jW+kvhiz/sMJIGedxkalxpNJKWVf9ThOslmwnI5Tg1a96kWyoaRfEevx3Lvt3vubNbJ+b5H1gV2qUryzxKn+XsDS1BifbJ+RaZKMkjcEeojf5HSFjpdRzYTGY/+sTz+YGGersbmKAFxMSbxoiXDWDzNr4lZkTvCN75e/ncw8DtZqp3LVDpSlMjzaOHK16DuJAG4uAtsU8KFMQiOfc9Wg0PfympuAsoeyS/kySJM1LOcqDerVuz+g+FWe3GvZhUI2g9sdz3jF1BiV+N0osSnQwJd2iUK9mBVfdcBlTP/Hfw8tznCnVM14Rky3VW9htCrvh3ZmOhf6P43m/er6K+wqu7KDvPsFpvxL2s0RhXAiL/jbTPzhXELYPS+54lf5haGczIpoZa1cKstLMo/Wdd2U3VZ578GkUvOV6G9ZF8Atb58o0dQlhNXxzYiNZeU3nKo6mUxjMqfHTddE+BIFWXSToPxUpoNZZL2aWTc+Es4PdyQra8XF5 88pk9ktJ zNEWjDuPMFfAe2WDiwRVqbjsFJueIg0T4vs5Ceg2V0wrHNW3eC9njT6kdk+yApHdTu0lCMEONJnn9wcoFKvJMMClHYNfrzeNpRUezsh/j3jAOpdQ3/+jUDrjl0o1I4cf5vjHwS33taYDkIl8dOgBrUAUZHU8UPTRri7ZSTDD5u6cvMwBE4LQ1Qc7kUOzAHq+o2yned6+GmyMPnKgJ+KgI/r9fJ/cCRvDcR5ZRoetUTeToDNnafAZJo1Le9yv28cp5DOx32EY/y8gX2Tn4WoymdYkhsTIsrFn2Ix6x X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 31, 2024, at 10:52, Yann Sionneau wrote: > On 22/01/2023 12:54, Krzysztof Kozlowski wrote: >> On 20/01/2023 15:10, Yann Sionneau wrote: >>> + >>> +int __init kvx_ipi_ctrl_probe(irqreturn_t (*ipi_irq_handler)(int, v= oid *)) >>> +{ >>> + struct device_node *np; >>> + int ret; >>> + unsigned int ipi_irq; >>> + void __iomem *ipi_base; >>> + >>> + np =3D of_find_compatible_node(NULL, NULL, "kalray,kvx-ipi-ctrl"); >> Nope, big no. >> >> Drivers go to drivers, not to arch code. Use proper driver infrastruc= ture. > Thank you for your review. > > It raises questions on our side about how to handle this change. > > First let me describe the hardware: > > The coolidge ipi controller device handles IPI communication between c= pus > inside a cluster. > > Each cpu has 8 of its dedicated irq lines (24 to 31) hard-wired to the= ipi. > The ipi controller has 8 sets of 2 registers: > - a 17-bit "interrupt" register > - a 17-bit "mask" register > > Each couple of register is dedicated to 1 of the 8 irqlines. > Each of the 17 bits of interrupt/mask register > identifies a cpu (cores 0 to 15 + secure_core). > Writing bit i in interrupt register sends an irq to cpu i, according t= o the mask > in mask register. > Writing in interrupt/mask register couple N targets irq line N of the = core. > > - Ipi generates an interrupt to the cpu when message is ready. > - Messages are delivered via Axi. > - Ipi does not have any interrupt input lines. > > > =C2=A0 +---------------+=C2=A0=C2=A0 irq=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 axi_w > =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 i=C2=A0= |<--/--- ipi <------ > =C2=A0 | CPU=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 n=C2=A0 |=C2=A0 x8 > =C2=A0 |=C2=A0 core0=C2=A0 |=C2=A0 t=C2=A0 | > =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 c=C2=A0= |=C2=A0 irq=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 irq=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 msi > =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 t=C2=A0= |<--/--- apic <----- mbox <------- > =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 l=C2=A0= |=C2=A0 x4 > =C2=A0 +---------------+ > =C2=A0 with intctl =3D core-irq controller >=C2=A0=C2=A0 =C2=A0 > > We analyzed how other Linux ports are handling this situation (IPI) an= d=20 > here are several possible solutions: > > 1/ put everything in smp.c like what longarch is doing. > =C2=A0 * Except that IPI in longarch seems to involve writing to a spe= cial=20 > purpose CPU register and not doing a memory mapped write like kvx. > > 2/ write a device driver in drivers/xxx/ with the content from ipi.c > =C2=A0 * the probe would just ioremap the reg from DT and register the= irq=20 > using request_percpu_irq() > =C2=A0 * it would export a function "kvx_ipi_send()" that would direct= ly be=20 > called by smp.c > =C2=A0 * Question : where would this driver be placed in drivers/ ?=20 > drivers/irqchip/ ? Even if this is not per-se an interrupt-controller=20 > driver? This looks like it's close enough to the irqchip driver that you can just have it in the same file as the 'intctl' portion. Top-level irqchip implementations tend to be rather architecture specific, as does the IPI mechanism. Depending on the register layout, I think you can have a single devicetree node for the combination of the core-irq (for managing your own interrupts) and the ipi (for receiving interrupts from others), and then have a driver in drivers/irqchip to deal with both. For the ipi mechanism, trying to abstract it too much generally makes it too slow, so I would not go through a nested irqchip or a mailbox driver etc. I don't know what the 'apic' in your diagram is, so that would be either a nested irqchip or could be part of the same driver as well. Arnd