From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0CABC05027 for ; Wed, 8 Feb 2023 18:04:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 304B16B0072; Wed, 8 Feb 2023 13:04:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B5E46B0074; Wed, 8 Feb 2023 13:04:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17CCD6B0075; Wed, 8 Feb 2023 13:04:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 090506B0072 for ; Wed, 8 Feb 2023 13:04:16 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A794EABC47 for ; Wed, 8 Feb 2023 18:04:15 +0000 (UTC) X-FDA: 80444898870.24.6AF8C8A Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) by imf19.hostedemail.com (Postfix) with ESMTP id A8FF51A000A for ; Wed, 8 Feb 2023 18:04:11 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=YQt7VUiE; spf=pass (imf19.hostedemail.com: domain of viacheslav.dubeyko@bytedance.com designates 209.85.219.49 as permitted sender) smtp.mailfrom=viacheslav.dubeyko@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675879452; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KF7M+Kj3n0+D3QyEmfqAMrT5IWsgyIgzncbmovOdPhI=; b=d7YdakPut2I2ZRi5n4FI78eVwt3rnu+Goer1ND9ZFt6EONnFWVeK8VlDGDbXmoa+XIF/L4 /UZ7GnvlGA6cuj2pQNutuaYoGfp5QIsmwxeKTwzjka+1UPxa5AeBYYakcqQNUG7bHo+v1z /AodCkhuKt61wy4JoCidZPXZ8yRJH9Q= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=YQt7VUiE; spf=pass (imf19.hostedemail.com: domain of viacheslav.dubeyko@bytedance.com designates 209.85.219.49 as permitted sender) smtp.mailfrom=viacheslav.dubeyko@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675879452; a=rsa-sha256; cv=none; b=o5WkNbve8hp8Sb1+Zrs5xWHpmkW7ZzBBeE8n9WDUGjYXUQhFQTpnWHHinNA2UrBr2cV7KS ticui/eZ04U5gvBX2r5R7WNXaRJQ78TK461UTUDdFzuIQujdw/KNylB9yE8t6l/izMmHLE DTLa+C0alR/66HF5eIwa+jesUJ1JDEM= Received: by mail-qv1-f49.google.com with SMTP id i1so8275156qvo.9 for ; Wed, 08 Feb 2023 10:04:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KF7M+Kj3n0+D3QyEmfqAMrT5IWsgyIgzncbmovOdPhI=; b=YQt7VUiE3eFSEFVaeUdNcB1FuoIgWysGFfLvmDD3t/ImN+bvSKA41/y7wy+oxAHQpY EYyOrs2l29SHFuR1GWTo8RvLw9t9pmr7nnUc98Zd9l8HnAlBY5LRPpeIiXkMMxwmcaBG 20NRE5zi8QcDIeTUTzSI5uDkA1mL1PnRHX35tnxtZqeBUPg874Q8AtHuJdIPiHQG4pmw xoCXIBi+kC8cX0w2EhXZ4dhK9oOHi4mmxOfFtGJ/T5zSd0YfiISO6OPh+upkozeUfeHF bW6AvMqPYfkAMf25JdDvlwf+nxtgPVT1MRxb+sSLDFMSEwFO9KRiIITQAoxxk582hTms CEzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KF7M+Kj3n0+D3QyEmfqAMrT5IWsgyIgzncbmovOdPhI=; b=jnU148beeTowvrkvfiy6srXFj2MUeEavdlu0l8Md1qeiCTTIdfOPtUwENUQWWlztsU nYDYTKILj2EP4Gi8JsNhvHGLnwSdMWb4VDlGw/L9UBngv9nDSxSHNtDAi7lL+MCnA65v UlbzpCegYqzQgh4WOD1oFL1USh1C3xUMhD+nLdpbpRLdzRkXMYYS4OOYxplS1/fUkbZs gMuks8qjbwMY934fkgLl8VoPr9QQHqjDNYfkxN+2oCzHOiH52zTzN5yXkjRh9biVhesC xFpprZFGwHHhN/eI04CTi0gTcgMRdEJvcdmJh5nq3xSK0SqInnItHIPWAYkBbJsUYa7h NZ/g== X-Gm-Message-State: AO0yUKWZc9Wy8rlp4gkZ6XcfMEZiGvm7JLm0ExW83YFbGy0pdfGV/Lyk J4HcDJM9yjUGwsrUWWwIMe6WNsKu0cE1Xgmv X-Google-Smtp-Source: AK7set94oQ93TcgiV4kB2SnWYkbxOL75GEa3U+AN9OGYy6F76zXtm1RRbjgjsNT/gUZJ8pZ8FXbrcw== X-Received: by 2002:a0c:c683:0:b0:56c:1b8e:826e with SMTP id d3-20020a0cc683000000b0056c1b8e826emr2562806qvj.15.1675879450780; Wed, 08 Feb 2023 10:04:10 -0800 (PST) Received: from smtpclient.apple (172-125-78-211.lightspeed.sntcca.sbcglobal.net. [172.125.78.211]) by smtp.gmail.com with ESMTPSA id n2-20020a05620a294200b006f9ddaaf01esm12544061qkp.102.2023.02.08.10.04.09 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 08 Feb 2023 10:04:10 -0800 (PST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: Re: [External] [LSF/MM/BPF TOPIC] CXL Fabric Manager (FM) architecture From: "Viacheslav A.Dubeyko" In-Reply-To: <20230208163844.GA407917@bgt-140510-bm01> Date: Wed, 8 Feb 2023 10:03:57 -0800 Cc: Jonathan Cameron , "lsf-pc@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-cxl@vger.kernel.org" , Dan Williams , Cong Wang , Viacheslav Dubeyko Content-Transfer-Encoding: quoted-printable Message-Id: <7E864E85-A36F-487B-8B70-C8C49FBECD73@bytedance.com> References: <7F001EAF-C512-436A-A9DD-E08730C91214@bytedance.com> <20230131174115.00007493@Huawei.com> <5671D3B3-83B3-49FF-A662-509648E6D297@bytedance.com> <20230202095402.0000585d@Huawei.com> <20230208163844.GA407917@bgt-140510-bm01> To: Adam Manzanares X-Mailer: Apple Mail (2.3731.300.101.1.3) X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A8FF51A000A X-Stat-Signature: j5ab661z6fm1ncwxttq5engkc6bjesb1 X-HE-Tag: 1675879451-257097 X-HE-Meta: U2FsdGVkX18h8vj3LqdRZvX9AEbhi/1xWFk7l7Ij3fe5nXR6uHy6HQXDNXemV7wsKQ0/uavju3WqnakqhWCRkb1A0dnd5pTiuS9xST1w2Fwhyza1JttDadT0EmTL/pah+Yu+zIqFHdOn0TvHclT3ykoTQmifUxca1+SLaRxOEKkVDJ59NE1jOFpHihDorm8EQnrq73meII7UqW9MFyMCM0BJc/sd4SIo6S8olzfSKsJoBy72/GaIxZ8yp0Dn5k92OSZCb4VlfS6rbWPNglUHap2JiQkHs9vyyWi2iVzWD1PRNho7NZ/jJ2ArFQPglMeiS92Ae52JqUyFs0x1IHXaotKC09edmMxiZIpjYVR1qQHUX2zJIwp0vdTOi+ySWE5eVBLJOnxTEJ6rqnD2lMsnevyWH9K58bxtCfH8SdlakGecdhuj4Tb7R96DT613t6YyEh2hD9U9MVq9BhQmxkjdlBcPCVBojdekEpyHgKMJ0/Cv55RNT9AjgUjE+hvnJuvTQRo3Zt0i8vfxP6vfAtPcmNFM2FGP+RgWQoL5xr4i5jAVHe5gRVD+zCQe2Uriwr/wJz/xthkoFozclYT7SURjUo8Vjxpf3zrQ0K0BLWsING9skGIboFvFfSjmAYQnDKDVwU7o1qv9fW2CBEa1FAJfq4tHgXOoEWFZccS9T4uiSs52ue4aPVUsjjp3lPLFsVPgCUPOMM6LrDLs4bvfh2B1og/lxjEs3q49L4OTN8m/kd8Xme7lv0rw1++IkfoeNuqSi6i3dPdPS9CVmheD89j2OsQ4TtYYz0am4VR0lN1KCEE8MZLspj0XPe2HmeWGTa/z2TPZAKD2V1D995avCXVNvJaOBFdLvUfetZ1jo69vpvwNkzpulJCHbAawxBt370DrfwP6KpTckv8SMA9TbxJhYULqCpa4ozmEp8SJOI2bdS7K9bEwypXw7CiQbZouqX0PFFt/ypnO21VePX41uSq Zkt1jdLo Y3CjbQVelRFMtAcpBmfkj1FYGi1HFxJAwW2oMysDvMhCyqpQL0nyTBba2mFWnrhl7bxCKBKlNmkU0R0cHxn0FdaMkwQn3kkaWCqmTh3l3PNMwHVyODbxx0Z2kII1HfA7EjCmDbGugfQIJPQerLUtsKOiSD0RUh8ZIGvxs1SriZLicnXMfwXrUNTSVXTThe5mXzNusCSYuwHhFfdsCpITp/7KoqP3Yu7uaF5uXoFLAPvac0DCiHkuc3rpJBzBwqtcV9NbDdAqi4pp+4aVP+w13DOSjcz4JL2COGD84O+AGnj+BQbqXjixD06WQ4cLcYz35jNRqGP+bVepFNP1z5Z9MOJPw6dsbdZQgkISU2URDi1MsW8uHQjsTgEcs6ke1+dPxTMR4tcrKUBqw7of/vU/JGQquk5z7VBbVkTFSMvjb7w/89Pddu9xxsH02W8mDwFOA3F589Pt0nnD+LpQ1PFuLd9ayqdpzexMT/Y95YT8f47pOGDtcvVcvqUZIPjjydU/MYkqcypVSrr8i6262rdyHfl0RhxUMhNdCBqRHfiM6y0EYJFp3zcqrhbvZf1e72k0I/VZOEJreofsQ7Hc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Feb 8, 2023, at 8:38 AM, Adam Manzanares = wrote: >=20 > On Thu, Feb 02, 2023 at 09:54:02AM +0000, Jonathan Cameron wrote: >> On Wed, 1 Feb 2023 12:04:56 -0800 >> "Viacheslav A.Dubeyko" wrote: >>=20 >>>>=20 >>>=20 >>> Most probably, we will have multiple FM implementations in firmware. >>> Yes, FM on host could be important for debug and to verify = correctness >>> firmware-based implementations. But FM daemon on host could be = important >>> to receive notifications and react somehow on these events. Also, = journalling >>> of events/messages/events could be important responsibility of FM = daemon >>> on host.=20 >>=20 >> I agree with an FM daemon somewhere (potentially running on the BMC = type chip >> that also has the lower level FM-API access). I think it is somewhat >> separate from the rest of this on basis it may well just be talking = redfish >> to the FM and there are lots of tools for that sort of handling = already. >>=20 >=20 > I would be interested in particpating in a BOF about this topic. I = wonder what > happens when we have multiple switches with multiple FMs each on a = separate BMC. > In this case, does it make more sense to have an owner of the global = FM state=20 > be a user space application. Is this the job of the orchestrator? >=20 > The BMC based FM seems to have scalability issues, but will we hit = them in > practice any time soon. I had discussion recently and it looks like there are interesting = points: (1) If we have multiple CXL switches (especially with complex = hierarchy), then it is very compute-intensive activity. So, potentially, FM on firmware side = could be not capable to digest and executes all responsibilities without potential = performance degradation. (2) However, if we have FM on host side, then there is security concerns = because FM sees everything and all details of multiple hosts and subsystems. (3) Technically speaking, there is one potential capability that = user-space FM daemon can run as on host side as on CXL switch side. I mean here that if we = implement user-space FM daemon, then it could be used to execute FM functionality = on CXL switch side (maybe????). :) >>>>> - Manage surprise removal of devices =20 >>>>=20 >>>> Likewise, beyond reporting I wouldn't expect the FM daemon to have = any idea >>>> what to do in the way of managing this. Scream loudly? >>>>=20 >>>=20 >>> Maybe, it could require application(s) notification. Let=E2=80=99s = imagine that application >>> uses some resources from removed device. Maybe, FM can manage = kernel-space >>> metadata correction and helping to manage application requests to = not existing >>> entities. >>=20 >> Notifications for the host are likely to come via inband means - so = type3 driver >> handling rather than related to FM. As far as the host is concerned = this is the >> same as case where there is no FM and someone ripped a device out. >>=20 >> There might indeed be meta data to manage, but doubt it will have = anything to >> do with kernel. >>=20 >=20 > I've also had similar thoughts, I think the OS responds to = notifications that > are generated in-band after changes to the state of the FM are made = through=20 > OOB means. >=20 > I envision the host sends REDFISH requests to a switch BMC that has an = FM > implementation. Once the changes are implemented by the FM it would = show up > as changes to the PCIe hierarchy on a host, which is capable of = responding to > such changes. >=20 I think I am not completely follow your point. :) First of all, I assume = that if host sends REDFISH request, then it will be expected the confirmation of = request execution. It means for me that host needs to receive some packet that informs that = request executed successfully or failed. It means that some subsystem or = application requested this change and only after receiving the confirmation requested = capabilities can be used. And if FM is on CXL switch side, then how FM will show up the changes? = It sounds for me that some FM subsystem should be on the host side to receive = confirmation/notification and to execute the real changes in PCIe hierarchy. Am missing something = here? Thanks, Slava.