From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84C54C76196 for ; Mon, 3 Apr 2023 08:34:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0911D6B0072; Mon, 3 Apr 2023 04:34:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 040586B0074; Mon, 3 Apr 2023 04:34:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4A456B0075; Mon, 3 Apr 2023 04:34:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D62966B0072 for ; Mon, 3 Apr 2023 04:34:29 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B0BA61C5EF8 for ; Mon, 3 Apr 2023 08:34:29 +0000 (UTC) X-FDA: 80639418258.30.A5BFA23 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 94A5E8000B for ; Mon, 3 Apr 2023 08:34:26 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aewv40S9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680510866; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FcZLYmIqRCN99tYF4yEMgjqA6ESHr46ONZQ166uj3mY=; b=mfpksZSLuw5bWHwjxuJgHtHtpoFQYUhBlWL7sEUIbHYKYKsosBuV0EzJ2MRiztBZEFZv9C o0JpE6ryf9yzrO4UF4boWoniMRmIbSLtMdnwQ9zf8hLPkvR266lUNPNtwTFJ6ImdpdbaXr DauHKyPtG7QcOkNyESJRfEJ6eIqN2l0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=aewv40S9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680510866; a=rsa-sha256; cv=none; b=rQmkOKG3aj2jI6530wQUCco3826GmI4BrWtWSaONbCrHFW3OMji3HwoeljuQfSQlMdrBTs cZtW2c2Y2DhdL+CdRoezp1pkhpGiA1695X5A2bbaavQ+4LpWioCPEekEf09nboSJSzkBBp XM3IQRlWmfpi812TZpzggQHSY70/KZA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680510866; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FcZLYmIqRCN99tYF4yEMgjqA6ESHr46ONZQ166uj3mY=; b=aewv40S9/MdBHRBRlvmg+b+laFBiu67RRkIj1WpuqZVSX07UiNtFDiUJNU0Fv8WUIPDQbC Q7jn4w6o3Fb00gu+OeNFc/N1LLw5g6Hn8U968ziTqpeSNRVX4mMFdd6+qgRp1CIzjhB4/5 RmOHf97s6Zh9yQU178ApJDRPhS1bWGo= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-647-XQERHbJqO2WUkwP2-8t40g-1; Mon, 03 Apr 2023 04:34:23 -0400 X-MC-Unique: XQERHbJqO2WUkwP2-8t40g-1 Received: by mail-wr1-f70.google.com with SMTP id i19-20020adfa513000000b002dc1cdac53fso3001719wrb.5 for ; Mon, 03 Apr 2023 01:34:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680510862; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FcZLYmIqRCN99tYF4yEMgjqA6ESHr46ONZQ166uj3mY=; b=RVrYpkO3a1lq9DWE/THu04FpNMpIve4DCSXQ/8WwFOkkSbXcAHzkLTEihvjfB3Gin9 W0gjm/XfgDHjFHNiKfcCx/YaE2zSMTwTBjMNb3KYEa0E6KtAOATD78n86n1EBwD9t4zv p7jgJu7Uru8P+lo1B/hAIMbX3I3PHMbkccTG6SPi4bDT1X24or1dF0XLuuQI3vlKIRaZ R3wXKSisc1dO+WWuV0QYKvz2+EAcc0vhWJ4eO2xkdgSMYkn6vyEvpr/6brY91IBVuBVl A72mPFjMbhjb10Kl9NpMlHIQ9Kh8cPjpzJZxxY84ZNSSXcp+A/A/xkATzN8uDeBQ4s2C zqmA== X-Gm-Message-State: AAQBX9eBxOk2VCxTBBZXwu8SzB+bNvPNl2tBp1BY7+9cCWxQozyObqdt PfzIYHHe1V5a6IjesNNGlnNMxx0hyfCcvzr1SYvlyGRVCwaLjiPWFchLVd2q5Hshqw7PGvQhOS5 In0v6KqT6W8A= X-Received: by 2002:adf:f8c2:0:b0:2c3:e7d8:245c with SMTP id f2-20020adff8c2000000b002c3e7d8245cmr25962122wrq.13.1680510861946; Mon, 03 Apr 2023 01:34:21 -0700 (PDT) X-Google-Smtp-Source: AKy350buGrvo6nZ4OBkdPpnnJzG9dcpRSvtFybHJQ8Jw4OJPMiVMYDLKKgmCWXOMOoPKkQciPT5ozA== X-Received: by 2002:adf:f8c2:0:b0:2c3:e7d8:245c with SMTP id f2-20020adff8c2000000b002c3e7d8245cmr25962099wrq.13.1680510861580; Mon, 03 Apr 2023 01:34:21 -0700 (PDT) Received: from ?IPV6:2003:cb:c702:5e00:8e78:71f3:6243:77f0? (p200300cbc7025e008e7871f3624377f0.dip0.t-ipconnect.de. [2003:cb:c702:5e00:8e78:71f3:6243:77f0]) by smtp.gmail.com with ESMTPSA id d16-20020a5d4f90000000b002d51d10a3fasm9159738wru.55.2023.04.03.01.34.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 03 Apr 2023 01:34:21 -0700 (PDT) Message-ID: <5d6a35c8-94cd-5968-3110-7ea4737e728b@redhat.com> Date: Mon, 3 Apr 2023 10:34:19 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Subject: Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL To: Frank van der Linden , Matthew Wilcox Cc: Kyungsan Kim , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com, dan.j.williams@intel.com, seungjun.ha@samsung.com, wj28.lee@samsung.com References: <7c7933df-43da-24e3-2144-0551cde05dcd@redhat.com> <20230331114220.400297-1-ks0204.kim@samsung.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 94A5E8000B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qrdzphy8iuiawrso8b1gx7ydjrjzf9ze X-HE-Tag: 1680510866-396315 X-HE-Meta: U2FsdGVkX1/4wgV+fyVwyaT73PosPqcZ0uoowSFePUwn+G83eWMno2KiD5fPFwa04f9yqA05xk8xgLNHePP5aQwjkgsLxI411JCIFGbx6HrsIsF1IQzDXwl6MyIH8+ba6a7ioEVkfjxaj1BXgCaDj4YyfK51mSrthDkE9RiMr+9QnxohGBTquitfgZcZI8U1dz+QN+kdlo5DNfxNHPsQWVFpd1SQdBdHMPe1/d4jJqhVIwbdZf9WMUpv/puxv41PZkjeZBj2/yp9goF5qdWY9F5Jybl4Bgy+FBGy0HDDG01kTAFMKlfZ8wnR4rBKUkBMFFCdicC4KJd3uGestpUIjDSNH7PMejz60GwLaycRLgioUYHAP1Bgx6tlr3kgO2QGa+Ds0ibzWTnr4arJaaJzTyz5nDM8ckwjtjBzv3olMgeyrmNCXhr0zC4wUy2lt/F2TWyZVBXlo9ZjosEX7iPz+Bfvxt+gFAwNINkGV3+fEQFDybF4IrQ6I0VtUPP8O1WqyfpBkWfLTiFF2gsiYxgogm9lIkLKYn2x0rPTGAVjI33arTGTOubfWy/OxYCEt0Tza39nZVPHhXYKUvPXPtEq0pInL6q/5ApjQzl9iU/lN8vz+NFKR5DEDPIcdZDs7bTO2SktBl1utZEC/RdKwp7Qh+5ynriXkIOoCpIuMdJOIyFjNleYZyC6oTpdDSnbgEmo5ARm9cJOPVRX39+8Ums/rOKWq0bc4wYKdT/ppQDRNDYjtS06Zj/UmMVRyudoCB0WP4DZVt5676bQca5nHqyInxHYhW8DszVDJdqlZ9WNCWJMioH31/cMyqMbOJsEc/B/cJFonR9OnvnqFbQEOZ1XjMDABROXQjfgxJL8zk+z+/83S1diuX+CDAIQTThPWNCWNHGgUUroavB5fQJNOEJW/3yxLuKE4OdVup+VV7Hm8ABIekD8CngGAIPKt0Jwsr0Un2SwyPNFOWIFyOY4jF/ oXwAgL+L Fgj70cD/Zc6ck31Kb/UyA1NF+ESXZnWLPbFqwEymf8tsVSBVWY1nN23wsc3tKA5uNT0CNqzdEFfzNV61s1A+ByICKXSjEQS01lnWv3dBVL5FNo/zS7kenR1U0W81EEcnYpJHGy5jzntwusuidVVRNDw29daM/4tzKc/RbA+YvTex0V1vaY4hXsAANVbMQwJJw1rspoGpxVK5XG0cqbcWAzV7ntyT/fQa78VdwjwyMzDjYEHImdvdTkSijMpM2BJBkhXPOSkGs49FbmKtKCNCDLbdYG8z13y4KI1BXvJE5osxU23NtuCSRyAUmWgl5D8uZkRxYGcpF5kfNBahmIqzwa5Y3tHITC3G73Da2B2mqvKCxUHP6VVJt6uSLIrLMIAVx2eUKj1764JLf5fd9PB65CstbXZ5lYE6Hwaq5sUd/b4yw41TTSpSo0/RaBtLAbTAilkCDG4qMeO6/7upnhSndBVZ68J4DOCTyaBlwTqSXSkUMXqY+tEy8D7SHwFyj6Eon+Rq3ZySLiMnLycf3uQ/SC9KowWM6Dp0jQZD4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000125, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 31.03.23 17:56, Frank van der Linden wrote: > On Fri, Mar 31, 2023 at 6:42 AM Matthew Wilcox wrote: >> >> On Fri, Mar 31, 2023 at 08:42:20PM +0900, Kyungsan Kim wrote: >>> Given our experiences/design and industry's viewpoints/inquiries, >>> I will prepare a few slides in the session to explain >>> 1. Usecase - user/kernespace memory tiering for near/far placement, memory virtualization between hypervisor/baremetal OS >>> 2. Issue - movability(movable/unmovable), allocation(explicit/implicit), migration(intented/unintended) >>> 3. HW - topology(direct, switch, fabric), feature(pluggability,error-handling,etc) >> >> I think you'll find everybody else in the room understands these issues >> rather better than you do. This is hardly the first time that we've >> talked about CXL, and CXL is not the first time that people have >> proposed disaggregated memory, nor heterogenous latency/bandwidth >> systems. All the previous attempts have failed, and I expect this >> one to fail too. Maybe there's something novel that means this time >> it really will work, so any slides you do should focus on that. >> >> A more profitable discussion might be: >> >> 1. Should we have the page allocator return pages from CXL or should >> CXL memory be allocated another way? >> 2. Should there be a way for userspace to indicate that it prefers CXL >> memory when it calls mmap(), or should it always be at the discretion >> of the kernel? >> 3. Do we continue with the current ZONE_DEVICE model, or do we come up >> with something new? >> >> > > Point 2 is what I proposed talking about here: > https://lore.kernel.org/linux-mm/a80a4d4b-25aa-a38a-884f-9f119c03a1da@google.com/T/ > > With the current cxl-as-numa-node model, an application can express a > preference through mbind(). But that also means that mempolicy and > madvise (e.g. MADV_COLD) are starting to overlap if the intention is > to use cxl as a second tier for colder memory. Are these the right > abstractions? Might it be more flexible to attach properties to memory > ranges, and have applications hint which properties they prefer? I think history told us that the discussions always go like "but user space wants more control, let's give user space all the power", and a couple of months later we get "but we cannot possibly enlighten all applications, and user space does not have sufficient information: we need the kernel to handle this transparently." It seems to be a steady back and forth. Most probably we want something in between: cxl-as-numa-node model is already a pretty good and simplistic abstractions. Avoid too many new special user-space knobs is most probably the way to go. Interesting discussion, I agree. And we had plenty of similar ones already with PMEM and NUMA in general. -- Thanks, David / dhildenb