From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E96AC76195 for ; Fri, 24 Mar 2023 22:33:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07BAD900004; Fri, 24 Mar 2023 18:33:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02B4F900002; Fri, 24 Mar 2023 18:33:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0E9E900004; Fri, 24 Mar 2023 18:33:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D166D900002 for ; Fri, 24 Mar 2023 18:33:20 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 937CE160BAE for ; Fri, 24 Mar 2023 22:33:20 +0000 (UTC) X-FDA: 80605244160.09.AB6B8F4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 9AA2640003 for ; Fri, 24 Mar 2023 22:33:17 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZvnZkyYw; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679697198; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M2Gd048jUU5nxLZZ0U+ngFbATTz0dYwE/LMy6TKJlvM=; b=n+wrxHadBZtFzIgMF7vBun9J1HyL+E8Baw3n+A8OiSUgFfiCwhigaPFdQ/bL5C9rrD1lqq zz+Euor3TB7yrOVYkXzC6X3wsEk5rmWnnZgAvhQnbM80o8RUBORNdFHG1GUS4H5bD1+pgb fli85/GEfvzlZBZKokCnux3QofBRIuw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZvnZkyYw; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679697198; a=rsa-sha256; cv=none; b=03f21Vi0leSHXtVMmy0ZeqYNOvFxub5jn6Hv87cXmKXaj6YKPgUUek2aZgpHLiuUAonMZu 8CzKRC6H/rMu7bjdr9p8MCL9J/Zw3oSupGJPjsVZHr7DvYCScPI5At3Sq53p1+xfPvHd9c zSF39iLAbqmMjrda4FukvuN9k74mGUA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1679697196; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=M2Gd048jUU5nxLZZ0U+ngFbATTz0dYwE/LMy6TKJlvM=; b=ZvnZkyYwt+IDeI84mORHnfIKDEYlkN5VgUZJD6qCJQrRFa1noTS2QG83ykEzjSgjfXFnm8 t0ez9l7GzQtrIa7+hxnsIFLBcQExs6EDa15QilFIU3K8+uAlk+PvIEFO8rMmmL+5nbvIah 39+8kI2oaH7LZ1GHTS3s+Mgc1DrVY8Q= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-34-3u3EAsctO0ibx77R3S6g2g-1; Fri, 24 Mar 2023 18:33:15 -0400 X-MC-Unique: 3u3EAsctO0ibx77R3S6g2g-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-54463468d06so31723207b3.7 for ; Fri, 24 Mar 2023 15:33:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679697195; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=M2Gd048jUU5nxLZZ0U+ngFbATTz0dYwE/LMy6TKJlvM=; b=SVDgwIiKB3O7EG2iDNgxpETw38xxyZqnyYz8yyI//bHY4BckzzSy/IUe2kVRO6CmGg zn//uG3IBL3bPAQTpgHwD2KUwTR23ZBG5NKB4H/17htz2tb4ic4asxlpJgYckK03GU2u /+VH20Zc0QE0zodlLIALxGNOiJ+DhxyoaEYGBdEQDRxRd5XKoX5sP+2LnOR47U8uCy5F QMfVOuZbvHMqWrGYCe6U9tM7hdEJz5S2ugITsPuXlzxKBMU2XGV2FXGjN7bpPaZZco3k ML7AJtGoIoDKEFcolH/dUrb4SZl9sbOG6uGq/VPGoKKk3YUyvxZpHXWHndzanTHE8QRn DvnA== X-Gm-Message-State: AAQBX9dagIkQxVizIBkuSA+nHaBVof5zMAZwGJcCLgD3uDrSxgScrCCA MsjUgfgKh+204sTvrYasGcqAEKICCaf61vBexTQVkTskemSgnzxYv7/obL2bax0oNsrGdoBf1y/ OU/RheLwv+zU= X-Received: by 2002:a81:6ac5:0:b0:541:894d:9360 with SMTP id f188-20020a816ac5000000b00541894d9360mr3452804ywc.21.1679697194984; Fri, 24 Mar 2023 15:33:14 -0700 (PDT) X-Google-Smtp-Source: AKy350ZxDFkfYE1yVhUtHbiwatfrVOh987ysutLClRdPWjsYtqHOWAfDK8VXxXnoVK8wOnV2R2YvYw== X-Received: by 2002:a81:6ac5:0:b0:541:894d:9360 with SMTP id f188-20020a816ac5000000b00541894d9360mr3452775ywc.21.1679697194417; Fri, 24 Mar 2023 15:33:14 -0700 (PDT) Received: from [100.69.142.128] ([206.173.106.22]) by smtp.gmail.com with ESMTPSA id t4-20020a81b504000000b00545a0818493sm664253ywh.35.2023.03.24.15.33.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 24 Mar 2023 15:33:14 -0700 (PDT) Message-ID: <7c7933df-43da-24e3-2144-0551cde05dcd@redhat.com> Date: Fri, 24 Mar 2023 23:33:12 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL To: =?UTF-8?Q?J=c3=b8rgen_Hansen?= , Kyungsan Kim Cc: "lsf-pc@lists.linux-foundation.org" , "linux-mm@kvack.org" , "linux-fsdevel@vger.kernel.org" , "linux-cxl@vger.kernel.org" , "a.manzanares@samsung.com" , "viacheslav.dubeyko@bytedance.com" , "dan.j.williams@intel.com" References: <91d02705-1c3f-5f55-158a-1a68120df2f4@redhat.com> <20230324095031.148164-1-ks0204.kim@samsung.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9AA2640003 X-Stat-Signature: p4scxwmtqds9kymce8rhbk5xcnz8ps1t X-HE-Tag: 1679697197-223553 X-HE-Meta: U2FsdGVkX1/57ATh707one1ZXKVCgwtDfqQDav0awPZ/KFNVd8QJj2e/top882XKqMk/KMra41SoWUDHG0UTHPOeaJEPrf1ZqpNT0Yx6GQKjBJEVK6d8u0XctqolYtvnSIBHQ1GDjRVfbAai9RFQjjWr3xKvVLK7FX2RIs+oDic7mikw+ywG8OYulgBMIZSlOZ6ovL5q2Zm3ajlI2AlcyBWfMTwq8v+3mhiJy9LFURr24+FDQL4LjvA2alBFzuKr+rqsO61yDndtlHGv63bq8254EBhao4lbCyLHfXSqIulVPBIVfuj86Ondt7C2vgNvWsA4X6Ldfi6+GpyoElVV6gG69RilXpiOddRWG/SGXKEyGu2dNbmBSSbkr3Hnz9WXBT9ibD0Dsy7NbhgFAeUnHZsQ2IZfjhnabtcUmn+zBp0f916TJXPignzyW7Ct0TIno67lRJTWxh7NBSzvo3qS+vzgruSuil8onRA5aRxn+je1el8nQ8DBKRNtEadk3bvuGOQyoMckD8Xh3yjsF6cWP7fPTKbgin/Sjt8182LXqpQk0RHBa1V0L3JucbOZSlg6H2llMtPoAAp8gb7aIHBVVpGOEkIxPQrPR3DhS5bHj2dUL4H7aMaBY6hXpOiKgqKr51orK1GgySrwyjJXrl6MZZ247q8SwxZwMY94T3dwWbe/++HSYqnGIfAQ3qq+VmtXSYuL4zIkJRwZSiCLgNcmxk8oJqnC/Yx2iiHgX8hsAWvQkgGC3zDozst4e5gGDh4ZnFmT/ihuaSb2W+O2wplyuW9GhWxYT3hOWWKGGtQRJ8qlqesTh5AZSEQkLn7cvKwgSPuDOEovFQkZglIKMKl215Tyxn3gq4O1ufkxzKyvzIruMkIiGiPDJpUtI6qu4tqS/hFovJnmuATaCdIsP1RI5q3xDMyEJIgDXVN7cWZ38lWikYZGLts8RggejShlqK1kTCZj2vsd8VRdDS4l+VW wkm1BFBv HTN4/82kL+0NWVf+HVpkyAkeO+YX9QMqLuvSCsTK8fC6eaTOc6Ri8KLqNYlmo+DppHxEJusZnRFDqp9Yp2JTMQZsFMN6IjRDf4xrKpqMvtLVcYwlwZw5X582UVVVBXy7HCzAcHHDvoJ0PEHb3ao7Vzg0XoDBQheRqPgMk4sqjjCKusiHm98p6LvFTRJ/AmIstnQUV7Z984W1tvv6OjlLf3VLKbCM+VUEww7u858+iM0kFkqTsvKgHF60TM2GRS4GBEQpsJU8cnJBEavcg2MfFVSLFfXB0GM+y5+eDnnthTZcJhmXMuCMXGVvDATJNDSdCBMIKBvv5bKFeVh4zSyaNybH4ChKwg4XgA7hlf7V/dfDWjYs/z4nmHgCPvi25EtGB8IQz6n6jWxeuOM/7R6E1PpSSpejpPDVdbkwrd1+YLAVYMoTxcSa7O+sCazuKBUZVQiJcepb4PgPqxtDbq/bTmbTcL+236Iomb0s5XdqPXnoGgHg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 24.03.23 14:08, Jørgen Hansen wrote: > >> On 24 Mar 2023, at 10.50, Kyungsan Kim wrote: >> >>> On 24.03.23 10:27, Kyungsan Kim wrote: >>>>> On 24.03.23 10:09, Kyungsan Kim wrote: >>>>>> Thank you David Hinderbrand for your interest on this topic. >>>>>> >>>>>>>> >>>>>>>>> Kyungsan Kim wrote: >>>>>>>>> [..] >>>>>>>>>>> In addition to CXL memory, we may have other kind of memory in the >>>>>>>>>>> system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >>>>>>>>>>> memory in GPU card, etc. I guess that we need to consider them >>>>>>>>>>> together. Do we need to add one zone type for each kind of memory? >>>>>>>>>> >>>>>>>>>> We also don't think a new zone is needed for every single memory >>>>>>>>>> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >>>>>>>>>> manage multiple volatile memory devices due to the increased device >>>>>>>>>> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >>>>>>>>>> represent extended volatile memories that have different HW >>>>>>>>>> characteristics. >>>>>>>>> >>>>>>>>> Some advice for the LSF/MM discussion, the rationale will need to be >>>>>>>>> more than "we think the ZONE_EXMEM can be used to represent extended >>>>>>>>> volatile memories that have different HW characteristics". It needs to >>>>>>>>> be along the lines of "yes, to date Linux has been able to describe DDR >>>>>>>>> with NUMA effects, PMEM with high write overhead, and HBM with improved >>>>>>>>> bandwidth not necessarily latency, all without adding a new ZONE, but a >>>>>>>>> new ZONE is absolutely required now to enable use case FOO, or address >>>>>>>>> unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >>>>>>>>> maintainability concern of "fewer degress of freedom in the ZONE >>>>>>>>> dimension" starts to dominate. >>>>>>>> >>>>>>>> One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. >>>>>>>> ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. >>>>>>>> ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. >>>>>>>> Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. >>>>>> >>>>>>> That sounds like a bad hack :) . >>>>>> I consent you. >>>>>> >>>>>>>> In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. >>>>>>>> We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. >>>>>> >>>>>>> I once raised the idea of a ZONE_PREFER_MOVABLE [1], maybe that's >>>>>>> similar to what you have in mind here. In general, adding new zones is >>>>>>> frowned upon. >>>>>> >>>>>> Actually, we have already studied your idea and thought it is similar with us in 2 aspects. >>>>>> 1. ZONE_PREFER_MOVABLE allows a kernelspace allocation using a new zone >>>>>> 2. ZONE_PREFER_MOVABLE helps less fragmentation by splitting zones, and ordering allocation requests from the zones. >>>>>> >>>>>> We think ZONE_EXMEM also helps less fragmentation. >>>>>> Because it is a separated zone and handles a page allocation as movable by default. >>>>> >>>>> So how is it different that it would justify a different (more confusing >>>>> IMHO) name? :) Of course, names don't matter that much, but I'd be >>>>> interested in which other aspect that zone would be "special". >>>> >>>> FYI for the first time I named it as ZONE_CXLMEM, but we thought it would be needed to cover other extended memory types as well. >>>> So I changed it as ZONE_EXMEM. >>>> We also would like to point out a "special" zone aspeact, which is different from ZONE_NORMAL for tranditional DDR DRAM. >>>> Of course, a symbol naming is important more or less to represent it very nicely, though. >>>> Do you prefer ZONE_SPECIAL? :) >>> >>> I called it ZONE_PREFER_MOVABLE. If you studied that approach there must >>> be a good reason to name it differently? >>> >> >> The intention of ZONE_EXMEM is a separated logical management dimension originated from the HW diffrences of extended memory devices. >> Althought the ZONE_EXMEM considers the movable and frementation aspect, it is not all what ZONE_EXMEM considers. >> So it is named as it. > > Given that CXL memory devices can potentially cover a wide range of technologies with quite different latency and bandwidth metrics, will one zone serve as the management vehicle that you seek? If a system contains both CXL attached DRAM and, let say, a byte-addressable CXL SSD - both used as (different) byte addressable tiers in a tiered memory hierarchy, allocating memory from the ZONE_EXMEM doesn’t really tell you much about what you get. So the client would still need an orthogonal method to characterize the desired performance characteristics. This method could be combined with a fabric independent zone such as ZONE_PREFER_MOVABLE to address the kernel allocation issue. At the same time, this new zone could also be useful in other cases, such as virtio-mem. Yes. I still did not get a satisfying answer to my original question: what would be the differences between both zones from a MM point of view? We can discuss that in the session, of course. Regarding performance differences, I thought the idea was to go with different nodes to express (and model) such. -- Thanks, David / dhildenb