From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E376C7619A for ; Fri, 24 Mar 2023 00:42:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 629EB6B0072; Thu, 23 Mar 2023 20:42:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D9C76B0074; Thu, 23 Mar 2023 20:42:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 456546B0075; Thu, 23 Mar 2023 20:42:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 32B936B0072 for ; Thu, 23 Mar 2023 20:42:15 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D317F160850 for ; Fri, 24 Mar 2023 00:42:14 +0000 (UTC) X-FDA: 80601940188.07.27D7A33 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf06.hostedemail.com (Postfix) with ESMTP id 0B977180008 for ; Fri, 24 Mar 2023 00:42:11 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F4yIgRTs; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679618533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QKyhW8FOe598XPaom1Roy9TL4SvCQzlbPT9lc7YLVao=; b=t04i0gwGLOCT3HVi5YcVVRz2iJ3GHDNp/GytwFHUZloVjmPGUuT1ZRzPKRZlmXeeZJS3vr tPWHcPnfHWu3lTeLZy186dQHeMtSzP4vs284MVBemRENIKbLzHsIaItoUF8g2mcVenge2f Yb+5Kcn3xDmWks189Do2r4U3eloJBXA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=F4yIgRTs; spf=pass (imf06.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679618533; a=rsa-sha256; cv=none; b=KnWKK0tOeVmKoAJKNwCkVgnZlXvNUjDZpyBKWqqb+cxOPECvzMFwCfkplBt0DNfb+9S+NZ dzR5DEk/75Z3pH8LX7K+xqT9cC2De9ZkjXVR5ziXx/hGNbuPHT/QjJ1gDqlBST48gbdvpA QQ7zasCynH3n1Rm4o/htQRf5f7Aw2HA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1679618532; x=1711154532; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=AnfAZEwkzaV+MXfEzXGLd84me6iNy8c8SDgIW7uWzHs=; b=F4yIgRTseFFN2eAVR0UOiL4xcoSt2s+xXTYejaXFKWs8eblpkvl/MbKk oJe9l+L70TCbZ440tVuV0m50DHIx3LEZhFhv/kMZVX2AmgDhO8yfsawqn aUJVsn0YV8l1c3YLYT0BIybTLJ+sCmkTTYh4CI2uq+xjjgliwCiG+WWy/ PW5N6Z3Mt8dLPLuIomqTxK372A1PPQRHrXFf7dlmnjXbIWxYPmpivul0m ipqI9LRZWYtwEcxzIECUOIhfNR3ju2X2BiXwJ0qvL+TnurppHyfkI0veV b/MQ3YQqMf6tjaqhQq4FinArMbg1JKZuwWAF9cUYvn8UiwDrhbm4WAsPW g==; X-IronPort-AV: E=McAfee;i="6600,9927,10658"; a="404577357" X-IronPort-AV: E=Sophos;i="5.98,286,1673942400"; d="scan'208";a="404577357" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Mar 2023 17:42:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10658"; a="684958656" X-IronPort-AV: E=Sophos;i="5.98,286,1673942400"; d="scan'208";a="684958656" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Mar 2023 17:42:08 -0700 From: "Huang, Ying" To: Kyungsan Kim Cc: dan.j.williams@intel.com, lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-cxl@vger.kernel.org, a.manzanares@samsung.com, viacheslav.dubeyko@bytedance.com Subject: Re: RE(2): FW: [LSF/MM/BPF TOPIC] SMDK inspired MM changes for CXL References: <641b7b2117d02_1b98bb294cb@dwillia2-xfh.jf.intel.com.notmuch> <20230323105105.145783-1-ks0204.kim@samsung.com> Date: Fri, 24 Mar 2023 08:41:02 +0800 In-Reply-To: <20230323105105.145783-1-ks0204.kim@samsung.com> (Kyungsan Kim's message of "Thu, 23 Mar 2023 19:51:05 +0900") Message-ID: <87wn37q8v5.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 0B977180008 X-Stat-Signature: a8c9ffmuh5az4cy574wdhigfz5mbzwcb X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1679618531-624534 X-HE-Meta: U2FsdGVkX1/XhoogW08FJWpmRo5DA6crK0O7zNH+82dm7M9DZN3NxbiyUFrIHCV+iwj+GnkT7cm3/FBYVLCR8qOcdArRdEL6v3mgRWcW5pTp5XcPNXkSzQKWNsYNuojE+hhTyG+8buAvLb0SEQWqSNkVVcHaHUhaAVDHpR/IVDe9aVPiTPD2/hM5NtvI6cNoRk6H5CVSb/i33hX98vUMatwnuzVUao10KBpZoJThHK+iUbS7Eh/nz1N6aq6mNXJwf3XCIf1NrGo//brwc039Z2uJxwXgeMth0IVl8+FVL24+IqVwBlsy7spvIvsVqqCDBfb7hv6VvggUkqqT7IcFNVMoHG2Q4wezl6yO4/+fI/25IVQk8nJEcgbd0DpFC+icfVgnrMLBcnOn6uIb2o7LAaua5oyvyKu8tHsUfoMi3AbkOQ02OS6QlrcXodgKTNYo+kig4LWa3QmmqKWBmmGiEngxN5Ta7XO48Qre/wR4z19W8gY34PjQr9gH2C68ImG3OHwdxb7aLmMlPwBWuF3lFdBwEeAGBDjM+GzoAN+w+JHSndpCDZb2GzsSKjj1V1SqxDidWKqGQyWxD4cFx8CmfzC/6tNcwDsmp3CfMywx2JjQiryEzm8S7qeKHpBVoY4qKnKWUBxw5erjlSHYF9gw2NQqunUoUwcoZI01P+rQYO0w1mjX3ATkWxztzPA6/M4CgMMQyVRZNrQ/oSF4T8vWfK0kX1D8PbXirM7Vm3EBJHI9G1rciyEb4AwlQU3Aclm5BdsVdNDz51ZDdMQK76nw2VYGSGX51NFvUfiOFBxKBuC1YuuysZUt33q8hKLJA2KkbVrGcxOlL5c9znHuhqRuL4/EENV7Ciy9PhmYfr5HoRkO0Kps5Z3r3nuaROvW7kvY3Orsh3ms6cF0rqMgaNXEYWCxu8AmrtChjvLEU6jY5FmUKV7/ak4M5C4hXAtxeUkoxT5NMIAsi/X/9RLE7SC j1imFvqs a+J8rC+eDie0XCvJfOsSpBqNF7uews2oroobt8UoxnqlCBHWF1tzr73pTsh9/v4KUVVoxe7KJXeHVmimZ4+PuGFE+czWNWq8sfIDKOpVnWbdEY2qTGPzM3ppkgmbMVRqIbJZhfmbiE+oaXZrzbhVpiA83pvKOTLZfDCZNo5yupTWfNmfbzpJMK+p5CH2PcQL7hDku X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Kyungsan Kim writes: > I appreciate dan for the careful advice. > >>Kyungsan Kim wrote: >>[..] >>> >In addition to CXL memory, we may have other kind of memory in the >>> >system, for example, HBM (High Bandwidth Memory), memory in FPGA card, >>> >memory in GPU card, etc. I guess that we need to consider them >>> >together. Do we need to add one zone type for each kind of memory? >>> >>> We also don't think a new zone is needed for every single memory >>> device. Our viewpoint is the sole ZONE_NORMAL becomes not enough to >>> manage multiple volatile memory devices due to the increased device >>> types. Including CXL DRAM, we think the ZONE_EXMEM can be used to >>> represent extended volatile memories that have different HW >>> characteristics. >> >>Some advice for the LSF/MM discussion, the rationale will need to be >>more than "we think the ZONE_EXMEM can be used to represent extended >>volatile memories that have different HW characteristics". It needs to >>be along the lines of "yes, to date Linux has been able to describe DDR >>with NUMA effects, PMEM with high write overhead, and HBM with improved >>bandwidth not necessarily latency, all without adding a new ZONE, but a >>new ZONE is absolutely required now to enable use case FOO, or address >>unfixable NUMA problem BAR." Without FOO and BAR to discuss the code >>maintainability concern of "fewer degress of freedom in the ZONE >>dimension" starts to dominate. > > One problem we experienced was occured in the combination of hot-remove and kerelspace allocation usecases. > ZONE_NORMAL allows kernel context allocation, but it does not allow hot-remove because kernel resides all the time. > ZONE_MOVABLE allows hot-remove due to the page migration, but it only allows userspace allocation. > Alternatively, we allocated a kernel context out of ZONE_MOVABLE by adding GFP_MOVABLE flag. > In case, oops and system hang has occasionally occured because ZONE_MOVABLE can be swapped. > We resolved the issue using ZONE_EXMEM by allowing seletively choice of the two usecases. Sorry, I don't get your idea. You want the memory range 1. can be hot-removed 2. allow kernel context allocation This appears impossible for me. Why cannot you just use ZONE_MOVABLE? Best Regards, Huang, Ying > As you well know, among heterogeneous DRAM devices, CXL DRAM is the first PCIe basis device, which allows hot-pluggability, different RAS, and extended connectivity. > So, we thought it could be a graceful approach adding a new zone and separately manage the new features. > > Kindly let me know any advice or comment on our thoughts.