From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04CC8C6FA82 for ; Mon, 5 Sep 2022 01:53:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25B258D004D; Sun, 4 Sep 2022 21:53:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 209B78D0031; Sun, 4 Sep 2022 21:53:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0842B8D004D; Sun, 4 Sep 2022 21:53:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E63F38D0031 for ; Sun, 4 Sep 2022 21:53:18 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ACEB3140376 for ; Mon, 5 Sep 2022 01:53:18 +0000 (UTC) X-FDA: 79876359276.26.4FBB2AA Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf21.hostedemail.com (Postfix) with ESMTP id A39841C006F for ; Mon, 5 Sep 2022 01:53:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1662342797; x=1693878797; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=3cGUgcQzZq9BOyICE83jr0jcitIR/Hn4WKTkEyaaacw=; b=lUjm4T+cHgxgC8F3zwycf4AXuaOACZ66D+jX59od9QzIJ1Ko2VxtVuTn ob6yHtFyhNO7KWEHdaAMX8zWcND58oM6NouImykUMKXkgB2l9EDYJ/y8i Is5CI+N4Nx6CmpBLGPT/mztbC3W70B4w+XstctTpztXU0r1KSn7aEJwvj MzM69viV44gWo6ZQgdeco0NWqpMXq3D/nA4DZfW4jKbS4TQ3MP6d8xF+e GaOfI9xAT82he4AVHBOfZVHsDxNtTzpscCWBBX8UZCdBcUZOc4BzUrxV5 C2eGEErVAtrpsARdPGP8P/tOL48nwSDSqH25NCL/ARyyAHdt7Z8wo4pza w==; X-IronPort-AV: E=McAfee;i="6500,9779,10460"; a="295027564" X-IronPort-AV: E=Sophos;i="5.93,290,1654585200"; d="scan'208";a="295027564" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Sep 2022 18:53:14 -0700 X-IronPort-AV: E=Sophos;i="5.93,290,1654585200"; d="scan'208";a="643620420" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Sep 2022 18:53:09 -0700 From: "Huang, Ying" To: Aneesh Kumar K V Cc: Wei Xu , Johannes Weiner , Linux MM , Andrew Morton , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , jvgediya.oss@gmail.com, Bharata B Rao , Greg Thelen , Greg Kroah-Hartman , "Rafael J. Wysocki" Subject: Re: [PATCH v3 updated] mm/demotion: Expose memory tier details via sysfs References: <20220830081736.119281-1-aneesh.kumar@linux.ibm.com> <87tu5rzigc.fsf@yhuang6-desk2.ccr.corp.intel.com> <87pmgezkhp.fsf@yhuang6-desk2.ccr.corp.intel.com> <87fshaz63h.fsf@yhuang6-desk2.ccr.corp.intel.com> <698120ce-d4df-3d13-dea9-a8f5c298783c@linux.ibm.com> <87bkryz4nh.fsf@yhuang6-desk2.ccr.corp.intel.com> <2b4ddc45-74ae-27df-d973-6724f61f4e18@linux.ibm.com> <877d2mz3c1.fsf@yhuang6-desk2.ccr.corp.intel.com> <45488760-02b5-115b-c16d-5219303f2f33@linux.ibm.com> <871qsuyzr2.fsf@yhuang6-desk2.ccr.corp.intel.com> <672e528d-40b7-fc12-9b0c-1591d586c079@linux.ibm.com> <87wnamxi30.fsf@yhuang6-desk2.ccr.corp.intel.com> <5aaf395d-514a-2717-58c6-3845b97692bd@linux.ibm.com> Date: Mon, 05 Sep 2022 09:52:43 +0800 In-Reply-To: <5aaf395d-514a-2717-58c6-3845b97692bd@linux.ibm.com> (Aneesh Kumar K. V.'s message of "Fri, 2 Sep 2022 15:14:14 +0530") Message-ID: <87sfl6y4d0.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=lUjm4T+c; spf=softfail (imf21.hostedemail.com: 192.55.52.120 is neither permitted nor denied by domain of ying.huang@intel.com) smtp.mailfrom=ying.huang@intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1662342798; a=rsa-sha256; cv=none; b=HGGGrJj+dOIe3BYm2wojrefJQ22IJI/gPc/TMo26DBITzB9guXwbe+ffQWzIe0IMbUNIjE wJRVUmywyouvJbUcgY/OxNPWk/V/q/sD5UDwRq8etizHFh8MroaOe5fahyn0Ak8A+kJiMG eo/41YJtSt8nZ+C1STZ9dCVYlaprpWs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1662342798; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FxETs9J6PLDJROlEdvwxIA3ZUg3o/B6MVSQyXSgo3L4=; b=a+ZAHFlh4bnps+/xwiWMlpiaKfbu6j0wcq2hh+CvVwHGlkMUOsAjLM68Na4ZCrP9ot+hBe RNWuHObEn8v15EpHMZjSqwxaDP2onEVLtWYcr246FfucwoT0aEaTrYG2avZgPn3qyJBWeA odonDxRYkXjMlUg0q6ksFWY3qkUQ8SA= Authentication-Results: imf21.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=lUjm4T+c; spf=softfail (imf21.hostedemail.com: 192.55.52.120 is neither permitted nor denied by domain of ying.huang@intel.com) smtp.mailfrom=ying.huang@intel.com; dmarc=fail reason="No valid SPF" header.from=intel.com (policy=none) X-Rspamd-Server: rspam06 X-Stat-Signature: 8ucocgq3bh4spcpiyw81a1oozztfjjr9 X-Rspam-User: X-Rspamd-Queue-Id: A39841C006F X-HE-Tag: 1662342797-814271 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Aneesh Kumar K V writes: > On 9/2/22 2:34 PM, Huang, Ying wrote: >> Aneesh Kumar K V writes: >> >>> On 9/2/22 1:27 PM, Huang, Ying wrote: >>>> Wei Xu writes: >>>> >>>>> On Thu, Sep 1, 2022 at 11:44 PM Aneesh Kumar K V >>>>> wrote: >>>>>> >>>>>> On 9/2/22 12:10 PM, Huang, Ying wrote: >>>>>>> Aneesh Kumar K V writes: >>>>>>> >>>>>>>> On 9/2/22 11:42 AM, Huang, Ying wrote: >>>>>>>>> Aneesh Kumar K V writes: >>>>>>>>> >>>>>>>>>> On 9/2/22 11:10 AM, Huang, Ying wrote: >>>>>>>>>>> Aneesh Kumar K V writes: >>>>>>>>>>> >>>>>>>>>>>> On 9/2/22 10:39 AM, Wei Xu wrote: >>>>>>>>>>>>> On Thu, Sep 1, 2022 at 5:33 PM Huang, Ying wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Aneesh Kumar K V writes: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 9/1/22 12:31 PM, Huang, Ying wrote: >>>>>>>>>>>>>>>> "Aneesh Kumar K.V" writes: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> This patch adds /sys/devices/virtual/memory_tiering/ where all memory tier >>>>>>>>>>>>>>>>> related details can be found. All allocated memory tiers will be listed >>>>>>>>>>>>>>>>> there as /sys/devices/virtual/memory_tiering/memory_tierN/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The nodes which are part of a specific memory tier can be listed via >>>>>>>>>>>>>>>>> /sys/devices/virtual/memory_tiering/memory_tierN/nodes >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think "memory_tier" is a better subsystem/bus name than >>>>>>>>>>>>>>>> memory_tiering. Because we have a set of memory_tierN devices inside. >>>>>>>>>>>>>>>> "memory_tier" sounds more natural. I know this is subjective, just my >>>>>>>>>>>>>>>> preference. >>>>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I missed replying to this earlier. I will keep memory_tiering as subsystem name in v4 >>>>>>>>>>>> because we would want it to a susbsystem where all memory tiering related details can be found >>>>>>>>>>>> including memory type in the future. This is as per discussion >>>>>>>>>>>> >>>>>>>>>>>> https://lore.kernel.org/linux-mm/CAAPL-u9TKbHGztAF=r-io3gkX7gorUunS2UfstudCWuihrA=0g@mail.gmail.com >>>>>>>>>>> >>>>>>>>>>> I don't think that it's a good idea to mix 2 types of devices in one >>>>>>>>>>> subsystem (bus). If my understanding were correct, that breaks the >>>>>>>>>>> driver core convention. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> All these are virtual devices .I am not sure i follow what you mean by 2 types of devices. >>>>>>>>>> memory_tiering is a subsystem that represents all the details w.r.t memory tiering. It shows >>>>>>>>>> details of memory tiers and can possibly contain details of different memory types . >>>>>>>>> >>>>>>>>> IMHO, memory_tier and memory_type are 2 kind of devices. They have >>>>>>>>> almost totally different attributes (sysfs file). So, we should create >>>>>>>>> 2 buses for them. Each has its own attribute group. "virtual" itself >>>>>>>>> isn't a subsystem. >>>>>>>> >>>>>>>> Considering both the details are related to memory tiering, wouldn't it be much simpler we consolidate >>>>>>>> them within the same subdirectory? I am still not clear why you are suggesting they need to be in different >>>>>>>> sysfs hierarchy. It doesn't break any driver core convention as you mentioned earlier. >>>>>>>> >>>>>>>> /sys/devices/virtual/memory_tiering/memory_tierN >>>>>>>> /sys/devices/virtual/memory_tiering/memory_typeN >>>>>>> >>>>>>> I think we should add >>>>>>> >>>>>>> /sys/devices/virtual/memory_tier/memory_tierN >>>>>>> /sys/devices/virtual/memory_type/memory_typeN >>>>>>> >>>>>> >>>>>> I am trying to find if there is a technical reason to do the same? >>>>>> >>>>>>> I don't think this is complex. Devices of same bus/subsystem should >>>>>>> have mostly same attributes. This is my understanding of driver core >>>>>>> convention. >>>>>>> >>>>>> >>>>>> I was not looking at this from code complexity point. Instead of having multiple directories >>>>>> with details w.r.t memory tiering, I was looking at consolidating the details >>>>>> within the directory /sys/devices/virtual/memory_tiering. (similar to all virtual devices >>>>>> are consolidated within /sys/devics/virtual/). >>>>>> >>>>>> -aneesh >>>>> >>>>> Here is an example of /sys/bus/nd/devices (I know it is not under >>>>> /sys/devices/virtual, but it can still serve as a reference): >>>>> >>>>> ls -1 /sys/bus/nd/devices >>>>> >>>>> namespace2.0 >>>>> namespace3.0 >>>>> ndbus0 >>>>> nmem0 >>>>> nmem1 >>>>> region0 >>>>> region1 >>>>> region2 >>>>> region3 >>>>> >>>>> So I think it is not unreasonable if we want to group memory tiering >>>>> related interfaces within a single top directory. >>>> >>>> Thanks for pointing this out. My original understanding of driver core >>>> isn't correct. >>>> >>>> But I still think it's better to separate instead of mixing memory_tier >>>> and memory_type. Per my understanding, memory_type shows information >>>> (abstract distance, latency, bandwidth, etc.) of memory types (and >>>> nodes), it can be useful even without memory tiers. That is, memory >>>> types describes the physical characteristics, while memory tier reflects >>>> the policy. >>>> >>> >>> The latency and bandwidth details are already exposed via >>> >>> /sys/devices/system/node/nodeY/access0/initiators/ >>> >>> Documentation/admin-guide/mm/numaperf.rst >>> >>> That is the interface that libraries like libmemkind will look at for finding >>> details w.r.t latency/bandwidth >> >> Yes. Only with that, it's still inconvenient to find out which nodes >> belong to same memory type (has same performance, same topology, managed >> by same driver, etc). So memory types can still provide useful >> information even without memory tiering. >> > > I am not sure i quiet follow what to conclude from your reply. I used the subsystem name > "memory_tiering" so that all memory tiering related information can be consolidated there. > I guess you agreed to the above part that we can consolidated things like that. I just prefer to separate memory_tier and memory_type sysfs directories personally. Because memory_type describes the physical memory types and performance, while memory_tier is more about the policy to group memory_types. > We might end up adding memory_type there if we allow changing "abstract distance" of a > memory type from userspace later. Otherwise, I don't see a reason for memory type to be > exposed. But then we don't have to decide on this now. As above, because I think memory_type can provide value even outside of memory_tier, I prefer to add memory_type sysfs interface anyway personally. Best Regards, Huang, Ying