RE: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory tiers

[Srinivasulu Thanneeru <sthanneeru@micron.com>]

Micron Confidential



Micron Confidential
> -----Original Message-----
> From: Huang, Ying <ying.huang@intel.com>
> Sent: Wednesday, January 3, 2024 11:38 AM
> To: Srinivasulu Thanneeru <sthanneeru@micron.com>
> Cc: gregory.price <gregory.price@memverge.com>; Srinivasulu Opensrc
> <sthanneeru.opensrc@micron.com>; linux-cxl@vger.kernel.org; linux-
> mm@kvack.org; aneesh.kumar@linux.ibm.com; dan.j.williams@intel.com;
> mhocko@suse.com; tj@kernel.org; john@jagalactic.com; Eishan Mirakhur
> <emirakhur@micron.com>; Vinicius Tavares Petrucci
> <vtavarespetr@micron.com>; Ravis OpenSrc <Ravis.OpenSrc@micron.com>;
> Jonathan.Cameron@huawei.com; linux-kernel@vger.kernel.org; Johannes
> Weiner <hannes@cmpxchg.org>; Wei Xu <weixugc@google.com>
> Subject: Re: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
> tiers
>
> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
> you recognize the sender and were expecting this message.
>
>
> Srinivasulu Thanneeru <sthanneeru@micron.com> writes:
>
> > Micron Confidential
> >
> > Hi Huang, Ying,
> >
> > My apologies for wrong mail reply format, my mail client settings got
> changed on my PC.
> > Please find comments bellow inline.
> >
> > Regards,
> > Srini
> >
> >
> > Micron Confidential
> >> -----Original Message-----
> >> From: Huang, Ying <ying.huang@intel.com>
> >> Sent: Monday, December 18, 2023 11:26 AM
> >> To: gregory.price <gregory.price@memverge.com>
> >> Cc: Srinivasulu Opensrc <sthanneeru.opensrc@micron.com>; linux-
> >> cxl@vger.kernel.org; linux-mm@kvack.org; Srinivasulu Thanneeru
> >> <sthanneeru@micron.com>; aneesh.kumar@linux.ibm.com;
> >> dan.j.williams@intel.com; mhocko@suse.com; tj@kernel.org;
> >> john@jagalactic.com; Eishan Mirakhur <emirakhur@micron.com>; Vinicius
> >> Tavares Petrucci <vtavarespetr@micron.com>; Ravis OpenSrc
> >> <Ravis.OpenSrc@micron.com>; Jonathan.Cameron@huawei.com; linux-
> >> kernel@vger.kernel.org; Johannes Weiner <hannes@cmpxchg.org>; Wei Xu
> >> <weixugc@google.com>
> >> Subject: [EXT] Re: [RFC PATCH v2 0/2] Node migration between memory
> tiers
> >>
> >> CAUTION: EXTERNAL EMAIL. Do not click links or open attachments unless
> >> you recognize the sender and were expecting this message.
> >>
> >>
> >> Gregory Price <gregory.price@memverge.com> writes:
> >>
> >> > On Fri, Dec 15, 2023 at 01:02:59PM +0800, Huang, Ying wrote:
> >> >> <sthanneeru.opensrc@micron.com> writes:
> >> >>
> >> >> > =============
> >> >> > Version Notes:
> >> >> >
> >> >> > V2 : Changed interface to memtier_override from adistance_offset.
> >> >> > memtier_override was recommended by
> >> >> > 1. John Groves <john@jagalactic.com>
> >> >> > 2. Ravi Shankar <ravis.opensrc@micron.com>
> >> >> > 3. Brice Goglin <Brice.Goglin@inria.fr>
> >> >>
> >> >> It appears that you ignored my comments for V1 as follows ...
> >> >>
> >> >>
> >>
> https://lore.k/
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
> served=0
> >> ernel.org%2Flkml%2F87o7f62vur.fsf%40yhuang6-
> >>
> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
> >>
> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
> >>
> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
> >>
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >>
> D%7C3000%7C%7C%7C&sdata=OpMkYCar%2Fv8uHb7AvXbmaNltnXeTvcNUTi
> >> bLhwV12Fg%3D&reserved=0
> >
> > Thank you, Huang, Ying for pointing to this.
> >
> https://lpc.ev/
> ents%2Fevent%2F16%2Fcontributions%2F1209%2Fattachments%2F1042%2F1
> 995%2FLive%2520In%2520a%2520World%2520With%2520Multiple%2520Me
> mory%2520Types.pdf&data=05%7C02%7Csthanneeru%40micron.com%7C3e
> 5d38eb47be463c295c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806
> f%7C0%7C0%7C638398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJW
> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3
> 000%7C%7C%7C&sdata=1fGraxff7%2F1hNaE0an0xEudSKSUvaF3HgClMkmdC7
> n8%3D&reserved=0
> >
> > In the presentation above, the adistance_offsets are per memtype.
> > We believe that adistance_offset per node is more suitable and flexible.
> > since we can change it per node. If we keep adistance_offset per memtype,
> > then we cannot change it for a specific node of a given memtype.
> >
> >> >>
> >>
> https://lore.k/
> %2F&data=05%7C02%7Csthanneeru%40micron.com%7C3e5d38eb47be463c2
> 95c08dc0c229d22%7Cf38a5ecd28134862b11bac1d563c806f%7C0%7C0%7C63
> 8398590664228240%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMD
> AiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C
> &sdata=7fPxb1YYR2tZ0v2FB1vlXnMJFcI%2Fr9HT2%2BUD1MNUd%2FI%3D&re
> served=0
> >> ernel.org%2Flkml%2F87jzpt2ft5.fsf%40yhuang6-
> >>
> desk2.ccr.corp.intel.com%2F&data=05%7C02%7Csthanneeru%40micron.com
> >>
> %7C5e614e5f028342b6b59c08dbff8e3e37%7Cf38a5ecd28134862b11bac1d56
> >>
> 3c806f%7C0%7C0%7C638384758666895965%7CUnknown%7CTWFpbGZsb3d
> >>
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >>
> D%7C3000%7C%7C%7C&sdata=O0%2B6T%2FgU0TicCEYBac%2FAyjOLwAeouh
> >> D%2BcMI%2BflOsI1M%3D&reserved=0
> >
> > Yes, memory_type would be grouping the related memories together as
> single tier.
> > We should also have a flexibility to move nodes between tiers, to address
> the issues.
> > described in use cases above.
>
> We don't pursue absolute flexibility.  We add necessary flexibility
> only.  Why do you need this kind of flexibility?  Can you provide some
> use cases where memory_type based "adistance_offset" doesn't work?

- /sys/devices/virtual/memory_type/memory_type/ adistance_offset
memory_type based "adistance_offset will provide a way to move all nodes of same memory_type (e.g. all cxl nodes)
to different tier.

Whereas /sys/devices/system/node/node2/memtier_override provide a way migrate a node from one tier to another.
Considering a case where we would like to move two cxl nodes into two different tiers in future.
So, I thought it would be good to have flexibility at node level instead of at memory_type.

>
> --
> Best Regards,
> Huang, Ying