[Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Dan Williams]

On Mon, Oct 12, 2015 at 12:01 PM, Theodore Ts'o <tytso@mit.edu> wrote:
> This is an initial draft for the upcoming kernel summit.
>
> It is still very drafty; in particular, just because a topic has been
> listed, it may turn out that the topic has been resolved off-line, or
> otherwise overtaken by events, in which case we will drop it.  In
> addition, we're also in the process of sending queries to the proposed
> topic leaders to see if they are willing to kick off the discussion,
> and so that is subject to change as well.
>
> I'd appreciate comments about any topics that you think might be
> missing and that would be worth our discussing.

I am wondering if it would be productive / good use of time to do a
direction check on the mm changes being done in support of large
persistent memory devices.  The new ZONE_DEVICE mm-zone was added in a
basic form in 4.3 [1].  The follow-on patches to make use of it have
not enjoyed much feedback to date, especially compared to the debate
that occurred when removing struct page from the block-layer was being
considered [2].

I'm assuming either the new approach is sufficiently non-controversial
and this is just normal patch review latency, or there are some
brewing discussion topics that might make forward progress in person.
One example might be unintended usages of devm_memremap_pages() as
some are experimenting with it to enable peer-to-peer PCI-E transfers.
Another might be the future of the effort to reduce the usage of, or
redefine, struct page.

[1]: https://git.kernel.org/cgit/linux/kernel/git/nvdimm/nvdimm.git/tag/?id=libnvdimm-for-4.3
[2]: https://lists.01.org/pipermail/linux-nvdimm/2015-May/000748.html

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Dave Hansen]

On 10/14/2015 05:17 PM, Dan Williams wrote:
> I am wondering if it would be productive / good use of time to do a
> direction check on the mm changes being done in support of large
> persistent memory devices.

I think there's probably an even wider discussion that we should have here.

Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
fast, and our current solutions are, at best, inconsistent. We currently
handle memory types as new zones, repurposed zones, pageblocks inside
zones, or faux NUMA nodes.

Are our current solutions too erratic?
Do we need to solve these problems generally, or are we going to kill
ourselves trying to make everyone happy?
What types do we ignore today, but shouldn't?
What types are coming down the pike?

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Jerome Glisse]

On Thu, Oct 15, 2015 at 09:49:49AM -0700, Dave Hansen wrote:
> On 10/14/2015 05:17 PM, Dan Williams wrote:
> > I am wondering if it would be productive / good use of time to do a
> > direction check on the mm changes being done in support of large
> > persistent memory devices.
> 
> I think there's probably an even wider discussion that we should have here.
> 
> Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
> fast, and our current solutions are, at best, inconsistent. We currently
> handle memory types as new zones, repurposed zones, pageblocks inside
> zones, or faux NUMA nodes.
> 
> Are our current solutions too erratic?
> Do we need to solve these problems generally, or are we going to kill
> ourselves trying to make everyone happy?
> What types do we ignore today, but shouldn't?
> What types are coming down the pike?

I think what i am working on (HMM and how to leverage GPU memory that is
not accessible by the CPU) apply here too. All features i am working on
imply that i have to dig through layers of code seeing if i can abuse an
existing mechanism to achieve something new.

So i definitly think we should discuss where we are and where we want to
be.

Cheers,
Jérôme

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Theodore Ts'o]

On Fri, Oct 16, 2015 at 05:08:21PM -0400, Jerome Glisse wrote:
> On Thu, Oct 15, 2015 at 09:49:49AM -0700, Dave Hansen wrote:
> > On 10/14/2015 05:17 PM, Dan Williams wrote:
> > > I am wondering if it would be productive / good use of time to do a
> > > direction check on the mm changes being done in support of large
> > > persistent memory devices.
> > 
> > I think there's probably an even wider discussion that we should have here.
> > 
> > Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
> > fast, and our current solutions are, at best, inconsistent. We currently
> > handle memory types as new zones, repurposed zones, pageblocks inside
> > zones, or faux NUMA nodes.
> > 
> > Are our current solutions too erratic?
> > Do we need to solve these problems generally, or are we going to kill
> > ourselves trying to make everyone happy?
> > What types do we ignore today, but shouldn't?
> > What types are coming down the pike?
> 
> I think what i am working on (HMM and how to leverage GPU memory that is
> not accessible by the CPU) apply here too. All features i am working on
> imply that i have to dig through layers of code seeing if i can abuse an
> existing mechanism to achieve something new.
> 
> So i definitly think we should discuss where we are and where we want to
> be.

Dan, would you be willing to kick off the discussion?

							- Ted
							

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Theodore Ts'o]

On Mon, Oct 19, 2015 at 02:29:23PM -0400, Theodore Ts'o wrote:
> On Fri, Oct 16, 2015 at 05:08:21PM -0400, Jerome Glisse wrote:
> > On Thu, Oct 15, 2015 at 09:49:49AM -0700, Dave Hansen wrote:
> > > On 10/14/2015 05:17 PM, Dan Williams wrote:
> > > > I am wondering if it would be productive / good use of time to do a
> > > > direction check on the mm changes being done in support of large
> > > > persistent memory devices.
> > > 
> > > I think there's probably an even wider discussion that we should have here.
> > > 
> > > Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
> > > fast, and our current solutions are, at best, inconsistent. We currently
> > > handle memory types as new zones, repurposed zones, pageblocks inside
> > > zones, or faux NUMA nodes.
> > > 
> > > Are our current solutions too erratic?
> > > Do we need to solve these problems generally, or are we going to kill
> > > ourselves trying to make everyone happy?
> > > What types do we ignore today, but shouldn't?
> > > What types are coming down the pike?
> > 
> > I think what i am working on (HMM and how to leverage GPU memory that is
> > not accessible by the CPU) apply here too. All features i am working on
> > imply that i have to dig through layers of code seeing if i can abuse an
> > existing mechanism to achieve something new.
> > 
> > So i definitly think we should discuss where we are and where we want to
> > be.
> 
> Dan, would you be willing to kick off the discussion?

Oops, sorry, it was pointed out to me that Dan isn't going to be at
the Kernel Summit.  Jerome, would you be willing to kick off the
discussion and report back to Dan?

						- Ted

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Dan Williams]

On Mon, Oct 19, 2015 at 11:52 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> On Mon, Oct 19, 2015 at 02:29:23PM -0400, Theodore Ts'o wrote:
>> On Fri, Oct 16, 2015 at 05:08:21PM -0400, Jerome Glisse wrote:
>> > On Thu, Oct 15, 2015 at 09:49:49AM -0700, Dave Hansen wrote:
>> > > On 10/14/2015 05:17 PM, Dan Williams wrote:
>> > > > I am wondering if it would be productive / good use of time to do a
>> > > > direction check on the mm changes being done in support of large
>> > > > persistent memory devices.
>> > >
>> > > I think there's probably an even wider discussion that we should have here.
>> > >
>> > > Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
>> > > fast, and our current solutions are, at best, inconsistent. We currently
>> > > handle memory types as new zones, repurposed zones, pageblocks inside
>> > > zones, or faux NUMA nodes.
>> > >
>> > > Are our current solutions too erratic?
>> > > Do we need to solve these problems generally, or are we going to kill
>> > > ourselves trying to make everyone happy?
>> > > What types do we ignore today, but shouldn't?
>> > > What types are coming down the pike?
>> >
>> > I think what i am working on (HMM and how to leverage GPU memory that is
>> > not accessible by the CPU) apply here too. All features i am working on
>> > imply that i have to dig through layers of code seeing if i can abuse an
>> > existing mechanism to achieve something new.
>> >
>> > So i definitly think we should discuss where we are and where we want to
>> > be.
>>
>> Dan, would you be willing to kick off the discussion?
>
> Oops, sorry, it was pointed out to me that Dan isn't going to be at
> the Kernel Summit.  Jerome, would you be willing to kick off the
> discussion and report back to Dan?

Well, you were right the first time ;-), I'm booked and registered.

Yes, I'd be willing to kick off the discussion.

Re: [Ksummit-discuss] ZONE_DEVICE and Persistent Memory (was: Re: Draft agenda for the kernel summit)

[Jerome Glisse]

On Mon, Oct 19, 2015 at 12:23:47PM -0700, Dan Williams wrote:
> On Mon, Oct 19, 2015 at 11:52 AM, Theodore Ts'o <tytso@mit.edu> wrote:
> > On Mon, Oct 19, 2015 at 02:29:23PM -0400, Theodore Ts'o wrote:
> >> On Fri, Oct 16, 2015 at 05:08:21PM -0400, Jerome Glisse wrote:
> >> > On Thu, Oct 15, 2015 at 09:49:49AM -0700, Dave Hansen wrote:
> >> > > On 10/14/2015 05:17 PM, Dan Williams wrote:
> >> > > > I am wondering if it would be productive / good use of time to do a
> >> > > > direction check on the mm changes being done in support of large
> >> > > > persistent memory devices.
> >> > >
> >> > > I think there's probably an even wider discussion that we should have here.
> >> > >
> >> > > Beyond just ZONE_DEVICE, the sheer number of memory types is increasing
> >> > > fast, and our current solutions are, at best, inconsistent. We currently
> >> > > handle memory types as new zones, repurposed zones, pageblocks inside
> >> > > zones, or faux NUMA nodes.
> >> > >
> >> > > Are our current solutions too erratic?
> >> > > Do we need to solve these problems generally, or are we going to kill
> >> > > ourselves trying to make everyone happy?
> >> > > What types do we ignore today, but shouldn't?
> >> > > What types are coming down the pike?
> >> >
> >> > I think what i am working on (HMM and how to leverage GPU memory that is
> >> > not accessible by the CPU) apply here too. All features i am working on
> >> > imply that i have to dig through layers of code seeing if i can abuse an
> >> > existing mechanism to achieve something new.
> >> >
> >> > So i definitly think we should discuss where we are and where we want to
> >> > be.
> >>
> >> Dan, would you be willing to kick off the discussion?
> >
> > Oops, sorry, it was pointed out to me that Dan isn't going to be at
> > the Kernel Summit.  Jerome, would you be willing to kick off the
> > discussion and report back to Dan?
> 
> Well, you were right the first time ;-), I'm booked and registered.
> 
> Yes, I'd be willing to kick off the discussion.

I can help keeping the discussion going on :)