Re: [Ksummit-discuss] [CORE TOPIC] Redesign Memory Management layer and more core subsystem

[Benjamin Herrenschmidt <benh@au1.ibm.com>]

On Fri, 2014-06-13 at 12:02 -0500, Christoph Lameter wrote:
> On Thu, 12 Jun 2014, Phillip Lougher wrote:
> 
> > > 1. The need to use larger order pages, and the resulting problems with
> > > fragmentation. Memory sizes grow and therefore the number of page structs
> > > where state has to be maintained. Maybe there is something different? If
> > > we use hugepages then we have 511 useless page structs. Some apps need
> > > linear memory where we have trouble and are creating numerous memory
> > > allocators (recently the new bootmem allocator and CMA. Plus lots of
> > > specialized allocators in various subsystems).
> > >
> >
> > This was never solved to my knowledge, there is no panacea here.
> > Even in the 90s we had video subsystems wanting to allocate in units
> > of 1Mbyte, and others in units of 4k.  The "solution" was so called
> > split-level allocators, each specialised to deal with a particular
> > "first class media", with them giving back memory to the underlying
> > allocator when memory got tight in another specialised allocator.
> > Not much different to the ad-hoc solutions being adopted in Linux,
> > except the general idea was each specialised allocator had the same
> > API.
> 
> It is solvable if the objects are inherent movable. If any object
> allocated provides a function that makes an object movable then
> defragmentation is possible and therefore large contiguous area of memory
> can be created at any time.

Another interesting thing is migration of pages with mapped DMA on
them :-)

Our IOMMUs support that, but there isn't a way to hook that up into
Linux page migration that wouldn't suck massively at this point.

> > > Can we develop the notion that subsystems own certain cores so that their
> > > execution is restricted to a subset of the system avoiding data
> > > replication and keeping subsystem data hot? I.e. have a device driver
> > > and subsystems driving those devices just run on the NUMA node to which
> > > the PCI-E root complex is attached. Restricting to NUMA node reduces data
> > > locality complexity and increases performance due to cache hot data.
> >
> > Lots of academic hot-air was expended here when designing distributed
> > systems which could scale seamlessly across heterogeneous CPUs connected
> > via different levels of interconnects (bus, ATM, ethernet etc.), zoning,
> > migration, replication etc.  The "solution" is probably out there somewhere
> > forgotten about.
> 
> We have the issue with homogenous cpus due to the proliferation of cores
> on processors now. Maybe that is solvable?
> 
> > Case in point, many years ago I was the lead Linux guy for a company
> > designing a SOC for digital TV.  Just before I left I had an interesting
> > "conversation" with the chief hardware guy of the team who designed the SOC.
> > Turns out they'd budgeted for the RAM bandwidth needed to decode a typical
> > MPEG stream, but they'd not reckoned on all the memcopies Linux needs to do
> > between its "separate address space" processes.  He'd been used to embedded
> > oses which run in a single address space.
> 
> Well maybe that is appropriate for some processes? And we could carve out
> subsections of the hardware where single adress space stuff is possible?
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss