Re: Ipmi modules and linux-4.19.1

["Paul E. McKenney" <paulmck@linux.ibm.com>]

Also adding Corey.  ;-)

On Thu, Dec 20, 2018 at 08:03:13AM -0800, Paul E. McKenney wrote:
> On Thu, Dec 20, 2018 at 07:42:17AM -0800, Tejun Heo wrote:
> > Hello, Angel.
> > 
> > (cc'ing Paul for SRCU)
> > 
> > On Thu, Dec 20, 2018 at 09:55:10AM +0200, Angel Shtilianov wrote:
> > > Hi everybody.
> > > A couple of days I've decided to migrate several servers on
> > > linux-4.19. What I've observed is that I have no /dev/ipmi. After
> > > taking a look into the boot log I've found that ipmi modules are
> > > complaining about percpu memory allocation failures:
> > > https://pastebin.com/MCDssZzV
> > ...
> > > -#define PERCPU_DYNAMIC_RESERVE         (28 << 10)
> > > +#define PERCPU_DYNAMIC_RESERVE         (28 << 11)
> > 
> > So, you prolly just needed to bump this number.  The reserved percpu
> > area is used to accommodate static percpu variables used by modules.
> > They are special because code generation assumes static symbols aren't
> > too far from the program counter.  The usual dynamic percpu area is
> > way high up in vmalloc area, so if we put static percpu allocations
> > there, they go out of range for module symbol relocations.
> > 
> > The reserved area has some issues.
> > 
> > 1. The area is not dynamically mapped, meaning that however much we
> >    reserve is hard allocated on boot for future module uses, so we
> >    don't can't increase it willy-nilly.
> > 
> > 2. There is no mechanism to adjust the size dynamically.  28k is just
> >    a number I pulled out of my ass after looking at some common
> >    configs like a decade ago, so it being low now isn't too
> >    surprising.  Provided that we can't make it run-time dynamic (and I
> >    can't think of a way to do that), the right thing to do would be
> >    sizing it during build with some buffer and allow it to be
> >    overridden boot time.  This is definitely doable.
> > 
> > BTW, ipmi's extra usage, 8k, is coming from the use of static SRCU.
> > Paul, that's quite a bit of percpu memory to reserve statically.
> > Would it be possible to make srcu_struct init dynamic so that it can
> > use the normal percpu_alloc?  That way, this problem can be completely
> > side-stepped and it only occupies percpu memory which tends to be
> > pretty expensive unless ipmi is actually initialized.
> 
> Yes, it is possible.  Just do something like this:
> 
> 	struct srcu_struct my_srcu_struct;
> 
> And before the first use of my_srcu_struct, do this:
> 
> 	init_srcu_struct(&my_srcu_struct);
> 
> This will result in alloc_percpu() being invoked to allocate the
> needed per-CPU space.
> 
> If my_srcu_struct is used in a module or some such, then to avoid memory
> leaks, after the last use of my_srcu_struct, do this:
> 
> 	cleanup_srcu_struct(&my_srcu_struct);
> 
> There are several places in the kernel that take this approach.
> 
> 							Thanx, Paul