[LSF/MM/BPF TOPIC] persistent memory, tracing, KASLR, Oh my!

[Steven Rostedt <rostedt@goodmis.org>]

This is different than Mike's "memory persistence over kexec" topic which
has to do with Kexec Hand Over (KHO). This is about tracing and generic
pstore itself.

Here's the problem statement:

To enable tracing over persistent memory in the field we add to the kernel
command line:

  reserve_mem=20M:12M:trace trace_instance=boot_mapped^traceoff@trace

The "reserve_mem=20M:12M:trace" will reserve 20 megabytes of memory that is
aligned by 12 megabytes and label it "trace". The
"trace_instance=boot_mapped^traceoff@trace" will look for memory from the
"reserve_mem" that has the label "trace", assign it to the boot_mapped
instance (found in /sys/kernel/tracing/instances/boot_mapped) and it will
be have tracing disabled by default on boot. (Note that pstore can also get
its persistent memory from the "reserve_mem" option as well).

On boot up, various events will be started in the "boot_mapped" trace
instance. If there's a crash, on the next boot, the "boot_mapped" trace
instance will be read which will contain all the events that led up to the
crash (this can also trace reboots too). Obviously this depends on the
machine not clearing memory and only doing a soft reset where the memory is
still persistent from the last boot.

This also only works if the "reserve_mem=20M:12M:trace" reserves the memory
in the same location as it did from the previous boot. Hence the problem I
have here. Namely KASLR.

If I disable KASLR (nokaslr on the kernel command line) then I found that
the memory reserved is consistent between boots and I have not hit any
issues while debugging in the lab, or on my own devices, as I can safely
add "nokaslr" on test machines. But for productions machines and for
machines in the field, that is not an option.

KASLR can move the placement of the kernel where it overlaps the previous
saved buffer, or what I found is even more common, it causes something that
was reserved before the "reserve_mem" was called to be allocated in a
different location which caused the memory reserved by reserve_mem to move.
I also found (at least on a few machines I played with), adding a large
alignment (like 12M) helps with that from happening. That is, things that
were allocated before "reserve_mem" was parsed can move around and the
memory reserved stays consistent. That helps, but does not prevent it.

My question:

Would it be possible to have some communication between KASLR and the
kernel to know about this reserved memory and work to keep it from moving?

Since movement of allocations that happen before "reserve_mem" is parsed
can affect where the reserved memory is allocated, I'm not sure just
keeping KASLR away from the reserved memory is enough. Perhaps, after a
successful allocation, there can be some memory range that can communicate
back to KASLR on the next boot to actually reserve that memory location
again, and then communicate back to the booting kernel that this memory was
reserved there and the "reserve_mem" will just used that memory?

Really, I'm looking for any ideas that can help keep "reserve_mem"
allocating memory at the same location after each boot.

Thoughts?

-- Steve