Re: Deferred Memory Init: How to bring rest of memory online after limiting it with `mem=XG`?

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear David, dear Daniel,


Am 03.12.20 um 21:58 schrieb Daniel Jordan:
> David Hildenbrand writes:
> 
>> On 03.12.20 13:52, Paul Menzel wrote:

>>> Am 03.12.20 um 13:25 schrieb David Hildenbrand:
>>>> On 03.12.20 11:51, Paul Menzel wrote:
>>>
>>>>> I am trying to reduce the startup time of Debian’s Linux 5.9.9 on a
>>>>> Intel Kaby Lake system with 32 GB of memory (TUXEDO Book BU1406 (Clevo
>>>>> N240BU)).
>>>
>>> […]
>>>
>>>> We do have deferred meminit in the kernel during boot that can
>>>> initialize memory in parallel.
>>>
>>> Is that used automatically, or do I need to activate it somehow?
>>
>> If your kernel is compiled with
>>
>> CONFIG_DEFERRED_STRUCT_PAGE_INIT
>>
>> it should be enabled automatically.

[…]

Yes, in Debian’s Linux kernel configuration, that option is selected.

> If it is enabled, what does
> 
> dmesg | grep 'deferred pages'
> 
> give you?

     $ grep 'deferred pages' dmesg-full.txt
     [    0.140199] node 0 deferred pages initialised in 40ms

     $ grep 'deferred pages' dmesg-2g.txt
     [    0.077892] node 0 deferred pages initialised in 4ms

> And assuming you're running systemd, what does systemd-analyze show you?
Please find it below. In my experience, unpacking the initrd is also 
good measurement point, and the time to write protect the kernel 
read-only data.

Without `mem=` (32 GB):

     $ dmesg
     […]
     [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.9.0-3-amd64 
root=UUID=d23ce27e-5c5c-45fb-bfa8-79a87caff13f ro quiet 
cryptomgr.notests ipv6.disable=1 log_buf_len=2M initcall_debug 
init=/lib/systemd/systemd-bootchart
     […]
     [    0.266513] Trying to unpack rootfs image as initramfs...
     [    0.275829] Freeing initrd memory: 4468K
     […]
     [    0.295890] Freeing unused kernel image (initmem) memory: 1640K
     [    0.335585] Write protecting the kernel read-only data: 18432k
     [    0.336261] Freeing unused kernel image (text/rodata gap) 
memory: 2044K
     [    0.336406] Freeing unused kernel image (rodata/data gap) 
memory: 292K
     [    0.392213] x86/mm: Checked W+X mappings: passed, no W+X pages 
found.
     [    0.392213] x86/mm: Checking user space page tables
     [    0.432697] x86/mm: Checked W+X mappings: passed, no W+X pages 
found.
     [    0.432701] Run /init as init process
     […]
     $ systemd-analyze time
     Startup finished in 3.792s (firmware) + 4.116s (loader) + 767ms 
(kernel) + 1.414s (userspace) = 10.091s
     graphical.target reached after 1.401s in userspace

With mem=2G

     $ dmesg
     […]
     [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.9.0-3-amd64 
root=UUID=d23ce27e-5c5c-45fb-bfa8-79a87caff13f ro quiet 
cryptomgr.notests ipv6.disable=1 log_buf_len=2M initcall_debug 
init=/lib/systemd/systemd-bootchart mem=2G
     […]
     [    0.199720] Trying to unpack rootfs image as initramfs...
     [    0.209058] Freeing initrd memory: 4468K
     […]
     [    0.227433] Freeing unused kernel image (initmem) memory: 1640K
     [    0.253494] Write protecting the kernel read-only data: 18432k
     [    0.253898] Freeing unused kernel image (text/rodata gap) 
memory: 2044K
     [    0.253987] Freeing unused kernel image (rodata/data gap) 
memory: 292K
     [    0.297996] x86/mm: Checked W+X mappings: passed, no W+X pages 
found.
     [    0.297997] x86/mm: Checking user space page tables
     [    0.338337] x86/mm: Checked W+X mappings: passed, no W+X pages 
found.
     [    0.338341] Run /init as init process
     $ sytsemd-analyze time
     Startup finished in 650ms (kernel) + 3.009s (userspace) = 3.659s
     graphical.target reached after 2.979s in userspace

(No idea why the firmware and loader timestamps are sometimes not 
available.)

So the different memory sizes result in an almost 100 ms difference 
during start-up, and initializing 32 GB results in a 30 percent longer 
boottime up to starting the init process. (Unfortunately userspace and 
several drivers later on take also quite some time.)

If you have ideas how to reduce the boot time of the full 32 GB, that’d 
be great.


Kind regards,

Paul