ping...... 2013/9/16 Zhang Yanfei > Hello tejun, > > Could you please help reviewing the patchset? As you suggested, > we've make the patchset much simpler and cleaner. > > Thanks in advance! > > On 09/13/2013 05:30 PM, Tang Chen wrote: > > This patch-set is based on tj's suggestion, and not fully tested. > > Just for review and discussion. > > > > This patch-set is based on the latest kernel (3.11) > > HEAD is: > > commit d5d04bb48f0eb89c14e76779bb46212494de0bec > > Author: Linus Torvalds > > Date: Wed Sep 11 19:55:12 2013 -0700 > > > > > > [Problem] > > > > The current Linux cannot migrate pages used by the kerenl because > > of the kernel direct mapping. In Linux kernel space, va = pa + > PAGE_OFFSET. > > When the pa is changed, we cannot simply update the pagetable and > > keep the va unmodified. So the kernel pages are not migratable. > > > > There are also some other issues will cause the kernel pages not > migratable. > > For example, the physical address may be cached somewhere and will be > used. > > It is not to update all the caches. > > > > When doing memory hotplug in Linux, we first migrate all the pages in one > > memory device somewhere else, and then remove the device. But if pages > are > > used by the kernel, they are not migratable. As a result, memory used by > > the kernel cannot be hot-removed. > > > > Modifying the kernel direct mapping mechanism is too difficult to do. And > > it may cause the kernel performance down and unstable. So we use the > following > > way to do memory hotplug. > > > > > > [What we are doing] > > > > In Linux, memory in one numa node is divided into several zones. One of > the > > zones is ZONE_MOVABLE, which the kernel won't use. > > > > In order to implement memory hotplug in Linux, we are going to arrange > all > > hotpluggable memory in ZONE_MOVABLE so that the kernel won't use these > memory. > > To do this, we need ACPI's help. > > > > In ACPI, SRAT(System Resource Affinity Table) contains NUMA info. The > memory > > affinities in SRAT record every memory range in the system, and also, > flags > > specifying if the memory range is hotpluggable. > > (Please refer to ACPI spec 5.0 5.2.16) > > > > With the help of SRAT, we have to do the following two things to achieve > our > > goal: > > > > 1. When doing memory hot-add, allow the users arranging hotpluggable as > > ZONE_MOVABLE. > > (This has been done by the MOVABLE_NODE functionality in Linux.) > > > > 2. when the system is booting, prevent bootmem allocator from allocating > > hotpluggable memory for the kernel before the memory initialization > > finishes. > > > > The problem 2 is the key problem we are going to solve. But before > solving it, > > we need some preparation. Please see below. > > > > > > [Preparation] > > > > Bootloader has to load the kernel image into memory. And this memory > must be > > unhotpluggable. We cannot prevent this anyway. So in a memory hotplug > system, > > we can assume any node the kernel resides in is not hotpluggable. > > > > Before SRAT is parsed, we don't know which memory ranges are > hotpluggable. But > > memblock has already started to work. In the current kernel, memblock > allocates > > the following memory before SRAT is parsed: > > > > setup_arch() > > |->memblock_x86_fill() /* memblock is ready */ > > |...... > > |->early_reserve_e820_mpc_new() /* allocate memory under 1MB */ > > |->reserve_real_mode() /* allocate memory under 1MB */ > > |->init_mem_mapping() /* allocate page tables, about 2MB to > map 1GB memory */ > > |->dma_contiguous_reserve() /* specified by user, should be low */ > > |->setup_log_buf() /* specified by user, several mega > bytes */ > > |->relocate_initrd() /* could be large, but will be freed > after boot, should reorder */ > > |->acpi_initrd_override() /* several mega bytes */ > > |->reserve_crashkernel() /* could be large, should reorder */ > > |...... > > |->initmem_init() /* Parse SRAT */ > > > > According to Tejun's advice, before SRAT is parsed, we should try our > best to > > allocate memory near the kernel image. Since the whole node the kernel > resides > > in won't be hotpluggable, and for a modern server, a node may have at > least 16GB > > memory, allocating several mega bytes memory around the kernel image > won't cross > > to hotpluggable memory. > > > > > > [About this patch-set] > > > > So this patch-set does the following: > > > > 1. Make memblock be able to allocate memory from low address to high > address. > > 1) Keep all the memblock APIs' prototype unmodified. > > 2) When the direction is bottom up, keep the start address greater > than the > > end of kernel image. > > > > 2. Improve init_mem_mapping() to support allocate page tables in bottom > up direction. > > > > 3. Introduce "movablenode" boot option to enable and disable this > functionality. > > > > PS: Reordering of relocate_initrd() has not been done yet. > acpi_initrd_override() > > needs to access initrd with virtual address. So relocate_initrd() > must be done > > before acpi_initrd_override(). > > > > > > Change log v2 -> v3: > > 1. According to Toshi's suggestion, move the direction checking logic > into memblock. > > And simply the code more. > > > > Change log v1 -> v2: > > 1. According to tj's suggestion, implemented a new function > memblock_alloc_bottom_up() > > to allocate memory from bottom upwards, whihc can simplify the code. > > > > > > Tang Chen (5): > > memblock: Introduce allocation direction to memblock. > > memblock: Improve memblock to support allocation from lower address. > > x86, acpi, crash, kdump: Do reserve_crashkernel() after SRAT is > > parsed. > > x86, mem-hotplug: Support initialize page tables from low to high. > > mem-hotplug: Introduce movablenode boot option to control memblock > > allocation direction. > > > > Documentation/kernel-parameters.txt | 15 ++++ > > arch/x86/kernel/setup.c | 44 ++++++++++++- > > arch/x86/mm/init.c | 121 > ++++++++++++++++++++++++++-------- > > include/linux/memblock.h | 22 ++++++ > > include/linux/memory_hotplug.h | 5 ++ > > mm/memblock.c | 120 > +++++++++++++++++++++++++++++++---- > > mm/memory_hotplug.c | 9 +++ > > 7 files changed, 293 insertions(+), 43 deletions(-) > > > > > > > -- > Thanks. > Zhang Yanfei > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ >