* [PATCH RESEND 1/3] driver core: export lock_device_hotplug/unlock_device_hotplug
2015-02-12 10:23 [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Vitaly Kuznetsov
@ 2015-02-12 10:23 ` Vitaly Kuznetsov
2015-02-12 10:23 ` [PATCH RESEND 2/3] memory_hotplug: add note about holding device_hotplug_lock and add_memory() Vitaly Kuznetsov
` (3 subsequent siblings)
4 siblings, 0 replies; 16+ messages in thread
From: Vitaly Kuznetsov @ 2015-02-12 10:23 UTC (permalink / raw)
To: linux-kernel
Cc: Greg Kroah-Hartman, K. Y. Srinivasan, Haiyang Zhang,
Andrew Morton, Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka,
David Rientjes, Fabian Frederick, Zhang Zhen, Vladimir Davydov,
Wang Nan, Rafael J. Wysocki, devel, linux-mm
add_memory() is supposed to be run with device_hotplug_lock grabbed, otherwise
it can race with e.g. device_online(). Allow external modules (hv_balloon for
now) to lock device hotplug.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/base/core.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 97e2baf..b3073af 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -55,11 +55,13 @@ void lock_device_hotplug(void)
{
mutex_lock(&device_hotplug_lock);
}
+EXPORT_SYMBOL_GPL(lock_device_hotplug);
void unlock_device_hotplug(void)
{
mutex_unlock(&device_hotplug_lock);
}
+EXPORT_SYMBOL_GPL(unlock_device_hotplug);
int lock_device_hotplug_sysfs(void)
{
--
1.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH RESEND 2/3] memory_hotplug: add note about holding device_hotplug_lock and add_memory()
2015-02-12 10:23 [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Vitaly Kuznetsov
2015-02-12 10:23 ` [PATCH RESEND 1/3] driver core: export lock_device_hotplug/unlock_device_hotplug Vitaly Kuznetsov
@ 2015-02-12 10:23 ` Vitaly Kuznetsov
2015-02-12 10:23 ` [PATCH RESEND 3/3] Drivers: hv: balloon: fix deadlock between memory adding and onlining Vitaly Kuznetsov
` (2 subsequent siblings)
4 siblings, 0 replies; 16+ messages in thread
From: Vitaly Kuznetsov @ 2015-02-12 10:23 UTC (permalink / raw)
To: linux-kernel
Cc: Greg Kroah-Hartman, K. Y. Srinivasan, Haiyang Zhang,
Andrew Morton, Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka,
David Rientjes, Fabian Frederick, Zhang Zhen, Vladimir Davydov,
Wang Nan, Rafael J. Wysocki, devel, linux-mm
add_memory() is supposed to be run with device_hotplug_lock grabbed, otherwise
it can race with e.g. device_online(). ACPI memory hotplug does that already
but e.g. Hyper-V ballooning driver doesn't.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
mm/memory_hotplug.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 9fab107..41638eb 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1213,7 +1213,11 @@ int zone_for_memory(int nid, u64 start, u64 size, int zone_default)
return zone_default;
}
-/* we are OK calling __meminit stuff here - we have CONFIG_MEMORY_HOTPLUG */
+/*
+ * NOTE: The caller must call lock_device_hotplug() to serialize hotplug
+ * and online/offline operations before this call.
+ * We are OK calling __meminit stuff here - we have CONFIG_MEMORY_HOTPLUG.
+ */
int __ref add_memory(int nid, u64 start, u64 size)
{
pg_data_t *pgdat = NULL;
--
1.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH RESEND 3/3] Drivers: hv: balloon: fix deadlock between memory adding and onlining
2015-02-12 10:23 [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Vitaly Kuznetsov
2015-02-12 10:23 ` [PATCH RESEND 1/3] driver core: export lock_device_hotplug/unlock_device_hotplug Vitaly Kuznetsov
2015-02-12 10:23 ` [PATCH RESEND 2/3] memory_hotplug: add note about holding device_hotplug_lock and add_memory() Vitaly Kuznetsov
@ 2015-02-12 10:23 ` Vitaly Kuznetsov
2015-02-12 15:38 ` [PATCH RESEND 0/3] memory_hotplug: hyperv: " KY Srinivasan
2015-03-06 15:50 ` Michal Hocko
4 siblings, 0 replies; 16+ messages in thread
From: Vitaly Kuznetsov @ 2015-02-12 10:23 UTC (permalink / raw)
To: linux-kernel
Cc: Greg Kroah-Hartman, K. Y. Srinivasan, Haiyang Zhang,
Andrew Morton, Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka,
David Rientjes, Fabian Frederick, Zhang Zhen, Vladimir Davydov,
Wang Nan, Rafael J. Wysocki, devel, linux-mm
If newly added memory is brought online with e.g. udev rule:
SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
the following deadlock is observed (and easily reproducable):
First participant, worker thread doing add_memory():
...
[ 725.491469] 6 locks held by kworker/0:1/27:
[ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
[ 725.533370] #1: ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
[ 725.565580] #2: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
[ 725.594369] #3: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
[ 725.628554] #4: (mem_sysfs_mutex){......}, at: [<ffffffff81601873>] register_new_memory+0x33/0xd0
[ 725.658519] #5: (&dev->mutex){......}, at: [<ffffffff815ed773>] device_attach+0x23/0xb0
Second participant, udev:
...
[ 726.150691] 7 locks held by systemd-udevd/888:
[ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>] vfs_write+0x1b3/0x1f0
[ 726.192422] #1: (&of->mutex){......}, at: [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0
[ 726.220289] #2: (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0
[ 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>] lock_device_hotplug_sysfs+0x15/0x50
[ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>] device_online+0x23/0xa0
[ 726.308619] #5: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
[ 726.337994] #6: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
Solve the issue bu grabbing device_hotplug_lock before doing add_memory(). If
we do that, lock_device_hotplug_sysfs() will cause syscall retry which will
eventually succeed.
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
drivers/hv/hv_balloon.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/drivers/hv/hv_balloon.c b/drivers/hv/hv_balloon.c
index b958ded..0af1aa2 100644
--- a/drivers/hv/hv_balloon.c
+++ b/drivers/hv/hv_balloon.c
@@ -592,9 +592,19 @@ static void hv_mem_hot_add(unsigned long start, unsigned long size,
dm_device.ha_waiting = true;
nid = memory_add_physaddr_to_nid(PFN_PHYS(start_pfn));
+
+ /*
+ * Grab hotplug lock as we'll be doing device_register() and we
+ * need to protect against someone (e.g. udev doing memory
+ * onlining) locking it before we're done.
+ */
+ lock_device_hotplug();
+
ret = add_memory(nid, PFN_PHYS((start_pfn)),
(HA_CHUNK << PAGE_SHIFT));
+ unlock_device_hotplug();
+
if (ret) {
pr_info("hot_add memory failed error is %d\n", ret);
if (ret == -EEXIST) {
--
1.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* RE: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 10:23 [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Vitaly Kuznetsov
` (2 preceding siblings ...)
2015-02-12 10:23 ` [PATCH RESEND 3/3] Drivers: hv: balloon: fix deadlock between memory adding and onlining Vitaly Kuznetsov
@ 2015-02-12 15:38 ` KY Srinivasan
2015-02-12 15:50 ` Vitaly Kuznetsov
2015-02-12 22:12 ` Rafael J. Wysocki
2015-03-06 15:50 ` Michal Hocko
4 siblings, 2 replies; 16+ messages in thread
From: KY Srinivasan @ 2015-02-12 15:38 UTC (permalink / raw)
To: Vitaly Kuznetsov, linux-kernel
Cc: Greg Kroah-Hartman, Haiyang Zhang, Andrew Morton,
Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka, David Rientjes,
Fabian Frederick, Zhang Zhen, Vladimir Davydov, Wang Nan,
Rafael J. Wysocki, devel, linux-mm
> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Thursday, February 12, 2015 2:24 AM
> To: linux-kernel@vger.kernel.org
> Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
> Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes; Fabian
> Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J. Wysocki;
> devel@linuxdriverproject.org; linux-mm@kvack.org
> Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> between memory adding and onlining
>
> RESEND (with no changes) because Rafael J. Wysocki was missing in
> recepients.
>
> If newly added memory is brought online with e.g. udev rule:
> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> the following deadlock is observed (and easily reproducable):
>
> First participant, worker thread doing add_memory():
>
> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
> [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [ 724.991736]
> ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 [
> 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
> ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
> ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> [ 725.082690] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [ 725.140992]
> [<ffffffff815ed773>] device_attach+0x23/0xb0 [ 725.159131]
> [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [ 725.177055]
> [<ffffffff815ea693>] device_add+0x443/0x650 [ 725.195558]
> [<ffffffff815ea8be>] device_register+0x1e/0x30 [ 725.213133]
> [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [ 725.231533]
> [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [ 725.250769]
> [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [ 725.269642]
> [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [ 725.288764]
> [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [ 725.306117]
> [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [ 725.322466]
> [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon] [ 725.342777]
> [<ffffffff8109509f>] process_one_work+0x1df/0x4e0 [ 725.361459]
> [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0 [ 725.380390]
> [<ffffffff810954bb>] worker_thread+0x11b/0x450 [ 725.397684]
> [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0 [ 725.416533]
> [<ffffffff8109ac33>] kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>]
> ? kthread_create_on_node+0x240/0x240
> [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [ 725.470994]
> [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
> [ 725.491469] 6 locks held by kworker/0:1/27:
> [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
> process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> register_new_memory+0x33/0xd0 [ 725.658519] #5: (&dev->mutex){......},
> at: [<ffffffff815ed773>] device_attach+0x23/0xb0
>
> Second participant, udev:
>
> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> 0x00000004
> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
> 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
> 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
> 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call Trace:
> [ 725.858987] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
> 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
> 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
> 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [ 725.971542]
> [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [ 725.989207]
> [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [ 726.008513]
> [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [ 726.025579]
> [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [ 726.043400]
> [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [ 726.060506]
> [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [ 726.077940]
> [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [ 726.099416]
> [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748] [<ffffffff811fabf8>]
> SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by systemd-
> udevd/888:
> [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>]
> vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0 [
> 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> lock_device_hotplug_sysfs+0x15/0x50
> [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
> device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......}, at:
> [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> mem_hotplug_begin+0x4f/0x80
>
> In short: onlining grabs device lock and then tries to do
> mem_hotplug_begin() while add_memory() is between
> mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
> device lock.
>
> To my understanding ACPI memory hotplug doesn't have the same issue as
> device_hotplug_lock is being grabbed when the ACPI device is added.
>
> Solve the issue by grabbing device_hotplug_lock before doing
> add_memory(). If we do that, lock_device_hotplug_sysfs() will cause syscall
> retry which will eventually succeed. To support the change we need to
> export lock_device_hotplug/ unlock_device_hotplug. This approach can be
> completely wrong though.
This issue was first discovered by Andy Whitcroft: https://lkml.org/lkml/2014/3/14/451
I had sent patches based on Andy's analysis that did not affect the users of the kernel hot-add
memory APIs: https://lkml.org/lkml/2014/12/2/662
This patch puts the burden where it needs to be and can address the issue for all clients.
K. Y
>
> Vitaly Kuznetsov (3):
> driver core: export lock_device_hotplug/unlock_device_hotplug
> memory_hotplug: add note about holding device_hotplug_lock and
> add_memory()
> Drivers: hv: balloon: fix deadlock between memory adding and onlining
>
> drivers/base/core.c | 2 ++
> drivers/hv/hv_balloon.c | 10 ++++++++++
> mm/memory_hotplug.c | 6 +++++-
> 3 files changed, 17 insertions(+), 1 deletion(-)
>
> --
> 1.9.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 15:38 ` [PATCH RESEND 0/3] memory_hotplug: hyperv: " KY Srinivasan
@ 2015-02-12 15:50 ` Vitaly Kuznetsov
2015-02-12 17:40 ` KY Srinivasan
2015-02-12 22:12 ` Rafael J. Wysocki
1 sibling, 1 reply; 16+ messages in thread
From: Vitaly Kuznetsov @ 2015-02-12 15:50 UTC (permalink / raw)
To: KY Srinivasan
Cc: linux-kernel, Greg Kroah-Hartman, Haiyang Zhang, Andrew Morton,
Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka, David Rientjes,
Fabian Frederick, Zhang Zhen, Vladimir Davydov, Wang Nan,
Rafael J. Wysocki, devel, linux-mm
KY Srinivasan <kys@microsoft.com> writes:
>> -----Original Message-----
>> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
>> Sent: Thursday, February 12, 2015 2:24 AM
>> To: linux-kernel@vger.kernel.org
>> Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
>> Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes; Fabian
>> Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J. Wysocki;
>> devel@linuxdriverproject.org; linux-mm@kvack.org
>> Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
>> between memory adding and onlining
>>
>> RESEND (with no changes) because Rafael J. Wysocki was missing in
>> recepients.
>>
>> If newly added memory is brought online with e.g. udev rule:
>> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
>> the following deadlock is observed (and easily reproducable):
>>
>> First participant, worker thread doing add_memory():
>>
>> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
>> [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [ 724.991736]
>> ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 [
>> 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
>> ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
>> ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
>> [ 725.082690] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
>> [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
>> 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [ 725.140992]
>> [<ffffffff815ed773>] device_attach+0x23/0xb0 [ 725.159131]
>> [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [ 725.177055]
>> [<ffffffff815ea693>] device_add+0x443/0x650 [ 725.195558]
>> [<ffffffff815ea8be>] device_register+0x1e/0x30 [ 725.213133]
>> [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [ 725.231533]
>> [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [ 725.250769]
>> [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [ 725.269642]
>> [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [ 725.288764]
>> [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [ 725.306117]
>> [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [ 725.322466]
>> [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon] [ 725.342777]
>> [<ffffffff8109509f>] process_one_work+0x1df/0x4e0 [ 725.361459]
>> [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0 [ 725.380390]
>> [<ffffffff810954bb>] worker_thread+0x11b/0x450 [ 725.397684]
>> [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0 [ 725.416533]
>> [<ffffffff8109ac33>] kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>]
>> ? kthread_create_on_node+0x240/0x240
>> [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [ 725.470994]
>> [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
>> [ 725.491469] 6 locks held by kworker/0:1/27:
>> [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
>> process_one_work+0x16d/0x4e0 [ 725.533370] #1:
>> ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
>> process_one_work+0x16d/0x4e0 [ 725.565580] #2:
>> (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
>> mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
>> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
>> mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
>> (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
>> register_new_memory+0x33/0xd0 [ 725.658519] #5: (&dev->mutex){......},
>> at: [<ffffffff815ed773>] device_attach+0x23/0xb0
>>
>> Second participant, udev:
>>
>> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
>> 0x00000004
>> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
>> 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
>> 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
>> 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call Trace:
>> [ 725.858987] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
>> [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
>> 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
>> 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
>> 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
>> 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [ 725.971542]
>> [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [ 725.989207]
>> [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [ 726.008513]
>> [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [ 726.025579]
>> [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [ 726.043400]
>> [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [ 726.060506]
>> [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [ 726.077940]
>> [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [ 726.099416]
>> [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748] [<ffffffff811fabf8>]
>> SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
>> system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by systemd-
>> udevd/888:
>> [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>]
>> vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
>> [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
>> (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0 [
>> 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
>> lock_device_hotplug_sysfs+0x15/0x50
>> [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
>> device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......}, at:
>> [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
>> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
>> mem_hotplug_begin+0x4f/0x80
>>
>> In short: onlining grabs device lock and then tries to do
>> mem_hotplug_begin() while add_memory() is between
>> mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
>> device lock.
>>
>> To my understanding ACPI memory hotplug doesn't have the same issue as
>> device_hotplug_lock is being grabbed when the ACPI device is added.
>>
>> Solve the issue by grabbing device_hotplug_lock before doing
>> add_memory(). If we do that, lock_device_hotplug_sysfs() will cause syscall
>> retry which will eventually succeed. To support the change we need to
>> export lock_device_hotplug/ unlock_device_hotplug. This approach can be
>> completely wrong though.
>
> This issue was first discovered by Andy Whitcroft: https://lkml.org/lkml/2014/3/14/451
> I had sent patches based on Andy's analysis that did not affect the users of the kernel hot-add
> memory APIs: https://lkml.org/lkml/2014/12/2/662
Sorry I missed that.
>
> This patch puts the burden where it needs to be and can address the issue for all clients.
>
There is also one good solution from David Rientjes:
https://lkml.org/lkml/2015/2/12/57
I just tested it and it also solves the issue.
> K. Y
>>
>> Vitaly Kuznetsov (3):
>> driver core: export lock_device_hotplug/unlock_device_hotplug
>> memory_hotplug: add note about holding device_hotplug_lock and
>> add_memory()
>> Drivers: hv: balloon: fix deadlock between memory adding and onlining
>>
>> drivers/base/core.c | 2 ++
>> drivers/hv/hv_balloon.c | 10 ++++++++++
>> mm/memory_hotplug.c | 6 +++++-
>> 3 files changed, 17 insertions(+), 1 deletion(-)
>>
>> --
>> 1.9.3
--
Vitaly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* RE: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 15:50 ` Vitaly Kuznetsov
@ 2015-02-12 17:40 ` KY Srinivasan
0 siblings, 0 replies; 16+ messages in thread
From: KY Srinivasan @ 2015-02-12 17:40 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-kernel, Greg Kroah-Hartman, Haiyang Zhang, Andrew Morton,
Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka, David Rientjes,
Fabian Frederick, Zhang Zhen, Vladimir Davydov, Wang Nan,
Rafael J. Wysocki, devel, linux-mm
> -----Original Message-----
> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> Sent: Thursday, February 12, 2015 7:50 AM
> To: KY Srinivasan
> Cc: linux-kernel@vger.kernel.org; Greg Kroah-Hartman; Haiyang Zhang;
> Andrew Morton; Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David
> Rientjes; Fabian Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael
> J. Wysocki; devel@linuxdriverproject.org; linux-mm@kvack.org
> Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> between memory adding and onlining
>
> KY Srinivasan <kys@microsoft.com> writes:
>
> >> -----Original Message-----
> >> From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> >> Sent: Thursday, February 12, 2015 2:24 AM
> >> To: linux-kernel@vger.kernel.org
> >> Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
> >> Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes; Fabian
> >> Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J. Wysocki;
> >> devel@linuxdriverproject.org; linux-mm@kvack.org
> >> Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> >> between memory adding and onlining
> >>
> >> RESEND (with no changes) because Rafael J. Wysocki was missing in
> >> recepients.
> >>
> >> If newly added memory is brought online with e.g. udev rule:
> >> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> >> the following deadlock is observed (and easily reproducable):
> >>
> >> First participant, worker thread doing add_memory():
> >>
> >> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
> >> [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [
> >> 724.991736]
> >> ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 [
> >> 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
> >> ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
> >> ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> >> [ 725.082690] [<ffffffff81aac0a9>]
> >> schedule_preempt_disabled+0x29/0x70
> >> [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> >> 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [
> >> 725.140992] [<ffffffff815ed773>] device_attach+0x23/0xb0 [
> >> 725.159131] [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [
> >> 725.177055] [<ffffffff815ea693>] device_add+0x443/0x650 [
> >> 725.195558] [<ffffffff815ea8be>] device_register+0x1e/0x30 [
> >> 725.213133] [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [
> >> 725.231533] [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [
> >> 725.250769] [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [
> >> 725.269642] [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [
> >> 725.288764] [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [
> >> 725.306117] [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [ 725.322466]
> >> [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon] [
> >> 725.342777] [<ffffffff8109509f>] process_one_work+0x1df/0x4e0 [
> >> 725.361459] [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0 [
> >> 725.380390] [<ffffffff810954bb>] worker_thread+0x11b/0x450 [
> >> 725.397684] [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0 [
> >> 725.416533] [<ffffffff8109ac33>] kthread+0xf3/0x110 [ 725.433372]
> >> [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
> >> [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [
> >> 725.470994] [<ffffffff8109ab40>] ?
> kthread_create_on_node+0x240/0x240
> >> [ 725.491469] 6 locks held by kworker/0:1/27:
> >> [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
> >> process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> >> ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> >> process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> >> (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> >> mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> >> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> >> mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> >> (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> >> register_new_memory+0x33/0xd0 [ 725.658519] #5:
> >> (&dev->mutex){......},
> >> at: [<ffffffff815ed773>] device_attach+0x23/0xb0
> >>
> >> Second participant, udev:
> >>
> >> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> >> 0x00000004
> >> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
> >> 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
> >> 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
> >> 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call
> Trace:
> >> [ 725.858987] [<ffffffff81aac0a9>]
> >> schedule_preempt_disabled+0x29/0x70
> >> [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> >> 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
> >> 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
> >> 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
> >> 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [
> >> 725.971542] [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [
> >> 725.989207] [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [
> >> 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [
> >> 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [
> >> 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [
> >> 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [
> >> 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [
> >> 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748]
> >> [<ffffffff811fabf8>]
> >> SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> >> system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by
> >> systemd-
> >> udevd/888:
> >> [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>]
> >> vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> >> [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> >> (s_active#60){......}, at: [<ffffffff81279c4e>]
> >> kernfs_fop_write+0x6e/0x1a0 [ 726.249382] #3:
> >> (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> >> lock_device_hotplug_sysfs+0x15/0x50
> >> [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
> >> device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......},
> at:
> >> [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
> >> (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> >> mem_hotplug_begin+0x4f/0x80
> >>
> >> In short: onlining grabs device lock and then tries to do
> >> mem_hotplug_begin() while add_memory() is between
> >> mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
> >> device lock.
> >>
> >> To my understanding ACPI memory hotplug doesn't have the same issue
> >> as device_hotplug_lock is being grabbed when the ACPI device is added.
> >>
> >> Solve the issue by grabbing device_hotplug_lock before doing
> >> add_memory(). If we do that, lock_device_hotplug_sysfs() will cause
> >> syscall retry which will eventually succeed. To support the change we
> >> need to export lock_device_hotplug/ unlock_device_hotplug. This
> >> approach can be completely wrong though.
> >
> > This issue was first discovered by Andy Whitcroft:
> > https://lkml.org/lkml/2014/3/14/451
> > I had sent patches based on Andy's analysis that did not affect the
> > users of the kernel hot-add memory APIs:
> > https://lkml.org/lkml/2014/12/2/662
>
> Sorry I missed that.
>
> >
> > This patch puts the burden where it needs to be and can address the issue
> for all clients.
> >
>
> There is also one good solution from David Rientjes:
> https://lkml.org/lkml/2015/2/12/57
>
> I just tested it and it also solves the issue.
Thanks; looks good.
K. Y
>
>
> > K. Y
> >>
> >> Vitaly Kuznetsov (3):
> >> driver core: export lock_device_hotplug/unlock_device_hotplug
> >> memory_hotplug: add note about holding device_hotplug_lock and
> >> add_memory()
> >> Drivers: hv: balloon: fix deadlock between memory adding and
> >> onlining
> >>
> >> drivers/base/core.c | 2 ++
> >> drivers/hv/hv_balloon.c | 10 ++++++++++
> >> mm/memory_hotplug.c | 6 +++++-
> >> 3 files changed, 17 insertions(+), 1 deletion(-)
> >>
> >> --
> >> 1.9.3
>
> --
> Vitaly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 15:38 ` [PATCH RESEND 0/3] memory_hotplug: hyperv: " KY Srinivasan
2015-02-12 15:50 ` Vitaly Kuznetsov
@ 2015-02-12 22:12 ` Rafael J. Wysocki
2015-02-12 22:01 ` KY Srinivasan
1 sibling, 1 reply; 16+ messages in thread
From: Rafael J. Wysocki @ 2015-02-12 22:12 UTC (permalink / raw)
To: KY Srinivasan
Cc: Vitaly Kuznetsov, linux-kernel, Greg Kroah-Hartman,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, devel, linux-mm
On Thursday, February 12, 2015 03:38:42 PM KY Srinivasan wrote:
>
> > -----Original Message-----
> > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> > Sent: Thursday, February 12, 2015 2:24 AM
> > To: linux-kernel@vger.kernel.org
> > Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
> > Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes; Fabian
> > Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J. Wysocki;
> > devel@linuxdriverproject.org; linux-mm@kvack.org
> > Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > between memory adding and onlining
> >
> > RESEND (with no changes) because Rafael J. Wysocki was missing in
> > recepients.
> >
> > If newly added memory is brought online with e.g. udev rule:
> > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> > the following deadlock is observed (and easily reproducable):
> >
> > First participant, worker thread doing add_memory():
> >
> > [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
> > [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [ 724.991736]
> > ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 [
> > 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
> > ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
> > ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> > [ 725.082690] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> > [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [ 725.140992]
> > [<ffffffff815ed773>] device_attach+0x23/0xb0 [ 725.159131]
> > [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [ 725.177055]
> > [<ffffffff815ea693>] device_add+0x443/0x650 [ 725.195558]
> > [<ffffffff815ea8be>] device_register+0x1e/0x30 [ 725.213133]
> > [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [ 725.231533]
> > [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [ 725.250769]
> > [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [ 725.269642]
> > [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [ 725.288764]
> > [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [ 725.306117]
> > [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [ 725.322466]
> > [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon] [ 725.342777]
> > [<ffffffff8109509f>] process_one_work+0x1df/0x4e0 [ 725.361459]
> > [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0 [ 725.380390]
> > [<ffffffff810954bb>] worker_thread+0x11b/0x450 [ 725.397684]
> > [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0 [ 725.416533]
> > [<ffffffff8109ac33>] kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>]
> > ? kthread_create_on_node+0x240/0x240
> > [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [ 725.470994]
> > [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
> > [ 725.491469] 6 locks held by kworker/0:1/27:
> > [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
> > process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> > ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> > process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> > (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> > mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> > (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> > register_new_memory+0x33/0xd0 [ 725.658519] #5: (&dev->mutex){......},
> > at: [<ffffffff815ed773>] device_attach+0x23/0xb0
> >
> > Second participant, udev:
> >
> > [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> > 0x00000004
> > [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
> > 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
> > 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
> > 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call Trace:
> > [ 725.858987] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> > [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
> > 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
> > 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
> > 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [ 725.971542]
> > [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [ 725.989207]
> > [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [ 726.008513]
> > [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [ 726.025579]
> > [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [ 726.043400]
> > [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [ 726.060506]
> > [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [ 726.077940]
> > [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [ 726.099416]
> > [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748] [<ffffffff811fabf8>]
> > SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> > system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by systemd-
> > udevd/888:
> > [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>]
> > vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> > [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> > (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0 [
> > 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> > lock_device_hotplug_sysfs+0x15/0x50
> > [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
> > device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......}, at:
> > [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
> > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > mem_hotplug_begin+0x4f/0x80
> >
> > In short: onlining grabs device lock and then tries to do
> > mem_hotplug_begin() while add_memory() is between
> > mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
> > device lock.
> >
> > To my understanding ACPI memory hotplug doesn't have the same issue as
> > device_hotplug_lock is being grabbed when the ACPI device is added.
> >
> > Solve the issue by grabbing device_hotplug_lock before doing
> > add_memory(). If we do that, lock_device_hotplug_sysfs() will cause syscall
> > retry which will eventually succeed. To support the change we need to
> > export lock_device_hotplug/ unlock_device_hotplug. This approach can be
> > completely wrong though.
>
> This issue was first discovered by Andy Whitcroft: https://lkml.org/lkml/2014/3/14/451
> I had sent patches based on Andy's analysis that did not affect the users of the kernel hot-add
> memory APIs: https://lkml.org/lkml/2014/12/2/662
>
> This patch puts the burden where it needs to be and can address the issue for all clients.
That seems to mean that this series is not needed. Is that correct?
Rafael
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* RE: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 22:12 ` Rafael J. Wysocki
@ 2015-02-12 22:01 ` KY Srinivasan
2015-02-12 22:27 ` Rafael J. Wysocki
0 siblings, 1 reply; 16+ messages in thread
From: KY Srinivasan @ 2015-02-12 22:01 UTC (permalink / raw)
To: Rafael J. Wysocki
Cc: Vitaly Kuznetsov, linux-kernel, Greg Kroah-Hartman,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, devel, linux-mm
> -----Original Message-----
> From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> Sent: Thursday, February 12, 2015 2:13 PM
> To: KY Srinivasan
> Cc: Vitaly Kuznetsov; linux-kernel@vger.kernel.org; Greg Kroah-Hartman;
> Haiyang Zhang; Andrew Morton; Yasuaki Ishimatsu; Tang Chen; Vlastimil
> Babka; David Rientjes; Fabian Frederick; Zhang Zhen; Vladimir Davydov;
> Wang Nan; devel@linuxdriverproject.org; linux-mm@kvack.org
> Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> between memory adding and onlining
>
> On Thursday, February 12, 2015 03:38:42 PM KY Srinivasan wrote:
> >
> > > -----Original Message-----
> > > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> > > Sent: Thursday, February 12, 2015 2:24 AM
> > > To: linux-kernel@vger.kernel.org
> > > Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
> > > Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes;
> > > Fabian Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J.
> > > Wysocki; devel@linuxdriverproject.org; linux-mm@kvack.org
> > > Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > > between memory adding and onlining
> > >
> > > RESEND (with no changes) because Rafael J. Wysocki was missing in
> > > recepients.
> > >
> > > If newly added memory is brought online with e.g. udev rule:
> > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> > > the following deadlock is observed (and easily reproducable):
> > >
> > > First participant, worker thread doing add_memory():
> > >
> > > [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2
> 0x00000000
> > > [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [
> > > 724.991736]
> > > ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0
> > > [ 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
> > > ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
> > > ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> > > [ 725.082690] [<ffffffff81aac0a9>]
> > > schedule_preempt_disabled+0x29/0x70
> > > [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > > 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [
> > > 725.140992] [<ffffffff815ed773>] device_attach+0x23/0xb0 [
> > > 725.159131] [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [
> > > 725.177055] [<ffffffff815ea693>] device_add+0x443/0x650 [
> > > 725.195558] [<ffffffff815ea8be>] device_register+0x1e/0x30 [
> > > 725.213133] [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [
> > > 725.231533] [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [
> > > 725.250769] [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [
> > > 725.269642] [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [
> > > 725.288764] [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [
> > > 725.306117] [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [
> > > 725.322466] [<ffffffffa00293af>] hot_add_req+0x33f/0xf90
> > > [hv_balloon] [ 725.342777] [<ffffffff8109509f>]
> > > process_one_work+0x1df/0x4e0 [ 725.361459] [<ffffffff8109502d>] ?
> > > process_one_work+0x16d/0x4e0 [ 725.380390] [<ffffffff810954bb>]
> > > worker_thread+0x11b/0x450 [ 725.397684] [<ffffffff810953a0>] ?
> > > process_one_work+0x4e0/0x4e0 [ 725.416533] [<ffffffff8109ac33>]
> > > kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>] ?
> > > kthread_create_on_node+0x240/0x240
> > > [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [
> > > 725.470994] [<ffffffff8109ab40>] ?
> > > kthread_create_on_node+0x240/0x240
> > > [ 725.491469] 6 locks held by kworker/0:1/27:
> > > [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
> > > process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> > > ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> > > process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> > > (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> > > mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> > > (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> > > register_new_memory+0x33/0xd0 [ 725.658519] #5:
> > > (&dev->mutex){......},
> > > at: [<ffffffff815ed773>] device_attach+0x23/0xb0
> > >
> > > Second participant, udev:
> > >
> > > [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> > > 0x00000004
> > > [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
> > > 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
> > > 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
> > > 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call
> Trace:
> > > [ 725.858987] [<ffffffff81aac0a9>]
> > > schedule_preempt_disabled+0x29/0x70
> > > [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > > 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
> > > 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
> > > 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
> > > 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [
> > > 725.971542] [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [
> > > 725.989207] [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [
> > > 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [
> > > 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [
> > > 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [
> > > 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [
> > > 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [
> > > 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748]
> > > [<ffffffff811fabf8>]
> > > SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> > > system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by
> > > systemd-
> > > udevd/888:
> > > [ 726.165044] #0: (sb_writers#3){......}, at:
> > > [<ffffffff811fa063>]
> > > vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> > > [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> > > (s_active#60){......}, at: [<ffffffff81279c4e>]
> > > kernfs_fop_write+0x6e/0x1a0 [ 726.249382] #3:
> > > (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> > > lock_device_hotplug_sysfs+0x15/0x50
> > > [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
> > > device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......},
> at:
> > > [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
> > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > mem_hotplug_begin+0x4f/0x80
> > >
> > > In short: onlining grabs device lock and then tries to do
> > > mem_hotplug_begin() while add_memory() is between
> > > mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
> > > device lock.
> > >
> > > To my understanding ACPI memory hotplug doesn't have the same issue
> > > as device_hotplug_lock is being grabbed when the ACPI device is added.
> > >
> > > Solve the issue by grabbing device_hotplug_lock before doing
> > > add_memory(). If we do that, lock_device_hotplug_sysfs() will cause
> > > syscall retry which will eventually succeed. To support the change
> > > we need to export lock_device_hotplug/ unlock_device_hotplug. This
> > > approach can be completely wrong though.
> >
> > This issue was first discovered by Andy Whitcroft:
> > https://lkml.org/lkml/2014/3/14/451
> > I had sent patches based on Andy's analysis that did not affect the
> > users of the kernel hot-add memory APIs:
> > https://lkml.org/lkml/2014/12/2/662
> >
> > This patch puts the burden where it needs to be and can address the issue
> for all clients.
>
> That seems to mean that this series is not needed. Is that correct?
This patch was never committed upstream and so the issue still is there.
K. Y
>
> Rafael
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 22:01 ` KY Srinivasan
@ 2015-02-12 22:27 ` Rafael J. Wysocki
2015-02-12 22:10 ` KY Srinivasan
0 siblings, 1 reply; 16+ messages in thread
From: Rafael J. Wysocki @ 2015-02-12 22:27 UTC (permalink / raw)
To: KY Srinivasan
Cc: Vitaly Kuznetsov, linux-kernel, Greg Kroah-Hartman,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, devel, linux-mm
On Thursday, February 12, 2015 10:01:48 PM KY Srinivasan wrote:
>
> > -----Original Message-----
> > From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> > Sent: Thursday, February 12, 2015 2:13 PM
> > To: KY Srinivasan
> > Cc: Vitaly Kuznetsov; linux-kernel@vger.kernel.org; Greg Kroah-Hartman;
> > Haiyang Zhang; Andrew Morton; Yasuaki Ishimatsu; Tang Chen; Vlastimil
> > Babka; David Rientjes; Fabian Frederick; Zhang Zhen; Vladimir Davydov;
> > Wang Nan; devel@linuxdriverproject.org; linux-mm@kvack.org
> > Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > between memory adding and onlining
> >
> > On Thursday, February 12, 2015 03:38:42 PM KY Srinivasan wrote:
> > >
> > > > -----Original Message-----
> > > > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> > > > Sent: Thursday, February 12, 2015 2:24 AM
> > > > To: linux-kernel@vger.kernel.org
> > > > Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton;
> > > > Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes;
> > > > Fabian Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J.
> > > > Wysocki; devel@linuxdriverproject.org; linux-mm@kvack.org
> > > > Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > > > between memory adding and onlining
> > > >
> > > > RESEND (with no changes) because Rafael J. Wysocki was missing in
> > > > recepients.
> > > >
> > > > If newly added memory is brought online with e.g. udev rule:
> > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> > > > the following deadlock is observed (and easily reproducable):
> > > >
> > > > First participant, worker thread doing add_memory():
> > > >
> > > > [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2
> > 0x00000000
> > > > [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [
> > > > 724.991736]
> > > > ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0
> > > > [ 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0
> > > > ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000
> > > > ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> > > > [ 725.082690] [<ffffffff81aac0a9>]
> > > > schedule_preempt_disabled+0x29/0x70
> > > > [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > > > 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [
> > > > 725.140992] [<ffffffff815ed773>] device_attach+0x23/0xb0 [
> > > > 725.159131] [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [
> > > > 725.177055] [<ffffffff815ea693>] device_add+0x443/0x650 [
> > > > 725.195558] [<ffffffff815ea8be>] device_register+0x1e/0x30 [
> > > > 725.213133] [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [
> > > > 725.231533] [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [
> > > > 725.250769] [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [
> > > > 725.269642] [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [
> > > > 725.288764] [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [
> > > > 725.306117] [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [
> > > > 725.322466] [<ffffffffa00293af>] hot_add_req+0x33f/0xf90
> > > > [hv_balloon] [ 725.342777] [<ffffffff8109509f>]
> > > > process_one_work+0x1df/0x4e0 [ 725.361459] [<ffffffff8109502d>] ?
> > > > process_one_work+0x16d/0x4e0 [ 725.380390] [<ffffffff810954bb>]
> > > > worker_thread+0x11b/0x450 [ 725.397684] [<ffffffff810953a0>] ?
> > > > process_one_work+0x4e0/0x4e0 [ 725.416533] [<ffffffff8109ac33>]
> > > > kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>] ?
> > > > kthread_create_on_node+0x240/0x240
> > > > [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [
> > > > 725.470994] [<ffffffff8109ab40>] ?
> > > > kthread_create_on_node+0x240/0x240
> > > > [ 725.491469] 6 locks held by kworker/0:1/27:
> > > > [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>]
> > > > process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> > > > ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> > > > process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> > > > (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> > > > mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> > > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > > mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> > > > (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> > > > register_new_memory+0x33/0xd0 [ 725.658519] #5:
> > > > (&dev->mutex){......},
> > > > at: [<ffffffff815ed773>] device_attach+0x23/0xb0
> > > >
> > > > Second participant, udev:
> > > >
> > > > [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> > > > 0x00000004
> > > > [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0
> > > > 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8
> > > > 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0
> > > > 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call
> > Trace:
> > > > [ 725.858987] [<ffffffff81aac0a9>]
> > > > schedule_preempt_disabled+0x29/0x70
> > > > [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470 [
> > > > 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [
> > > > 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [
> > > > 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [
> > > > 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520 [
> > > > 725.971542] [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [
> > > > 725.989207] [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [
> > > > 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [
> > > > 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [
> > > > 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [
> > > > 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [
> > > > 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [
> > > > 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [ 726.115748]
> > > > [<ffffffff811fabf8>]
> > > > SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> > > > system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by
> > > > systemd-
> > > > udevd/888:
> > > > [ 726.165044] #0: (sb_writers#3){......}, at:
> > > > [<ffffffff811fa063>]
> > > > vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> > > > [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> > > > (s_active#60){......}, at: [<ffffffff81279c4e>]
> > > > kernfs_fop_write+0x6e/0x1a0 [ 726.249382] #3:
> > > > (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> > > > lock_device_hotplug_sysfs+0x15/0x50
> > > > [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>]
> > > > device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......},
> > at:
> > > > [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6:
> > > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > > mem_hotplug_begin+0x4f/0x80
> > > >
> > > > In short: onlining grabs device lock and then tries to do
> > > > mem_hotplug_begin() while add_memory() is between
> > > > mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing
> > > > device lock.
> > > >
> > > > To my understanding ACPI memory hotplug doesn't have the same issue
> > > > as device_hotplug_lock is being grabbed when the ACPI device is added.
> > > >
> > > > Solve the issue by grabbing device_hotplug_lock before doing
> > > > add_memory(). If we do that, lock_device_hotplug_sysfs() will cause
> > > > syscall retry which will eventually succeed. To support the change
> > > > we need to export lock_device_hotplug/ unlock_device_hotplug. This
> > > > approach can be completely wrong though.
> > >
> > > This issue was first discovered by Andy Whitcroft:
> > > https://lkml.org/lkml/2014/3/14/451
> > > I had sent patches based on Andy's analysis that did not affect the
> > > users of the kernel hot-add memory APIs:
> > > https://lkml.org/lkml/2014/12/2/662
> > >
> > > This patch puts the burden where it needs to be and can address the issue
> > for all clients.
> >
> > That seems to mean that this series is not needed. Is that correct?
>
> This patch was never committed upstream and so the issue still is there.
Well, I'm not sure what to do now to be honest.
Is this series regarded as the right way to address the problem that
everybody is comfortable with? Or is it still under discussion?
Rafael
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* RE: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 22:27 ` Rafael J. Wysocki
@ 2015-02-12 22:10 ` KY Srinivasan
2015-02-12 22:43 ` Rafael J. Wysocki
0 siblings, 1 reply; 16+ messages in thread
From: KY Srinivasan @ 2015-02-12 22:10 UTC (permalink / raw)
To: Rafael J. Wysocki
Cc: Vitaly Kuznetsov, linux-kernel, Greg Kroah-Hartman,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, devel, linux-mm
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 10173 bytes --]
> -----Original Message-----
> From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> Sent: Thursday, February 12, 2015 2:28 PM
> To: KY Srinivasan
> Cc: Vitaly Kuznetsov; linux-kernel@vger.kernel.org; Greg Kroah-Hartman;
> Haiyang Zhang; Andrew Morton; Yasuaki Ishimatsu; Tang Chen; Vlastimil
> Babka; David Rientjes; Fabian Frederick; Zhang Zhen; Vladimir Davydov;
> Wang Nan; devel@linuxdriverproject.org; linux-mm@kvack.org
> Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> between memory adding and onlining
>
> On Thursday, February 12, 2015 10:01:48 PM KY Srinivasan wrote:
> >
> > > -----Original Message-----
> > > From: Rafael J. Wysocki [mailto:rjw@rjwysocki.net]
> > > Sent: Thursday, February 12, 2015 2:13 PM
> > > To: KY Srinivasan
> > > Cc: Vitaly Kuznetsov; linux-kernel@vger.kernel.org; Greg
> > > Kroah-Hartman; Haiyang Zhang; Andrew Morton; Yasuaki Ishimatsu; Tang
> > > Chen; Vlastimil Babka; David Rientjes; Fabian Frederick; Zhang Zhen;
> > > Vladimir Davydov; Wang Nan; devel@linuxdriverproject.org;
> > > linux-mm@kvack.org
> > > Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > > between memory adding and onlining
> > >
> > > On Thursday, February 12, 2015 03:38:42 PM KY Srinivasan wrote:
> > > >
> > > > > -----Original Message-----
> > > > > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com]
> > > > > Sent: Thursday, February 12, 2015 2:24 AM
> > > > > To: linux-kernel@vger.kernel.org
> > > > > Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew
> > > > > Morton; Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David
> > > > > Rientjes; Fabian Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan;
> Rafael J.
> > > > > Wysocki; devel@linuxdriverproject.org; linux-mm@kvack.org
> > > > > Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock
> > > > > between memory adding and onlining
> > > > >
> > > > > RESEND (with no changes) because Rafael J. Wysocki was missing
> > > > > in recepients.
> > > > >
> > > > > If newly added memory is brought online with e.g. udev rule:
> > > > > SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> > > > > the following deadlock is observed (and easily reproducable):
> > > > >
> > > > > First participant, worker thread doing add_memory():
> > > > >
> > > > > [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2
> > > 0x00000000
> > > > > [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [
> > > > > 724.991736]
> > > > > ffff88000412f9c8 0000000000000000 ffff88003fa1dc30
> > > > > 00000000000151c0 [ 725.019725] 0000000000000246
> > > > > ffff88000412ffd8 00000000000151c0
> > > > > ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30
> > > > > 00000001032a6000
> > > > > ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace:
> > > > > [ 725.082690] [<ffffffff81aac0a9>]
> > > > > schedule_preempt_disabled+0x29/0x70
> > > > > [ 725.103799] [<ffffffff81aae33b>]
> > > > > mutex_lock_nested+0x14b/0x470 [ 725.122367]
> > > > > [<ffffffff815ed773>] ? device_attach+0x23/0xb0 [ 725.140992]
> > > > > [<ffffffff815ed773>] device_attach+0x23/0xb0 [ 725.159131]
> > > > > [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0 [ 725.177055]
> > > > > [<ffffffff815ea693>] device_add+0x443/0x650 [ 725.195558]
> > > > > [<ffffffff815ea8be>] device_register+0x1e/0x30 [ 725.213133]
> > > > > [<ffffffff81601790>] init_memory_block+0xd0/0xf0 [ 725.231533]
> > > > > [<ffffffff816018f1>] register_new_memory+0xb1/0xd0 [ 725.250769]
> > > > > [<ffffffff81a961cf>] __add_pages+0x13f/0x250 [ 725.269642]
> > > > > [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0 [ 725.288764]
> > > > > [<ffffffff81063770>] arch_add_memory+0x70/0xf0 [ 725.306117]
> > > > > [<ffffffff81a95f8f>] add_memory+0xef/0x1f0 [ 725.322466]
> > > > > [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon] [
> > > > > 725.342777] [<ffffffff8109509f>]
> > > > > process_one_work+0x1df/0x4e0 [ 725.361459] [<ffffffff8109502d>] ?
> > > > > process_one_work+0x16d/0x4e0 [ 725.380390] [<ffffffff810954bb>]
> > > > > worker_thread+0x11b/0x450 [ 725.397684] [<ffffffff810953a0>] ?
> > > > > process_one_work+0x4e0/0x4e0 [ 725.416533] [<ffffffff8109ac33>]
> > > > > kthread+0xf3/0x110 [ 725.433372] [<ffffffff8109ab40>] ?
> > > > > kthread_create_on_node+0x240/0x240
> > > > > [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0 [
> > > > > 725.470994] [<ffffffff8109ab40>] ?
> > > > > kthread_create_on_node+0x240/0x240
> > > > > [ 725.491469] 6 locks held by kworker/0:1/27:
> > > > > [ 725.505037] #0: ("events"){......}, at:
> > > > > [<ffffffff8109502d>]
> > > > > process_one_work+0x16d/0x4e0 [ 725.533370] #1:
> > > > > ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>]
> > > > > process_one_work+0x16d/0x4e0 [ 725.565580] #2:
> > > > > (mem_hotplug.lock){......}, at: [<ffffffff811e6525>]
> > > > > mem_hotplug_begin+0x5/0x80 [ 725.594369] #3:
> > > > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > > > mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4:
> > > > > (mem_sysfs_mutex){......}, at: [<ffffffff81601873>]
> > > > > register_new_memory+0x33/0xd0 [ 725.658519] #5:
> > > > > (&dev->mutex){......},
> > > > > at: [<ffffffff815ed773>] device_attach+0x23/0xb0
> > > > >
> > > > > Second participant, udev:
> > > > >
> > > > > [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530
> > > > > 0x00000004
> > > > > [ 725.773767] ffff88003b94fc68 0000000000000000
> > > > > ffff8800034949c0
> > > > > 00000000000151c0 [ 725.798332] ffffffff8210d980
> > > > > ffff88003b94ffd8
> > > > > 00000000000151c0 ffff880037a69270 [ 725.822841]
> > > > > ffff8800034949c0
> > > > > 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [
> > > > > 725.849184] Call
> > > Trace:
> > > > > [ 725.858987] [<ffffffff81aac0a9>]
> > > > > schedule_preempt_disabled+0x29/0x70
> > > > > [ 725.879231] [<ffffffff81aae33b>]
> > > > > mutex_lock_nested+0x14b/0x470 [ 725.897860]
> > > > > [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80 [ 725.916698]
> > > > > [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80 [ 725.935064]
> > > > > [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80 [ 725.953464]
> > > > > [<ffffffff81a9631b>] online_pages+0x3b/0x520 [ 725.971542]
> > > > > [<ffffffff815eb0b3>] ? device_online+0x23/0xa0 [ 725.989207]
> > > > > [<ffffffff81601524>] memory_subsys_online+0x64/0xc0 [
> > > > > 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0 [
> > > > > 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0 [
> > > > > 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30 [
> > > > > 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60 [
> > > > > 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0 [
> > > > > 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0 [
> > > > > 726.115748] [<ffffffff811fabf8>]
> > > > > SyS_write+0x58/0xd0 [ 726.131933] [<ffffffff81ab1ea9>]
> > > > > system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by
> > > > > systemd-
> > > > > udevd/888:
> > > > > [ 726.165044] #0: (sb_writers#3){......}, at:
> > > > > [<ffffffff811fa063>]
> > > > > vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at:
> > > > > [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2:
> > > > > (s_active#60){......}, at: [<ffffffff81279c4e>]
> > > > > kernfs_fop_write+0x6e/0x1a0 [ 726.249382] #3:
> > > > > (device_hotplug_lock){......}, at: [<ffffffff815e9c15>]
> > > > > lock_device_hotplug_sysfs+0x15/0x50
> > > > > [ 726.281901] #4: (&dev->mutex){......}, at:
> > > > > [<ffffffff815eb0b3>]
> > > > > device_online+0x23/0xa0 [ 726.308619] #5:
> > > > > (mem_hotplug.lock){......},
> > > at:
> > > > > [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80 [ 726.337994]
> #6:
> > > > > (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>]
> > > > > mem_hotplug_begin+0x4f/0x80
> > > > >
> > > > > In short: onlining grabs device lock and then tries to do
> > > > > mem_hotplug_begin() while add_memory() is between
> > > > > mem_hotplug_begin() and mem_hotplug_done() and it tries
> grabbing
> > > > > device lock.
> > > > >
> > > > > To my understanding ACPI memory hotplug doesn't have the same
> > > > > issue as device_hotplug_lock is being grabbed when the ACPI device
> is added.
> > > > >
> > > > > Solve the issue by grabbing device_hotplug_lock before doing
> > > > > add_memory(). If we do that, lock_device_hotplug_sysfs() will
> > > > > cause syscall retry which will eventually succeed. To support
> > > > > the change we need to export lock_device_hotplug/
> > > > > unlock_device_hotplug. This approach can be completely wrong
> though.
> > > >
> > > > This issue was first discovered by Andy Whitcroft:
> > > > https://lkml.org/lkml/2014/3/14/451
> > > > I had sent patches based on Andy's analysis that did not affect
> > > > the users of the kernel hot-add memory APIs:
> > > > https://lkml.org/lkml/2014/12/2/662
> > > >
> > > > This patch puts the burden where it needs to be and can address
> > > > the issue
> > > for all clients.
> > >
> > > That seems to mean that this series is not needed. Is that correct?
> >
> > This patch was never committed upstream and so the issue still is there.
>
> Well, I'm not sure what to do now to be honest.
>
> Is this series regarded as the right way to address the problem that
> everybody is comfortable with? Or is it still under discussion?
We need to solve this problem and that is not under discussion. I also believe this problem
needs to be solved in a way that addresses the problem where it belongs - not in the users of
the hot_add API. Both my solution and the one proposed by David https://lkml.org/lkml/2015/2/12/57
address this issue. You can select either patch and check it in. I just want the issue addressed and I am not
married to the solution I proposed.
K. Y
>
> Rafael
N‹§²æìr¸›zǧu©ž²Æ {\b†éì¹»\x1c®&Þ–)îÆi¢žØ^n‡r¶‰šŽŠÝ¢j$½§$¢¸\x05¢¹¨è§~Š'.)îÄÃ,yèm¶ŸÿÃ\f%Š{±šj+ƒðèž×¦j)Z†·Ÿ
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 22:10 ` KY Srinivasan
@ 2015-02-12 22:43 ` Rafael J. Wysocki
2015-02-12 22:25 ` Andrew Morton
0 siblings, 1 reply; 16+ messages in thread
From: Rafael J. Wysocki @ 2015-02-12 22:43 UTC (permalink / raw)
To: KY Srinivasan, Andrew Morton
Cc: Vitaly Kuznetsov, linux-kernel, Greg Kroah-Hartman,
Haiyang Zhang, Yasuaki Ishimatsu, Tang Chen, Vlastimil Babka,
David Rientjes, Fabian Frederick, Zhang Zhen, Vladimir Davydov,
Wang Nan, devel, linux-mm
On Thursday, February 12, 2015 10:10:30 PM KY Srinivasan wrote:
[cut]
> > > > >
> > > > > This issue was first discovered by Andy Whitcroft:
> > > > > https://lkml.org/lkml/2014/3/14/451
> > > > > I had sent patches based on Andy's analysis that did not affect
> > > > > the users of the kernel hot-add memory APIs:
> > > > > https://lkml.org/lkml/2014/12/2/662
> > > > >
> > > > > This patch puts the burden where it needs to be and can address
> > > > > the issue
> > > > for all clients.
> > > >
> > > > That seems to mean that this series is not needed. Is that correct?
> > >
> > > This patch was never committed upstream and so the issue still is there.
> >
> > Well, I'm not sure what to do now to be honest.
> >
> > Is this series regarded as the right way to address the problem that
> > everybody is comfortable with? Or is it still under discussion?
>
> We need to solve this problem and that is not under discussion. I also believe this problem
> needs to be solved in a way that addresses the problem where it belongs - not in the users of
> the hot_add API. Both my solution and the one proposed by David https://lkml.org/lkml/2015/2/12/57
> address this issue. You can select either patch and check it in. I just want the issue addressed and I am not
> married to the solution I proposed.
OK, thanks!
So having looked at both your patch and the David's one I think that
the Andrew's tree is appropriate for any of them.
Andrew?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 22:43 ` Rafael J. Wysocki
@ 2015-02-12 22:25 ` Andrew Morton
0 siblings, 0 replies; 16+ messages in thread
From: Andrew Morton @ 2015-02-12 22:25 UTC (permalink / raw)
To: Rafael J. Wysocki
Cc: KY Srinivasan, Vitaly Kuznetsov, linux-kernel,
Greg Kroah-Hartman, Haiyang Zhang, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, devel, linux-mm
On Thu, 12 Feb 2015 23:43:17 +0100 "Rafael J. Wysocki" <rjw@rjwysocki.net> wrote:
> On Thursday, February 12, 2015 10:10:30 PM KY Srinivasan wrote:
>
> [cut]
yay!
> > > > > >
> > > > > > This issue was first discovered by Andy Whitcroft:
> > > > > > https://lkml.org/lkml/2014/3/14/451
> > > > > > I had sent patches based on Andy's analysis that did not affect
> > > > > > the users of the kernel hot-add memory APIs:
> > > > > > https://lkml.org/lkml/2014/12/2/662
> > > > > >
> > > > > > This patch puts the burden where it needs to be and can address
> > > > > > the issue
> > > > > for all clients.
> > > > >
> > > > > That seems to mean that this series is not needed. Is that correct?
> > > >
> > > > This patch was never committed upstream and so the issue still is there.
> > >
> > > Well, I'm not sure what to do now to be honest.
> > >
> > > Is this series regarded as the right way to address the problem that
> > > everybody is comfortable with? Or is it still under discussion?
> >
> > We need to solve this problem and that is not under discussion. I also believe this problem
> > needs to be solved in a way that addresses the problem where it belongs - not in the users of
> > the hot_add API. Both my solution and the one proposed by David https://lkml.org/lkml/2015/2/12/57
> > address this issue. You can select either patch and check it in. I just want the issue addressed and I am not
> > married to the solution I proposed.
>
> OK, thanks!
>
> So having looked at both your patch and the David's one I think that
> the Andrew's tree is appropriate for any of them.
>
> Andrew?
OK, I'll wake up and take a look. Hopefully as 3.21 material but I
need to to back and reread everything. Is it more urgent than that?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-02-12 10:23 [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Vitaly Kuznetsov
` (3 preceding siblings ...)
2015-02-12 15:38 ` [PATCH RESEND 0/3] memory_hotplug: hyperv: " KY Srinivasan
@ 2015-03-06 15:50 ` Michal Hocko
2015-03-09 8:40 ` Vitaly Kuznetsov
4 siblings, 1 reply; 16+ messages in thread
From: Michal Hocko @ 2015-03-06 15:50 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-kernel, Greg Kroah-Hartman, K. Y. Srinivasan,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, Rafael J. Wysocki, devel, linux-mm
[Sorry for the late response]
This is basically the same code posted by KY Srinivasan posted late last
year (http://marc.info/?l=linux-mm&m=141782228129426&w=2). I had
objections to the implementation
http://marc.info/?l=linux-mm&m=141805109216700&w=2
On Thu 12-02-15 11:23:51, Vitaly Kuznetsov wrote:
> RESEND (with no changes) because Rafael J. Wysocki was missing in recepients.
>
> If newly added memory is brought online with e.g. udev rule:
> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
> the following deadlock is observed (and easily reproducable):
>
> First participant, worker thread doing add_memory():
>
> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
> [ 724.973543] Workqueue: events hot_add_req [hv_balloon]
> [ 724.991736] ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0
> [ 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0 ffff88003a77a4e0
> [ 725.046486] ffff88003fa1dc30 00000001032a6000 ffff88003a7ca838 ffff88003a7ca898
> [ 725.072969] Call Trace:
> [ 725.082690] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470
> [ 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0
> [ 725.140992] [<ffffffff815ed773>] device_attach+0x23/0xb0
> [ 725.159131] [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0
> [ 725.177055] [<ffffffff815ea693>] device_add+0x443/0x650
> [ 725.195558] [<ffffffff815ea8be>] device_register+0x1e/0x30
> [ 725.213133] [<ffffffff81601790>] init_memory_block+0xd0/0xf0
> [ 725.231533] [<ffffffff816018f1>] register_new_memory+0xb1/0xd0
> [ 725.250769] [<ffffffff81a961cf>] __add_pages+0x13f/0x250
> [ 725.269642] [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0
> [ 725.288764] [<ffffffff81063770>] arch_add_memory+0x70/0xf0
> [ 725.306117] [<ffffffff81a95f8f>] add_memory+0xef/0x1f0
> [ 725.322466] [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon]
> [ 725.342777] [<ffffffff8109509f>] process_one_work+0x1df/0x4e0
> [ 725.361459] [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0
> [ 725.380390] [<ffffffff810954bb>] worker_thread+0x11b/0x450
> [ 725.397684] [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0
> [ 725.416533] [<ffffffff8109ac33>] kthread+0xf3/0x110
> [ 725.433372] [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
> [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0
> [ 725.470994] [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
> [ 725.491469] 6 locks held by kworker/0:1/27:
> [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
> [ 725.533370] #1: ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
> [ 725.565580] #2: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
> [ 725.594369] #3: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
> [ 725.628554] #4: (mem_sysfs_mutex){......}, at: [<ffffffff81601873>] register_new_memory+0x33/0xd0
> [ 725.658519] #5: (&dev->mutex){......}, at: [<ffffffff815ed773>] device_attach+0x23/0xb0
>
> Second participant, udev:
>
> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530 0x00000004
> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0 00000000000151c0
> [ 725.798332] ffffffff8210d980 ffff88003b94ffd8 00000000000151c0 ffff880037a69270
> [ 725.822841] ffff8800034949c0 0000000100000001 ffff8800034949c0 ffffffff81ff2b48
> [ 725.849184] Call Trace:
> [ 725.858987] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
> [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470
> [ 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80
> [ 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
> [ 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80
> [ 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520
> [ 725.971542] [<ffffffff815eb0b3>] ? device_online+0x23/0xa0
> [ 725.989207] [<ffffffff81601524>] memory_subsys_online+0x64/0xc0
> [ 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0
> [ 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0
> [ 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30
> [ 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60
> [ 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0
> [ 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0
> [ 726.115748] [<ffffffff811fabf8>] SyS_write+0x58/0xd0
> [ 726.131933] [<ffffffff81ab1ea9>] system_call_fastpath+0x12/0x17
> [ 726.150691] 7 locks held by systemd-udevd/888:
> [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>] vfs_write+0x1b3/0x1f0
> [ 726.192422] #1: (&of->mutex){......}, at: [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0
> [ 726.220289] #2: (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0
> [ 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>] lock_device_hotplug_sysfs+0x15/0x50
> [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>] device_online+0x23/0xa0
> [ 726.308619] #5: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
> [ 726.337994] #6: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
>
> In short: onlining grabs device lock and then tries to do mem_hotplug_begin()
> while add_memory() is between mem_hotplug_begin() and mem_hotplug_done() and it
> tries grabbing device lock.
>
> To my understanding ACPI memory hotplug doesn't have the same issue as
> device_hotplug_lock is being grabbed when the ACPI device is added.
>
> Solve the issue by grabbing device_hotplug_lock before doing add_memory(). If
> we do that, lock_device_hotplug_sysfs() will cause syscall retry which will
> eventually succeed. To support the change we need to export lock_device_hotplug/
> unlock_device_hotplug. This approach can be completely wrong though.
>
> Vitaly Kuznetsov (3):
> driver core: export lock_device_hotplug/unlock_device_hotplug
> memory_hotplug: add note about holding device_hotplug_lock and
> add_memory()
> Drivers: hv: balloon: fix deadlock between memory adding and onlining
>
> drivers/base/core.c | 2 ++
> drivers/hv/hv_balloon.c | 10 ++++++++++
> mm/memory_hotplug.c | 6 +++++-
> 3 files changed, 17 insertions(+), 1 deletion(-)
>
> --
> 1.9.3
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-03-06 15:50 ` Michal Hocko
@ 2015-03-09 8:40 ` Vitaly Kuznetsov
2015-03-12 16:18 ` Michal Hocko
0 siblings, 1 reply; 16+ messages in thread
From: Vitaly Kuznetsov @ 2015-03-09 8:40 UTC (permalink / raw)
To: Michal Hocko
Cc: linux-kernel, Greg Kroah-Hartman, K. Y. Srinivasan,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, Rafael J. Wysocki, devel, linux-mm
Michal Hocko <mhocko@suse.cz> writes:
> [Sorry for the late response]
>
> This is basically the same code posted by KY Srinivasan posted late last
> year (http://marc.info/?l=linux-mm&m=141782228129426&w=2). I had
> objections to the implementation
> http://marc.info/?l=linux-mm&m=141805109216700&w=2
Np, David's alternative fix is already in -mm:
https://lkml.org/lkml/2015/2/12/655
>
> On Thu 12-02-15 11:23:51, Vitaly Kuznetsov wrote:
>> RESEND (with no changes) because Rafael J. Wysocki was missing in recepients.
>>
>> If newly added memory is brought online with e.g. udev rule:
>> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online"
>> the following deadlock is observed (and easily reproducable):
>>
>> First participant, worker thread doing add_memory():
>>
>> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000
>> [ 724.973543] Workqueue: events hot_add_req [hv_balloon]
>> [ 724.991736] ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0
>> [ 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0 ffff88003a77a4e0
>> [ 725.046486] ffff88003fa1dc30 00000001032a6000 ffff88003a7ca838 ffff88003a7ca898
>> [ 725.072969] Call Trace:
>> [ 725.082690] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
>> [ 725.103799] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470
>> [ 725.122367] [<ffffffff815ed773>] ? device_attach+0x23/0xb0
>> [ 725.140992] [<ffffffff815ed773>] device_attach+0x23/0xb0
>> [ 725.159131] [<ffffffff815ecba0>] bus_probe_device+0xb0/0xe0
>> [ 725.177055] [<ffffffff815ea693>] device_add+0x443/0x650
>> [ 725.195558] [<ffffffff815ea8be>] device_register+0x1e/0x30
>> [ 725.213133] [<ffffffff81601790>] init_memory_block+0xd0/0xf0
>> [ 725.231533] [<ffffffff816018f1>] register_new_memory+0xb1/0xd0
>> [ 725.250769] [<ffffffff81a961cf>] __add_pages+0x13f/0x250
>> [ 725.269642] [<ffffffff81063770>] ? arch_add_memory+0x70/0xf0
>> [ 725.288764] [<ffffffff81063770>] arch_add_memory+0x70/0xf0
>> [ 725.306117] [<ffffffff81a95f8f>] add_memory+0xef/0x1f0
>> [ 725.322466] [<ffffffffa00293af>] hot_add_req+0x33f/0xf90 [hv_balloon]
>> [ 725.342777] [<ffffffff8109509f>] process_one_work+0x1df/0x4e0
>> [ 725.361459] [<ffffffff8109502d>] ? process_one_work+0x16d/0x4e0
>> [ 725.380390] [<ffffffff810954bb>] worker_thread+0x11b/0x450
>> [ 725.397684] [<ffffffff810953a0>] ? process_one_work+0x4e0/0x4e0
>> [ 725.416533] [<ffffffff8109ac33>] kthread+0xf3/0x110
>> [ 725.433372] [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
>> [ 725.453749] [<ffffffff81ab1dfc>] ret_from_fork+0x7c/0xb0
>> [ 725.470994] [<ffffffff8109ab40>] ? kthread_create_on_node+0x240/0x240
>> [ 725.491469] 6 locks held by kworker/0:1/27:
>> [ 725.505037] #0: ("events"){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
>> [ 725.533370] #1: ((&dm_device.ha_wrk.wrk)){......}, at: [<ffffffff8109502d>] process_one_work+0x16d/0x4e0
>> [ 725.565580] #2: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
>> [ 725.594369] #3: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
>> [ 725.628554] #4: (mem_sysfs_mutex){......}, at: [<ffffffff81601873>] register_new_memory+0x33/0xd0
>> [ 725.658519] #5: (&dev->mutex){......}, at: [<ffffffff815ed773>] device_attach+0x23/0xb0
>>
>> Second participant, udev:
>>
>> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530 0x00000004
>> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0 00000000000151c0
>> [ 725.798332] ffffffff8210d980 ffff88003b94ffd8 00000000000151c0 ffff880037a69270
>> [ 725.822841] ffff8800034949c0 0000000100000001 ffff8800034949c0 ffffffff81ff2b48
>> [ 725.849184] Call Trace:
>> [ 725.858987] [<ffffffff81aac0a9>] schedule_preempt_disabled+0x29/0x70
>> [ 725.879231] [<ffffffff81aae33b>] mutex_lock_nested+0x14b/0x470
>> [ 725.897860] [<ffffffff811e656f>] ? mem_hotplug_begin+0x4f/0x80
>> [ 725.916698] [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
>> [ 725.935064] [<ffffffff811e6525>] ? mem_hotplug_begin+0x5/0x80
>> [ 725.953464] [<ffffffff81a9631b>] online_pages+0x3b/0x520
>> [ 725.971542] [<ffffffff815eb0b3>] ? device_online+0x23/0xa0
>> [ 725.989207] [<ffffffff81601524>] memory_subsys_online+0x64/0xc0
>> [ 726.008513] [<ffffffff815eb0fd>] device_online+0x6d/0xa0
>> [ 726.025579] [<ffffffff816012eb>] store_mem_state+0x5b/0xe0
>> [ 726.043400] [<ffffffff815e8258>] dev_attr_store+0x18/0x30
>> [ 726.060506] [<ffffffff8127a808>] sysfs_kf_write+0x48/0x60
>> [ 726.077940] [<ffffffff81279d1b>] kernfs_fop_write+0x13b/0x1a0
>> [ 726.099416] [<ffffffff811f9f67>] vfs_write+0xb7/0x1f0
>> [ 726.115748] [<ffffffff811fabf8>] SyS_write+0x58/0xd0
>> [ 726.131933] [<ffffffff81ab1ea9>] system_call_fastpath+0x12/0x17
>> [ 726.150691] 7 locks held by systemd-udevd/888:
>> [ 726.165044] #0: (sb_writers#3){......}, at: [<ffffffff811fa063>] vfs_write+0x1b3/0x1f0
>> [ 726.192422] #1: (&of->mutex){......}, at: [<ffffffff81279c46>] kernfs_fop_write+0x66/0x1a0
>> [ 726.220289] #2: (s_active#60){......}, at: [<ffffffff81279c4e>] kernfs_fop_write+0x6e/0x1a0
>> [ 726.249382] #3: (device_hotplug_lock){......}, at: [<ffffffff815e9c15>] lock_device_hotplug_sysfs+0x15/0x50
>> [ 726.281901] #4: (&dev->mutex){......}, at: [<ffffffff815eb0b3>] device_online+0x23/0xa0
>> [ 726.308619] #5: (mem_hotplug.lock){......}, at: [<ffffffff811e6525>] mem_hotplug_begin+0x5/0x80
>> [ 726.337994] #6: (mem_hotplug.lock#2){......}, at: [<ffffffff811e656f>] mem_hotplug_begin+0x4f/0x80
>>
>> In short: onlining grabs device lock and then tries to do mem_hotplug_begin()
>> while add_memory() is between mem_hotplug_begin() and mem_hotplug_done() and it
>> tries grabbing device lock.
>>
>> To my understanding ACPI memory hotplug doesn't have the same issue as
>> device_hotplug_lock is being grabbed when the ACPI device is added.
>>
>> Solve the issue by grabbing device_hotplug_lock before doing add_memory(). If
>> we do that, lock_device_hotplug_sysfs() will cause syscall retry which will
>> eventually succeed. To support the change we need to export lock_device_hotplug/
>> unlock_device_hotplug. This approach can be completely wrong though.
>>
>> Vitaly Kuznetsov (3):
>> driver core: export lock_device_hotplug/unlock_device_hotplug
>> memory_hotplug: add note about holding device_hotplug_lock and
>> add_memory()
>> Drivers: hv: balloon: fix deadlock between memory adding and onlining
>>
>> drivers/base/core.c | 2 ++
>> drivers/hv/hv_balloon.c | 10 ++++++++++
>> mm/memory_hotplug.c | 6 +++++-
>> 3 files changed, 17 insertions(+), 1 deletion(-)
>>
>> --
>> 1.9.3
>>
>> --
>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>> the body to majordomo@kvack.org. For more info on Linux MM,
>> see: http://www.linux-mm.org/
>> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Vitaly
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining
2015-03-09 8:40 ` Vitaly Kuznetsov
@ 2015-03-12 16:18 ` Michal Hocko
0 siblings, 0 replies; 16+ messages in thread
From: Michal Hocko @ 2015-03-12 16:18 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: linux-kernel, Greg Kroah-Hartman, K. Y. Srinivasan,
Haiyang Zhang, Andrew Morton, Yasuaki Ishimatsu, Tang Chen,
Vlastimil Babka, David Rientjes, Fabian Frederick, Zhang Zhen,
Vladimir Davydov, Wang Nan, Rafael J. Wysocki, devel, linux-mm
On Mon 09-03-15 09:40:43, Vitaly Kuznetsov wrote:
> Michal Hocko <mhocko@suse.cz> writes:
>
> > [Sorry for the late response]
> >
> > This is basically the same code posted by KY Srinivasan posted late last
> > year (http://marc.info/?l=linux-mm&m=141782228129426&w=2). I had
> > objections to the implementation
> > http://marc.info/?l=linux-mm&m=141805109216700&w=2
>
> Np, David's alternative fix is already in -mm:
>
> https://lkml.org/lkml/2015/2/12/655
Thanks for the pointer. I have missed this one. This is definitely a
better approach than cluttering around exporting device lock.
--
Michal Hocko
SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 16+ messages in thread