From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pa0-f54.google.com (mail-pa0-f54.google.com [209.85.220.54]) by kanga.kvack.org (Postfix) with ESMTP id C8CC16B0038 for ; Thu, 12 Feb 2015 10:38:46 -0500 (EST) Received: by mail-pa0-f54.google.com with SMTP id kx10so12102949pab.13 for ; Thu, 12 Feb 2015 07:38:46 -0800 (PST) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0116.outbound.protection.outlook.com. [157.56.111.116]) by mx.google.com with ESMTPS id wp3si5564243pab.1.2015.02.12.07.38.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 12 Feb 2015 07:38:45 -0800 (PST) From: KY Srinivasan Subject: RE: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining Date: Thu, 12 Feb 2015 15:38:42 +0000 Message-ID: References: <1423736634-338-1-git-send-email-vkuznets@redhat.com> In-Reply-To: <1423736634-338-1-git-send-email-vkuznets@redhat.com> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Vitaly Kuznetsov , "linux-kernel@vger.kernel.org" Cc: Greg Kroah-Hartman , Haiyang Zhang , Andrew Morton , Yasuaki Ishimatsu , Tang Chen , Vlastimil Babka , David Rientjes , Fabian Frederick , Zhang Zhen , Vladimir Davydov , Wang Nan , "Rafael J. Wysocki" , "devel@linuxdriverproject.org" , "linux-mm@kvack.org" > -----Original Message----- > From: Vitaly Kuznetsov [mailto:vkuznets@redhat.com] > Sent: Thursday, February 12, 2015 2:24 AM > To: linux-kernel@vger.kernel.org > Cc: Greg Kroah-Hartman; KY Srinivasan; Haiyang Zhang; Andrew Morton; > Yasuaki Ishimatsu; Tang Chen; Vlastimil Babka; David Rientjes; Fabian > Frederick; Zhang Zhen; Vladimir Davydov; Wang Nan; Rafael J. Wysocki; > devel@linuxdriverproject.org; linux-mm@kvack.org > Subject: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock > between memory adding and onlining >=20 > RESEND (with no changes) because Rafael J. Wysocki was missing in > recepients. >=20 > If newly added memory is brought online with e.g. udev rule: > SUBSYSTEM=3D=3D"memory", ACTION=3D=3D"add", ATTR{state}=3D"online" > the following deadlock is observed (and easily reproducable): >=20 > First participant, worker thread doing add_memory(): >=20 > [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00= 000000 > [ 724.973543] Workqueue: events hot_add_req [hv_balloon] [ 724.991736] > ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 [ > 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0 > ffff88003a77a4e0 [ 725.046486] ffff88003fa1dc30 00000001032a6000 > ffff88003a7ca838 ffff88003a7ca898 [ 725.072969] Call Trace: > [ 725.082690] [] schedule_preempt_disabled+0x29/0x70 > [ 725.103799] [] mutex_lock_nested+0x14b/0x470 [ > 725.122367] [] ? device_attach+0x23/0xb0 [ 725.140992= ] > [] device_attach+0x23/0xb0 [ 725.159131] > [] bus_probe_device+0xb0/0xe0 [ 725.177055] > [] device_add+0x443/0x650 [ 725.195558] > [] device_register+0x1e/0x30 [ 725.213133] > [] init_memory_block+0xd0/0xf0 [ 725.231533] > [] register_new_memory+0xb1/0xd0 [ 725.250769] > [] __add_pages+0x13f/0x250 [ 725.269642] > [] ? arch_add_memory+0x70/0xf0 [ 725.288764] > [] arch_add_memory+0x70/0xf0 [ 725.306117] > [] add_memory+0xef/0x1f0 [ 725.322466] > [] hot_add_req+0x33f/0xf90 [hv_balloon] [ 725.342777] > [] process_one_work+0x1df/0x4e0 [ 725.361459] > [] ? process_one_work+0x16d/0x4e0 [ 725.380390] > [] worker_thread+0x11b/0x450 [ 725.397684] > [] ? process_one_work+0x4e0/0x4e0 [ 725.416533] > [] kthread+0xf3/0x110 [ 725.433372] [] > ? kthread_create_on_node+0x240/0x240 > [ 725.453749] [] ret_from_fork+0x7c/0xb0 [ 725.47099= 4] > [] ? kthread_create_on_node+0x240/0x240 > [ 725.491469] 6 locks held by kworker/0:1/27: > [ 725.505037] #0: ("events"){......}, at: [] > process_one_work+0x16d/0x4e0 [ 725.533370] #1: > ((&dm_device.ha_wrk.wrk)){......}, at: [] > process_one_work+0x16d/0x4e0 [ 725.565580] #2: > (mem_hotplug.lock){......}, at: [] > mem_hotplug_begin+0x5/0x80 [ 725.594369] #3: > (mem_hotplug.lock#2){......}, at: [] > mem_hotplug_begin+0x4f/0x80 [ 725.628554] #4: > (mem_sysfs_mutex){......}, at: [] > register_new_memory+0x33/0xd0 [ 725.658519] #5: (&dev->mutex){......}, > at: [] device_attach+0x23/0xb0 >=20 > Second participant, udev: >=20 > [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530 > 0x00000004 > [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0 > 00000000000151c0 [ 725.798332] ffffffff8210d980 ffff88003b94ffd8 > 00000000000151c0 ffff880037a69270 [ 725.822841] ffff8800034949c0 > 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 [ 725.849184] Call Tr= ace: > [ 725.858987] [] schedule_preempt_disabled+0x29/0x70 > [ 725.879231] [] mutex_lock_nested+0x14b/0x470 [ > 725.897860] [] ? mem_hotplug_begin+0x4f/0x80 [ > 725.916698] [] mem_hotplug_begin+0x4f/0x80 [ > 725.935064] [] ? mem_hotplug_begin+0x5/0x80 [ > 725.953464] [] online_pages+0x3b/0x520 [ 725.971542] > [] ? device_online+0x23/0xa0 [ 725.989207] > [] memory_subsys_online+0x64/0xc0 [ 726.008513] > [] device_online+0x6d/0xa0 [ 726.025579] > [] store_mem_state+0x5b/0xe0 [ 726.043400] > [] dev_attr_store+0x18/0x30 [ 726.060506] > [] sysfs_kf_write+0x48/0x60 [ 726.077940] > [] kernfs_fop_write+0x13b/0x1a0 [ 726.099416] > [] vfs_write+0xb7/0x1f0 [ 726.115748] [] > SyS_write+0x58/0xd0 [ 726.131933] [] > system_call_fastpath+0x12/0x17 [ 726.150691] 7 locks held by systemd- > udevd/888: > [ 726.165044] #0: (sb_writers#3){......}, at: [] > vfs_write+0x1b3/0x1f0 [ 726.192422] #1: (&of->mutex){......}, at: > [] kernfs_fop_write+0x66/0x1a0 [ 726.220289] #2: > (s_active#60){......}, at: [] kernfs_fop_write+0x6e/0x1= a0 [ > 726.249382] #3: (device_hotplug_lock){......}, at: [] > lock_device_hotplug_sysfs+0x15/0x50 > [ 726.281901] #4: (&dev->mutex){......}, at: [] > device_online+0x23/0xa0 [ 726.308619] #5: (mem_hotplug.lock){......}, = at: > [] mem_hotplug_begin+0x5/0x80 [ 726.337994] #6: > (mem_hotplug.lock#2){......}, at: [] > mem_hotplug_begin+0x4f/0x80 >=20 > In short: onlining grabs device lock and then tries to do > mem_hotplug_begin() while add_memory() is between > mem_hotplug_begin() and mem_hotplug_done() and it tries grabbing > device lock. >=20 > To my understanding ACPI memory hotplug doesn't have the same issue as > device_hotplug_lock is being grabbed when the ACPI device is added. >=20 > Solve the issue by grabbing device_hotplug_lock before doing > add_memory(). If we do that, lock_device_hotplug_sysfs() will cause sysca= ll > retry which will eventually succeed. To support the change we need to > export lock_device_hotplug/ unlock_device_hotplug. This approach can be > completely wrong though. This issue was first discovered by Andy Whitcroft: https://lkml.org/lkml/20= 14/3/14/451 I had sent patches based on Andy's analysis that did not affect the users o= f the kernel hot-add memory APIs: https://lkml.org/lkml/2014/12/2/662 This patch puts the burden where it needs to be and can address the issue f= or all clients. K. Y >=20 > Vitaly Kuznetsov (3): > driver core: export lock_device_hotplug/unlock_device_hotplug > memory_hotplug: add note about holding device_hotplug_lock and > add_memory() > Drivers: hv: balloon: fix deadlock between memory adding and onlining >=20 > drivers/base/core.c | 2 ++ > drivers/hv/hv_balloon.c | 10 ++++++++++ > mm/memory_hotplug.c | 6 +++++- > 3 files changed, 17 insertions(+), 1 deletion(-) >=20 > -- > 1.9.3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org