From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-f174.google.com (mail-we0-f174.google.com [74.125.82.174]) by kanga.kvack.org (Postfix) with ESMTP id F00F66B0032 for ; Mon, 9 Mar 2015 04:41:17 -0400 (EDT) Received: by wesw55 with SMTP id w55so9585176wes.2 for ; Mon, 09 Mar 2015 01:41:17 -0700 (PDT) Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id k6si18269372wif.35.2015.03.09.01.41.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 09 Mar 2015 01:41:15 -0700 (PDT) From: Vitaly Kuznetsov Subject: Re: [PATCH RESEND 0/3] memory_hotplug: hyperv: fix deadlock between memory adding and onlining References: <1423736634-338-1-git-send-email-vkuznets@redhat.com> <20150306155002.GB23443@dhcp22.suse.cz> Date: Mon, 09 Mar 2015 09:40:43 +0100 In-Reply-To: <20150306155002.GB23443@dhcp22.suse.cz> (Michal Hocko's message of "Fri, 6 Mar 2015 16:50:02 +0100") Message-ID: <871tkyy35g.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "K. Y. Srinivasan" , Haiyang Zhang , Andrew Morton , Yasuaki Ishimatsu , Tang Chen , Vlastimil Babka , David Rientjes , Fabian Frederick , Zhang Zhen , Vladimir Davydov , Wang Nan , "Rafael J. Wysocki" , devel@linuxdriverproject.org, linux-mm@kvack.org Michal Hocko writes: > [Sorry for the late response] > > This is basically the same code posted by KY Srinivasan posted late last > year (http://marc.info/?l=linux-mm&m=141782228129426&w=2). I had > objections to the implementation > http://marc.info/?l=linux-mm&m=141805109216700&w=2 Np, David's alternative fix is already in -mm: https://lkml.org/lkml/2015/2/12/655 > > On Thu 12-02-15 11:23:51, Vitaly Kuznetsov wrote: >> RESEND (with no changes) because Rafael J. Wysocki was missing in recepients. >> >> If newly added memory is brought online with e.g. udev rule: >> SUBSYSTEM=="memory", ACTION=="add", ATTR{state}="online" >> the following deadlock is observed (and easily reproducable): >> >> First participant, worker thread doing add_memory(): >> >> [ 724.948846] kworker/0:1 D ffff88000412f9c8 13248 27 2 0x00000000 >> [ 724.973543] Workqueue: events hot_add_req [hv_balloon] >> [ 724.991736] ffff88000412f9c8 0000000000000000 ffff88003fa1dc30 00000000000151c0 >> [ 725.019725] 0000000000000246 ffff88000412ffd8 00000000000151c0 ffff88003a77a4e0 >> [ 725.046486] ffff88003fa1dc30 00000001032a6000 ffff88003a7ca838 ffff88003a7ca898 >> [ 725.072969] Call Trace: >> [ 725.082690] [] schedule_preempt_disabled+0x29/0x70 >> [ 725.103799] [] mutex_lock_nested+0x14b/0x470 >> [ 725.122367] [] ? device_attach+0x23/0xb0 >> [ 725.140992] [] device_attach+0x23/0xb0 >> [ 725.159131] [] bus_probe_device+0xb0/0xe0 >> [ 725.177055] [] device_add+0x443/0x650 >> [ 725.195558] [] device_register+0x1e/0x30 >> [ 725.213133] [] init_memory_block+0xd0/0xf0 >> [ 725.231533] [] register_new_memory+0xb1/0xd0 >> [ 725.250769] [] __add_pages+0x13f/0x250 >> [ 725.269642] [] ? arch_add_memory+0x70/0xf0 >> [ 725.288764] [] arch_add_memory+0x70/0xf0 >> [ 725.306117] [] add_memory+0xef/0x1f0 >> [ 725.322466] [] hot_add_req+0x33f/0xf90 [hv_balloon] >> [ 725.342777] [] process_one_work+0x1df/0x4e0 >> [ 725.361459] [] ? process_one_work+0x16d/0x4e0 >> [ 725.380390] [] worker_thread+0x11b/0x450 >> [ 725.397684] [] ? process_one_work+0x4e0/0x4e0 >> [ 725.416533] [] kthread+0xf3/0x110 >> [ 725.433372] [] ? kthread_create_on_node+0x240/0x240 >> [ 725.453749] [] ret_from_fork+0x7c/0xb0 >> [ 725.470994] [] ? kthread_create_on_node+0x240/0x240 >> [ 725.491469] 6 locks held by kworker/0:1/27: >> [ 725.505037] #0: ("events"){......}, at: [] process_one_work+0x16d/0x4e0 >> [ 725.533370] #1: ((&dm_device.ha_wrk.wrk)){......}, at: [] process_one_work+0x16d/0x4e0 >> [ 725.565580] #2: (mem_hotplug.lock){......}, at: [] mem_hotplug_begin+0x5/0x80 >> [ 725.594369] #3: (mem_hotplug.lock#2){......}, at: [] mem_hotplug_begin+0x4f/0x80 >> [ 725.628554] #4: (mem_sysfs_mutex){......}, at: [] register_new_memory+0x33/0xd0 >> [ 725.658519] #5: (&dev->mutex){......}, at: [] device_attach+0x23/0xb0 >> >> Second participant, udev: >> >> [ 725.750889] systemd-udevd D ffff88003b94fc68 14016 888 530 0x00000004 >> [ 725.773767] ffff88003b94fc68 0000000000000000 ffff8800034949c0 00000000000151c0 >> [ 725.798332] ffffffff8210d980 ffff88003b94ffd8 00000000000151c0 ffff880037a69270 >> [ 725.822841] ffff8800034949c0 0000000100000001 ffff8800034949c0 ffffffff81ff2b48 >> [ 725.849184] Call Trace: >> [ 725.858987] [] schedule_preempt_disabled+0x29/0x70 >> [ 725.879231] [] mutex_lock_nested+0x14b/0x470 >> [ 725.897860] [] ? mem_hotplug_begin+0x4f/0x80 >> [ 725.916698] [] mem_hotplug_begin+0x4f/0x80 >> [ 725.935064] [] ? mem_hotplug_begin+0x5/0x80 >> [ 725.953464] [] online_pages+0x3b/0x520 >> [ 725.971542] [] ? device_online+0x23/0xa0 >> [ 725.989207] [] memory_subsys_online+0x64/0xc0 >> [ 726.008513] [] device_online+0x6d/0xa0 >> [ 726.025579] [] store_mem_state+0x5b/0xe0 >> [ 726.043400] [] dev_attr_store+0x18/0x30 >> [ 726.060506] [] sysfs_kf_write+0x48/0x60 >> [ 726.077940] [] kernfs_fop_write+0x13b/0x1a0 >> [ 726.099416] [] vfs_write+0xb7/0x1f0 >> [ 726.115748] [] SyS_write+0x58/0xd0 >> [ 726.131933] [] system_call_fastpath+0x12/0x17 >> [ 726.150691] 7 locks held by systemd-udevd/888: >> [ 726.165044] #0: (sb_writers#3){......}, at: [] vfs_write+0x1b3/0x1f0 >> [ 726.192422] #1: (&of->mutex){......}, at: [] kernfs_fop_write+0x66/0x1a0 >> [ 726.220289] #2: (s_active#60){......}, at: [] kernfs_fop_write+0x6e/0x1a0 >> [ 726.249382] #3: (device_hotplug_lock){......}, at: [] lock_device_hotplug_sysfs+0x15/0x50 >> [ 726.281901] #4: (&dev->mutex){......}, at: [] device_online+0x23/0xa0 >> [ 726.308619] #5: (mem_hotplug.lock){......}, at: [] mem_hotplug_begin+0x5/0x80 >> [ 726.337994] #6: (mem_hotplug.lock#2){......}, at: [] mem_hotplug_begin+0x4f/0x80 >> >> In short: onlining grabs device lock and then tries to do mem_hotplug_begin() >> while add_memory() is between mem_hotplug_begin() and mem_hotplug_done() and it >> tries grabbing device lock. >> >> To my understanding ACPI memory hotplug doesn't have the same issue as >> device_hotplug_lock is being grabbed when the ACPI device is added. >> >> Solve the issue by grabbing device_hotplug_lock before doing add_memory(). If >> we do that, lock_device_hotplug_sysfs() will cause syscall retry which will >> eventually succeed. To support the change we need to export lock_device_hotplug/ >> unlock_device_hotplug. This approach can be completely wrong though. >> >> Vitaly Kuznetsov (3): >> driver core: export lock_device_hotplug/unlock_device_hotplug >> memory_hotplug: add note about holding device_hotplug_lock and >> add_memory() >> Drivers: hv: balloon: fix deadlock between memory adding and onlining >> >> drivers/base/core.c | 2 ++ >> drivers/hv/hv_balloon.c | 10 ++++++++++ >> mm/memory_hotplug.c | 6 +++++- >> 3 files changed, 17 insertions(+), 1 deletion(-) >> >> -- >> 1.9.3 >> >> -- >> To unsubscribe, send a message with 'unsubscribe linux-mm' in >> the body to majordomo@kvack.org. For more info on Linux MM, >> see: http://www.linux-mm.org/ >> Don't email: email@kvack.org -- Vitaly -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org