From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 091FEC3ABCB for ; Wed, 14 May 2025 06:47:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BFC106B00D1; Wed, 14 May 2025 02:47:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BABB46B00D2; Wed, 14 May 2025 02:47:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A731F6B00D3; Wed, 14 May 2025 02:47:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8A21A6B00D1 for ; Wed, 14 May 2025 02:47:38 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 88E281CD481 for ; Wed, 14 May 2025 06:47:39 +0000 (UTC) X-FDA: 83440582638.09.ECA8C48 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf15.hostedemail.com (Postfix) with ESMTP id D259CA0007 for ; Wed, 14 May 2025 06:47:36 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf15.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747205257; a=rsa-sha256; cv=none; b=dFZG4CCuuL8ZKBeG63/+0N8173iFNGJIkBzsKCm6BswaDAQkVwRjqlAukGzf9wwcI6pOYu OruLJiwuU4hdDNgQYLjCcRXcLUeMduCmKQSSFXu24wVsIcUGqoUNVDGxaHPmXzyYziU2dX EbYwdHPavTU4CjeC9Ovm/6p6Q4fo09M= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf15.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747205257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kBMCheO81hKCa03XH8hoZAErKFuobi+TUcf5rFjgLG4=; b=6/WSIFRqWWE17SAacIxcloxfsxMApw/IvENfEICsaA0KiIgunx18exosXYcfl0eEA+Ts9R EU39P0/7rtgvrUVEWMTNJqqJBgUtuoTuI2/dAG2aXzmiYzz+07eBpuECzumdMMKvsrQDk2 YjuqeyTgRDe2seB+CjH8mYxHJ0bgfFc= X-AuditID: a67dfc5b-681ff7000002311f-2b-68243c86168a Date: Wed, 14 May 2025 15:47:29 +0900 From: Byungchul Park To: Gavin Guo Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, muchun.song@linux.dev, osalvador@suse.de, akpm@linux-foundation.org, mike.kravetz@oracle.com, kernel-dev@igalia.com, stable@vger.kernel.org, Hugh Dickins , Florent Revest , Gavin Shan , kernel_team@skhynix.com Subject: Re: [PATCH] mm/hugetlb: fix a deadlock with pagecache_folio and hugetlb_fault_mutex_table Message-ID: <20250514064729.GA17622@system.software.com> References: <20250513093448.592150-1-gavinguo@igalia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20250513093448.592150-1-gavinguo@igalia.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprOIsWRmVeSWpSXmKPExsXC9ZZnkW6bjUqGwd+VUhZz1q9hs1iy9gyz xctd25gsnn7qY7E49+I7k8XlXXPYLO6t+c9q8XF/sMWynQ9ZLM5MK7LonvmD1WLBxkeMDjwe CzaVekyY3c3msenTJHaPEzN+s3gsbJjK7PHx6S0Wj/f7rrJ5bD5d7fF5k1wAZxSXTUpqTmZZ apG+XQJXxv6rt9kLlvtV7OmaydzA2G3XxcjJISFgIrFt60IWGPvNq4mMIDaLgKrEvXP7wOJs AuoSN278ZAaxRQSUJT5MOQgU5+JgFjjBJHFx3iJ2kISwQIrEtQPbwBp4BSwkDr+9AGRzcAgJ WEpcn+8MERaUODnzCVgJM9DMP/MuMYOUMAtISyz/xwERlpdo3jobbBWngJXEsq0Q5aJAaw9s O84EslZC4DubxJFjr5ghbpaUOLjiBssERsFZSFbMQrJiFsKKWUhWLGBkWcUolJlXlpuYmWOi l1GZl1mhl5yfu4kRGF3Lav9E72D8dCH4EKMAB6MSD6+FrnKGEGtiWXFl7iFGCQ5mJRHe61lA Id6UxMqq1KL8+KLSnNTiQ4zSHCxK4rxG38pThATSE0tSs1NTC1KLYLJMHJxSDYw1Gbnfk4WV mUzsdrz3qzhk8TlGpPfQ5iQT1+gXIslnf+SePL71x6xHsvumHQhXvXtOfdn5RyJi0syPFqYv /j6z3PvA5m73rENTrxzp3mLhV7C1UN5Ix7iXu/vy0sZFfROvdEYwlc2unaecJniZffphqVn7 fR4kNF1LPCIkfezUzhf/bX47+ixVYinOSDTUYi4qTgQAwToKqKoCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrHLMWRmVeSWpSXmKPExsXC5WfdrNtmo5Jh0HeV22LO+jVsFkvWnmG2 eLlrG5PF0099LBbnXnxnsjg89ySrxeVdc9gs7q35z2rxcX+wxbKdD1kszkwrsuie+YPVYsHG R4wOvB4LNpV6TJjdzeax6dMkdo8TM36zeCxsmMrs8fHpLRaP9/uusnksfvGByWPz6WqPz5vk AriiuGxSUnMyy1KL9O0SuDL2X73NXrDcr2JP10zmBsZuuy5GTg4JAROJN68mMoLYLAKqEvfO 7WMBsdkE1CVu3PjJDGKLCChLfJhyECjOxcEscIJJ4uK8RewgCWGBFIlrB7aBNfAKWEgcfnsB yObgEBKwlLg+3xkiLChxcuYTsBJmoJl/5l1iBilhFpCWWP6PAyIsL9G8dTbYKk4BK4llWyHK RYHWHth2nGkCI98sJJNmIZk0C2HSLCSTFjCyrGIUycwry03MzDHVK87OqMzLrNBLzs/dxAiM lWW1fybuYPxy2f0QowAHoxIPr4WucoYQa2JZcWXuIUYJDmYlEd7rWUAh3pTEyqrUovz4otKc 1OJDjNIcLErivF7hqQlCAumJJanZqakFqUUwWSYOTqkGxvOss2zEJCc92nC5Ztk9n01tu4Ju zQ9+O72i66GSqNG2Ns+CblUVp9ZDKVufX311gseU8f2fY1xuD78umv3jftge/YcqGw4lBCQc 6fy4+4Ct79xLn5eYqUU5a/cl1Yi/qozc8cL4qYlta/K3f/cN605c3XrboJT7wkaT3EvFszlu N0foT3vr0afEUpyRaKjFXFScCABBztgikQIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D259CA0007 X-Stat-Signature: f5e3qpzuduz6eqfne449i6idbe3tf8if X-HE-Tag: 1747205256-934227 X-HE-Meta: U2FsdGVkX18VAavqKNxz8X/zxv/C77NjAOhFajpWOy/SKdoTlf6nrZZ1WNYqh2hZU9KiM88UblP7hFEzaY9fFiawlIn4J2OC9AX68SVCmm7TVmXN1uKbtmhrjqVmOVeX9BqEml7ZHIr3WxsTJ7RKaTEBzlO6Qe5rPfpStncEHWZwe1WI/iGJcSpGPCA+mCntyYECIqKYBMndIBB8eTOWt0yCXcq+1Wh317nj6Xlqi8+saR2epWrPQjzEO8zvQEWoXPYZk5F6xjU/o9HZ0CiVJhOadmuBOnhC3bSytDpTql4Y5Pf+VE7ynoiDVynCgPcOkSCyKasJIcClvNG6uQ60aNFFsTl4p3lzSk7g99lrfcsW9PVR3sHwvspQ8rJjhn/+vAxfU0i1Iu15BJVnVeQ4jiD1j2JGzJWSmZBXOc+wQLj9ktjMQtqSvTFjESumdAH8pY2zfPRfcs95hz895+gAQfIXroyjUPjEJi4E0UtImg1TR//pwButbQgq5r+6+LoauTXBKbvBIwQPS6rA1OMf7jyBw0Mp4uWJqyFoDNZ0AftfV4vgnRWqRu3S08dKRYqxWYf5MAPjWfyQDimT9HbPVkgouvOvGPjfOSaNzq7TK74o3dd/fCRTUhGZqUdHzhOWKaMg/bYFudWZb912HqQa96Q1KeGe9lAWQiubQXzctUvrCZKCMF4KjzGOLYTMWWuqNZ8Z8XZqFSxhbx8JlLkZBcpVB8vAqbrqV8gOQN52jMl/8uNnArG6ExzY4bUeUcVtQAcl3N+51hG+mGzbXt/eU18ELBYlUoGj/FC1r9ekXi3NSR8BGZbPtox1Kd/jzLCiVVewgMoYCm1hsAClcadvWPK5bXLzCOUzRt2gDwgKhDxGVpuUo9HVk8iKEf3eE65mcGltWp6zyAd31+1Uwo4Lz9xwepybMyKCgKAr4dMIO6/kREc9gRXo/CUX7zAY/7MfwdfmkMgSnKPKV206QKV Hqar+7RS SgnhfQKRfB3kEGjNt9HId3J4bbKtdjrtDq+G9/7YC0FDWNR6UJSJE9O/89JxYVAuCZvtS3sS6a9zPl9yCnnEuMj1C2l17ivVm7pmfzXZUEfK7ERXq31xdv+8TQ2G61jO8wJ52yrgZKbznUVtCR6j9Q8oLyyPyvFzomWte1s37e9K5HQR5BANeWqmof+/usemWz/rYcGko907toyZGxIz6w4TXFBKNiMmWQGlkbM+NzED/xwE93YiqoeZUemKtaQcBJo2ZE65sdXA9C1ATvEwz7YitbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, May 13, 2025 at 05:34:48PM +0800, Gavin Guo wrote: > The patch fixes a deadlock which can be triggered by an internal > syzkaller [1] reproducer and captured by bpftrace script [2] and its log Hi, I'm trying to reproduce using the test program [1]. But not yet produced. I see a lot of segfaults while running [1]. I guess something goes wrong. Is there any prerequisite condition to reproduce it? Lemme know if any. Or can you try DEPT15 with your config and environment by the following steps: 1. Apply the patchset on v6.15-rc6. https://lkml.kernel.org/r/20250513100730.12664-1-byungchul@sk.com 2. Turn on CONFIG_DEPT. 3. Run test program reproducing the deadlock. 4. Check dmesg to see if dept reported the dependency. Byungchul > [3] in this scenario: > > Process 1 Process 2 > --- --- > hugetlb_fault > mutex_lock(B) // take B > filemap_lock_hugetlb_folio > filemap_lock_folio > __filemap_get_folio > folio_lock(A) // take A > hugetlb_wp > mutex_unlock(B) // release B > ... hugetlb_fault > ... mutex_lock(B) // take B > filemap_lock_hugetlb_folio > filemap_lock_folio > __filemap_get_folio > folio_lock(A) // blocked > unmap_ref_private > ... > mutex_lock(B) // retake and blocked > > This is a ABBA deadlock involving two locks: > - Lock A: pagecache_folio lock > - Lock B: hugetlb_fault_mutex_table lock > > The deadlock occurs between two processes as follows: > 1. The first process (let’s call it Process 1) is handling a > copy-on-write (COW) operation on a hugepage via hugetlb_wp. Due to > insufficient reserved hugetlb pages, Process 1, owner of the reserved > hugetlb page, attempts to unmap a hugepage owned by another process > (non-owner) to satisfy the reservation. Before unmapping, Process 1 > acquires lock B (hugetlb_fault_mutex_table lock) and then lock A > (pagecache_folio lock). To proceed with the unmap, it releases Lock B > but retains Lock A. After the unmap, Process 1 tries to reacquire Lock > B. However, at this point, Lock B has already been acquired by another > process. > > 2. The second process (Process 2) enters the hugetlb_fault handler > during the unmap operation. It successfully acquires Lock B > (hugetlb_fault_mutex_table lock) that was just released by Process 1, > but then attempts to acquire Lock A (pagecache_folio lock), which is > still held by Process 1. > > As a result, Process 1 (holding Lock A) is blocked waiting for Lock B > (held by Process 2), while Process 2 (holding Lock B) is blocked waiting > for Lock A (held by Process 1), constructing a ABBA deadlock scenario. > > The solution here is to unlock the pagecache_folio and provide the > pagecache_folio_unlocked variable to the caller to have the visibility > over the pagecache_folio status for subsequent handling. > > The error message: > INFO: task repro_20250402_:13229 blocked for more than 64 seconds. > Not tainted 6.15.0-rc3+ #24 > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > task:repro_20250402_ state:D stack:25856 pid:13229 tgid:13228 ppid:3513 task_flags:0x400040 flags:0x00004006 > Call Trace: > > __schedule+0x1755/0x4f50 > schedule+0x158/0x330 > schedule_preempt_disabled+0x15/0x30 > __mutex_lock+0x75f/0xeb0 > hugetlb_wp+0xf88/0x3440 > hugetlb_fault+0x14c8/0x2c30 > trace_clock_x86_tsc+0x20/0x20 > do_user_addr_fault+0x61d/0x1490 > exc_page_fault+0x64/0x100 > asm_exc_page_fault+0x26/0x30 > RIP: 0010:__put_user_4+0xd/0x20 > copy_process+0x1f4a/0x3d60 > kernel_clone+0x210/0x8f0 > __x64_sys_clone+0x18d/0x1f0 > do_syscall_64+0x6a/0x120 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x41b26d > > INFO: task repro_20250402_:13229 is blocked on a mutex likely owned by task repro_20250402_:13250. > task:repro_20250402_ state:D stack:28288 pid:13250 tgid:13228 ppid:3513 task_flags:0x400040 flags:0x00000006 > Call Trace: > > __schedule+0x1755/0x4f50 > schedule+0x158/0x330 > io_schedule+0x92/0x110 > folio_wait_bit_common+0x69a/0xba0 > __filemap_get_folio+0x154/0xb70 > hugetlb_fault+0xa50/0x2c30 > trace_clock_x86_tsc+0x20/0x20 > do_user_addr_fault+0xace/0x1490 > exc_page_fault+0x64/0x100 > asm_exc_page_fault+0x26/0x30 > RIP: 0033:0x402619 > > INFO: task repro_20250402_:13250 blocked for more than 65 seconds. > Not tainted 6.15.0-rc3+ #24 > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > task:repro_20250402_ state:D stack:28288 pid:13250 tgid:13228 ppid:3513 task_flags:0x400040 flags:0x00000006 > Call Trace: > > __schedule+0x1755/0x4f50 > schedule+0x158/0x330 > io_schedule+0x92/0x110 > folio_wait_bit_common+0x69a/0xba0 > __filemap_get_folio+0x154/0xb70 > hugetlb_fault+0xa50/0x2c30 > trace_clock_x86_tsc+0x20/0x20 > do_user_addr_fault+0xace/0x1490 > exc_page_fault+0x64/0x100 > asm_exc_page_fault+0x26/0x30 > RIP: 0033:0x402619 > > > Showing all locks held in the system: > 1 lock held by khungtaskd/35: > #0: ffffffff879a7440 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x30/0x180 > 2 locks held by repro_20250402_/13229: > #0: ffff888017d801e0 (&mm->mmap_lock){++++}-{4:4}, at: lock_mm_and_find_vma+0x37/0x300 > #1: ffff888000fec848 (&hugetlb_fault_mutex_table[i]){+.+.}-{4:4}, at: hugetlb_wp+0xf88/0x3440 > 3 locks held by repro_20250402_/13250: > #0: ffff8880177f3d08 (vm_lock){++++}-{0:0}, at: do_user_addr_fault+0x41b/0x1490 > #1: ffff888000fec848 (&hugetlb_fault_mutex_table[i]){+.+.}-{4:4}, at: hugetlb_fault+0x3b8/0x2c30 > #2: ffff8880129500e8 (&resv_map->rw_sema){++++}-{4:4}, at: hugetlb_fault+0x494/0x2c30 > > Link: https://drive.google.com/file/d/1DVRnIW-vSayU5J1re9Ct_br3jJQU6Vpb/view?usp=drive_link [1] > Link: https://github.com/bboymimi/bpftracer/blob/master/scripts/hugetlb_lock_debug.bt [2] > Link: https://drive.google.com/file/d/1bWq2-8o-BJAuhoHWX7zAhI6ggfhVzQUI/view?usp=sharing [3] > Fixes: 40549ba8f8e0 ("hugetlb: use new vma_lock for pmd sharing synchronization") > Cc: > Cc: Hugh Dickins > Cc: Florent Revest > Cc: Gavin Shan > Signed-off-by: Gavin Guo > --- > mm/hugetlb.c | 33 ++++++++++++++++++++++++++++----- > 1 file changed, 28 insertions(+), 5 deletions(-) > > diff --git a/mm/hugetlb.c b/mm/hugetlb.c > index e3e6ac991b9c..ad54a74aa563 100644 > --- a/mm/hugetlb.c > +++ b/mm/hugetlb.c > @@ -6115,7 +6115,8 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma, > * Keep the pte_same checks anyway to make transition from the mutex easier. > */ > static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, > - struct vm_fault *vmf) > + struct vm_fault *vmf, > + bool *pagecache_folio_unlocked) > { > struct vm_area_struct *vma = vmf->vma; > struct mm_struct *mm = vma->vm_mm; > @@ -6212,6 +6213,22 @@ static vm_fault_t hugetlb_wp(struct folio *pagecache_folio, > u32 hash; > > folio_put(old_folio); > + /* > + * The pagecache_folio needs to be unlocked to avoid > + * deadlock and we won't re-lock it in hugetlb_wp(). The > + * pagecache_folio could be truncated after being > + * unlocked. So its state should not be relied > + * subsequently. > + * > + * Setting *pagecache_folio_unlocked to true allows the > + * caller to handle any necessary logic related to the > + * folio's unlocked state. > + */ > + if (pagecache_folio) { > + folio_unlock(pagecache_folio); > + if (pagecache_folio_unlocked) > + *pagecache_folio_unlocked = true; > + } > /* > * Drop hugetlb_fault_mutex and vma_lock before > * unmapping. unmapping needs to hold vma_lock > @@ -6566,7 +6583,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, > hugetlb_count_add(pages_per_huge_page(h), mm); > if ((vmf->flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) { > /* Optimization, do the COW without a second fault */ > - ret = hugetlb_wp(folio, vmf); > + ret = hugetlb_wp(folio, vmf, NULL); > } > > spin_unlock(vmf->ptl); > @@ -6638,6 +6655,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, > struct hstate *h = hstate_vma(vma); > struct address_space *mapping; > int need_wait_lock = 0; > + bool pagecache_folio_unlocked = false; > struct vm_fault vmf = { > .vma = vma, > .address = address & huge_page_mask(h), > @@ -6792,7 +6810,8 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, > > if (flags & (FAULT_FLAG_WRITE|FAULT_FLAG_UNSHARE)) { > if (!huge_pte_write(vmf.orig_pte)) { > - ret = hugetlb_wp(pagecache_folio, &vmf); > + ret = hugetlb_wp(pagecache_folio, &vmf, > + &pagecache_folio_unlocked); > goto out_put_page; > } else if (likely(flags & FAULT_FLAG_WRITE)) { > vmf.orig_pte = huge_pte_mkdirty(vmf.orig_pte); > @@ -6809,10 +6828,14 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, > out_ptl: > spin_unlock(vmf.ptl); > > - if (pagecache_folio) { > + /* > + * If the pagecache_folio is unlocked in hugetlb_wp(), we skip > + * folio_unlock() here. > + */ > + if (pagecache_folio && !pagecache_folio_unlocked) > folio_unlock(pagecache_folio); > + if (pagecache_folio) > folio_put(pagecache_folio); > - } > out_mutex: > hugetlb_vma_unlock_read(vma); > > > base-commit: d76bb1ebb5587f66b0f8b8099bfbb44722bc08b3 > -- > 2.43.0 >