From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BD10C47DA9 for ; Mon, 29 Jan 2024 12:56:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2E4D8D0007; Mon, 29 Jan 2024 07:56:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDDA48D0001; Mon, 29 Jan 2024 07:56:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA5668D0007; Mon, 29 Jan 2024 07:56:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9A0798D0001 for ; Mon, 29 Jan 2024 07:56:36 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5465B409F4 for ; Mon, 29 Jan 2024 12:56:36 +0000 (UTC) X-FDA: 81732347592.11.06AE1DF Received: from szxga05-in.huawei.com (szxga05-in.huawei.com [45.249.212.191]) by imf06.hostedemail.com (Postfix) with ESMTP id D1D2D180017 for ; Mon, 29 Jan 2024 12:56:31 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706532994; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dNlKeFdo2bBfweOGktYFXFmFANNhUADwjpLaKwBn3Hw=; b=j8XiKXEkLR4eZ/utzSiGgZpNCqOn74U6ZJto5IqjWlDdFRM9CQ4dbqnJz1vdXifvkeI823 wDKk3EYQkE0XhDaipwtHFL7f3ykW2vuo8zrWjjBFjLqjO/w9rMjnN/8p3aT0Qgi4xnH0zV MbEK+dgzNr2zoDpWc6LPSO9zxD4AsoE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706532994; a=rsa-sha256; cv=none; b=IlJliukEfK7vqcM7buZQ4xJmxODSwZOdkTEdcvGncK7kN/Woxo+kaZRYpgWsaHXMK8indk ZuB6Utb8GfXfAbGt7HhIY0LF/yBVlavBqhjSf+SMA0YIPIqLBisqxOCCtxFXHOVVz7PaWM 7jxmgRDGgub92y3U8u6CiVkmvsfF/VA= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.191 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.19.162.112]) by szxga05-in.huawei.com (SkyGuard) with ESMTP id 4TNpCy3VKLz1gy4D; Mon, 29 Jan 2024 20:54:38 +0800 (CST) Received: from canpemm500002.china.huawei.com (unknown [7.192.104.244]) by mail.maildlp.com (Postfix) with ESMTPS id 13EE91404DA; Mon, 29 Jan 2024 20:56:27 +0800 (CST) Received: from [10.173.135.154] (10.173.135.154) by canpemm500002.china.huawei.com (7.192.104.244) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Mon, 29 Jan 2024 20:56:26 +0800 Subject: Re: hugetlbfs: WARNING: bad unlock balance detected during MADV_REMOVE From: Miaohe Lin To: Muchun Song , Thorvald Natvig CC: Linux-MM References: <42788ABD-99AE-4AEF-B543-C0FABAFA0464@linux.dev> <4780b0e3-42e1-9099-d010-5a1793b6cbd3@huawei.com> Message-ID: <531195fb-b642-2bc1-3a07-4944ee5d8664@huawei.com> Date: Mon, 29 Jan 2024 20:56:26 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <4780b0e3-42e1-9099-d010-5a1793b6cbd3@huawei.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.173.135.154] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To canpemm500002.china.huawei.com (7.192.104.244) X-Rspamd-Queue-Id: D1D2D180017 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 7y56kyd9iutwkp39re74m61q61we41s7 X-HE-Tag: 1706532991-910428 X-HE-Meta: U2FsdGVkX1/2YDFBdL54/FkFYvCTQcd0h3Ogxy1wloNs5wfIYNToogYvfS3QlPnNbv6bKZ6qza5VQT8nAVtDuRqKHCwrPQT5/tRib7b1GfJ8IdI6t9W3ncX7PhlgOw/SyLStZD7k8pkTl7v4Vb2XXqk/bHyIWZ+TmcNAzyR/Tb2vav2msro9cVfeVs0CxCZyrsLdwTnBBdFnGztwNqs7B8t1AjOjiVrxeJxy8aY/Q9BWMhHXYsE0qN/iCIuROe7x8nPWfM5QJp/Pupo1X6wXL/WkbAXn6NbsFPn6Ou912LgWVuvTGiCNeelEqF+u/gEdVnJu6KZHK9E7D2b+Y6oHGf4/HhlCWq7fsyhQyZYscwBdjO9pe3mewwR7KWzz+0Jdn+mJx0sIPXH6M+aU/HOmY4HC5XHgllsXG1YZwmFeUlIk3dpEt7w4ABma6Cx6Pf8SexMJY899yzUM+FTzvbt9DiKyIJjYx8yQBQe+47ig3kYZcqUY2zJ6V+ZFEYpgjPar2x1hNEzKI3qEc1sxj0OVBAR3GfBxVKd4s1nxb6CTH7KN3HvuuNsefBT7SKj6gsSOD+nj4xhSDRLdB4SOxdwAo4i94SFzgyR+y+7VrBOVlYQwOz04qdbl5hdok/ECYf6ITGOZfPpAXvwuqa+GMmXSmgNgDz2Up/0tu7GJWTnBnpUTTDILGzekJF1+DtfvgFhZ2jzNgXUPVlrAyVCfSNQgzFf3UrefscETE5MQ+U8a3GYdJzWuvxPGL6Et9JbrvHrKuvUtsdwaA0jURlZk+3wMys8Q6vr53Uw5S3NspDhtyVWKtj15MLLuZQGz35mUAf7EKXMmyxyvrdJXpXw8vgMqRHm9gvNidb0eKJqFIXuwa7e+122tSt4xRgIXtMM6VFDY/peX8YPOAXuLxTrOt11w6tTBA6UaaZHq4W2glH29hBsm4N4EuXSm6XGN2QfGiJsc+LIQtuPlXDOVIlADO9a oxFaxnW7 dCKO6H8h5JVSKySt7a8fSl1rvbiGMIgxeuR/j72nSGGRXrQg09jwEdSzSfonjUhiLm8r2XrDIbLTGAN0XdyQQkB4b7hdcnkFAy8b6txGkXxaQfVo9dYTmplWrz3kMGvEAhAlT9wCyQYIwPfKBsK6h0KfQ9ugAORTEvEC4CnnP+nwROaWm/CiqC++NqCb+CRW1nScOrW4h8iRlHbyrFCxTWEYR8mw067MZDZFPKQLWrQ/893B/JwE0nJWbBUTEeVfhZC5rZsQkWjBDNVWx6QfNqNpN2g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/1/27 18:13, Miaohe Lin wrote: > On 2024/1/26 15:50, Muchun Song wrote: >> >> >>> On Jan 26, 2024, at 04:28, Thorvald Natvig wrote: >>> >>> We've found what appears to be a lock issue that results in a blocked >>> process somewhere in hugetlbfs for shared maps; seemingly from an >>> interaction between hugetlb_vm_op_open and hugetlb_vmdelete_list. >>> >>> Based on some added pr_warn, we believe the following is happening: >>> When hugetlb_vmdelete_list is entered from the child process, >>> vma->vm_private_data is NULL, and hence hugetlb_vma_trylock_write does >>> not lock, since neither __vma_shareable_lock nor __vma_private_lock >>> are true. >>> >>> While hugetlb_vmdelete_list is executing, the parent process does >>> fork(), which ends up in hugetlb_vm_op_open, which in turn allocates a >>> lock for the same vma. >>> >>> Thus, when the hugetlb_vmdelete_list in the child reaches the end of >>> the function, vma->vm_private_data is now populated, and hence >>> hugetlb_vma_unlock_write tries to unlock the vma_lock, which it does >>> not hold. >> >> Thanks for your report. ->vm_private_data was introduced since the >> series [1]. So I suspect it was caused by this. But I haven't reviewed >> that at that time (actually, it is a little complex in pmd sharing >> case). I saw Miaohe had reviewed many of those. >> >> CC Miaohe, maybe he has some ideas on this. >> >> [1] https://lore.kernel.org/all/20220914221810.95771-7-mike.kravetz@oracle.com/T/#m2141e4bc30401a8ce490b1965b9bad74e7f791ff >> >> Thanks. >> >>> >>> dmesg: >>> WARNING: bad unlock balance detected! >>> 6.8.0-rc1+ #24 Not tainted >>> ------------------------------------- >>> lock/2613 is trying to release lock (&vma_lock->rw_sema) at: >>> [] hugetlb_vma_unlock_write+0x48/0x60 >>> but there are no more locks to release! > > Thanks for your report. It seems there's a race: > > CPU 1 CPU 2 > fork hugetlbfs_fallocate > dup_mmap hugetlbfs_punch_hole > i_mmap_lock_write(mapping); > vma_interval_tree_insert_after -- Child vma is visible through i_mmap tree. > i_mmap_unlock_write(mapping); > hugetlb_dup_vma_private -- Clear vma_lock outside i_mmap_rwsem! i_mmap_lock_write(mapping); > hugetlb_vmdelete_list > vma_interval_tree_foreach > hugetlb_vma_trylock_write -- Vma_lock is cleared. > tmp->vm_ops->open -- Alloc new vma_lock outside i_mmap_rwsem! > hugetlb_vma_unlock_write -- Vma_lock is assigned!!! > i_mmap_unlock_write(mapping); > > hugetlb_dup_vma_private and hugetlb_vm_op_open are called outside i_mmap_rwsem lock. So there will be another bugs behind it. > But I'm not really sure. I will take a more closed look at next week. This can be fixed by deferring vma_interval_tree_insert_after() until vma is fully initialized. But I'm not sure whether there're side effects with this patch. linux-UJMmTI:/home/linmiaohe/mm # git diff diff --git a/kernel/fork.c b/kernel/fork.c index 47ff3b35352e..2ef2711452e0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -712,21 +712,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, } else if (anon_vma_fork(tmp, mpnt)) goto fail_nomem_anon_vma_fork; vm_flags_clear(tmp, VM_LOCKED_MASK); - file = tmp->vm_file; - if (file) { - struct address_space *mapping = file->f_mapping; - - get_file(file); - i_mmap_lock_write(mapping); - if (vma_is_shared_maywrite(tmp)) - mapping_allow_writable(mapping); - flush_dcache_mmap_lock(mapping); - /* insert tmp into the share list, just after mpnt */ - vma_interval_tree_insert_after(tmp, mpnt, - &mapping->i_mmap); - flush_dcache_mmap_unlock(mapping); - i_mmap_unlock_write(mapping); - } /* * Copy/update hugetlb private vma information. @@ -747,6 +732,22 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (tmp->vm_ops && tmp->vm_ops->open) tmp->vm_ops->open(tmp); + file = tmp->vm_file; + if (file) { + struct address_space *mapping = file->f_mapping; + + get_file(file); + i_mmap_lock_write(mapping); + if (vma_is_shared_maywrite(tmp)) + mapping_allow_writable(mapping); + flush_dcache_mmap_lock(mapping); + /* insert tmp into the share list, just after mpnt. */ + vma_interval_tree_insert_after(tmp, mpnt, + &mapping->i_mmap); + flush_dcache_mmap_unlock(mapping); + i_mmap_unlock_write(mapping); + } + if (retval) { mpnt = vma_next(&vmi); goto loop_out; root@qemu:~# ./hugetlb_vma_lock Clean exit > > Thanks. >