From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD915C46CD2 for ; Sat, 27 Jan 2024 10:13:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BA776B0074; Sat, 27 Jan 2024 05:13:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 169B56B0078; Sat, 27 Jan 2024 05:13:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 05A1D6B007B; Sat, 27 Jan 2024 05:13:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EB76F6B0074 for ; Sat, 27 Jan 2024 05:13:43 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8A61B140737 for ; Sat, 27 Jan 2024 10:13:43 +0000 (UTC) X-FDA: 81724679526.26.26E6FA0 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf13.hostedemail.com (Postfix) with ESMTP id 59AC62000A for ; Sat, 27 Jan 2024 10:13:39 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706350421; a=rsa-sha256; cv=none; b=lQi+mAGemq0fMEvS8jTpDwMlj2BK5a0Jvg+6hh4ULqEa0CsHpCwrw9VB1iuLUkDe6EpVXm dEpP5eIOswjSJIO2u2CbxlGvnnT7T4TdKx7w3Mv1j4Sro5Plq1xCY/5yZR/2/WTkwL9QNO HUUCOQaxM6LG4q805g+MgMKslJXJOzU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of linmiaohe@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=linmiaohe@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706350421; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YDFqGEDlHMt94c2OQ+i0JXGUXimWvw/6HSxBWxFj1+M=; b=kZHwb9Uns2PJbmnQLGqyeTIfiPSflNZAjx1k1nVVegMfhypjzDk1wL5YHsvVNHEvZT/QPg 71F3F9lHlEOC54ZALtIpc/p+8cEI/DFoO8A3ljNkkihboAU2aZh8SfRdmHcg56Jf1VGA+N QOczF9zo3m7bQoap+YhPGH3HYDZMTEA= Received: from mail.maildlp.com (unknown [172.19.163.48]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4TMVjY6zzszXgkB; Sat, 27 Jan 2024 18:12:17 +0800 (CST) Received: from canpemm500002.china.huawei.com (unknown [7.192.104.244]) by mail.maildlp.com (Postfix) with ESMTPS id BCC9718006C; Sat, 27 Jan 2024 18:13:34 +0800 (CST) Received: from [10.173.135.154] (10.173.135.154) by canpemm500002.china.huawei.com (7.192.104.244) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 27 Jan 2024 18:13:34 +0800 Subject: Re: hugetlbfs: WARNING: bad unlock balance detected during MADV_REMOVE To: Muchun Song , Thorvald Natvig CC: Linux-MM References: <42788ABD-99AE-4AEF-B543-C0FABAFA0464@linux.dev> From: Miaohe Lin Message-ID: <4780b0e3-42e1-9099-d010-5a1793b6cbd3@huawei.com> Date: Sat, 27 Jan 2024 18:13:34 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <42788ABD-99AE-4AEF-B543-C0FABAFA0464@linux.dev> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.173.135.154] X-ClientProxiedBy: dggems704-chm.china.huawei.com (10.3.19.181) To canpemm500002.china.huawei.com (7.192.104.244) X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 59AC62000A X-Stat-Signature: 5fwq8bwnqmup33yfenczujapni64ydf8 X-HE-Tag: 1706350419-212946 X-HE-Meta: U2FsdGVkX19X7mvDyTWrJiqvcPG6pmz1de8MFGr9MEUKo7OwJNh2FlIs7rUbg9+o9X58nZwu+L1D+Z6dUopUuJVbgkqEW9DEF7iXF7hNnLshsFK2r6Fm1d0jzzwDu3nwfE0H32a/ai4Now/cw/SQZA1ZX+QkTPndhe4ftKnvZYICVLp/Ih9Z4szffyJ+yRaVB+X6pnWQKrla3YclFkM1uNijeP282w3Hw+/TMJfjEpYfDnXRM7ln6V1RRYf5yh65zTBACt7aKvrdljZFiSWgafaR35F5FMB6bFYmw5Rog7rKcngJZa5/e1cVEhOPtqH6gssdfEhig8QS1S88MYxuJ7S45zo4Xhj2zBgWTMuESVQunODZa+HmuvE7FJxHOz6gId7DgT7dRmN9r/m+NVzwcVnKKho4945yoF56I4w7UMybQpqC4TMul50UEHPY+j8wzdYjLRwtyWfoLHnxiZmqdNLvZ1Dv6Pm0vM/7Tq7nAIHR0DBvsafyebEhAaOt/1gzbV76s4dObWnvlD2jaQ+Fjpx2tTqL+hu4wVkx5fQJ3GNqQyeO2aLXVbYUyORe1o9Vhp6z+vqu0llSRtaUJoynDRQ6JdEaNvln0VYonIe8E6m/rR1z5Alt+hCAYSXur4fatBSgB7T0GhH8DoGW/8t0UF8CGN7e2PUmJ6cNho8N5k/wUAr828L4ElRbIFdkwfHzRGDZjir9jFj2DSUKjCSLSGObREuoDgBmwsEnyBdf0yRJcKgOB3lK9a3t4FT+MK4RU0/RhVckmcTw71b/6jxC9yfzl32LO2NDiEQFbIc5PIlL/3d3aWXKi9CtCvZtI9N/twH1f9PY6gmjsnOip0TCitDgHNFQWdSUoGZlH7kctmtx1v1t3PbMDY0xPGWX6QKvcoH3EA45j1f2cIpoB+OePscC1wXIcC7bN+jeHF5eoUP8WXCgBMuF9s7M3lMfOdDq3GDUf/wYb4SsD/nfEsg yG3Gw8fu tRQqSTv5Ce6f2p5UJGZI5sJ8Ix3iwhnePS2gw3kXMBwvyGqGdDoNsFidIBKpADwab77/VvDFYJI5jyFFJOxP2V7LeHUka7p+ZmCAYIA1Uk7zyXt5fTcA2XfnGu7WyBfX3O+G2EwKQt7fSNLuzh9DmWM2VN2CsgDuzn1XdDlx844sytQMOU/T55pGUgNz/jpkb4QdqkY/9nGiaygh+bKs99+WhHCnNuAoucH4Uy5fq7GNEnpVb16xvHKahKfuJ//GLI9S7IrCLYaRCMypP8+qOyaaDWsJmRL/J8Rhv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/1/26 15:50, Muchun Song wrote: > > >> On Jan 26, 2024, at 04:28, Thorvald Natvig wrote: >> >> We've found what appears to be a lock issue that results in a blocked >> process somewhere in hugetlbfs for shared maps; seemingly from an >> interaction between hugetlb_vm_op_open and hugetlb_vmdelete_list. >> >> Based on some added pr_warn, we believe the following is happening: >> When hugetlb_vmdelete_list is entered from the child process, >> vma->vm_private_data is NULL, and hence hugetlb_vma_trylock_write does >> not lock, since neither __vma_shareable_lock nor __vma_private_lock >> are true. >> >> While hugetlb_vmdelete_list is executing, the parent process does >> fork(), which ends up in hugetlb_vm_op_open, which in turn allocates a >> lock for the same vma. >> >> Thus, when the hugetlb_vmdelete_list in the child reaches the end of >> the function, vma->vm_private_data is now populated, and hence >> hugetlb_vma_unlock_write tries to unlock the vma_lock, which it does >> not hold. > > Thanks for your report. ->vm_private_data was introduced since the > series [1]. So I suspect it was caused by this. But I haven't reviewed > that at that time (actually, it is a little complex in pmd sharing > case). I saw Miaohe had reviewed many of those. > > CC Miaohe, maybe he has some ideas on this. > > [1] https://lore.kernel.org/all/20220914221810.95771-7-mike.kravetz@oracle.com/T/#m2141e4bc30401a8ce490b1965b9bad74e7f791ff > > Thanks. > >> >> dmesg: >> WARNING: bad unlock balance detected! >> 6.8.0-rc1+ #24 Not tainted >> ------------------------------------- >> lock/2613 is trying to release lock (&vma_lock->rw_sema) at: >> [] hugetlb_vma_unlock_write+0x48/0x60 >> but there are no more locks to release! Thanks for your report. It seems there's a race: CPU 1 CPU 2 fork hugetlbfs_fallocate dup_mmap hugetlbfs_punch_hole i_mmap_lock_write(mapping); vma_interval_tree_insert_after -- Child vma is visible through i_mmap tree. i_mmap_unlock_write(mapping); hugetlb_dup_vma_private -- Clear vma_lock outside i_mmap_rwsem! i_mmap_lock_write(mapping); hugetlb_vmdelete_list vma_interval_tree_foreach hugetlb_vma_trylock_write -- Vma_lock is cleared. tmp->vm_ops->open -- Alloc new vma_lock outside i_mmap_rwsem! hugetlb_vma_unlock_write -- Vma_lock is assigned!!! i_mmap_unlock_write(mapping); hugetlb_dup_vma_private and hugetlb_vm_op_open are called outside i_mmap_rwsem lock. So there will be another bugs behind it. But I'm not really sure. I will take a more closed look at next week. Thanks.