From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DA77C021AA for ; Wed, 19 Feb 2025 15:03:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0965280235; Wed, 19 Feb 2025 10:03:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9938028022F; Wed, 19 Feb 2025 10:03:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BDC2280235; Wed, 19 Feb 2025 10:03:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4091528022F for ; Wed, 19 Feb 2025 10:03:17 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6712E161A40 for ; Wed, 19 Feb 2025 15:02:48 +0000 (UTC) X-FDA: 83137011216.23.18010EE Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf18.hostedemail.com (Postfix) with ESMTP id E3BE01C000B for ; Wed, 19 Feb 2025 15:02:42 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BOCIU2/K"; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739977363; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mZHxmk9Ox5llPkI9w3/SQ5aztAbCrTpqRs4EJf1BNlA=; b=SHmqeply1T/Fp92yEUVMtzIE0xBoVuceyUrjzFz5qOigFMXTWlYkc2H+M+FtujZVE/BkGK BbEeNRCDdhknj0YNb2CSSzyUXr9Ji61QG7K138MkSLoExX8iMVZVZZhmTYhxUBT/2Rswmr VOCw9wDNETdHq1HtniC48kh1wdJl+7w= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BOCIU2/K"; spf=pass (imf18.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739977363; a=rsa-sha256; cv=none; b=yU6TZGWOtrCpVPrQcm9oIi3HBOzDfuaZI1f67g6ZWpy+A0bx697ufgVKirRS2de8KdTZxK gDWc84d0wBwqLmV6CHIgoDHqEmzUsjeodyWoe+0SwDoSqUQzAp0kDJL8XOZKezzZmfznrS ISpKEgPegfv3SAyPPiOkZlB10EBq19U= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-5e0452f859cso6065720a12.2 for ; Wed, 19 Feb 2025 07:02:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739977360; x=1740582160; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=mZHxmk9Ox5llPkI9w3/SQ5aztAbCrTpqRs4EJf1BNlA=; b=BOCIU2/KGhx7ZEBehj035WWVDrKcAYeYX49xTwNgfZxK7iaoxahcFP3GmDftDx4GSe g/Khm+Z61Jrm66THrn7sQFqjhwxurEcP2LxLL03n7uOPA+spf1g03fAL0pLvBkHSO+Oa 1F0k1YP0rrAVREsF5JBlB/zvsSoqvIoUGZh3YVmkwfx1t2/JthB9iNbtwYQZnMVWPMLn OI+p3bLFbG3L1kNrHYgDaHzEIUT4rkRqjP++tAQdbM8VWXAYEFDesc+yg6Yvtm5GZOda Zx8rbKLqj6A34VddI27xSML0YkZRarK8USPpTVjLUbTG7FDEToI2dsh2IGjzHkv+7W4y vP1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739977360; x=1740582160; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mZHxmk9Ox5llPkI9w3/SQ5aztAbCrTpqRs4EJf1BNlA=; b=sp5fY+Z58kR+JCmuVMTsw+NwV2AhO02e3sWUBZtcYG+0V3pG8DE5zvDBFolN+z648b kgxJtRdYJgwS0rnIEWJhw8j3JQu5PzFf18q56C5okgTxxIyx5YvxIrvQcOvIyt7VrtA2 gZfBovgOBPRUhwcu2WEsriAbWa2+JUpcsT4YmXLqG7Hzw5n4dLJ+bbpVt9zLMrM8GW7G PUZwV63El484ZfCwRfWw6nhKKoe/1eAzEqqPusVFNoWy2WPUmH1IKGijFlAtk2a/x1Al 2BU9hIGpJQp7c1ystFSD1gRoTQkZ6deMx7k6NNr0wmSdQkIuQ2D8pDnHuEyIo2FkrnLf 4Jkg== X-Forwarded-Encrypted: i=1; AJvYcCVDkEKbpFbrByuYa458627wubnRxfzQMXXY/azMdc+P6w9fxRjF4wj9TtLURlLDPkM1oGy3igOxqQ==@kvack.org X-Gm-Message-State: AOJu0YxRLyVd6VbtNelDMvWuUnzGdDa0Jdh81GpI7siX4LxFH3W8kcDg IkYa5F4PwU98AuXibvFz9GkQrmWNkzOZDoV03tC6kfSrFMZNxGA2b60xrfvp+f6OJIs8JiydQnh 31x9YIahNQmkSXdS9e6qb7/juvCE= X-Gm-Gg: ASbGncvmB4T2wKvzIu450StlK5Lr+APX4IttxDYZbJTDWi9L1gVt+1ScT5kQ/WHiysJ r1wdxf2tI/WSEkPft94QuPbL2mfY+0QgJliImThDG9IuayVdGrXQuElRC84BfJUxlhsfecZxI X-Google-Smtp-Source: AGHT+IHq4IO2JlnOqG7659shJfdrV3OkM/TNkV5y/AM1wIU+x9ko/zSrRY+nfZq394aLcLTbFpCDw4LEKK3G94anSiI= X-Received: by 2002:a05:6402:26cd:b0:5e0:6770:2b6a with SMTP id 4fb4d7f45d1cf-5e089d316demr3345670a12.23.1739977358654; Wed, 19 Feb 2025 07:02:38 -0800 (PST) MIME-Version: 1.0 References: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> In-Reply-To: From: Lance Yang Date: Wed, 19 Feb 2025 23:02:01 +0800 X-Gm-Features: AWEUYZkdZAHl4TKYpUipGVKNVSOrlB6SVRk-Qdb4IJkgGdtuOP47nBGCQssv9Xw Message-ID: Subject: Re: [PATCH 0/2] hung_task: Dump the blocking task stacktrace To: mhiramat@kernel.org Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Andrew Morton , Boqun Feng , Waiman Long , Joel Granados , Anna Schumaker , Kent Overstreet , Yongliang Gao , Steven Rostedt , Tomasz Figa , Sergey Senozhatsky , linux-kernel@vger.kernel.org, Linux Memory Management List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: E3BE01C000B X-Stat-Signature: h1ariusxmzzrs49r3x3ezio34qj665n8 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739977362-265264 X-HE-Meta: U2FsdGVkX1/HxWCC5BDEA/0QdVTUG1vG5wSoyzVGhqkbc1WTH6uFp5mDvLsieMXNeILXCw9ZOrDAC51D4Z3mfte0ReNYZywSnk1uUsCphFoyTNUsN7eOgphKWyMqx3lTl2Ke6Luyl0Fl24LGo1YeYlfj0Gz5+k+hwZ9r3iZFPp8ri32InlbnLHpRBEASw9a59bPqPs8qrGnIcWq9+ECV90BXJHqmu3DOFhhd8iyAXY5gttG9dh/LjGJbqNAm4ibEC4hla7gtSZrxPk5CYvGKjBP3adaJz/0zP5LSwWrQk5LS3tvY5TZYQjqLNVJCyUEB3XjzoJNdwmtH3swW0YrG3z99B8XQ95j2ZEn2JooeXb0UiwKmPxez7aRIpiCyW38U5d7pTFYrxQCAkUhxkgqnIbcdYCGBWAv/WInKg9R2Q5nHVKj5H9sJ6Gb23EPAPTu0FeM/rvq5kpYHALLoqqc2PXGLQUk6QjiEWev9u7VZxDZT8NPqzdYXxHq3g15soM7UPaNq58vcM9BBVLASwU4ErPI+LqShejvPFSJPli0aF1wYwOGbDO++pBu70fEZ9XRL2ahWPS7r0sfx7/ZCOeIkSAGXljwkt6B/rQqmcAnZM9a3Wk2Zo8FXAlkmAoS7oUTKOYSNxOkwHLDt2lF4MJCLWLQZ4JRLSrNCTcOhKJ8z/6OSsbIj//PPibRZkPMluXQd0lycxdDNUAhXUIXztq27pB0DQVwSWGXI/d4wJy91bOg1LixVUidHmTx90djBwFr+ss7hur9nTHc1N3lPq41AlLfHGtUtx/fZ5yGEEEqv4s4/rmUz+/PyauVHw3jO8c1FXBtEizyVZP6d69yZ3A+/Q1sv2J1eO06HOi0/XhhYGZJoijecl9G7m/xUgFakYSuUQClcork22zI65ILD86qn+l2JMFsa35K5maVBBbioaTERKrEYMh+TnaPBCwaxHTJNkn8ZPv2ch+AQ7ZMTdzs xOaOMpmr 2PY8RTcnExL53d1C8PLFWhKKWrEA7vzQu0cCJ0AfVUPXuXTDMxZkNfG14Wn3R9tX4PhmOHri6utGT5W0Nlj7DKZUOyFwggjABKX0he7iXDCN1nw9cg0sdj8o8i4BYvhdFjxU/ohxmniChFCVL805L2EWo8OUOTsxf22yB4zFL1OSurH8NGiR50AnTbUV5cvsQIRT3Fa8jLUrn1XNAZ5Le30AwXrBnDh8DiBgwVlb4hXKqzG0l2Csrg5meDoo/SnIuPyiIwj51EAP3RaI7lfOrGMpqvF6fxJfoR3jNNwiINv7R0KiLULnPyg3XRjE4I6Hgm1lHbi8SZLxD+5KfPv0dkFdZbDnxaiL7VvvogNOZyLiw3E8Bveo4QKnDg4wOki2Api+wCc6yYS9JQUfLMAaewkSBMdwZPzad/lGWTbsoL9zdtSYW1ZHRjXDuRvpp4yrbJvRJR1fYX9UWjjiP3rk83rJDYZc4L/H89iDUE92PuSMg/FE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Feb 19, 2025 at 9:33=E2=80=AFPM Lance Yang wr= ote: > > CC linux-mm > > On Wed, Feb 19, 2025 at 9:00=E2=80=AFPM Masami Hiramatsu (Google) > wrote: > > > > Hi, > > > > The hung_task detector is very useful for detecting the lockup. > > However, since it only dumps the blocked (uninterruptible sleep) > > processes, it is not enough to identify the root cause of that > > lockup. > > > > For example, if a process holds a mutex and sleep an event in > > interruptible state long time, the other processes will wait on > > the mutex in uninterruptible state. In this case, the waiter > > processes are dumped, but the blocker process is not shown > > because it is sleep in interruptible state. Cool! I just ran into something similar today, but with rwsem. In that case, the blocked process was locked up, and we could not identify the root cause either ;( Thanks, Lance > > > > This adds a feature to dump the blocker task which holds a mutex > > when detecting a hung task. e.g. > > > > INFO: task cat:113 blocked for more than 122 seconds. > > Not tainted 6.14.0-rc3-00002-g6afe972e1b9b #152 > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this messa= ge. > > task:cat state:D stack:13432 pid:113 tgid:113 ppid:103= task_flags:0x400100 flags:0x00000002 > > Call Trace: > > > > __schedule+0x731/0x960 > > ? schedule_preempt_disabled+0x54/0xa0 > > schedule+0xb7/0x140 > > ? __mutex_lock+0x51d/0xa50 > > ? __mutex_lock+0x51d/0xa50 > > schedule_preempt_disabled+0x54/0xa0 > > __mutex_lock+0x51d/0xa50 > > ? current_time+0x3a/0x120 > > read_dummy+0x23/0x70 > > full_proxy_read+0x6a/0xc0 > > vfs_read+0xc2/0x340 > > ? __pfx_direct_file_splice_eof+0x10/0x10 > > ? do_sendfile+0x1bd/0x2e0 > > ksys_read+0x76/0xe0 > > do_syscall_64+0xe3/0x1c0 > > ? exc_page_fault+0xa9/0x1d0 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > RIP: 0033:0x4840cd > > RSP: 002b:00007ffe632b76c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 > > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd > > RDX: 0000000000001000 RSI: 00007ffe632b7710 RDI: 0000000000000003 > > RBP: 00007ffe632b7710 R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 > > R13: 000000003a8b63a0 R14: 0000000000000001 R15: ffffffffffffffff > > > > INFO: task cat:113 is blocked on a mutex owned by task cat:112. > > task:cat state:S stack:13432 pid:112 tgid:112 ppid:103= task_flags:0x400100 flags:0x00000002 > > Call Trace: > > > > __schedule+0x731/0x960 > > ? schedule_timeout+0xa8/0x120 > > schedule+0xb7/0x140 > > schedule_timeout+0xa8/0x120 > > ? __pfx_process_timeout+0x10/0x10 > > msleep_interruptible+0x3e/0x60 > > read_dummy+0x2d/0x70 > > full_proxy_read+0x6a/0xc0 > > vfs_read+0xc2/0x340 > > ? __pfx_direct_file_splice_eof+0x10/0x10 > > ? do_sendfile+0x1bd/0x2e0 > > ksys_read+0x76/0xe0 > > do_syscall_64+0xe3/0x1c0 > > ? exc_page_fault+0xa9/0x1d0 > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > RIP: 0033:0x4840cd > > RSP: 002b:00007ffd69513748 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 > > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd > > RDX: 0000000000001000 RSI: 00007ffd69513790 RDI: 0000000000000003 > > RBP: 00007ffd69513790 R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 > > R13: 0000000029d8d3a0 R14: 0000000000000001 R15: ffffffffffffffff > > > > > > Thank you, > > > > --- > > > > Masami Hiramatsu (Google) (2): > > hung_task: Show the blocker task if the task is hung on mutex > > samples: Add hung_task detector mutex blocking sample > > > > > > kernel/hung_task.c | 38 ++++++++++++++++++++ > > kernel/locking/mutex-debug.c | 1 + > > kernel/locking/mutex.c | 9 +++++ > > kernel/locking/mutex.h | 6 +++ > > samples/Kconfig | 9 +++++ > > samples/Makefile | 1 + > > samples/hung_task/Makefile | 2 + > > samples/hung_task/hung_task_mutex.c | 66 +++++++++++++++++++++++++++= ++++++++ > > 8 files changed, 132 insertions(+) > > create mode 100644 samples/hung_task/Makefile > > create mode 100644 samples/hung_task/hung_task_mutex.c > > > > -- > > Masami Hiramatsu (Google)