From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91701C021AA for ; Wed, 19 Feb 2025 13:34:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FF50280230; Wed, 19 Feb 2025 08:34:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B03328022F; Wed, 19 Feb 2025 08:34:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E437B280230; Wed, 19 Feb 2025 08:34:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BE54928022F for ; Wed, 19 Feb 2025 08:34:24 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1279E8191E for ; Wed, 19 Feb 2025 13:34:07 +0000 (UTC) X-FDA: 83136787734.13.CF34853 Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by imf30.hostedemail.com (Postfix) with ESMTP id 2150180014 for ; Wed, 19 Feb 2025 13:34:04 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=XBNBYaeL; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.43 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739972045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=25REnXQLxgZn05Nqwdk7rSgxrWQCeKo0uID/S1aXIUY=; b=4LCKCRI+6aN/N3KM5L2O+siIV36el9m6ystJwuniSwLpn3LGZ5giyotdV2HUaiqNsFeI8k 2K7cAJTWRkwQ0QsYu7B019NyZO6Z64mynQMLAmpEAOhC3XKEm1wh/mNUwd7qD/Oi2n9S/s 7Hydc7Jg3TiyZfVF/WinrSt+xWut5/Q= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=XBNBYaeL; spf=pass (imf30.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.208.43 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739972045; a=rsa-sha256; cv=none; b=c9TfU/l8OhRYj/n2jufoxbSLs1w6Pc67WGVYgNvNG9ryNeU7VwnVX45AVk9k6J6SX5WfBo NrZIj1Hn6g29AnJrIfXtWr2lysCxcV+gavZbU+BTgwpyfET5vwd/OtbfrEg8b8p9I7urhQ Tsm0OeS+UFfDC+C9fFQGjAzq1iieZRQ= Received: by mail-ed1-f43.google.com with SMTP id 4fb4d7f45d1cf-5ded46f323fso8895885a12.1 for ; Wed, 19 Feb 2025 05:34:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1739972043; x=1740576843; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=25REnXQLxgZn05Nqwdk7rSgxrWQCeKo0uID/S1aXIUY=; b=XBNBYaeLGsyTylTy9Ejt7MRsJPx45LnKXY5uJogbcNPh7NX3DV+xC55dghfIIROtWa Q8BNl5lRMtFXkBggidoo4hI5xu3YvAm1cYJs2cXcmRlnDv9V8S8hsCb5Ko6mvOcevN0I p2wiN1ifzNLk25/k6Q3qtGJ/fbgW5uS2+gpawFFTd+v16GqBRtwIXf/WNfJuc/il6yFT +z66A4SEgHfhUR7fFDF7XAJgVOGJGRlTtZiIi3osZkOE7KsqVzDy4DnWPFSfw4HZsD4S ELrzpa2f3HBrUY8PcpsxfB6O816wTHfQ+yIhftXPaR2t55+WP8MdV83lurJGq75Ym/Xf /u2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739972043; x=1740576843; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=25REnXQLxgZn05Nqwdk7rSgxrWQCeKo0uID/S1aXIUY=; b=ax7f6UfflqaCa3uPh6nqzqwomiZ+SINg5SRVl0a3A2h6fpX8XaXabbZoIxJk4ig2ky 0Bix1F/oJfE4Q02NsJvX+0HbEJdMuVjGgZLaiB8VenwSHgChX1huQq7gPfgOMuUFt+i7 AnS+jjXD14KCB/uW9d5v2SL+geZtrXxUmk915Tn9lFyfT3YbNDVxQyw5UCxK/cYi5kDz 7CXPd5YG8VpQ7lr9kjMcRXZYP32Nq9yJQct8A4iH/VtVaLgAqMdnumiqBiXs/SsO2YMR DTXwMHcectq9mtaTH8lj55iK6dhIUaUd5DRJ/2+3MQeYfKs1UvwcaTbyCvQdu3z7bRio z6+g== X-Forwarded-Encrypted: i=1; AJvYcCUHDM9YnQI/4MjqFQHG+OiBC1rf88pBhyIjkdQr64f/33JL2HMMz3qiDoAVht5QfP+otvj4kwCGIw==@kvack.org X-Gm-Message-State: AOJu0YwUkwn5pscFkE/rxlf4JGItbHaoTkKcKEWtFiRwWwDpauFantBT QI3k5PATCQoRUNr8i6fBO6PWVVMvyFqGcZx+4ljJO6yU9JOVMMPcU7rebsH8QUzj47mLz52y+AG Hy4QR8JS8x799e6/9ZJ67PLY/Saw= X-Gm-Gg: ASbGncuDwGIB6sFswuN/eZ9gssA6vMIn+Udgj7qVs10hMKH8MrSYgj7Pt84WN1sb9Gq fy2nQKahc459rZdfSQ9jim8LsaSWxHQzCsNKAl4fpvD1sqIyG8JOEkbqc3UwO5rh3Rcwg6Pxe X-Google-Smtp-Source: AGHT+IHOusADMstgVk/7CL7pb+x/4cTH/2aIvBkQyTGGtQrOgnoTh3jlohJ13hFDLOOohFvbSBzVzXjhmHPe5OEwsRk= X-Received: by 2002:a05:6402:5244:b0:5e0:7ff3:20c with SMTP id 4fb4d7f45d1cf-5e0895248ebmr3346441a12.17.1739972043037; Wed, 19 Feb 2025 05:34:03 -0800 (PST) MIME-Version: 1.0 References: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> In-Reply-To: <173997003868.2137198.9462617208992136056.stgit@mhiramat.tok.corp.google.com> From: Lance Yang Date: Wed, 19 Feb 2025 21:33:26 +0800 X-Gm-Features: AWEUYZlHThszyuGsJB_bdNtLWfXi6Mc4cFvllrrOIEFuOJKhgJa6EbXlQ-qZC2w Message-ID: Subject: Re: [PATCH 0/2] hung_task: Dump the blocking task stacktrace To: mhiramat@kernel.org Cc: Peter Zijlstra , Ingo Molnar , Will Deacon , Andrew Morton , Boqun Feng , Waiman Long , Joel Granados , Anna Schumaker , Kent Overstreet , Yongliang Gao , Steven Rostedt , Tomasz Figa , Sergey Senozhatsky , linux-kernel@vger.kernel.org, Linux Memory Management List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2150180014 X-Stat-Signature: 7rkkckpk13dzj7g33ayrgs88j8uda7py X-HE-Tag: 1739972044-486634 X-HE-Meta: U2FsdGVkX1/0pDrCCYeOsLk67KF5SvE0MzYLDdtKGrAKt6p+XLD39pCUoHFKvrjB9HBOnBRs4KIc2OjR5v8/KMNmKhz+rhXBPmzf5GzK+oEbLbFcFjUNPTY3mQVIwZLGsvkiIn3BH/ugoeljiT6j7RUXIGoKPLPnDmLYR7ZQPzdPMLylIdKLuZ/FI3t+tOWDUDs4gBiUj+fOCTNp4MMBO4SVIZY+aEINnMIJhFL15fd2DXy4Waf6gtot6eihaZY2Wy/ykkfPbYJcvUmps4q5s8c3y2SO8YHx0/BI4Uu54vX+BfsSSFTTSPAUQAa0Fqw3apjsh2jYTpxfTbgk1wBKBQd+QHdYLh2RqXcDYrSqHPxQphtkRW+ifC8C61uiMF/Q2JK77w/0iHVozBgMHq0Fjr50bZ3I01fZFy38CcawNiD/OwUkVGsuL/lxPzoEiU0D7cKXFmeHhQAa5m4VmpFGF4MP5Fuxm5BX6ahKSsul+msJEVR6nAMDLNgDj2pC37QI5T1FtfxpH2+ekxCJDRVX+KfofvNrknhoir7LG751fUhYam71gLPpqh5Dm31ABo25M90gA2kIks06LQ0MnONLFkDeFs0/014ijRGAaqghYxFwMexVcSSMQ/30RFsvUVTKIpCAtjhygjskXG8MFp0YWaJzxxLPUTDdTW58cUzPjmW85+EuWpKxxNuWvhr83vmp8jtHY7z6lgDmVd+LwZEjgMvjpU97OCjCML1bnlQZ/j0wGwVzlg1WJHambpEBNvGPygh2pt/TqFBzYsIjG2CP6frI6ZXvbLaQXavDNrRtRK6pazc52IEzVPqX4TmtVfyyNPMHyerjOnInKiW7bhVn/QqsQiO2clwQzTapYNywz2m+ShvUQy1oj1X4W0WN7Ze7n++kkMdtVyuuct6KHYmmZji6qFr2oM4BAcEOQWTkvQtc1Xt6HO5PHLekmIdeLTMk8b8X+WFHWaZzxslibN2 BbtUcZ6y DMZllfgZswO4mmPkyIPx22K6DVObysih4H6jKJs2n02GrukNqNgtAAuzE8G9X24b+CqhXwBNmM2mUHkb8sUz1inS60Q4HIvB5c6m1E22GGpSD93H9QucgZIg0ij3qg2x9t3KMFWEKMKn3luoUGeTMt3CC2CAwUZtBW4dFtzJd++t68CKFaB69xMjw3jcfNZKAF6sBlHBAjVikkwTbWc/oNQClai50T1O0MI61v2LthhnN9GUx45bbNCEicTC/LqHw+3AufA0FAOpmTbIXxMoRnYS9SFL2nbBrbRQXu9M/MLFNTsZyWz31PKNBqnha2X8IWOMuMcCtFm34qNZdbAfwTYrXUjX640W+zBBAh/xeQ6GicU2a84p3jz18qyyeq7/yQQUw26qM5RmexMpc2SwkSPk+nwWXqG+FjiJKm7Qmr+M1Cz0wFqfSKp5jMf3roDVKUeUff7B6oKO48qs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: CC linux-mm On Wed, Feb 19, 2025 at 9:00=E2=80=AFPM Masami Hiramatsu (Google) wrote: > > Hi, > > The hung_task detector is very useful for detecting the lockup. > However, since it only dumps the blocked (uninterruptible sleep) > processes, it is not enough to identify the root cause of that > lockup. > > For example, if a process holds a mutex and sleep an event in > interruptible state long time, the other processes will wait on > the mutex in uninterruptible state. In this case, the waiter > processes are dumped, but the blocker process is not shown > because it is sleep in interruptible state. > > This adds a feature to dump the blocker task which holds a mutex > when detecting a hung task. e.g. > > INFO: task cat:113 blocked for more than 122 seconds. > Not tainted 6.14.0-rc3-00002-g6afe972e1b9b #152 > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message= . > task:cat state:D stack:13432 pid:113 tgid:113 ppid:103 = task_flags:0x400100 flags:0x00000002 > Call Trace: > > __schedule+0x731/0x960 > ? schedule_preempt_disabled+0x54/0xa0 > schedule+0xb7/0x140 > ? __mutex_lock+0x51d/0xa50 > ? __mutex_lock+0x51d/0xa50 > schedule_preempt_disabled+0x54/0xa0 > __mutex_lock+0x51d/0xa50 > ? current_time+0x3a/0x120 > read_dummy+0x23/0x70 > full_proxy_read+0x6a/0xc0 > vfs_read+0xc2/0x340 > ? __pfx_direct_file_splice_eof+0x10/0x10 > ? do_sendfile+0x1bd/0x2e0 > ksys_read+0x76/0xe0 > do_syscall_64+0xe3/0x1c0 > ? exc_page_fault+0xa9/0x1d0 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > RIP: 0033:0x4840cd > RSP: 002b:00007ffe632b76c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd > RDX: 0000000000001000 RSI: 00007ffe632b7710 RDI: 0000000000000003 > RBP: 00007ffe632b7710 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 > R13: 000000003a8b63a0 R14: 0000000000000001 R15: ffffffffffffffff > > INFO: task cat:113 is blocked on a mutex owned by task cat:112. > task:cat state:S stack:13432 pid:112 tgid:112 ppid:103 = task_flags:0x400100 flags:0x00000002 > Call Trace: > > __schedule+0x731/0x960 > ? schedule_timeout+0xa8/0x120 > schedule+0xb7/0x140 > schedule_timeout+0xa8/0x120 > ? __pfx_process_timeout+0x10/0x10 > msleep_interruptible+0x3e/0x60 > read_dummy+0x2d/0x70 > full_proxy_read+0x6a/0xc0 > vfs_read+0xc2/0x340 > ? __pfx_direct_file_splice_eof+0x10/0x10 > ? do_sendfile+0x1bd/0x2e0 > ksys_read+0x76/0xe0 > do_syscall_64+0xe3/0x1c0 > ? exc_page_fault+0xa9/0x1d0 > entry_SYSCALL_64_after_hwframe+0x77/0x7f > RIP: 0033:0x4840cd > RSP: 002b:00007ffd69513748 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 > RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004840cd > RDX: 0000000000001000 RSI: 00007ffd69513790 RDI: 0000000000000003 > RBP: 00007ffd69513790 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000001000000 R11: 0000000000000246 R12: 0000000000001000 > R13: 0000000029d8d3a0 R14: 0000000000000001 R15: ffffffffffffffff > > > Thank you, > > --- > > Masami Hiramatsu (Google) (2): > hung_task: Show the blocker task if the task is hung on mutex > samples: Add hung_task detector mutex blocking sample > > > kernel/hung_task.c | 38 ++++++++++++++++++++ > kernel/locking/mutex-debug.c | 1 + > kernel/locking/mutex.c | 9 +++++ > kernel/locking/mutex.h | 6 +++ > samples/Kconfig | 9 +++++ > samples/Makefile | 1 + > samples/hung_task/Makefile | 2 + > samples/hung_task/hung_task_mutex.c | 66 +++++++++++++++++++++++++++++= ++++++ > 8 files changed, 132 insertions(+) > create mode 100644 samples/hung_task/Makefile > create mode 100644 samples/hung_task/hung_task_mutex.c > > -- > Masami Hiramatsu (Google)