From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFF31FC618C for ; Sun, 15 Sep 2024 00:31:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 322CF6B007B; Sat, 14 Sep 2024 20:31:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D2846B0082; Sat, 14 Sep 2024 20:31:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C1776B0083; Sat, 14 Sep 2024 20:31:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F37756B007B for ; Sat, 14 Sep 2024 20:31:18 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 85E8FA0FF7 for ; Sun, 15 Sep 2024 00:31:18 +0000 (UTC) X-FDA: 82565093436.06.D594267 Received: from mail78-58.sinamail.sina.com.cn (mail78-58.sinamail.sina.com.cn [219.142.78.58]) by imf18.hostedemail.com (Postfix) with ESMTP id 42D1E1C0003 for ; Sun, 15 Sep 2024 00:31:14 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of hdanton@sina.com designates 219.142.78.58 as permitted sender) smtp.mailfrom=hdanton@sina.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726360222; a=rsa-sha256; cv=none; b=Bxn5u8Xcj8JocglACLD6sWC5ALtabX2K08wQwKWo6xgOrp7fH+1xjhgrvBEPZKStMIloz8 RLFVnxBxPUML/Qh0iwgLJdJKveeCqpfC97UCLCqOcqS7X5VbX7qO5OpRuio/ZVNVIbYXsX n8bpKdMUKTrKn2v5Wm/ZSQU4MeZBBAI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of hdanton@sina.com designates 219.142.78.58 as permitted sender) smtp.mailfrom=hdanton@sina.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726360222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EVsDuXvGWSokyBDPYbvkDwP5NDFZeEGbAY18gcm4Mys=; b=zob4qDckQGtbyItFtKpjw4b9Bx4ZiQCN0bZj0DjagWUI1YVgM8KF/dEviYeN0/tQyLAW6e FmOQymRmQdZMbj9w10tWkgVJ922WiAD/ge/+1yCkBax8qxwO/hJH7mUP6O+3axytedsxTT B39YsGqsYjt1ZJp7eIS+c0FxFq6VmvA= X-SMAIL-HELO: localhost.localdomain Received: from unknown (HELO localhost.localdomain)([113.118.64.223]) by sina.com (10.185.250.24) with ESMTP id 66E62ACC00002FE8; Sun, 15 Sep 2024 08:31:10 +0800 (CST) X-Sender: hdanton@sina.com X-Auth-ID: hdanton@sina.com X-SMAIL-MID: 63931610748365 X-SMAIL-UIID: C4817436ADED4F4E96A5E582EAA9C455-20240915-083110-1 From: Hillf Danton To: Marcelo Tosatti Cc: Leonardo Bras , Michal Hocko , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH v1 0/4] Introduce QPW for per-cpu operations Date: Sun, 15 Sep 2024 08:30:58 +0800 Message-Id: <20240915003058.478-1-hdanton@sina.com> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: zpk9nhbpcesr874n3xpee5tc3h71bbgk X-Rspamd-Queue-Id: 42D1E1C0003 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1726360274-660504 X-HE-Meta: U2FsdGVkX1/XTNoOF2gsGuM+eOurzPsV54SODL8uJ9U8gFluo1sW4SzDu2Lb5Gl0VGYp+bMUvO/kIlxPxeQJdgZVl6OH9cOhhtEz8GwytBsvNtC8AUB0iX9PfSxEyBfmiAbu2P54+D23ZvCMf7uKYL9WUOOUv/L2Kjy5MDKEH0rdGB3npj78vYV4QK1oPE0Mjfr7SkBJ9ItpCWT5lpn5oWXVxBNZ7eH90g47zhohaeivg/mMg6TSA48tv10JZZbJS+6O8aeEkLu/9W8iaHgX1TAJ5/LTK/DAmdtxvg6fbaTXsefVrHWnzOpUsn1WEHRwifAy6OS7xJ9ovm4HOionTBH+f+0SJfIcG4ybLzpqSggaur/NrZOrNrwjgcK4x1xEg8pUjomKGnVbaKLqSkyNUtGVc4P7urN0gLd53EBUihhPQvE0Rgcmosqp82jcRIF+VzJsSLStWziUuCgJ4uNJwMYzPbTkUBLrcAbfpFF3yEoqsW/iVgtSlAnpZCGqNvCsowVgzaTptNUcbiKVrfutc8SRN5j440pvRGvwT3TjEyPKMW2CzcJiAeBnYg1RUwdmVaKH8M0C88EFOrjC9RtOd1b6nUlT3+nuTZ0P/oj+SCrla3qrVnGR/Tws2OduzHUXhWHQ6tBLho6E9/pIYwiJkGo2/yHskrdCJpYsbyTqkKvqzyqkstz0mR1ikBO2AbpD5KFbXAF8McvGKmsZiEPqgxES33Ci5ZQO4RXh3nftPBsFotwWWpYRRnZkoIqmDCRkBiuPj4/vZuUy7C6AotDBBVrtlB3jMDYUiT/EEDn9sy+5WVNrk4xARnLIt7OF2lF0iyqLHIqDAvyiu/4jN8W62GZ7g6mcktckqGDzBVj7PTGg9dWA4hhff4pEsRFd9WC4Zg5O1ZeMEf/tIMqeDz99SoBg829EF1vZIvZquyGsrZYc75yxSuYyhv5COOwmuiRmuDvyAH7jjL3NAtmsNqt vCDo1yt4 18aSdCrapNh648SgEXWbE08PPUOMlCn2YamwfLIvG2da09btwVtm1kr/PG6HvtQcgo5dtwPU4qDm3AhZvz5KfoeARs14npcyy96YEizpaTnjWNTUZHY9iUDGoms2Qxxmj/h+ULE1XLO/1+hM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, 11 Sep 2024 00:04:46 -0300 Marcelo Tosatti > On Fri, Sep 06, 2024 at 06:19:08AM +0800, Hillf Danton wrote: > > On Tue, 23 Jul 2024 14:14:34 -0300 Marcelo Tosatti > > > On Sat, Jun 22, 2024 at 12:58:08AM -0300, Leonardo Bras wrote: > > > > The problem: > > > > Some places in the kernel implement a parallel programming strategy > > > > consisting on local_locks() for most of the work, and some rare remote > > > > operations are scheduled on target cpu. This keeps cache bouncing low since > > > > cacheline tends to be mostly local, and avoids the cost of locks in non-RT > > > > kernels, even though the very few remote operations will be expensive due > > > > to scheduling overhead. > > > > > > > > On the other hand, for RT workloads this can represent a problem: getting > > > > an important workload scheduled out to deal with remote requests is > > > > sure to introduce unexpected deadline misses. > > > > > > Another hang with a busy polling workload (kernel update hangs on > > > grub2-probe): > > > > > > [342431.665417] INFO: task grub2-probe:24484 blocked for more than 622 seconds. > > > [342431.665458] Tainted: G W X ------- --- 5.14.0-438.el9s.x86_64+rt #1 > > > [342431.665488] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > > [342431.665515] task:grub2-probe state:D stack:0 pid:24484 ppid:24455 flags:0x00004002 > > > [342431.665523] Call Trace: > > > [342431.665525] > > > [342431.665527] __schedule+0x22a/0x580 > > > [342431.665537] schedule+0x30/0x80 > > > [342431.665539] schedule_timeout+0x153/0x190 > > > [342431.665543] ? preempt_schedule_thunk+0x16/0x30 > > > [342431.665548] ? preempt_count_add+0x70/0xa0 > > > [342431.665554] __wait_for_common+0x8b/0x1c0 > > > [342431.665557] ? __pfx_schedule_timeout+0x10/0x10 > > > [342431.665560] __flush_work.isra.0+0x15b/0x220 > > > > The fresh new flush_percpu_work() is nop with CONFIG_PREEMPT_RT enabled, why > > are you testing it with 5.14.0-438.el9s.x86_64+rt instead of mainline? Or what > > are you testing? > > I am demonstrating a type of bug that can happen without Leo's patch. > > > BTW the hang fails to show the unexpected deadline misses. > > Yes, because in this case the realtime app with FIFO priority never > stops running, therefore grub2-probe hangs and is unable to execute: > Thanks, I see why it is a type of bug that can happen without Leo's patch. Because linux kernel is never the pill to kill all pains in the field, I prefer to think instead it represents no real idea of 5.14-xxx-rt at product designing stage - what is kernel reaction to 600s cpu hog for instance?. More interesting, what would you comment if task hang is replaced with oom? Given locality cut by this patchset, lock contention follows up and opens the window for priority inversion, right? > > > [342431.665417] INFO: task grub2-probe:24484 blocked for more than 622 seconds > > > > > [342431.665565] ? __pfx_wq_barrier_func+0x10/0x10 > > > [342431.665570] __lru_add_drain_all+0x17d/0x220 > > > [342431.665576] invalidate_bdev+0x28/0x40 > > > [342431.665583] blkdev_common_ioctl+0x714/0xa30 > > > [342431.665588] ? bucket_table_alloc.isra.0+0x1/0x150 > > > [342431.665593] ? cp_new_stat+0xbb/0x180 > > > [342431.665599] blkdev_ioctl+0x112/0x270 > > > [342431.665603] ? security_file_ioctl+0x2f/0x50 > > > [342431.665609] __x64_sys_ioctl+0x87/0xc0