From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DDF1CE7AA1 for ; Thu, 5 Sep 2024 22:19:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9D776B0083; Thu, 5 Sep 2024 18:19:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D4D346B0085; Thu, 5 Sep 2024 18:19:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BED0F6B0088; Thu, 5 Sep 2024 18:19:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9C8A96B0083 for ; Thu, 5 Sep 2024 18:19:29 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 0ADBE120C3A for ; Thu, 5 Sep 2024 22:19:29 +0000 (UTC) X-FDA: 82532102058.30.FA833E7 Received: from smtp134-32.sina.com.cn (smtp134-32.sina.com.cn [180.149.134.32]) by imf19.hostedemail.com (Postfix) with ESMTP id D14001A0004 for ; Thu, 5 Sep 2024 22:19:25 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of hdanton@sina.com designates 180.149.134.32 as permitted sender) smtp.mailfrom=hdanton@sina.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725574669; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RhsnrhI85DT10iEVSnAzaTsY9fBIC9XyUSjsbx6P7DI=; b=GKXPwC69CvTFqGXxu3lNgwexLFwAar/4jd5qexVLGpx7CXHV6jJTQnVz8QPcyXFwkOxnmp cpZ09fynGPq+EAHuQCebTZCfsH+zMg0iUG5kR6N3J500ZTleixXQSytNfdwkm/EttnrsUi 4ilNtLGs8xNp42JcpkOdNGIcTAMfPKY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725574669; a=rsa-sha256; cv=none; b=wdWivfqk+q3HPBH/mvyQSQmldInD9K9uFiS0I0DeW8xOOi0kckwsCWxw7EZWW8o/D+Z8a+ GxAHep2UR5RI/jO580SF2C2Spr3+YpjCp2/ofUyofthRZjSI766cFawzj99t41KTr4IFtg XMVKuG4ALZI4N31bBAueUpgs7gq33P0= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of hdanton@sina.com designates 180.149.134.32 as permitted sender) smtp.mailfrom=hdanton@sina.com; dmarc=none X-SMAIL-HELO: localhost.localdomain Received: from unknown (HELO localhost.localdomain)([116.24.11.67]) by sina.com (10.185.250.21) with ESMTP id 66DA2E65000011A1; Thu, 6 Sep 2024 06:19:19 +0800 (CST) X-Sender: hdanton@sina.com X-Auth-ID: hdanton@sina.com X-SMAIL-MID: 8518303408330 X-SMAIL-UIID: AC4A801E363B48AE88BEE6DFBA4036CA-20240906-061919-1 From: Hillf Danton To: Marcelo Tosatti Cc: Leonardo Bras , Michal Hocko , Roman Gushchin , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [RFC PATCH v1 0/4] Introduce QPW for per-cpu operations Date: Fri, 6 Sep 2024 06:19:08 +0800 Message-Id: <20240905221908.1960-1-hdanton@sina.com> In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D14001A0004 X-Stat-Signature: s8ady19x546y787ojc8zhmg6bchagpqa X-HE-Tag: 1725574765-350025 X-HE-Meta: U2FsdGVkX185tscdE1z6vaGFVKdyQTTDrpbKW29tNpfaMerS56Xj2DUb7ulQRB42Yuhri0XtYRZYD8yBgALtN8RjDWBReGgTdlq9HMAGfUEIzCtd6Y3PB2cxA6QCFKCdXoYWJxUBjNn2GNR6xA1EFWBaoLH6Q6/mpNmX2GhibglnITLlOJCbFEnEvKOaQB82IVoLIYc8pA330osdrTCDsS6ODlu+PCY7zbnbRHwHx2YLhV5L9YX+C/XxBNaARfUlyDdb6c+s/BSCubJu1ZSj3cAM58EJwMtdjiPn/Jo8vep/PreCG8NKVWtsa5rnSrUHCg9W6ZLDgQOy92BqGeHM68zFWt8ljFkxZnYT1S1KPUcrXq26KaPrsPtA7hU84EONlCUk+YUQ/GgHAEhjAzse9CyN3WE4sqo9nJwVAay2sGXU1lAaDdtnMNHp9lXwJK/dzcuFHPlzkEAkXOqwuSNEghhftWYg9QycTJa9XDMIJ3XehrQyoOgb1X7S8TwPdtaoVUbMwNppRDZ5yFWFSeL7hGp/788HlzWsjrLQjTzYNUZ57PYUfMFagGiHcG9jpPv63PVc+s95UWNwVOfd7SDCkEf8uZ4wxn+nLdSoclmH7qn14gPYxOl6SWRi+xBmR/Ps/twI/FcTAiK2fbWphc49ev+bC7fu7LU7l86FIBw1e0bNQnZ9pWmb5r0cZqZQiWyNPkVvh91tV2DQs5ZNX3P9pAnZouJ1HZ66LF6//gJHq520EtMBVQ10vgaJfqbFNLNxtxmIHN4palT+1/cq9JLU0mScxSLQil+lKuPrToo6cQpilACA4rBj6JoWzmwy92tnzKAidQN/WeOkg5y2etvHkkH0oIMOmL0Kg2pZAt/VZcalo4Ii0YpGhKNJGWGJ5TexcfU0C/OXASbZvjwLn+4hcxec2T+Njda5KcTfNj99aZ2f9SwbvfzzUyIssf28jq84oKQQIAskQyZbcNOjW1/ ydJuEKDy kTVoQ2bTJYMVc9612FdDwU3TiwblLilvW9S/3yDrPvvHixTe/2QmSK8IwT2xg5DT8y0EONlG8QWwYe6MstMtmy/CqrWeYcpnUFb/csNgpviPxQj99+IkMpYmn59wDLDSgiSy58znSCcwqAJ/JoFvbssi2R9w9GcU1tN7NakzgFaeH6mc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 23 Jul 2024 14:14:34 -0300 Marcelo Tosatti > On Sat, Jun 22, 2024 at 12:58:08AM -0300, Leonardo Bras wrote: > > The problem: > > Some places in the kernel implement a parallel programming strategy > > consisting on local_locks() for most of the work, and some rare remote > > operations are scheduled on target cpu. This keeps cache bouncing low since > > cacheline tends to be mostly local, and avoids the cost of locks in non-RT > > kernels, even though the very few remote operations will be expensive due > > to scheduling overhead. > > > > On the other hand, for RT workloads this can represent a problem: getting > > an important workload scheduled out to deal with remote requests is > > sure to introduce unexpected deadline misses. > > Another hang with a busy polling workload (kernel update hangs on > grub2-probe): > > [342431.665417] INFO: task grub2-probe:24484 blocked for more than 622 seconds. > [342431.665458] Tainted: G W X ------- --- 5.14.0-438.el9s.x86_64+rt #1 > [342431.665488] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [342431.665515] task:grub2-probe state:D stack:0 pid:24484 ppid:24455 flags:0x00004002 > [342431.665523] Call Trace: > [342431.665525] > [342431.665527] __schedule+0x22a/0x580 > [342431.665537] schedule+0x30/0x80 > [342431.665539] schedule_timeout+0x153/0x190 > [342431.665543] ? preempt_schedule_thunk+0x16/0x30 > [342431.665548] ? preempt_count_add+0x70/0xa0 > [342431.665554] __wait_for_common+0x8b/0x1c0 > [342431.665557] ? __pfx_schedule_timeout+0x10/0x10 > [342431.665560] __flush_work.isra.0+0x15b/0x220 The fresh new flush_percpu_work() is nop with CONFIG_PREEMPT_RT enabled, why are you testing it with 5.14.0-438.el9s.x86_64+rt instead of mainline? Or what are you testing? BTW the hang fails to show the unexpected deadline misses. > [342431.665565] ? __pfx_wq_barrier_func+0x10/0x10 > [342431.665570] __lru_add_drain_all+0x17d/0x220 > [342431.665576] invalidate_bdev+0x28/0x40 > [342431.665583] blkdev_common_ioctl+0x714/0xa30 > [342431.665588] ? bucket_table_alloc.isra.0+0x1/0x150 > [342431.665593] ? cp_new_stat+0xbb/0x180 > [342431.665599] blkdev_ioctl+0x112/0x270 > [342431.665603] ? security_file_ioctl+0x2f/0x50 > [342431.665609] __x64_sys_ioctl+0x87/0xc0