From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F162E77170 for ; Thu, 5 Dec 2024 15:22:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E677E6B00F4; Thu, 5 Dec 2024 10:19:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B24AA6B0133; Thu, 5 Dec 2024 10:19:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BD066B00A9; Thu, 5 Dec 2024 10:19:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 20A536B007B for ; Thu, 24 Oct 2024 02:10:01 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CE3A64114B for ; Thu, 24 Oct 2024 06:09:50 +0000 (UTC) X-FDA: 82707469614.26.F4513D3 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf22.hostedemail.com (Postfix) with ESMTP id AFB36C0012 for ; Thu, 24 Oct 2024 06:09:36 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ILJ3Tq3M; spf=pass (imf22.hostedemail.com: domain of jimzhao.ai@gmail.com designates 209.85.215.193 as permitted sender) smtp.mailfrom=jimzhao.ai@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729750045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3YtbyEaMsa4GAbWnsy2Tbe/1HjhrxOqJDOx9rhvgg2o=; b=nHZtv9wajwfLKP/z6d8MpEF7coitKIekpdeykTiYLEQNhj0kSQeJcamAm9FsEmqff3XGM0 ngy0r0Je4DiwtXOfikpb1UALObhHIFODbrYVfdEmWIXB+lQTmVzoBMddqAOPdVKBhg7XMB 5/YXFX1YdAISC8qVA9as/SoRn62/fk8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729750045; a=rsa-sha256; cv=none; b=AsVrPSp19xhB9ZRVD7LWSvjZRHCPRreqH35yf21Y7PQjuK1yXF1orFdtL0Pxmgu111j14E 3N2PsvMAhYHrNnFOZ+VxSP8OR2OqaiDuZvkcFtUUvDGQjbB9b3ycx647ycdv9trafWjL+5 /g/JEXyG3tBaeORyoBmmWIQ6DC999rU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ILJ3Tq3M; spf=pass (imf22.hostedemail.com: domain of jimzhao.ai@gmail.com designates 209.85.215.193 as permitted sender) smtp.mailfrom=jimzhao.ai@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f193.google.com with SMTP id 41be03b00d2f7-7ea8ecacf16so335687a12.1 for ; Wed, 23 Oct 2024 23:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729750197; x=1730354997; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3YtbyEaMsa4GAbWnsy2Tbe/1HjhrxOqJDOx9rhvgg2o=; b=ILJ3Tq3MGMBZSXvL8Duno5dPYCp7K5MiXBltDjtmLb+I3i6HDvNHXGO2DrkyAbm3IQ 2lyvpD32ScA8pPQx/1CKO/hwLZIhHmH6SYUrqnYyY3uVmMhxvEhQDYSitku41QgjJkur 4NntzhHARtRPUmtILDdv59x1MLuOokYJQyt8tfojHx6AwuIQ/LrMIt0MgRwGmNlDuqxN 3Mii3q8tSZjN8hF/67gGryaaPWFbG4XzwSg+Zzen16Cs5EuLYMuxUtU93EkM6i4XETN1 OwHI+sYUp9HPGDyZRV5t5RevjAtivDpRzDVMoh+KKB0JhzBLN0AzmIFCmeAUrdvR2Xur hvug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729750197; x=1730354997; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3YtbyEaMsa4GAbWnsy2Tbe/1HjhrxOqJDOx9rhvgg2o=; b=RXyPC/2L12KhUQaCez5oeTgE2z6n6JpCLK7XA334qpKHUGErAYfs4/F9WRpACBN61N Nb3nrq0PQEfmqyn5fVt70TNcX3CRyIlYTAnWXoIgK6D4R8p8qFvfJrr74BzpZgH+QhFg A6MUrDOkJ/1PlgEgz988wfVyMRNS1efXF3k9KdHGGtQynaR3ITeRO7+cc0lCyiWxG10Z 2VEE/bWLWJMD+ALSPMOA+vLKM7sSl+nD6h3Qc3gTzH2fuFplsS5ZPqRfBbidGAZeN/jI ivJGMZ2GL+JK39LMqClXJ0LjwOJ+R+CXL9HBbFq+njMXMYLEfNhbZPq94VjIROtgrjDv QF+Q== X-Forwarded-Encrypted: i=1; AJvYcCW4+klVXvSLKGFBSs5cAqnOynTskwfa9rp6w6nkZPWOicU37cqeqBRwyMLaa1VjbcKWgk+gasMjnA==@kvack.org X-Gm-Message-State: AOJu0YxpBqMMUdOdKvjvSK1OTksyMUnKL7xqMM3LlEN5Zg7RHAEYykl3 rqkn36CauH9SDxgwsf0GoHJXYlOzVoJFtX1kyA8bz7QCnE930F8YdMkQ9UMYglc6ng== X-Google-Smtp-Source: AGHT+IEb4DbHvwbycDEptOsdZXO6q1mXwibi0nQJsNWuqbQ3vGbu82WIslgzpjLzZdW8RPJK3z2x2A== X-Received: by 2002:a05:6a21:174d:b0:1cf:38cf:df92 with SMTP id adf61e73a8af0-1d978b2e2d8mr6172207637.30.1729750197500; Wed, 23 Oct 2024 23:09:57 -0700 (PDT) Received: from localhost.localdomain ([43.154.34.99]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71ec132ffb8sm7329074b3a.50.2024.10.23.23.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Oct 2024 23:09:57 -0700 (PDT) From: Jim Zhao To: akpm@linux-foundation.org Cc: jimzhao.ai@gmail.com, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org Subject: Re: [PATCH] mm/page-writeback: Raise wb_thresh to prevent write blocking with strictlimit Date: Thu, 24 Oct 2024 14:09:54 +0800 Message-Id: <20241024060954.443574-1-jimzhao.ai@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241023162447.2bf480b4ce590fdeb8b6c52d@linux-foundation.org> References: <20241023162447.2bf480b4ce590fdeb8b6c52d@linux-foundation.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AFB36C0012 X-Stat-Signature: up6biqffs3zpyn48iio8oudb68rc5emi X-HE-Tag: 1729750176-326931 X-HE-Meta: U2FsdGVkX1+vwCI3xCbT4mnyHgsSzoycvx4HYxRWB10qFCGNKS+9TViQKZozDV9p2tnYfEf6fCr488QyxMUEX6Bi8ZVIE6ZuN8Pi7Juc969f4CB4HYip9bQoKAtYu8pix47d0wDbRmqDQ0U72qHaxqTtsOs8w0gjzuBoAl/NCvfbLr2Fn40Zy11nMSgWe4jt59NfEepHDTURTCvTjCUILVeDUUzs85aXO6YN7ssUBomh6mHCoR6Q9bWQFsbDf3SFjUbQIoqk2oaNjvdcDdM/bc5qMxCon8zYv0OS9Nhbeb5KaDUNYhd1bNGWQMEvC4bK3evQONaRHdFsqhrBTEuPnNU1cjwKpC91w9wgCJ+8BoqHk8H68EzTp8xSTob56RTJgWt9aHrXBzkeeExM1kk2y2OZIQal+SQGtZg1Qd+GLx3cfzx3tJTKtwltw45pTUD04LkqP4Lm66fH6mZesSHQr11naVzbstA2ewMZ2mYnYCVTjjQXzXsS+bCqMdnGDKm53qVvggcuER1Zj1Zd6E36cG0V67lhp9EsTuQTzIi1zhTtXr1CPs+cuepJXMapwMnnDwVmYPITp5BJJPtV/nx/QqkLz5z1oAdT/awb96c1eGYbLiKTIuc1uXv0wcydz0jT8fsFLGBMOaB5gCoIAWdYYhNV76JM79O3pmmhIhsRQ076LkERkjfm9NyN7bNtLOM7pR7ZIwCFuQlVdpgxDwK1hGfJhAo2X16iiK0RNT6j/j/bT7JbhxyJyItSOFDvI6UK8KSAN4fjzaF7EyknPcUNLMS1U4en8OClzNNV+26XJXNk6fWjbas+dbR1lWrpKcctRgWEdgRPUAMqdTrycw9GT8IP4ftCqLnavZW7afyoQgWEmrmGHgUhrA5s2eDibuKT9CebIqkXlfo2JnaqQpGkpiP3Okoxf8FStwM1G92ozUKZESZf/5HVL4TAKU2acSjs4M3AOcHBAyynkajUhyq 04wCwZwe ILJTjBUWSRoidBg9YO7GqIO8lOrm5nF1zbVOQqWhCDs7WSepk0hCJ6SeaF7OtjH7zH/8mSQUyCdL0eRNbZhp/XazMWiudGmZTVa97n1tmDMKhuDJiFEm77brfdzx6h0iPx/NjKowLRWNat68Pgv40FXvqBzcyj1k6Zb8NBd/q4GKvZyvcthXsZvca5EqZzhkmzxMXphkjfj/Ji3F9XFq8BdNtOqd0qh1kf5MPjXrmORaXuXZN7lWx0ZX8YJr9PvWUTHiJY1e2bPPMugKyd6BmaRoRBjLlGy3AZfZIoUhRG5S55OOX8ImLasgdwNP/p29hwlJjRpD2t3Xs9/vA+ywqk8KqQ0MM9WomWyuO4R8ZSS53izUrRk4QpToVeD25JkBfTQJKrIPP7gqZiFJjJN47m8aOrVAwetM0QSSbKzU6l9dXCq9nO1+B3TcPqnf+PrFLX9QRafLESL0yjlVSnVu6GiER4S292SkxIv2ZEgatJqvI4dxzoOa52FF+jzk/8wU01B0gh9fyoSjv9pZGBVpW2kQg7I1c83LjxDb5+uNFLUvCIdIU5LPLMOBPNDPMzAU4ILvfCQl0C5tXKBN3aE0D7Ca5vuo0CLtCvzFr/nv96pLkZSl6ts6977lLOOrf4p+XQfXdohr/h/fRFeY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.258385, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > On Wed, 23 Oct 2024 18:00:32 +0800 Jim Zhao wrote: > > With the strictlimit flag, wb_thresh acts as a hard limit in > > balance_dirty_pages() and wb_position_ratio(). When device write > > operations are inactive, wb_thresh can drop to 0, causing writes to > > be blocked. The issue occasionally occurs in fuse fs, particularly > > with network backends, the write thread is blocked frequently during > > a period. To address it, this patch raises the minimum wb_thresh to a > > controllable level, similar to the non-strictlimit case. > Please tell us more about the userspace-visible effects of this. It > *sounds* like a serious (but occasional) problem, but that is unclear. > And, very much relatedly, do you feel this fix is needed in earlier > (-stable) kernels? The problem exists in two scenarios: 1. FUSE Write Transition from Inactive to Active sometimes, active writes require several pauses to ramp up to the appropriate wb_thresh. As shown in the trace below, both bdi_setpoint and task_ratelimit are 0, means wb_thresh is 0. The dd process pauses multiple times before reaching a normal state. dd-1206590 [003] .... 62988.324049: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259360 dirty=454 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=18716 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=4 period=4 think=0 cgroup_ino=1 dd-1206590 [003] .... 62988.332063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259453 dirty=454 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 dd-1206590 [003] .... 62988.340064: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259526 dirty=454 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 dd-1206590 [003] .... 62988.348061: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=489 bdi_setpoint=0 bdi_dirty=35 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 dd-1206590 [003] .... 62988.356063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=490 bdi_setpoint=0 bdi_dirty=36 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 ... 2. FUSE with Unstable Network Backends and Occasional Writes Not easy to reproduce, but when it occurs in this scenario, it causes the write thread to experience more pauses and longer durations. Currently, some code is in place to improve this situation, but seems insufficient: if (dtc->wb_dirty < 8) { // ... } So the patch raise min wb_thresh to keep the occasional writes won't be blocked and active writes can rampup the threshold quickly. -- Thanks, Jim Zhao