From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC1BBE7716D for ; Thu, 5 Dec 2024 15:22:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FBBE6B0133; Thu, 5 Dec 2024 10:19:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5A156B012A; Thu, 5 Dec 2024 10:19:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13A646B00C4; Thu, 5 Dec 2024 10:19:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 57EF06B0088 for ; Thu, 24 Oct 2024 03:29:26 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A039CA111D for ; Thu, 24 Oct 2024 07:28:52 +0000 (UTC) X-FDA: 82707669954.18.88902BD Received: from mail-oo1-f68.google.com (mail-oo1-f68.google.com [209.85.161.68]) by imf28.hostedemail.com (Postfix) with ESMTP id BD77BC0009 for ; Thu, 24 Oct 2024 07:29:05 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LKPRFreM; spf=pass (imf28.hostedemail.com: domain of jimzhao.ai@gmail.com designates 209.85.161.68 as permitted sender) smtp.mailfrom=jimzhao.ai@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729754912; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pYzWy6kio/5TGHmC4KnU0HU3SKktd6/yeFNR0szREPo=; b=vU5TgK9aATD3tzs2l8vmSNs4Hy28RLIz5p+o1rvk/gzVYC8hcI7/NvsALk5LeZZUuyx8Nl m8xkSHXkVZScnv17hP02pfj2N2L4+w4ElRonbgE+Wng5AKhK+C24qFD6CONiXFLJv6fWE4 426M0GmTrT9wahH4a/4jQ4AOMRroBGQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LKPRFreM; spf=pass (imf28.hostedemail.com: domain of jimzhao.ai@gmail.com designates 209.85.161.68 as permitted sender) smtp.mailfrom=jimzhao.ai@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729754912; a=rsa-sha256; cv=none; b=HGklnTPxl4vvrL+n6sXK8Dwu9iypdwRrbkzYusT6LQC73GI/wZV53/J9s3BSw9aFDlXs4w isDNmpnWoct+BeyqJBVRnAvkD9BQpNKAK0nXG1S2pfNyPcbT5juCbNdmLGzD5ZeTg7SaOC KWxbjJBaWgp/46vTDXeLUuWIHLeKtWI= Received: by mail-oo1-f68.google.com with SMTP id 006d021491bc7-5ebc27fdc30so269198eaf.2 for ; Thu, 24 Oct 2024 00:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729754963; x=1730359763; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pYzWy6kio/5TGHmC4KnU0HU3SKktd6/yeFNR0szREPo=; b=LKPRFreMWyamqlpZISYFSFrMOLIXzRsBYWHpC9axSXGiPI0oe/Djvz3Tn5GuqiyvHX l8YQQaZZRVh7Pb4CCwO1VOwZCjevwpH3+fvDqTwJdi6IKcQ4xZqLNcnwpsPYLl8ajLko o3KpEbhhyyQlCEuF3hSUgusLWxpvJuruxgslNtz7HPmw++a4hiMwNIhXO55X9RlecBfg tv3acmzYn9N8NG5QYV7cuIYB5IcyH7hd6xr8cfZCVltH/Bm9zLtKAd2deh8x85yK27n2 Q8cCTk8djyF/EQ1Ciw0Oj6e64H4G+wZLba0DABLyP/VkpvxIaRnuGGdSWymPeI7F3s77 aAzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729754963; x=1730359763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pYzWy6kio/5TGHmC4KnU0HU3SKktd6/yeFNR0szREPo=; b=Ju3dXXSFtI5SwgHX6zQA6IwfOX1LS2ew8x3cxR5TNfZq9dnsEuvBAE+VQGgyUbai8g hEKrQ41dHaOQyM+sD2B+VKbPj4w6KKPMukStEFUSrZrBLeoxIQ5g6P968GNzZJfKecKJ BtpU6Fg3Zi+d76Wxzu1eY0sQ7dawa/o7NClnjJeJ55m9U6ZHy55BfJTybPnTmWgGHKRu IgUnS7OwWvoZrmKLpKWUCzCmuFHfo9haw/5QsBphS93z+4j7fDf35TGG77JdTpjTcX+r 1O4gyH2uzXfzdhMNASDUpHzwZ7e89jX9QxBCAsbc2Lc5Yqqc24JpYBD3pdbE52u8vq/4 lVCw== X-Forwarded-Encrypted: i=1; AJvYcCUDx3d1Bqr3eRVC/LK6JsIbR2d3w0VGCCeA7xu+Gw5t9M5H8060tyBAdt3R4vhJ8cYc95+paccSCw==@kvack.org X-Gm-Message-State: AOJu0YxPCLIKghN7Fqv6KjGRzLVgEVJH3VsyvRjAzZYM2xWRnoh2AaG7 xyh3mg+VNB6KXWs10kjSDBVcgjtkr7dfICRt8yXl96cdAGPheyzm X-Google-Smtp-Source: AGHT+IFj/sLrVs1zpDnBVyrOd3V+tOWXLkNCExzIVHuijrdfZtdlUlBm11jYo8zP2Sw/Y2vR2Mnm+w== X-Received: by 2002:a05:6870:169b:b0:288:865e:1864 with SMTP id 586e51a60fabf-28cecdcfe8bmr1079911fac.0.1729754962806; Thu, 24 Oct 2024 00:29:22 -0700 (PDT) Received: from localhost.localdomain ([43.154.34.99]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71ec13ea1c4sm7393825b3a.152.2024.10.24.00.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Oct 2024 00:29:22 -0700 (PDT) From: Jim Zhao To: akpm@linux-foundation.org Cc: jimzhao.ai@gmail.com, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org Subject: Re: [PATCH] mm/page-writeback: Raise wb_thresh to prevent write blocking with strictlimit Date: Thu, 24 Oct 2024 15:29:19 +0800 Message-Id: <20241024072919.468589-1-jimzhao.ai@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20241023232042.f9373f9f826ceae2a4f4da35@linux-foundation.org> References: <20241023232042.f9373f9f826ceae2a4f4da35@linux-foundation.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: asqpn473rxrd1nnjszon7h4s75gbtib1 X-Rspamd-Queue-Id: BD77BC0009 X-Rspamd-Server: rspam11 X-HE-Tag: 1729754945-547747 X-HE-Meta: U2FsdGVkX19LxU+y3NUw2RiD/Nyd28J7Xn0DIqUwL1+MA+U1uz/2sbXaHHVmzjJY8xbWRjiSt4EQ/TA+B+8P91ywjH+3dOJuBl80w1J8KuC7yhh/8FQY4Zp3kgCLthBU7TykEVFvdxU4Io6dm5T/rMS9gSHzRqqZ3g9kIEeyyd08EU0VDnR2uG2aTH5vIHXLviPAnDwKo99UWuqiWIKya5zRnZu4t3WeQAokhHUIThpSGZQuayBo/2vFM4dDiDmfDTn8S7ndhyQIZpvI93xVQwmTK6YkJmcmmNj3ZhyK5SHSrMNVdIDka9o9a5u9HEzkinSb3lpM9OnAws0KEFDM7aU7GmxodrbrYsyq09MdKaAViIpTbb+85owCMSrlxEx38eqHNnvaXHd/fDt7hdNmyhlbqMZItyh4UF5KLxBidEeG74Oxa8PaPH9R1bZv8anLEnEqY+NyHCVhS8nDHOfyaX+6wG2suNY/772PN5wn/RI+tnAiQDuyPeqOLhaOYkGSEF+BvAOAEr+qac/t0WMWijIMhrbvrRQOFqEa0ZBYQEJ7+428EYM6YVozdXhVkjbU9+b9WGfGzYjiugBCdWvnYqv1Wg7CLHF9G3mhbVu9idbGq2/PazerjGE8o3KtD7Nff8ezKEj/mvkHm7ijg2pC6ZOg4rwxeiocUADu8lRrCwfQqtAx0wui+d25iM9Yl8hGoRcLUstS8A/E8gGXBW+R8EyxhC1tmR3LCbVnZEO40vp1nnwx74TGNirD39EAeMomE/EZ+n42169zZauvwSgdMLvSatLYOvxS5AVcLYpEsMPb108RVpnXQo8qhxP6BRcqym5q2WyIH4XBSuu1dMcnglo05RLZu6jqe4/Azl3RuydgUM+3zkS3skjSAWJv/IA5mc/7LkwjbCgyLve7Jy1M9JEqzAq/2VrwZcnwxbQZJHAaxqcY+OaQBfSozv2OzOjRUQJQGxaXPNFQp+SV93S 1XFKjFqj JPznTeLXohkklhepr0QLGk1OPy6mqogB1Va7tAsndyRaURTpG/410kfRAlRLhs4XxA4qa2TEUBvYjVGFJuu6kNMc7IipwBr85cIcgEtzCUfxY2g1xL3EtICZBmcewm/3ko5viXzf2b+qrHEN0SnfwEQCDIL2njtBlJS3uHDiVG1tk+5+P13v/mp1Xp3/vubqmv9aYscvCBsRv/2eLYeV/A+8T/SVEaagsBOMn70DFzrLj+yWkRl5aFU2bU1RRkH5Twr8oWQfISM8HR1w2oses+2bRdK9x0/+hb71UQDFZ4OIBYPwuBhjyVqu5gIwSqdOcHbdfVWDy2qxqtk8BpEtYbJgRJPBNLfIHPFqe7cb5E/m/14pvpblh5k7rEPTwQOmqS78Ux+KAO9ZaG9r+Na7WH01ixWmrVWfQ2cuhXYK+BZwcyWxmBvxhwhXzC5lYeNECWkYBUl/J+NJVXNtJpbqvcr14b5VrquQet7pt8To+PJeqlV7mo5jDQ6sIecNcHyy9mX1v1Bt2F+/aREUIjDgrczCAyUbFUzueP6nZLigVvC/adp78w8K+G9yDYnbe8tkuQT+MnGi0ZHzfLvHDtE/IzARp+AC67t2horeKEkBQC8PJIX43IngEFmE9Eo4iyuoqOlVqkJipGqoiG1o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001551, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > On Thu, 24 Oct 2024 14:09:54 +0800 Jim Zhao wrote: > > > On Wed, 23 Oct 2024 18:00:32 +0800 Jim Zhao wrote: > > > > > > With the strictlimit flag, wb_thresh acts as a hard limit in > > > > balance_dirty_pages() and wb_position_ratio(). When device write > > > > operations are inactive, wb_thresh can drop to 0, causing writes to > > > > be blocked. The issue occasionally occurs in fuse fs, particularly > > > > with network backends, the write thread is blocked frequently during > > > > a period. To address it, this patch raises the minimum wb_thresh to a > > > > controllable level, similar to the non-strictlimit case. > > > > > Please tell us more about the userspace-visible effects of this. It > > > *sounds* like a serious (but occasional) problem, but that is unclear. > > > > > And, very much relatedly, do you feel this fix is needed in earlier > > > (-stable) kernels? > > > > The problem exists in two scenarios: > > 1. FUSE Write Transition from Inactive to Active > > > > sometimes, active writes require several pauses to ramp up to the appropriate wb_thresh. > > As shown in the trace below, both bdi_setpoint and task_ratelimit are 0, means wb_thresh is 0. > > The dd process pauses multiple times before reaching a normal state. > > > > dd-1206590 [003] .... 62988.324049: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259360 dirty=454 bdi_setpoint=0 bdi_dirty=32 dirty_ratelimit=18716 task_ratelimit=0 dirtied=32 dirtied_pause=32 paused=0 pause=4 period=4 think=0 cgroup_ino=1 > > dd-1206590 [003] .... 62988.332063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259453 dirty=454 bdi_setpoint=0 bdi_dirty=33 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.340064: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259526 dirty=454 bdi_setpoint=0 bdi_dirty=34 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.348061: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=489 bdi_setpoint=0 bdi_dirty=35 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > dd-1206590 [003] .... 62988.356063: balance_dirty_pages: bdi 0:51: limit=295073 setpoint=259531 dirty=490 bdi_setpoint=0 bdi_dirty=36 dirty_ratelimit=18716 task_ratelimit=0 dirtied=1 dirtied_pause=0 paused=0 pause=4 period=4 think=4 cgroup_ino=1 > > ... > > > > 2. FUSE with Unstable Network Backends and Occasional Writes > > Not easy to reproduce, but when it occurs in this scenario, > > it causes the write thread to experience more pauses and longer durations. > Thanks, but it's still unclear how this impacts our users. How lenghty > are these pauses? The length is related to device writeback bandwidth. Under normal bandwidth, each pause may last around 4ms in several times as shown in the trace above(5 times). In extreme cases, fuse with unstable network backends, if pauses occur frequently and bandwidth is low, each pause can exceed 10ms, the total duration of pauses can accumulate to second. Thnaks, Jim Zhao