From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDF82C5478C for ; Mon, 26 Feb 2024 21:55:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F5A66B016A; Mon, 26 Feb 2024 16:55:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A59444017F; Mon, 26 Feb 2024 16:55:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F8D66B016A; Mon, 26 Feb 2024 16:55:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E60B06B015B for ; Mon, 26 Feb 2024 16:55:17 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AAE94A03D6 for ; Mon, 26 Feb 2024 21:55:17 +0000 (UTC) X-FDA: 81835311474.26.1DF86A1 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf26.hostedemail.com (Postfix) with ESMTP id 289DE14001C for ; Mon, 26 Feb 2024 21:55:14 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NfNJyr1c; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of "SRS0=kqK5=KD=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 145.40.73.55 as permitted sender) smtp.mailfrom="SRS0=kqK5=KD=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708984515; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4rAtxPAcknrq52StKgB4MKJgob5Vjx/85UUJSdMmJOA=; b=7whEALrPmoC5VQ3m4Hea1pLeWzx8YR5UFcPifHT+OXSDkQi79QUC9hHWmTGjDzQpQqbwvH gle9TGzbsu4vejk4mkH5xV2SUr0GKOFEgRGJ2Q9BNwaxd3PoqBhdWsKJArDuoLYEKp3Y9x nFz2vQtv1KB69fOAaPz2VYIcSH4BYnM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NfNJyr1c; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf26.hostedemail.com: domain of "SRS0=kqK5=KD=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" designates 145.40.73.55 as permitted sender) smtp.mailfrom="SRS0=kqK5=KD=paulmck-ThinkPad-P17-Gen-1.home=paulmck@kernel.org" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708984515; a=rsa-sha256; cv=none; b=tnG92EQg+V89zTkPVUjg9i8Sf+X8QxXyksO41NYSJQ6TenR1GYDe6sBDNWkPl40GHsz2uG FcMI/xqQi9JKtEd0/25ZT/CBJszx4HAUGrVKP5z7kIPz/oq37baU25nMOPckE/qmD26ycA MzbsJIbbACcKxRaijnZ0RCvtzIippNA= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id AE012CE1AA2; Mon, 26 Feb 2024 21:55:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6AEAC433F1; Mon, 26 Feb 2024 21:55:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708984511; bh=QSYRZy9FUoynzy1lLrNSLVKDLhKHxKpJtpQ/pB5dh7s=; h=Date:From:To:Cc:Subject:Reply-To:References:In-Reply-To:From; b=NfNJyr1ca61qNnElH4a4LKcqC9/Uvmept7lOTssDeHjaW6SXTNjDYpP39n51VkntJ 54Wu6Hc4t7+P13aAaBF6XHuGTBUYAI7WUQqMfmk5SRix8yH2xPiXHfO1sPykmI/hQw q/x6lw58BYkBA96dx7UERNmO3ATWUNyuWSi0sZug6zu3KvsETjnKMapjVGKFiEWToD IgcsdIrp6ItY/983qAj7GtcXf05RfIodQaBNqJq85FxkhvlHqJKKvBjrRPeqAf7EC6 0LJoZ/Es9G+FwwixfpefXajRi1Dy0pyU4xyA5ZmEhn6FQaoKq8nJb6c9CeQbeJiUo9 HdQbfpuzKw0aw== Received: by paulmck-ThinkPad-P17-Gen-1.home (Postfix, from userid 1000) id 8F33ECE0E27; Mon, 26 Feb 2024 13:55:10 -0800 (PST) Date: Mon, 26 Feb 2024 13:55:10 -0800 From: "Paul E. McKenney" To: Kent Overstreet Cc: Matthew Wilcox , Linus Torvalds , Al Viro , Luis Chamberlain , lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-mm , Daniel Gomez , Pankaj Raghav , Jens Axboe , Dave Chinner , Christoph Hellwig , Chris Mason , Johannes Weiner Subject: Re: [LSF/MM/BPF TOPIC] Measuring limits and enhancing buffered IO Message-ID: Reply-To: paulmck@kernel.org References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Stat-Signature: nm5zjkijkhoubtprb789pktwi17c5p4y X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 289DE14001C X-HE-Tag: 1708984514-684071 X-HE-Meta: U2FsdGVkX1/E2rpmh6kQk9w4Me9u+B92NlDJE32Q0i5Rn9OknTnVbbvbFrgLKmO5N4qtZDV7yeTCCzO9c6ljt8FDpr8DKv+VBGCoIZXcgRvcu89WCwRLyWx4WLhews/xUGClsG3DgnBUqLWFfg2imXrnqMOFRlcWEcKVvnZyupTtoPaE86/ArbFkXPnHIR94EeGmqhtRt339m7eNv5Hozvc0oPKLqWSbAju4pwXSnFRCRo4rHC2TtmIoqqz0NmK1ldAgR8r5V9n7RA44uqX+K+E0U8LGpvN4Xos57oNz493vNstJBzBZyG6/NILL8BnZhnhor1ppa6c9NWsufOhh/bcKdP6T23sksPfZxUb1PLXA6k6eumDIl+il84LG13oSSMFSozjXoBsjFqXFAAAPVBWLg+3isNslvmeFrsTfZND7XAEAgmhxq4ZjmA0JjYojYE9wd/aAzrzz5TgMPa80IRjXCC8z9XAHUf63XadJBTRLp9gfLiAlN/dEQoc4WkhENI2sYiPe3XOEPpisPfr/0ogQIBCllHcteZviqcoKNDB4LUjdxWVohWHHNSU++6pv9u9DBNZxU5UyAH8v3HLB+PCnSxEFFV2ETsqL+24tIpiJLSWUIaoEch/COWiGIhEuhqvNonpxUhRYIBZ41G62Ue//Jg371nIUaYyu6KtOE3Y0xDjd5yqdt4g+7/RrmBGcEQl97ELb3vHPC5GOHmNEtkAsg6kBhtgsqM0yEqKsHkbuF2OwzIEfUYVjebTmDdoUplK8p6tz5hUV+8C10gXPRqYHaXM6TPR91JaFoZ9w6EGWSlwqMsW5WGDjc382ihtbfPs6pi3d16y40OCiIEIncU3dJcarTxJx3W3Xih0DPvH08uRZU1S6f7ESqUWIUYosSfR0+ih0nCa8RMCf6K8pHubaraP2/1rmgkxFiPRTFLUm1Y7gD1GIXmm/Kg4Y47qPRt0Ir9iTyZ1u26e023n 35S1fyGH bQDj6D6PmhgZD/AeW1un8qBbb+zX4V8wsV6w00bO4AW/jCaJCcMpDLhKEkky3tjcw5YRajM9KhvfSY9404b2flQ9Ffq8wCiaSj5Z6OmT58HuqnorYrpeJK1q0zezoqkVai2w6vH0jtWpONat5s7a77K6pxpZCu9ARItlgXgR9DOL9IeuANjrsWCx9MD7t5LRfkCEKJu4hqF2CEsC3UZNDUHC7xV/hkHXq9tjg96JSMJtukgepWy3tPlew3S5vYru/kQCgMkUDe2A492jFSBNeqQXy5uVFCAhRNa99uctQ66phpadPr3ubdcj4urSYVabD3uRgFbocuh8ReE1gzfb0wnYWmjYHelUJM5Ao/wAuaR1AYvIiDDcBty8sfNZ8sERDA5m+2DKU+hv/9VQWykj9nWeps4Cnfp1i9QhBB0afD9kPDNgFLBfeamUN/g52aSBEphVIlCgCrh1bE7FyS4aGQTt7rR0t/sLixUVvqx5RZs5YMHpTtoynTanJnULhmyQkctiPeTFa+YAPHaAiI4EWh9bgE/j+kATRFSVf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Feb 26, 2024 at 04:19:14PM -0500, Kent Overstreet wrote: > +cc Paul > > On Mon, Feb 26, 2024 at 04:17:19PM -0500, Kent Overstreet wrote: > > On Mon, Feb 26, 2024 at 09:07:51PM +0000, Matthew Wilcox wrote: > > > On Mon, Feb 26, 2024 at 09:17:33AM -0800, Linus Torvalds wrote: > > > > Willy - tangential side note: I looked closer at the issue that you > > > > reported (indirectly) with the small reads during heavy write > > > > activity. > > > > > > > > Our _reading_ side is very optimized and has none of the write-side > > > > oddities that I can see, and we just have > > > > > > > > filemap_read -> > > > > filemap_get_pages -> > > > > filemap_get_read_batch -> > > > > folio_try_get_rcu() > > > > > > > > and there is no page locking or other locking involved (assuming the > > > > page is cached and marked uptodate etc, of course). > > > > > > > > So afaik, it really is just that *one* atomic access (and the matching > > > > page ref decrement afterwards). > > > > > > Yep, that was what the customer reported on their ancient kernel, and > > > we at least didn't make that worse ... > > > > > > > We could easily do all of this without getting any ref to the page at > > > > all if we did the page cache release with RCU (and the user copy with > > > > "copy_to_user_atomic()"). Honestly, anything else looks like a > > > > complete disaster. For tiny reads, a temporary buffer sounds ok, but > > > > really *only* for tiny reads where we could have that buffer on the > > > > stack. > > > > > > > > Are tiny reads (handwaving: 100 bytes or less) really worth optimizing > > > > for to that degree? > > > > > > > > In contrast, the RCU-delaying of the page cache might be a good idea > > > > in general. We've had other situations where that would have been > > > > nice. The main worry would be low-memory situations, I suspect. > > > > > > > > The "tiny read" optimization smells like a benchmark thing to me. Even > > > > with the cacheline possibly bouncing, the system call overhead for > > > > tiny reads (particularly with all the mitigations) should be orders of > > > > magnitude higher than two atomic accesses. > > > > > > Ah, good point about the $%^&^*^ mitigations. This was pre mitigations. > > > I suspect that this customer would simply disable them; afaik the machine > > > is an appliance and one interacts with it purely by sending transactions > > > to it (it's not even an SQL system, much less a "run arbitrary javascript" > > > kind of system). But that makes it even more special case, inapplicable > > > to the majority of workloads and closer to smelling like a benchmark. > > > > > > I've thought about and rejected RCU delaying of the page cache in the > > > past. With the majority of memory in anon memory & file memory, it just > > > feels too risky to have so much memory waiting to be reused. We could > > > also improve gup-fast if we could rely on RCU freeing of anon memory. > > > Not sure what workloads might benefit from that, though. > > > > RCU allocating and freeing of memory can already be fairly significant > > depending on workload, and I'd expect that to grow - we really just need > > a way for reclaim to kick RCU when needed (and probably add a percpu > > counter for "amount of memory stranded until the next RCU grace > > period"). There are some APIs for that, though the are sharp-edged and mainly intended for rcutorture, and there are some hooks for a CI Kconfig option called RCU_STRICT_GRACE_PERIOD that could be organized into something useful. Of course, if there is a long-running RCU reader, there is nothing RCU can do. By definition, it must wait on all pre-existing readers, no exceptions. But my guess is that you instead are thinking of memory-exhaustion emergencies where you would like RCU to burn more CPU than usual to reduce grace-period latency, there are definitely things that can be done. I am sure that there are more questions that I should ask, but the one that comes immediately to mind is "Is this API call an occasional thing, or does RCU need to tolerate many CPUs hammering it frequently?" Either answer is fine, I just need to know. ;-) Thanx, Paul