From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A51FC83F1A for ; Mon, 14 Jul 2025 20:13:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C0818D0002; Mon, 14 Jul 2025 16:13:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0716A8D0001; Mon, 14 Jul 2025 16:13:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC94C8D0002; Mon, 14 Jul 2025 16:13:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D9B868D0001 for ; Mon, 14 Jul 2025 16:13:10 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 85FA4140299 for ; Mon, 14 Jul 2025 20:13:10 +0000 (UTC) X-FDA: 83663969340.09.C00C37E Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf07.hostedemail.com (Postfix) with ESMTP id 8BA2F40009 for ; Mon, 14 Jul 2025 20:13:08 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=jUbqEtrI; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752523989; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8XoKnjcBylfGEBNdUuIZFyv37qBPEWmYJJGzJJn5qtA=; b=uPY7LObUdoOAjFzz5TV63yvVfUXR1r1ViGfENqdeec6bWVmn9Ui8o7BkVYR/w1jbid1Zs0 EaK0ZmHLTVCWQeG3K9JKjG0lK2CpSpomIe/6NheEjoXHM+OdpyaBau/cYdBrWiSGEmQJ1q q8Hg1Ua8RUldSZEZ6GGyMU9kRyCPvkM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752523989; a=rsa-sha256; cv=none; b=k9bSrZSeIjAtrgE6zXgTDSJ8Z1JEuvU3byw4hvPcfCpi5IdxSG3pz+xGVtfUS0r/798EGE 29Weyw2wTx1SXT9XrQ2JtRPvMlPhirJK9vtW7zJoTmriVgc+O1x615WdAP32jGaOgTV4VS fGOnj3LhCOYzHnOxjFLsu4ETA5LOGmM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=jUbqEtrI; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf07.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1752523984; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=8XoKnjcBylfGEBNdUuIZFyv37qBPEWmYJJGzJJn5qtA=; b=jUbqEtrIIB2vzENsalRF51bVNRuElUwz3OuvWK9eJUIEL9TU/4JvH0+0CUO0c+gxY2B0tq HWDjYCcu3tWqOmFriyfbEFcbh8HD+p6BypbIj9zo+yDUzwWWczYdj1+b240IKfnWvyCBtC btjTnmQBNMLb0kUvbv5O35xZ4Zesb4Q= From: Roman Gushchin To: Jan Kara Cc: Andrew Morton , Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Liu Shixin Subject: Re: [PATCH] mm: consider disabling readahead if there are signs of thrashing In-Reply-To: (Jan Kara's message of "Mon, 14 Jul 2025 17:16:51 +0200") References: <20250710195232.124790-1-roman.gushchin@linux.dev> Date: Mon, 14 Jul 2025 13:12:59 -0700 Message-ID: <87ms96v8dw.fsf@linux.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 8BA2F40009 X-Stat-Signature: be1gjqzerrqdgb986x49h3b59mhbss8c X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1752523988-512859 X-HE-Meta: U2FsdGVkX18aSIaWvnBXgB4YoJM/3ujgHG7Xi1rukBGjA7qdoo1HZpbrhHX8+cTiNQ4M5lI18X0z94CRNsvzYc7XDo8tJTCI1z8r7JRFtrpDPuJqxvB5KOXizHWcyFXBtZZzjQ3f1t2W5ssUj0yNJhe+tSzZwEVunpIPTBCeNir0gWija6kXq/20+BL88q4kMThE5mJZRZuyRUBBwGUZnL0GkVhSrjl9kD17H4Ffxvduk4+8Yx659JDY8IkVE/yeTD5Zz1aSue5onm8n3SstCuJktdghVUAvI9qcPIeqye4uCjRkymqvoUwj8DybQp/3t1DOtJy+I2MvZzp6ZaV51czCbrBIk3b4jQ0xOzUK6Vk983NPdAzt5mnKGrGVYwVpDsaoWP4faoIjOIOTlVVcKOyPhRVjD3bfsBBVbw8jYN36hkNYgolIPWDszwtxts/45oOr38FMdQz+ODw2HOPEjnXi/vJRC8PZUgj7Rq3gDV6KEQnSGzQWleEjXBoJa0+39i5SIX5RptFAfFNHLSXKR7HJs7wr+fXPG1/Ky3lnBQae8BCvNdo5IS1TsmUPRgB0m3yNi35RDc8/BJQqF+K51f8ZEnvlj+Z8oxGjKG5Fm/7JMRLwVZpHlS+Ubf9wwLP5iK41yTRHfJNYfqdZVHrj9RI88/qczcWvzUSqboRxI+sG2ALshZMbu2a3uO1Zy6qPYzu+CA7MMlGkrSs54K5PTH6U9BnQJyOUVXq+PaKEn4oWkqVxrvF63C6ZNtKqn5ly1MB921/vyrD0oRepIZheiVULcFP9klmpekQjDoEo4az0nzoxJRYZrnPZpm5sazUhtjFWBM9//Gd9QlU2vmXV5Osaj6cbwDyhv0Kj8adQp5biprvAd3XFUNIPGpUpx4vwuEdNI1SSwhbkZS/aPOEOPOVncIhPfaH4EyewPLnD4o2AWbfb7WELwTNvDOgRxtIsUZXDFppkExM2JiznyxH JdiNUUdI DxfoepUQOI/szhd2RKrzFN5g+hYbK9jJr42xRoGTsS6UGCSY9zoUrNfrDedbXGBO37jbrg52fQL9Ku87iyWbeagyeArEC5oUzcBgI+nEnenHFIJT3GJfTI4IvzlnVIHvg49Epml3rPRk2qeqhaZRpa5NuovpOlLoL/QoVjCja6u0e44Mxw481IA2ZvCBrj8kYEYFOv09yentgXji/ZsQFTCV1sg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Jan Kara writes: > On Thu 10-07-25 12:52:32, Roman Gushchin wrote: >> We've noticed in production that under a very heavy memory pressure >> the readahead behavior becomes unstable causing spikes in memory >> pressure and CPU contention on zone locks. >> >> The current mmap_miss heuristics considers minor pagefaults as a >> good reason to decrease mmap_miss and conditionally start async >> readahead. This creates a vicious cycle: asynchronous readahead >> loads more pages, which in turn causes more minor pagefaults. >> This problem is especially pronounced when multiple threads of >> an application fault on consecutive pages of an evicted executable, >> aggressively lowering the mmap_miss counter and preventing readahead >> from being disabled. > > I think you're talking about filemap_map_pages() logic of handling > mmap_miss. It would be nice to mention it in the changelog. There's one > thing that doesn't quite make sense to me: When there's memory pressure, > I'd expect the pages to be reclaimed from memory and not just unmapped. > Also given your solution uses !uptodate folios suggests the pages were > actually fully reclaimed and the problem really is that filemap_map_pages() > treats as minor page fault (i.e., cache hit) what is in fact a major page > fault (i.e., cache miss)? > > Actually, now that I digged deeper I've remembered that based on Liu > Shixin's report > (https://lore.kernel.org/all/20240201100835.1626685-1-liushixin2@huawei.com/) > which sounds a lot like what you're reporting, we have eventually merged his > fixes (ended up as commits 0fd44ab213bc ("mm/readahead: break read-ahead > loop if filemap_add_folio return -ENOMEM"), 5c46d5319bde ("mm/filemap: > don't decrease mmap_miss when folio has workingset flag")). Did you test a > kernel with these fixes (6.10 or later)? In particular after these fixes > the !folio_test_workingset() check in filemap_map_folio_range() and > filemap_map_order0_folio() should make sure we don't decrease mmap_miss > when faulting fresh pages. Or was in your case page evicted so long ago > that workingset bit is already clear? Hi Jan! I've tried to cherry-pick those changes into the kernel I'm looking at (it's 6.6-based), it didn't help much. I haven't looked why, but at some point I added traces into filemap_map_pages() and I don't remember seeing anything. Most likely because pages are not uptodate at the moment of fault. In my case I saw a large number of minor pagefaults on consequent pages. It seems like one thread is having a major pagefault and then a bunch of other threads are faulting into next pages, effectively breaking the mmap_miss heuristics. Sometimes it reaches 1000, but struggles to stay there. > > Once we better understand the situation, let me also mention that I have > two patches which I originally proposed to fix Liu's problems. They didn't > quite fix them so his patches got merged in the end but the problems > described there are still somewhat valid: > > mm/readahead: Improve page readaround miss detection > > filemap_map_pages() decreases ra->mmap_miss for every page it maps. This > however overestimates number of real cache hits because we have no idea > whether the application will use the pages we map or not. This is > problematic in particular in memory constrained situations where we > think we have great readahead success rate although in fact we are just > trashing page cache & disk. Change filemap_map_pages() to count only > success of mapping the page we are faulting in. This should be actually > enough to keep mmap_miss close to 0 for workloads doing sequential reads > because filemap_map_pages() does not map page with readahead flag and > thus these are going to contribute to decreasing the mmap_miss counter. > > Fixes: f1820361f83d ("mm: implement ->map_pages for page cache") > > - > mm/readahead: Fix readahead miss detection with FAULT_FLAG_RETRY_NOWAIT > > When the page fault happens with FAULT_FLAG_RETRY_NOWAIT (which is > common) we will bail out of the page fault after issuing reads and retry > the fault. That will then find the created pages in filemap_map_pages() > and hence will be treated as cache hit canceling out the cache miss in > do_sync_mmap_readahead(). Increment mmap_miss by two in > do_sync_mmap_readahead() in case FAULT_FLAG_RETRY_NOWAIT is set to > account for the following expected hit. If the page gets evicted even > before we manage to retry the fault, we are under so heavy memory > pressure that increasing mmap_miss by two is fine. > > Fixes: d065bd810b6d ("mm: retry page fault when blocking on disk transfer") Yeah, this looks interesting... Thanks!