From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C0ADC54E5D for ; Tue, 12 Mar 2024 20:11:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3092E6B018E; Tue, 12 Mar 2024 16:11:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B9406B01FC; Tue, 12 Mar 2024 16:11:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15B0D6B01FB; Tue, 12 Mar 2024 16:11:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 07A2B6B02B8 for ; Tue, 12 Mar 2024 16:11:33 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C3B45A084C for ; Tue, 12 Mar 2024 20:11:32 +0000 (UTC) X-FDA: 81889482024.19.E2C6F0F Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) by imf03.hostedemail.com (Postfix) with ESMTP id 16C4B20018 for ; Tue, 12 Mar 2024 20:11:30 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rIX0heMh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of yuzhao@google.com designates 209.85.166.181 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710274291; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gtyufjympz261LAYpMFsuxxNb0cIyOumJAAy8HMFSng=; b=Y7nX7YQ6wElhJiMTDZjY9qAEjRSCJR8Zs4qVvDkPyQ2MxtMilD+P56p3sxdV/Fklu8545L pks4abXVgJyPMw1LKorMdbI10h+mMFZGhsVmwNn+6L+UbiWgcEFLTx+4IeCb8vcTWuWZ74 4NmNHhBbE7Lh5SvNkhvvImUkpDpW0n4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=rIX0heMh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of yuzhao@google.com designates 209.85.166.181 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710274291; a=rsa-sha256; cv=none; b=uZ2zflbXK3reR1dV+6+SgFyPKoN2xr3VoyDPovaFlQ2j9vo5KMmC7eTeBOEW5S/FIQT7M0 5a705YSBB9i2qce61ZfCSW0ZcwmDZENjbfJaOSxm7hAQVS8m4rK/znZK+OYkGJyf6BmIQU 5FKuKS3fxeMittcZcFmNTY1eQAHIj94= Received: by mail-il1-f181.google.com with SMTP id e9e14a558f8ab-3666ffc850eso3915ab.1 for ; Tue, 12 Mar 2024 13:11:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710274290; x=1710879090; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=gtyufjympz261LAYpMFsuxxNb0cIyOumJAAy8HMFSng=; b=rIX0heMh+5i657lWfg5g3w9Gsi1zu0nHBu6+IENT8nugCEt+GGHK+aY6OkYWBwTk1p Hypr7UHxRzf9Qu0ltwdtO+EdhewyyTPS+FgKM8z8P7vYBAxEQUBg//b3X3iS3HHkgtFa 5VuTz/nAfzd5Y5WEey4ClkzZrtR4Gb+luzisqM2X2pXo4/hcfWsms0XQhiUKt9LpVgSv 7ozLYhxdEQnE9Lp2bpmeNgdFUAGewufDbBmsufiMwOvwuRmlAkN81R0f21ZGvk37wGVD 5XbWkslEhfgjkwO89hB200sn6OSGT9lLBtAY8HXBtUzWTUBbArQQkbL1x92BZeHTy3e1 SDYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710274290; x=1710879090; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gtyufjympz261LAYpMFsuxxNb0cIyOumJAAy8HMFSng=; b=HTlQNuJgOWW1bIZ1BQyWpyr3EdtX33qquyq4Jqao4fTomSDz2Q+uhVD8r187GtNlhG UEOSLwjEaIcWv1qcfQ+ge6x8ACrovj75cyAQ0TRnQUXLT6eg4U18s6Jh/V//NAIvNz3N KrjKyCHxPoPBLu6dIVIvKgJiwwjJDW+pblxNN5TywLGg4GwMckrkrmHzvnmpl7VSW3Nt PnUdjKh3Q0QVUs5dQVn5s98XWelXyiJ45gPPmlcLrgchjE9tI4actiZz9LR/sAIMLDz/ XZf1gqTqNe/QPVWX+BcheM0Ik+ahm8vls1/A488RgZ6pcq7kZxmcew9anEt+xmF7PaHx jpdQ== X-Forwarded-Encrypted: i=1; AJvYcCWpSuzKmTUp1V8AS3LxxnZpGFl/5IgNZ5AtW63pgTqtuixK7Ojr6/TwkEB58JB58u+4rs+Km/0HxKmjIKCBY4THAUk= X-Gm-Message-State: AOJu0YwOKQrhW12/B5JyzcM2z63OcRW2udFhRPgaiL4s4crRgA1wstvZ RlkkL5S51oMN5YUcuL5LUpi18Pnbk/IYwh5r4s8i0Ig0J1oT/pQFIynik604Yg== X-Google-Smtp-Source: AGHT+IGJ79aNGwwMiN/XE1qv+e3S3QKnXAHJweEMjS7wLLVhClpO1ctbpd91SmbTh3SMXUAE9E8ZXQ== X-Received: by 2002:a05:6e02:20ce:b0:366:5dc2:2843 with SMTP id 14-20020a056e0220ce00b003665dc22843mr79777ilq.14.1710274290030; Tue, 12 Mar 2024 13:11:30 -0700 (PDT) Received: from google.com ([100.64.188.49]) by smtp.gmail.com with ESMTPSA id fw18-20020a0566381d9200b00476018ee74csm2483211jab.20.2024.03.12.13.11.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Mar 2024 13:11:28 -0700 (PDT) Date: Tue, 12 Mar 2024 14:11:23 -0600 From: Yu Zhao To: Axel Rasmussen Cc: Yafang Shao , Chris Down , cgroups@vger.kernel.org, hannes@cmpxchg.org, kernel-team@fb.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: MGLRU premature memcg OOM on slow writes Message-ID: References: <20240229235134.2447718-1-axelrasmussen@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 16C4B20018 X-Stat-Signature: 9x8yprfi8op3yrdfmmc8btbx6r7wsi9k X-Rspam-User: X-HE-Tag: 1710274290-643390 X-HE-Meta: U2FsdGVkX1/wHDV83wtnWsm9Iz0jwxRZ/9Uhqd++S9KhRb4lWIu9bEHZLUTkwhgRv+AnaiaGH+1FLn5qxQAXbMYwOrJpDHwe1v7FYA3N3TeF8gDM4Hh9mWgPU+Le54QUXDMJR+BcC+S6Ok3CcS1nKR25pcTm3JkZxL2BT8rtLxZpazG39gLBeWU2/Yvnf+4GJc+6bLpzkczzialN2aPW6SYSFw2X4s6XSI+KK/8OVYBr4FUvDtYgn0sIFyge7wbgP2DAR5Q1SLuB0C/c7cWTbbuW1Cm4stRZp0UVOt7FhgGZzS5cJRWwnk5hu/so7Y+7Kf0K/CpuiPc4KepAiDYDHviiYblSjo55Qbzc2S/6UsVahz8BoJAG8QP+s7olemcGUtTdAdAO/B3DFJvyWt1HisgQx2dK7ICcRAf4X0qlzraPDEkxY9O0yMBIp/Sjm/5AvO2VR5XphnkBKv08L4307gr7CE6BEAFebnLN/o/yDbuqTEYXIEnVM2cUu7bGe+tjd6j9Xv/KnSHgUX7fcoSePlmh3XmiYpEUiKY481x+IfCmpTlgNrRraQYZ9DEvWZzEirzN3PYrOMpsythbxDwxVshMAHN8RTU1/sq2CmDW9pXDCylq63F4kVkAzbY3lPx9iYlwzpQNZNUoDow1vPiae0tgyAW6vqbdpjIbWp/WWhpecfbd15PS571In43yICV4JmQniJf3pJRYnsTKc1lbkax/z2HM+BlokN7zzvs2IYgBvuslc+dnoonIgZRMlHFk53Tk06f5vfVpGViSBQrp/MJT/Xjorv1nDfmwCG8hjGnEJxbJ2g2AdanpRUziQqpIoVW+poaUeWVthZ3TBXB9n1zFN63/GQge/UihAkcJF4b15V+CyDEQezYsPCotq0+6H/Ut4W0VZDcsCO9l2SLYZ4VH0EyfLqCE6mq6xT5LsyLms5S1Mtcj6/O+CaQC0I16J7/LQjQsX3SO285SPUq nW6ktbCo P1RVqIXKhjp7WG9SOjD7jy+uhZsKKW9oOKL6CvtJ/vqgIIhjrTroxTTc0Ut3dboID7kL5SeeCkLaSnc46V9Nq/6LCcBxhmtGMZcZX++6UI1vuDfM8LFuYxanZJLcwCI/306jAxRSnum/BwDWuEat6Yd7UyA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Mar 12, 2024 at 02:07:04PM -0600, Yu Zhao wrote: > On Tue, Mar 12, 2024 at 09:44:19AM -0700, Axel Rasmussen wrote: > > On Mon, Mar 11, 2024 at 2:11 AM Yafang Shao wrote: > > > > > > On Sat, Mar 9, 2024 at 3:19 AM Axel Rasmussen wrote: > > > > > > > > On Thu, Feb 29, 2024 at 4:30 PM Chris Down wrote: > > > > > > > > > > Axel Rasmussen writes: > > > > > >A couple of dumb questions. In your test, do you have any of the following > > > > > >configured / enabled? > > > > > > > > > > > >/proc/sys/vm/laptop_mode > > > > > >memory.low > > > > > >memory.min > > > > > > > > > > None of these are enabled. The issue is trivially reproducible by writing to > > > > > any slow device with memory.max enabled, but from the code it looks like MGLRU > > > > > is also susceptible to this on global reclaim (although it's less likely due to > > > > > page diversity). > > > > > > > > > > >Besides that, it looks like the place non-MGLRU reclaim wakes up the > > > > > >flushers is in shrink_inactive_list() (which calls wakeup_flusher_threads()). > > > > > >Since MGLRU calls shrink_folio_list() directly (from evict_folios()), I agree it > > > > > >looks like it simply will not do this. > > > > > > > > > > > >Yosry pointed out [1], where MGLRU used to call this but stopped doing that. It > > > > > >makes sense to me at least that doing writeback every time we age is too > > > > > >aggressive, but doing it in evict_folios() makes some sense to me, basically to > > > > > >copy the behavior the non-MGLRU path (shrink_inactive_list()) has. > > > > > > > > > > Thanks! We may also need reclaim_throttle(), depending on how you implement it. > > > > > Current non-MGLRU behaviour on slow storage is also highly suspect in terms of > > > > > (lack of) throttling after moving away from VMSCAN_THROTTLE_WRITEBACK, but one > > > > > thing at a time :-) > > > > > > > > > > > > Hmm, so I have a patch which I think will help with this situation, > > > > but I'm having some trouble reproducing the problem on 6.8-rc7 (so > > > > then I can verify the patch fixes it). > > > > > > We encountered the same premature OOM issue caused by numerous dirty pages. > > > The issue disappears after we revert the commit 14aa8b2d5c2e > > > "mm/mglru: don't sync disk for each aging cycle" > > > > > > To aid in replicating the issue, we've developed a straightforward > > > script, which consistently reproduces it, even on the latest kernel. > > > You can find the script provided below: > > > > > > ``` > > > #!/bin/bash > > > > > > MEMCG="/sys/fs/cgroup/memory/mglru" > > > ENABLE=$1 > > > > > > # Avoid waking up the flusher > > > sysctl -w vm.dirty_background_bytes=$((1024 * 1024 * 1024 *4)) > > > sysctl -w vm.dirty_bytes=$((1024 * 1024 * 1024 *4)) > > > > > > if [ ! -d ${MEMCG} ]; then > > > mkdir -p ${MEMCG} > > > fi > > > > > > echo $$ > ${MEMCG}/cgroup.procs > > > echo 1g > ${MEMCG}/memory.limit_in_bytes > > > > > > if [ $ENABLE -eq 0 ]; then > > > echo 0 > /sys/kernel/mm/lru_gen/enabled > > > else > > > echo 0x7 > /sys/kernel/mm/lru_gen/enabled > > > fi > > > > > > dd if=/dev/zero of=/data0/mglru.test bs=1M count=1023 > > > rm -rf /data0/mglru.test > > > ``` > > > > > > This issue disappears as well after we disable the mglru. > > > > > > We hope this script proves helpful in identifying and addressing the > > > root cause. We eagerly await your insights and proposed fixes. > > > > Thanks Yafang, I was able to reproduce the issue using this script. > > > > Perhaps interestingly, I was not able to reproduce it with cgroupv2 > > memcgs. I know writeback semantics are quite a bit different there, so > > perhaps that explains why. > > > > Unfortunately, it also reproduces even with the commit I had in mind > > (basically stealing the "if (all isolated pages are unqueued dirty) { > > wakeup_flusher_threads(); reclaim_throttle(); }" from > > shrink_inactive_list, and adding it to MGLRU's evict_folios()). So > > I'll need to spend some more time on this; I'm planning to send > > something out for testing next week. > > Hi Chris, > > My apologies for not getting back to you sooner. > > And thanks everyone for all the input! > > My take is that Chris' premature OOM kills were NOT really due to > the flusher not waking up or missing throttling. > > Yes, these two are among the differences between the active/inactive > LRU and MGLRU, but their roles, IMO, are not as important as the LRU > positions of dirty pages. The active/inactive LRU moves dirty pages > all the way to the end of the line (reclaim happens at the front) > whereas MGLRU moves them into the middle, during direct reclaim. The > rationale for MGLRU was that this way those dirty pages would still > be counted as "inactive" (or cold). > > This theory can be quickly verified by comparing how much > nr_vmscan_immediate_reclaim grows, i.e., > > Before the copy > grep nr_vmscan_immediate_reclaim /proc/vmstat > And then after the copy > grep nr_vmscan_immediate_reclaim /proc/vmstat > > The growth should be trivial for MGLRU and nontrivial for the > active/inactive LRU. > > If this is indeed the case, I'd appreciate very much if anyone could > try the following (I'll try it myself too later next week). > > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 4255619a1a31..020f5d98b9a1 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -4273,10 +4273,13 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c > } > > /* waiting for writeback */ > - if (folio_test_locked(folio) || folio_test_writeback(folio) || > - (type == LRU_GEN_FILE && folio_test_dirty(folio))) { > - gen = folio_inc_gen(lruvec, folio, true); > - list_move(&folio->lru, &lrugen->folios[gen][type][zone]); > + if (folio_test_writeback(folio) || (type == LRU_GEN_FILE && folio_test_dirty(folio))) { > + DEFINE_MAX_SEQ(lruvec); > + int old_gen, new_gen = lru_gen_from_seq(max_seq); > + > + old_gen = folio_update_gen(folio, new_gen); > + lru_gen_update_size(lruvec, folio, old_gen, new_gen); > + list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]); Sorry missing one line here: + folio_set_reclaim(folio); > return true; > } > > > > > If I understand the issue right, all we should need to do is get a > > > > slow filesystem, and then generate a bunch of dirty file pages on it, > > > > while running in a tightly constrained memcg. To that end, I tried the > > > > following script. But, in reality I seem to get little or no > > > > accumulation of dirty file pages. > > > > > > > > I thought maybe fio does something different than rsync which you said > > > > you originally tried, so I also tried rsync (copying /usr/bin into > > > > this loop mount) and didn't run into an OOM situation either. > > > > > > > > Maybe some dirty ratio settings need tweaking or something to get the > > > > behavior you see? Or maybe my test has a dumb mistake in it. :) > > > > > > > > > > > > > > > > #!/usr/bin/env bash > > > > > > > > echo 0 > /proc/sys/vm/laptop_mode || exit 1 > > > > echo y > /sys/kernel/mm/lru_gen/enabled || exit 1 > > > > > > > > echo "Allocate disk image" > > > > IMAGE_SIZE_MIB=1024 > > > > IMAGE_PATH=/tmp/slow.img > > > > dd if=/dev/zero of=$IMAGE_PATH bs=1024k count=$IMAGE_SIZE_MIB || exit 1 > > > > > > > > echo "Setup loop device" > > > > LOOP_DEV=$(losetup --show --find $IMAGE_PATH) || exit 1 > > > > LOOP_BLOCKS=$(blockdev --getsize $LOOP_DEV) || exit 1 > > > > > > > > echo "Create dm-slow" > > > > DM_NAME=dm-slow > > > > DM_DEV=/dev/mapper/$DM_NAME > > > > echo "0 $LOOP_BLOCKS delay $LOOP_DEV 0 100" | dmsetup create $DM_NAME || exit 1 > > > > > > > > echo "Create fs" > > > > mkfs.ext4 "$DM_DEV" || exit 1 > > > > > > > > echo "Mount fs" > > > > MOUNT_PATH="/tmp/$DM_NAME" > > > > mkdir -p "$MOUNT_PATH" || exit 1 > > > > mount -t ext4 "$DM_DEV" "$MOUNT_PATH" || exit 1 > > > > > > > > echo "Generate dirty file pages" > > > > systemd-run --wait --pipe --collect -p MemoryMax=32M \ > > > > fio -name=writes -directory=$MOUNT_PATH -readwrite=randwrite \ > > > > -numjobs=10 -nrfiles=90 -filesize=1048576 \ > > > > -fallocate=posix \ > > > > -blocksize=4k -ioengine=mmap \ > > > > -direct=0 -buffered=1 -fsync=0 -fdatasync=0 -sync=0 \ > > > > -runtime=300 -time_based