From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0B4CC27C75 for ; Tue, 11 Jun 2024 14:45:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 277BD6B0082; Tue, 11 Jun 2024 10:45:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 200846B0098; Tue, 11 Jun 2024 10:45:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 079C96B009F; Tue, 11 Jun 2024 10:45:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D926D6B0082 for ; Tue, 11 Jun 2024 10:45:11 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7866E1613A7 for ; Tue, 11 Jun 2024 14:45:11 +0000 (UTC) X-FDA: 82218880422.10.768D131 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf18.hostedemail.com (Postfix) with ESMTP id 05F001C0025 for ; Tue, 11 Jun 2024 14:45:07 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=GsWzJ9dP; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf18.hostedemail.com: domain of djwong@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=djwong@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718117108; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mInTx6MpNYPtHXz2SoI0fDvo6zXFh5neyFbDWLND4GU=; b=I8Yf/ZXDVo+BPl5Cq+6kSWc2SqLVkaZZjZUvMhmDZHMqGLOXcHBI/ZnpvfEscawdp7ev+h lfcWw4RLJIWuONIA3s3jvXHW/c7ZA0uXkYSpZSQs5quMbSw4iSG7GjPIQquq2gIbIWL/sY hFDUI3M+T3tuuuP/aajXUTgS80Labsc= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=GsWzJ9dP; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf18.hostedemail.com: domain of djwong@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=djwong@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718117108; a=rsa-sha256; cv=none; b=ZTEUvC6GT+wvt7fRxq8KDgdaeShf6OQr1wf/0GQfX8Xcnxw1V12n7QBzEqqPJLgE9HGafk xuc1FmVo5RWmUtJIW2JGNJH3t1iED9FIcToPzv1N11gMz1ZctLVGn8WQVei7pxdx3QnzXk yzMv0tfAsAUp96/fZdVhqMVP0Ee5HEw= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 8EBE0CE1C13; Tue, 11 Jun 2024 14:45:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B6BE2C2BD10; Tue, 11 Jun 2024 14:45:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1718117103; bh=/qy27D9TdU90zlrZxsGplfXoqcvHs7I9DYvfbjjLYMk=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=GsWzJ9dPmMw+/B24v3ppf4gZGMLRXx777bAh41V8/1KSDEI89CcpY3rt8hzNPc29m Cwph+hpLhU9IBl+laVqCCm26pQ5R+Ybq9LSmnIesVqtR1Zlsq/ipClRR1LJ/4GmLdx 0uHXt0SLE6to5uCBrYVT7dYY+ZtPzbSNMaBjERf8mbqJ2Pqc4z5wtxfFcjcgRmLf6Z XQsIb4DTQUSA5BHQQvfH3s7ygw+cQ2PGourE47hLwTef67A3siBwLDxMbg5GJiGi3U DjbkayV/a8qqtBaatR93bdeQacbtTa0NytzcRnV5BQY/lPnyj0xz7AusdAmgB6yHxy 4ikI2aCD3xqDg== Date: Tue, 11 Jun 2024 07:45:03 -0700 From: "Darrick J. Wong" To: Luis Chamberlain Cc: patches@lists.linux.dev, fstests@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, akpm@linux-foundation.org, ziy@nvidia.com, vbabka@suse.cz, seanjc@google.com, willy@infradead.org, david@redhat.com, hughd@google.com, linmiaohe@huawei.com, muchun.song@linux.dev, osalvador@suse.de, p.raghav@samsung.com, da.gomez@samsung.com, hare@suse.de, john.g.garry@oracle.com Subject: Re: [PATCH 5/5] fstests: add stress truncation + writeback test Message-ID: <20240611144503.GI52977@frogsfrogsfrogs> References: <20240611030203.1719072-1-mcgrof@kernel.org> <20240611030203.1719072-6-mcgrof@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240611030203.1719072-6-mcgrof@kernel.org> X-Rspamd-Queue-Id: 05F001C0025 X-Rspamd-Server: rspam10 X-Rspam-User: X-Stat-Signature: 8sto5byju8n53h3qo61a4es65f4ujuzi X-HE-Tag: 1718117107-792291 X-HE-Meta: U2FsdGVkX1/ZQq0Bpoudo/Veaqqr5rDnyBKiMpJ2YV5feFH2T6eF7atAJCsSUkQDbzfij/FheOTZY2eWzlGJi8lKXwmazjFTUjLkjbpzMf4MXLvHLe1dx3LUdQDCfWA756AP72rMxkoFuiqz4ECozdeUaDvcnRx9N6lr8iB6SO60JK5wM/Sczyol2NQSsbzcwAUVH8hEdi/lP3Qv48yQmnFOhhrIyohRWDVT2DwekmALSHNrRssw8vVdEupcTiDdjMcfh7Rpm97WCZOeJxCfvK1glWtptGy/LAj6mViam1IrQMCABVM0pW8Zm5Su7rOXpelTa9yt7ealUdscA6m5bKcgP4QlL0fp1xCGX+/iR/NFYpf3QPa9SBOFyIWSbrWFR33DQLP1I5plyWT4TiTTpEh0x4NsFkphdFzESR76x8lgjky+ZiP2sKSQJYozL/IFHpwL+ANRA1c+q9MGKJr02nx0Kx5jNntJg+0C3jG3b6m8R8Nu5GL0u+J95PuL7vhiMDVLqq4oZ7PI5YnQ08p16jCjpYNFcwsR9sAsrSkmiNuv8wnlshoRKHxj/pLJVwoKB8cA9UKpnlYBupoux9HYamHIRG8mZbJW9wMMVNRt/zOF/8w5jdh0CyIGo3jlw7aY0SITRFdtY3TgiXYh7lxXX4KKYpU7+Yly9jJe/zSZnvrfgCNbH0goqA9EPjW6oOjHd1WgA8Jyn6ZiOKz2KVvFPfjLHqOAVs/xVSEvwbnx8pNvFfl86Amx5pl4g8uGYdR/cvc2NwewQNwULZVwTGUlptuojJ4HPBbZ1H7Wf519jDoHqvDKRJUZDGXyuK1clAIUDe62dS/k1qIu+5g7XOtG50FIkTacVu+k2Zs1vDbjAw1UjY8e1UZ8lxLUzRKU7AErMvGuz+c1BZ6EIU2/6DfqyetGfjXYjFW4FKIJJaurXIJhYwdzCUJ0uNnf66uKc0waU0j9zz0eSASuvjlk7ju oVvuk/rP xL1zBelNosnBmSm3QGtz5TTMDOzxDE3V/jdMf1Y0REFNMTgI67FrRXOSg65lZvPz8mKimbmgu6taGwkFfyU27fqm9WzRzO7MhO/g9HsPO9HxVEsef8l2iwIKsrHvPq+Wiv5fDgxam6Goks9vEajPlnFRHUcdrcCVHZaGwpNDNH0ai0MXGicyPp+V3mnFSTg1M5Cee+aUcMz7TMSWyORWvkcvzbQiammcmjnctz7buKxLiKfBAJ51FkuQT+pN7sDB8jJbxy9hV8aZ/4Le7J+KPaT6G8BLu8rYqfrt0vWEeuEViGJ8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jun 10, 2024 at 08:02:02PM -0700, Luis Chamberlain wrote: > Stress test folio splits by using the new debugfs interface to a target > a new smaller folio order while triggering writeback at the same time. > > This is known to only creates a crash with min order enabled, so for example > with a 16k block sized XFS test profile, an xarray fix for that is merged > already. This issue is fixed by kernel commit 2a0774c2886d ("XArray: set the > marks correctly when splitting an entry"). > > If inspecting more closely, you'll want to enable on your kernel boot: > > dyndbg='file mm/huge_memory.c +p' > > Since we want to race large folio splits we also augment the full test > output log $seqres.full with the test specific number of successful > splits from vmstat thp_split_page and thp_split_page_failed. The larger > the vmstat thp_split_page the more we stress test this path. > > This test reproduces a really hard to reproduce crash immediately. > > Signed-off-by: Luis Chamberlain > --- > common/rc | 14 +++++ > tests/generic/751 | 127 ++++++++++++++++++++++++++++++++++++++++++ > tests/generic/751.out | 2 + > 3 files changed, 143 insertions(+) > create mode 100755 tests/generic/751 > create mode 100644 tests/generic/751.out > > diff --git a/common/rc b/common/rc > index 30beef4e5c02..60f572108818 100644 > --- a/common/rc > +++ b/common/rc > @@ -158,6 +158,20 @@ _require_vm_compaction() > _notrun "Need compaction enabled CONFIG_COMPACTION=y" > fi > } > + > +# Requires CONFIG_DEBUGFS and truncation knobs > +_require_split_debugfs() Er... I thought "split" referred to debugfs itself. _require_split_huge_pages_knob? > +{ > + if [ ! -f $DEBUGFS_MNT/split_huge_pages ]; then > + _notrun "Needs CONFIG_DEBUGFS and split_huge_pages" > + fi > +} > + > +_split_huge_pages_all() > +{ > + echo 1 > $DEBUGFS_MNT/split_huge_pages > +} > + > # Get hugepagesize in bytes > _get_hugepagesize() > { > diff --git a/tests/generic/751 b/tests/generic/751 > new file mode 100755 > index 000000000000..7cc96054a5a9 > --- /dev/null > +++ b/tests/generic/751 > @@ -0,0 +1,127 @@ > +#! /bin/bash > +# SPDX-License-Identifier: GPL-2.0 > +# Copyright (C) 2024 Luis Chamberlain. All Rights Reserved. > +# > +# FS QA Test No. 751 > +# > +# stress page cache truncation + writeback > +# > +# This aims at trying to reproduce a difficult to reproduce bug found with > +# min order. The issue was root caused to an xarray bug when we split folios > +# to another order other than 0. This functionality is used to support min > +# order. The crash: > +# > +# https://gist.github.com/mcgrof/d12f586ec6ebe32b2472b5d634c397df You might want to paste the stacktrace in here directly, in case the gist ever goes away. > +# > +# This may also find future truncation bugs in the future, as truncating any > +# mapped file through the collateral of using echo 1 > split_huge_pages will > +# always respect the min order. Truncating to a larger order then is excercised > +# when this test is run against any filesystem LBS profile or an LBS device. > +# > +# If you're enabling this and want to check underneath the hood you may want to > +# enable: > +# > +# dyndbg='file mm/huge_memory.c +p' > +# > +# This tests aims at increasing the rate of successful truncations so we want > +# to increase the value of thp_split_page in $seqres.full. Using echo 1 > > +# split_huge_pages is extremely aggressive, and even accounts for anonymous > +# memory on a system, however we accept that tradeoff for the efficiency of > +# doing the work in-kernel for any mapped file too. Our general goal here is to > +# race with folio truncation + writeback. > + > +. ./common/preamble > + > +_begin_fstest auto long_rw stress soak smoketest > + > +# Override the default cleanup function. > +_cleanup() > +{ > + cd / > + rm -f $tmp.* > + rm -f $runfile > + kill -9 $split_huge_pages_files_pid > /dev/null 2>&1 > +} > + > +fio_config=$tmp.fio > +fio_out=$tmp.fio.out > + > +# real QA test starts here > +_supported_fs generic > +_require_test > +_require_scratch > +_require_debugfs > +_require_split_debugfs > +_require_command "$KILLALL_PROG" "killall" > +_fixed_by_git_commit kernel 2a0774c2886d \ > + "XArray: set the marks correctly when splitting an entry" > + > +# we need buffered IO to force truncation races with writeback in the > +# page cache > +cat >$fio_config < +[force_large_large_folio_parallel_writes] > +nrfiles=10 > +direct=0 > +bs=4M > +group_reporting=1 > +filesize=1GiB > +readwrite=write > +fallocate=none > +numjobs=$(nproc) > +directory=$SCRATCH_MNT > +runtime=100*${TIME_FACTOR} > +time_based > +EOF > + > +_require_fio $fio_config > + > +echo "Silence is golden" > + > +_scratch_mkfs >>$seqres.full 2>&1 > +_scratch_mount >> $seqres.full 2>&1 > + > +# used to let our loops know when to stop > +runfile="$tmp.keep.running.loop" > +touch $runfile > + > +# The background ops are out of bounds, the goal is to race with fsstress. > + > +# Force folio split if possible, this seems to be screaming for MADV_NOHUGEPAGE > +# for large folios. > +while [ -e $runfile ]; do > + _split_huge_pages_all >/dev/null 2>&1 > +done & > +split_huge_pages_files_pid=$! > + > +split_count_before=0 > +split_count_failed_before=0 > + > +if grep -q thp_split_page /proc/vmstat; then > + split_count_before=$(grep ^thp_split_page /proc/vmstat | head -1 | awk '{print $2}') > + split_count_failed_before=$(grep ^thp_split_page_failed /proc/vmstat | head -1 | awk '{print $2}') > +else > + echo "no thp_split_page in /proc/vmstat" >> $seqres.full > +fi > + > +# we blast away with large writes to force large folio writes when > +# possible. > +echo -e "Running fio with config:\n" >> $seqres.full > +cat $fio_config >> $seqres.full > +$FIO_PROG $fio_config --alloc-size=$(( $(nproc) * 8192 )) --output=$fio_out > + > +rm -f $runfile > + > +wait > /dev/null 2>&1 > + > +if grep -q thp_split_page /proc/vmstat; then > + split_count_after=$(grep ^thp_split_page /proc/vmstat | head -1 | awk '{print $2}') > + split_count_failed_after=$(grep ^thp_split_page_failed /proc/vmstat | head -1 | awk '{print $2}') I think this ought to be a separate function for cleanliness? _proc_vmstat() { awk -v name="$1" '{if ($1 ~ name) {print($2)}}' /proc/vmstat } split_count_after=$(_proc_vmstat thp_split_page) Otherwise this test looks fine to me. --D > + thp_split_page=$((split_count_after - split_count_before)) > + thp_split_page_failed=$((split_count_failed_after - split_count_failed_before)) > + > + echo "vmstat thp_split_page: $thp_split_page" >> $seqres.full > + echo "vmstat thp_split_page_failed: $thp_split_page_failed" >> $seqres.full > +fi > + > +status=0 > +exit > diff --git a/tests/generic/751.out b/tests/generic/751.out > new file mode 100644 > index 000000000000..6479fa6f1404 > --- /dev/null > +++ b/tests/generic/751.out > @@ -0,0 +1,2 @@ > +QA output created by 751 > +Silence is golden > -- > 2.43.0 > >