From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E76D0C52D7C for ; Wed, 21 Aug 2024 05:49:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EFC576B0092; Wed, 21 Aug 2024 01:49:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EAC016B0093; Wed, 21 Aug 2024 01:49:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D4CA26B0095; Wed, 21 Aug 2024 01:49:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B86846B0092 for ; Wed, 21 Aug 2024 01:49:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 00DB8140939 for ; Wed, 21 Aug 2024 05:49:33 +0000 (UTC) X-FDA: 82475175468.14.1FA8120 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf28.hostedemail.com (Postfix) with ESMTP id 0DBC1C000B for ; Wed, 21 Aug 2024 05:49:31 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EwL2FANf; spf=pass (imf28.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724219292; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=epZcu6NqeywwM9Rp6su95R4Re49Q2nt588tKQWwC7ZM=; b=3gqwPHEoKFk0S2ZhQ28FkP9wl8ihDDHpc56kwl+OZPEQPuQkCzqgbeUBnzPlP37gWpV0iY xL2COyA0rWlJyBX+kTAYPxg2FdaKvm936czJHuORSYnyRSXjpkOLehndzuJRy3z7oN2gxK 4JO7HJckoAqOLEmgaMfJwcL2IHzEFFA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724219292; a=rsa-sha256; cv=none; b=lIyrwl5wg0k7ON2cRgjIaI9MtihFOBCWgbQNrNM9GRkgOtGNgzQAH1lprG8QLmsHDkX7WH 1CTLM5TqDs5R3LcbRs04z/gmQV9fWPlbf1e8NoA2EUFMxb/IclEx387BfaqmiHFkse48Ng hEbSXbuaw0ypc2oGAr9/ahwN4dKf+rM= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=EwL2FANf; spf=pass (imf28.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-7c3ebba7fbbso4876009a12.1 for ; Tue, 20 Aug 2024 22:49:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724219370; x=1724824170; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=epZcu6NqeywwM9Rp6su95R4Re49Q2nt588tKQWwC7ZM=; b=EwL2FANfft2P66Y6t03DMqIijib2JKbMNY/NuYpZGHLdwTXjUxDDEweWqYxoaWeZcT ORSIHIViy0TVZmt0x7e1UJs7DZIlJjwuu/6ebMiVArKdwpVU2Lg/1DaMmJk5/mBTwZVL y2AJUranMt79XUAIqzvdLNs0g9krKywGDp/kCaks0LVjq2QNg07we+IxOxU32Lb3RbBE UPUvzphNVNEp0ZGRi0aMhqosLWy6DgnxmwccgcqQile0TOUSzsZ+5BX4LcnbcHnZ8NTi ECQ7JySBVRjtDNdAKbVp10/p6Ju1u3VCtEclm9hBipr61bw1GVvlJEVuChmf4D/cmZn8 vsAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724219370; x=1724824170; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=epZcu6NqeywwM9Rp6su95R4Re49Q2nt588tKQWwC7ZM=; b=fKJEaH+XyMwypk6we0P49nc/Ek/wkSVs1LG/sQ/afnNoDGAGLud1fmV9kfUNpdvoI4 GkX0xw13Fy7PZeacVKih/UxV6JQCcxAH0fjobAVU3UcZ5lDQ3OEiUuZVlQadLwrMM2nN 2Ajke9f1x7/pJU5do+3iRwtLnzEpwwggEQeW2+GAWa8LBa6jkP51Uy66yEcIdcODQwHb 8O0HD3DHiOWc2/Qa4r1QaBIAnkCBl6DmN17awv9eGstOnugXDLSNClQZ42hV9g1RvPen 9mf0+9dyhP7VVwgHcFBITedXA/crXEYytd35gmv8KaMxb0e/W1RLTdS3sAzjOy5CTtoK F2rg== X-Forwarded-Encrypted: i=1; AJvYcCUe1fWkR0fr78DO0KiEBmFUXNPD2tWMecEpweGj0wkPrAngw130So2ghAcYUIbtJQjF7pRNe0U8fg==@kvack.org X-Gm-Message-State: AOJu0Yyshaq1m+I5Tzu6bTRWiIqUxIzsEkCgqIXX9zbcjTiHLleSG5Z3 Pcm4x7c1r6V8g/rzntMFuMbjK+EqMivGzdgqvKfNm0CXgT9tJtdh X-Google-Smtp-Source: AGHT+IEbb5CcwUtsPqQA8InXkVkQaWGp+zg2iM8CUmPCKElu58r8lQCccmP2IS2ZXjfGt70drxNiAQ== X-Received: by 2002:a05:6a20:9f0b:b0:1ca:cbf5:593 with SMTP id adf61e73a8af0-1cad7f8b771mr2137053637.20.1724219370327; Tue, 20 Aug 2024 22:49:30 -0700 (PDT) Received: from Barrys-MBP.hub ([2407:7000:8942:5500:9d62:3169:ab00:4dd1]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5c2e163a8sm1840894a91.1.2024.08.20.22.49.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 20 Aug 2024 22:49:29 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: ryncsn@gmail.com Cc: 21cnbao@gmail.com, akpm@linux-foundation.org, chengming.zhou@linux.dev, chrisl@kernel.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, ryan.roberts@arm.com, syzbot+ce6029250d7fd4d0476d@syzkaller.appspotmail.com, syzkaller-bugs@googlegroups.com, ying.huang@intel.com, yosryahmed@google.com Subject: Re: [syzbot] [mm?] WARNING in zswap_swapoff Date: Wed, 21 Aug 2024 17:49:21 +1200 Message-Id: <20240821054921.43468-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Stat-Signature: 68kp6qdnj67gu7kthtqceah8drhgfdk9 X-Rspamd-Queue-Id: 0DBC1C000B X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1724219371-190721 X-HE-Meta: U2FsdGVkX18xX47qsVFo7AJtrOn35GgbfL0B0p0045QjOxFryLubm3C+AIOcqhWC6CBP4zOo0cC9zH4QWKa8H144kYvrT46/eRmxxwlQfNGABo459CT2rc+sj7N2/qI2Cm6KQ/jg0qgB7OWuK2Pzp4YLbVg99rmXxEWK/1weaIq2mqlUMZcUkRCufmZxqOANqX7Gwj2FmNP/rx2bE8LVhzAvJ66oEwtL7P326yYLQlyVDM5JYEYzwF0QMmzK4iTty4sYudZFpnAJhkLn/WMpbTgh7XjAvEyfd6iV/wj0l17LkH2Y9UqXkIQ1DTa8UhsPboNei0OBi4mLo+ymzFvD+1EE9YQ1HldK1tm9hH+pGSLLQFiTvRONAda9TYgrANWqMO+I7fs6RsCrKBecR05UFHiJaWo/eX8Xaew5XzuLNCjhDTMzg3L7moDE3cjKNY4araeeKaz99/btbeShFf8kPsCUFWbskkh4Op1rdDIRCziUITjBmRUGhmLNpOAKXLrMaMZvyb+or3Y7JX0+bLn6LyPWxWwpH0gHlg94YZ3Q0IIzvrZ4csxGnX8Os6bYtzgmEjOuRGeD4zozY2X+/uyPeM+iZecg89Xs7UlxX6i/QYLt3J43GeuYgtr8B+Rgs1qLpF9Ib6/+ONaYgixVrf+8M3Uuz8e/sfuznJ+7AypxKh6eGS3ZIkLP+UcSJXFoOE/eurzizPI1J7PcnxrKofj+yZg/FpqBSy1oWYiLyyVH+2ZzcppuQ4QFmjxOjOGMk0C9lU6ALo5p5A2ajVw72Gv8HpY4tZJVLUiJypYPTethkHU1jHzYABA2z4ZaTzW+krzeeNvEPRSu9aHRDmOmSSqDLcLZU6gKvjTb1sbYbc4EAKhNj7r161MzS3gSKhIPZyj1FqmikmPeTJZmhG6Z2UajoXi/WCgqkHZlfWp/kdTUrj8Rx57EnjD5dkNfDglv8V0EpQRD8xdgQ8X2xaGOz9q u7PCCKQp HlRAd1HcDTGSz9OMVJRSniBlfW728kS/0K0QotFDgiEtSHVVI5s9bD+PodsjO1NeX91Bec7XAaXsnidQYdco+lnnlPFRMJHruqYkRNpoYQqKRC8zzDu+qXJ6K3vVVUBQIXYdQxkwkKjIUJpLViuqj7zcWkP6BVTrMzNAFm3MDlJLagocdcuzaf2yGEbOxcg7TmPekcVpH80NuJyjHGWR8AsRke9MU4iw6dNDDvLU/juEl0QvASdWgkEN4NviEHrCl9698XL58Z2gXn+w9dcF4Pn350E6B8fs/Y7W+Y87ryKc/IwQBR+cEdKwXGZhdt0PKDGuHxJD8xy89JOOHyU26yOKb0u3FtNVShoOjWDJzBfpQdFerhzMIlYzO+jbGgW/RedLGNScz8Uf5AvZ5FiQ8q3bKRO1FZA+UkMq+D7YxCaP55BHPSNMS4212MIod6eJZlzGcws8uieG39ULaOkiJbjC6c2FoQKEZJklXPE2zb0PDBOluPEuWlLfvr/1YAuRLxSZWsYm3ju1B6cY8Y1gk/fBuZ7gQuj3RwxqhbmVESot3sfMIozfS7nxT/IEnb4LXLdSv6tMivegA7SsDiogR/5IVI4OgPuGvYwdntrZH/HrFMrNib39e0+1bUsK5mJwlhU4TUPZF4PGjGw/0MiZTqcn9+Bb9+U3p6EgInOUI5YiP3bZhoHDSHHQl0MARvXyAZrf0Yg8ekVwbmzJPgw6bNoGiVn3JvZecgKpZDO0nZz18xiSjJk+9DCxDGjurTjr8doyRqofzuX5PUySE2D6vDhtYxw+M0Ejhbt9U/ECr5Gl1tICidZUVwrZnEHHVYhMw27huY/Rm3jC7VHD3j279shX0cfMi+EabX4fk4hamJcu6XOE40h8SnQB24aLnh0XdjRh20Q+vcQSugyAKFsEUiRIByRy9UIItLDoefeA/W7nfgGhJESIVAqn3TRSMR5mWJSB9yLfkGmoNdj25uqIm+QP+hCFB bpiHA1gt LBFQYNupghIJ9kmWG/mIvrFM6s50qTXzGqpM0M7LNhD40YH4nTYEmBpwmXv4vTLf2hP7sRk8LfiqVH4QbuDr1JvncUA+wc7EyGDEX4Hi9q7GiaaGDPmbRw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Aug 20, 2024 at 9:02 PM Kairui Song wrote: > > On Tue, Aug 20, 2024 at 4:47 PM Kairui Song wrote: > > > > On Tue, Aug 20, 2024 at 4:13 AM Yosry Ahmed wrote: > > > On Fri, Aug 16, 2024 at 12:52 PM syzbot > > > wrote: > > > > > > > > Hello, > > > > > > > > syzbot found the following issue on: > > > > > > > > HEAD commit:    367b5c3d53e5 Add linux-next specific files for 20240816 > > > > I can't find this commit, seems this commit is not in linux-next any more? > > > > > > git tree:       linux-next > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=12489105980000 > > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=61ba6f3b22ee5467 > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=ce6029250d7fd4d0476d > > > > compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40 > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > Downloadable assets: > > > > disk image: https://storage.googleapis.com/syzbot-assets/0b1b4e3cad3c/disk-367b5c3d.raw.xz > > > > vmlinux: https://storage.googleapis.com/syzbot-assets/5bb090f7813c/vmlinux-367b5c3d.xz > > > > kernel image: https://storage.googleapis.com/syzbot-assets/6674cb0709b1/bzImage-367b5c3d.xz > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > Reported-by: syzbot+ce6029250d7fd4d0476d@syzkaller.appspotmail.com > > > > > > > > ------------[ cut here ]------------ > > > > WARNING: CPU: 0 PID: 11298 at mm/zswap.c:1700 zswap_swapoff+0x11b/0x2b0 mm/zswap.c:1700 > > > > Modules linked in: > > > > CPU: 0 UID: 0 PID: 11298 Comm: swapoff Not tainted 6.11.0-rc3-next-20240816-syzkaller #0 > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024 > > > > RIP: 0010:zswap_swapoff+0x11b/0x2b0 mm/zswap.c:1700 > > > > Code: 74 05 e8 78 73 07 00 4b 83 7c 35 00 00 75 15 e8 1b bd 9e ff 48 ff c5 49 83 c6 50 83 7c 24 0c 17 76 9b eb 24 e8 06 bd 9e ff 90 <0f> 0b 90 eb e5 48 8b 0c 24 80 e1 07 80 c1 03 38 c1 7c 90 48 8b 3c > > > > RSP: 0018:ffffc9000302fa38 EFLAGS: 00010293 > > > > RAX: ffffffff81f4d66a RBX: dffffc0000000000 RCX: ffff88802c19bc00 > > > > RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff888015986248 > > > > RBP: 0000000000000000 R08: ffffffff81f4d620 R09: 1ffffffff1d476ac > > > > R10: dffffc0000000000 R11: fffffbfff1d476ad R12: dffffc0000000000 > > > > R13: ffff888015986200 R14: 0000000000000048 R15: 0000000000000002 > > > > FS:  00007f9e628a5380(0000) GS:ffff8880b9000000(0000) knlGS:0000000000000000 > > > > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > > CR2: 0000001b30f15ff8 CR3: 000000006c5f0000 CR4: 00000000003506f0 > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > > Call Trace: > > > >   > > > >  __do_sys_swapoff mm/swapfile.c:2837 [inline] > > > >  __se_sys_swapoff+0x4653/0x4cf0 mm/swapfile.c:2706 > > > >  do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > >  do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > > >  entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > > RIP: 0033:0x7f9e629feb37 > > > > Code: 73 01 c3 48 8b 0d f1 52 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 a8 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c1 52 0d 00 f7 d8 64 89 01 48 > > > > RSP: 002b:00007fff17734f68 EFLAGS: 00000246 ORIG_RAX: 00000000000000a8 > > > > RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9e629feb37 > > > > RDX: 00007f9e62a9e7e8 RSI: 00007f9e62b9beed RDI: 0000563090942a20 > > > > RBP: 0000563090942a20 R08: 0000000000000000 R09: 77872e07ed164f94 > > > > R10: 000000000000001f R11: 0000000000000246 R12: 00007fff17735188 > > > > R13: 00005630909422a0 R14: 0000563073724169 R15: 00007f9e62bdda80 > > > >   > > > > > > I am hoping syzbot would find a reproducer and bisect this for us. > > > Meanwhile, from a high-level it looks to me like we are missing a > > > zswap_invalidate() call in some paths. > > > > > > If I have to guess, I would say it's related to the latest mTHP swap > > > changes, but I am not following closely. Perhaps one of the following > > > things happened: > > > > > > (1) We are not calling zswap_invalidate() in some invalidation paths. > > > It used to not be called for the cluster freeing path, so maybe we end > > > up with some order-0 swap entries in a cluster? or maybe there is an > > > entirely new invalidation path that does not go through > > > free_swap_slot() for order-0 entries? > > > > > > (2) Some higher order swap entries (i.e. a cluster) end up in zswap > > > somehow. zswap_store() has a warning to cover that though. Maybe > > > somehow some swap entries are allocated as a cluster, but then pages > > > are swapped out one-by-one as order-0 (which can go to zswap), but > > > then we still free the swap entries as a cluster? > > > > Hi Yosry, thanks for the report. > > > > There are many mTHP related optimizations recently, for this problem I > > can reproduce this locally. Can confirm the problem is gone for me > > after reverting: > > > > "mm: attempt to batch free swap entries for zap_pte_range()" > > > > Hi Barry, > > > > If a set of continuous slots are having the same value, they are > > considered a mTHP and freed, bypassing the slot cache, and causing > > zswap leak. > > This didn't happen in put_swap_folio because that function is > > expecting an actual mTHP folio behind the slots but > > free_swap_and_cache_nr is simply walking the slots. > > > > For the testing, I actually have to disable mTHP, because linux-next > > will panic with mTHP due to lack of following fixes: > > https://lore.kernel.org/linux-mm/a4b1b34f-0d8c-490d-ab00-eaedbf3fe780@gmail.com/ > > https://lore.kernel.org/linux-mm/403b7f3c-6e5b-4030-ab1c-3198f36e3f73@gmail.com/ > > > > > > > > I am not closely following the latest changes so I am not sure. CCing > > > folks who have done work in that area recently. > > > > > > I am starting to think maybe it would be more reliable to just call > > > zswap_invalidate() for all freed swap entries anyway. Would that be > > > too expensive? We used to do that before the zswap_invalidate() call > > > was moved by commit 0827a1fb143f ("mm/zswap: invalidate zswap entry > > > when swap entry free"), and that was before we started using the > > > xarray (so it was arguably worse than it would be now). > > > > > > > That might be a good idea, I suggest moving zswap_invalidate to > > swap_range_free and call it for every freed slot. > > > > Below patch can be squash into or put before "mm: attempt to batch > > free swap entries for zap_pte_range()". > > Hmm, on second thought, the commit message in the attachment commit > might be not suitable, current zswap_invalidate is also designed to > only work for order 0 ZSWAP, so things are not clean even after this. Kairui, what about the below? we don't touch the path of __try_to_reclaim_swap() where you have one folio backed? diff --git a/mm/swapfile.c b/mm/swapfile.c index c1638a009113..8ff58be40544 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1514,6 +1514,8 @@ static bool __swap_entries_free(struct swap_info_struct *si, unlock_cluster_or_swap_info(si, ci); if (!has_cache) { + for (i = 0; i < nr; i++) + zswap_invalidate(swp_entry(si->type, offset + i)); spin_lock(&si->lock); swap_entry_range_free(si, entry, nr); spin_unlock(&si->lock); > > And for performance, it will cause unnecessary heavier contention for > the mTHP page on ZSWAP Xarray. It does fix the leak though, please > ignore this fix, let's try find a better fix.