From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99220C3DA4A for ; Tue, 20 Aug 2024 09:22:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 154D86B007B; Tue, 20 Aug 2024 05:22:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 104CB6B0082; Tue, 20 Aug 2024 05:22:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE8D96B0083; Tue, 20 Aug 2024 05:22:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CFA5D6B007B for ; Tue, 20 Aug 2024 05:22:29 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 59277A7D72 for ; Tue, 20 Aug 2024 09:22:29 +0000 (UTC) X-FDA: 82472083218.05.95B7EE1 Received: from mail-vk1-f171.google.com (mail-vk1-f171.google.com [209.85.221.171]) by imf19.hostedemail.com (Postfix) with ESMTP id 89AD61A0014 for ; Tue, 20 Aug 2024 09:22:27 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="RzjxDuT/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724145683; a=rsa-sha256; cv=none; b=JSRAfleybRolPcbBMeLxKyoewvif4XPdSr6u13+8/WRpD/ZjYPGPiJyG5lrRsWw7LHDhJ3 72yCg/GY/0yoMXD1B+N31WaHlh8VjgnhpugnxpkwCTkC2PVkeoXpHH8WZ4QHmkGdL/WZ4a +9DQpzE0Yu1KY0tvmxWGSswVDvw8xeo= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="RzjxDuT/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf19.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724145683; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=15otqUFjTUzokqAb6iLvUFCNyLARihL5UIkmuZY7HYU=; b=lAF9Jj9gl2AqEd/+aR8Qbo20fD2fyXNrj6oR+lgX4gmZgfIA2uKV1aHKlDyjAhFg5xmJbn D4EvACcy9AEOv2iN5P3raKiwXFm11QrJqi/n+fNaYoG+0apuWZdA0KIivZjYuUT1leK1R3 HfGD6ZNQuJUZ2bDlqF6Mp/UOZ0C6+IE= Received: by mail-vk1-f171.google.com with SMTP id 71dfb90a1353d-4f51c1f9372so2089783e0c.2 for ; Tue, 20 Aug 2024 02:22:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724145746; x=1724750546; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=15otqUFjTUzokqAb6iLvUFCNyLARihL5UIkmuZY7HYU=; b=RzjxDuT/AVC2aWw+tpayQ4RpD0xLg6/88m6jrvpxhh3zFD0XPczOz8gYFTFNINoG8k /26XAQo64CAgbq6l4ptcjqJi+ckjmIOrCa4Lf3LypkswYvVZeu1QzSQfYe5mPsTYoVUg ElDGO2O6827jSpzRX2TqbYbDzHZLWyj2IcWj45H5YLnsWn70IPkZ37pmec0rNa+avK2y D3aVfVm4wCokInrMoXw9qjbzeFjg4jJ/Ixx3TOKzQ9loIqL71aveCX6x5PtARdslLAw1 diQoUP1tHTmBddvOK64xpEBv6TwZH59so+9rW+ykcticy+e43Zme3zryu9rH73nwhZ3/ OP3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724145746; x=1724750546; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=15otqUFjTUzokqAb6iLvUFCNyLARihL5UIkmuZY7HYU=; b=vpyNPIK2Zp58d7tk3pZlUByY4FY4J/SZuEG/YTAl7fxbsRgPZYnfxnsJSedtfr3pWV dlQLnIwC4n6SOz+psKxonLsFqcQwQcZC6ES1c0/KEC725jcc1/07TYgITyp59q5BRrCf 7UHAzVf1PPWaSoAvh0mH6JidKr2TGnWU2UNLXRmRBkT591COooz89dLxRb+fzrkzka++ DkRRNNIP6gBM8tJCTj1LZAhN8hgFQ8SwwexS+lIHIWfk1vHRvj40iCbhq36EJQm/aud8 Wy1asRKScDysFu5HVkBc7WpxvQL4Jw23rW14CBzo56zSmPBNYdhD5uFFzsTgdvSUBe1Q w/GA== X-Forwarded-Encrypted: i=1; AJvYcCXCJ2QbfqJojRiYxjjN+Qc8qWM1ELqLTtXJxt04iKaP7altAhOCBpcW4PUVRumXAa8vsKCtUc00Tw==@kvack.org X-Gm-Message-State: AOJu0Yyl0ug6wBFLDbdFKtnH/RdkWvtfh2KaiqNos3BvjsLP6Z97yBXY xGt7I90LQLoG8h6Qd7cUAoytpc2kWtHaYAbofu+y0vH0ulSpYJQy0Yv4e2gmF5k8bf9E4kOz0Ev QKEgchvaOe/QlC0s2hHSYgW/RwbU= X-Google-Smtp-Source: AGHT+IG6xMyVhxSBrAmk+Z9rfB7BHDrPYhv1pDnr8OcABRKIMt1AeghI833YCaDOX3iF2O0i/x0PClR0DjCeOndcOcU= X-Received: by 2002:a05:6122:32cb:b0:4f5:20fb:4e46 with SMTP id 71dfb90a1353d-4fc6c721edcmr15940462e0c.6.1724145746510; Tue, 20 Aug 2024 02:22:26 -0700 (PDT) MIME-Version: 1.0 References: <00000000000060cf79061fd24ca8@google.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Tue, 20 Aug 2024 21:22:15 +1200 Message-ID: Subject: Re: [syzbot] [mm?] WARNING in zswap_swapoff To: Kairui Song Cc: Yosry Ahmed , syzbot , akpm@linux-foundation.org, chengming.zhou@linux.dev, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, syzkaller-bugs@googlegroups.com, Chris Li , Ying , Ryan Roberts Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 89AD61A0014 X-Stat-Signature: cw1kkdej1qcg1myz358y7spxh3w5zs9f X-Rspam-User: X-HE-Tag: 1724145747-264712 X-HE-Meta: U2FsdGVkX1/vl6mbBT1ZOj3Qg76QqOhNOQqJoK2qnxoXW8CTdlgTMi3NFfNKuejOrynfeATleE7MPcVm8qclxnuArivZ3j6A7UN0cE6PiIT09VeehoprLfzhnBLoofz9Ark3CE+9vNcmqyglcP8uJ8KS/aRzVNq6EwMJlG+0tEHdOchUoJP/5OE6cWFwPx0syxnmSKx/ZodyytAU/q+QkhHTh3FQFbyVIIepymPcUvzdIaPy4abCkQi7QCyxU7+8q+FUPKPPovqheIaw/p68KhwCg5XUm2euGiWcdy7uX/ygd8qYXhwd//6HarWyvNv8+XK2TW3xW0gd/YiYNQKFIcfY5WjEcENr2erQQSH0T7qDZw+0zbVFw1txVc2VlxBTpTt0RijqGX/9Aes3QQetYpsa7JXq53angdIXFO8Gb753WgfhFDxbEknTb6bMluz7zDYQbu3AqyTrHWYU4dJLM1D6QaDZ9WosBSMnfhDaOMcd/kIq98kfS1GBuvGo7lMy21VtvnuR3mJiAMP1I6HcsRFg3D2PzVlAX/7aLHPPYZLzUpATVIQP8tsTi3hPEbQIGIP3EUoSgBaynwLF9JYPhcwK85tVOaRiN6sgC5HCZVr3oMEvJ35K4clH4jwwmP2MLJ403JJuFCqPlBc/I/42hzLzGg54H3yOcdajiKAwN4KscrBriNxBaosbMdgKk/CMckng0c45sX8SkHDCMAL7i9FYG41WQMt7Us3P2yW2aQ+XFZy1r0s3q3ruwC/2IbeVyHoaKQwc7aqSkMVIuTecF/HhR4sL+/euSfIX9ysio4FAc1J9rRg4B2XPgh4GfmiMOBnubqTdZwoMLXzhOldeog6gUbuD67QyBZVAg/s99tgD4BDF1ntfvzIrfgyULa7OJlXwrWaxZQ3eSyNYG/DCDKYT27Kd+/ERggKHjdlinpR/CaXUqn4Rok/jPQ2VdC2b59BQCEgLWsOKG9lkerd v30Pbdi/ QlP/9UHfGsF037iIGQNTikjjz+/W6IAoRifb5jbDu3pmGrPnALSnQJc5qHUhrkH7K+iFgCrDs7/yBhejFjFMAv+GXrEfCCg+04A6+h8hzUKfryRPmZl51UnTCV2Ad3sTeyUtI2rE9izj1y8hKGLuohwwb6R9SyoJc9grAx5+0Is5NSudwBFYizMHfQegxmCIEo1x7LZmT9Xu2tatIJJU6onei6LOnsowQOsbo+FYPEdTNSZt5kwnXdr0l0bDDTVw2UoFvJ9N8rNBhjd/Tm95RkP2NaiYQzqNeZEbAa3K+He0CL++KjXg8F1/7gj2ZTnW+rk0g/4NnO3JzRzh/HwALcvstMAKiYT0D+BvUv3l7bZzI2YAjVDZFLQBilQDU/a0t5yFot6q5li+x+CaZbX2PXnjPzOhFAJlCIbl6RWLhWYGCivF2LPHTq+kTZgdgysRoLrW9gNdl5KLl3IQNEZfytDjaaXAFUPkzEmanCNY4Mq6+dyd6VrOAo7QTUPs9c9VnZzKoQMTnuNc/t6GS5mE/pjrRa8ST2dhDrxcLEGzIMhBE3KWZh+nbUiA30B0Jg6M4waP06BAnDOBfo/PCznQjrryMSB4eyWLFF69KKk6/3C1fB9/V+mf20NW/sKmUsDPEtcYIGhz48IGeAxVkKQhmRRr+zZyZZn2LjNQLl70AkWkPqWsTeUGDDlmj+2ISa6IRRbUwQE3EIDC59ysmrf8pWfmvFVq3whC8g9jxQJwTlRsW6jH942lstsq5J+O95q/ALLZ0FevERqx3kysYs0IAazpSDeJQ4LmRVoLkAhjefSK/fDmyvf4dSmXjrBBuVGVTttubsFx+ZeTmWqebCjYJiudH9FEEwzjFsgRjRyNg/AJQJdyVZTuGJvnbtRku2LElfCgJnHNYUouR5qIxeMUHobPUgkymAoQ8u/j085yCs0hV5Uxdca0UJO0FMuBAbvaJAXB6ipu/vfKbeq2MCsg+2aeshzJb Wp4LCbK+ TPzN7FOxQn7bVZa4ATp1RpUObs+IN17RIR4DZ4TVOCI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Aug 20, 2024 at 8:47=E2=80=AFPM Kairui Song wrot= e: > > On Tue, Aug 20, 2024 at 4:13=E2=80=AFAM Yosry Ahmed wrote: > > On Fri, Aug 16, 2024 at 12:52=E2=80=AFPM syzbot > > wrote: > > > > > > Hello, > > > > > > syzbot found the following issue on: > > > > > > HEAD commit: 367b5c3d53e5 Add linux-next specific files for 202408= 16 > > I can't find this commit, seems this commit is not in linux-next any more= ? > > > > git tree: linux-next > > > console output: https://syzkaller.appspot.com/x/log.txt?x=3D124891059= 80000 > > > kernel config: https://syzkaller.appspot.com/x/.config?x=3D61ba6f3b2= 2ee5467 > > > dashboard link: https://syzkaller.appspot.com/bug?extid=3Dce6029250d7= fd4d0476d > > > compiler: Debian clang version 15.0.6, GNU ld (GNU Binutils for= Debian) 2.40 > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > Downloadable assets: > > > disk image: https://storage.googleapis.com/syzbot-assets/0b1b4e3cad3c= /disk-367b5c3d.raw.xz > > > vmlinux: https://storage.googleapis.com/syzbot-assets/5bb090f7813c/vm= linux-367b5c3d.xz > > > kernel image: https://storage.googleapis.com/syzbot-assets/6674cb0709= b1/bzImage-367b5c3d.xz > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the = commit: > > > Reported-by: syzbot+ce6029250d7fd4d0476d@syzkaller.appspotmail.com > > > > > > ------------[ cut here ]------------ > > > WARNING: CPU: 0 PID: 11298 at mm/zswap.c:1700 zswap_swapoff+0x11b/0x2= b0 mm/zswap.c:1700 > > > Modules linked in: > > > CPU: 0 UID: 0 PID: 11298 Comm: swapoff Not tainted 6.11.0-rc3-next-20= 240816-syzkaller #0 > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BI= OS Google 06/27/2024 > > > RIP: 0010:zswap_swapoff+0x11b/0x2b0 mm/zswap.c:1700 > > > Code: 74 05 e8 78 73 07 00 4b 83 7c 35 00 00 75 15 e8 1b bd 9e ff 48 = ff c5 49 83 c6 50 83 7c 24 0c 17 76 9b eb 24 e8 06 bd 9e ff 90 <0f> 0b 90 e= b e5 48 8b 0c 24 80 e1 07 80 c1 03 38 c1 7c 90 48 8b 3c > > > RSP: 0018:ffffc9000302fa38 EFLAGS: 00010293 > > > RAX: ffffffff81f4d66a RBX: dffffc0000000000 RCX: ffff88802c19bc00 > > > RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffff888015986248 > > > RBP: 0000000000000000 R08: ffffffff81f4d620 R09: 1ffffffff1d476ac > > > R10: dffffc0000000000 R11: fffffbfff1d476ad R12: dffffc0000000000 > > > R13: ffff888015986200 R14: 0000000000000048 R15: 0000000000000002 > > > FS: 00007f9e628a5380(0000) GS:ffff8880b9000000(0000) knlGS:000000000= 0000000 > > > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > CR2: 0000001b30f15ff8 CR3: 000000006c5f0000 CR4: 00000000003506f0 > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > Call Trace: > > > > > > __do_sys_swapoff mm/swapfile.c:2837 [inline] > > > __se_sys_swapoff+0x4653/0x4cf0 mm/swapfile.c:2706 > > > do_syscall_x64 arch/x86/entry/common.c:52 [inline] > > > do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 > > > entry_SYSCALL_64_after_hwframe+0x77/0x7f > > > RIP: 0033:0x7f9e629feb37 > > > Code: 73 01 c3 48 8b 0d f1 52 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 = 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 a8 00 00 00 0f 05 <48> 3d 01 f= 0 ff ff 73 01 c3 48 8b 0d c1 52 0d 00 f7 d8 64 89 01 48 > > > RSP: 002b:00007fff17734f68 EFLAGS: 00000246 ORIG_RAX: 00000000000000a= 8 > > > RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9e629feb37 > > > RDX: 00007f9e62a9e7e8 RSI: 00007f9e62b9beed RDI: 0000563090942a20 > > > RBP: 0000563090942a20 R08: 0000000000000000 R09: 77872e07ed164f94 > > > R10: 000000000000001f R11: 0000000000000246 R12: 00007fff17735188 > > > R13: 00005630909422a0 R14: 0000563073724169 R15: 00007f9e62bdda80 > > > > > > > I am hoping syzbot would find a reproducer and bisect this for us. > > Meanwhile, from a high-level it looks to me like we are missing a > > zswap_invalidate() call in some paths. > > > > If I have to guess, I would say it's related to the latest mTHP swap > > changes, but I am not following closely. Perhaps one of the following > > things happened: > > > > (1) We are not calling zswap_invalidate() in some invalidation paths. > > It used to not be called for the cluster freeing path, so maybe we end > > up with some order-0 swap entries in a cluster? or maybe there is an > > entirely new invalidation path that does not go through > > free_swap_slot() for order-0 entries? > > > > (2) Some higher order swap entries (i.e. a cluster) end up in zswap > > somehow. zswap_store() has a warning to cover that though. Maybe > > somehow some swap entries are allocated as a cluster, but then pages > > are swapped out one-by-one as order-0 (which can go to zswap), but > > then we still free the swap entries as a cluster? > > Hi Yosry, thanks for the report. > > There are many mTHP related optimizations recently, for this problem I > can reproduce this locally. Can confirm the problem is gone for me > after reverting: > > "mm: attempt to batch free swap entries for zap_pte_range()" > > Hi Barry, > > If a set of continuous slots are having the same value, they are > considered a mTHP and freed, bypassing the slot cache, and causing > zswap leak. > This didn't happen in put_swap_folio because that function is > expecting an actual mTHP folio behind the slots but > free_swap_and_cache_nr is simply walking the slots. Hi Kairui, I don't understand, if anyone has a folio backend, the code will go fallback to __try_to_reclaim_swap(), it won't call swap_entry_range_free(). ci =3D lock_cluster_or_swap_info(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { unlock_cluster_or_swap_info(si, ci); goto fallback; } for (i =3D 0; i < nr; i++) WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); unlock_cluster_or_swap_info(si, ci); if (!has_cache) { spin_lock(&si->lock); swap_entry_range_free(si, entry, nr); spin_unlock(&si->lock); } return has_cache; Am i missing something? > > For the testing, I actually have to disable mTHP, because linux-next > will panic with mTHP due to lack of following fixes: > https://lore.kernel.org/linux-mm/a4b1b34f-0d8c-490d-ab00-eaedbf3fe780@gma= il.com/ > https://lore.kernel.org/linux-mm/403b7f3c-6e5b-4030-ab1c-3198f36e3f73@gma= il.com/ > > > > > I am not closely following the latest changes so I am not sure. CCing > > folks who have done work in that area recently. > > > > I am starting to think maybe it would be more reliable to just call > > zswap_invalidate() for all freed swap entries anyway. Would that be > > too expensive? We used to do that before the zswap_invalidate() call > > was moved by commit 0827a1fb143f ("mm/zswap: invalidate zswap entry > > when swap entry free"), and that was before we started using the > > xarray (so it was arguably worse than it would be now). > > > > That might be a good idea, I suggest moving zswap_invalidate to > swap_range_free and call it for every freed slot. > > Below patch can be squash into or put before "mm: attempt to batch > free swap entries for zap_pte_range()".