From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09FF7C636CD for ; Sun, 5 Feb 2023 14:38:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD1FC6B0072; Sun, 5 Feb 2023 09:38:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D5ABB6B0073; Sun, 5 Feb 2023 09:38:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFB3E6B0074; Sun, 5 Feb 2023 09:38:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A865C6B0072 for ; Sun, 5 Feb 2023 09:38:57 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7B578140109 for ; Sun, 5 Feb 2023 14:38:57 +0000 (UTC) X-FDA: 80433495114.27.1847597 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 9CE8340011 for ; Sun, 5 Feb 2023 14:38:55 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bxjgROKM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675607935; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wr9Yn75E/tlpsd1l3eBqYyWcWboYor9r5ZBi3BKnUJM=; b=VFl3Ln5cGWrkqJFdoVIlohZuXBncuRvQ/HSOuTwH02WbV2olrw8qq3Ed+iJV3crca+llel JeYccaNi8E9c4fsKO7AhHT4hpHFEA6yrYKegjbFuFGPQrlklGQD0hBjt9nUpaX+fYujquz OVMmXsjmN9SlTE2fWTIp6S0l3TSZhY8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bxjgROKM; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf12.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675607935; a=rsa-sha256; cv=none; b=PVeG7l6hcoewkTBKB6BxvJu1d23xvk9YsbjozFHpdprCMOYz5WrOr05T4KU5tGeM6MHJ5Z uZsuqznxT2c05G+dw2yFzFhfaAk4lyFZ+tnlDbAUmQnafRcqhtpKbB25yibWpO7RMgsz+G e5D1BHme9veNJQuFypkAq1dGFVpXc0E= Received: by mail-pf1-f172.google.com with SMTP id cr11so6728767pfb.1 for ; Sun, 05 Feb 2023 06:38:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=wr9Yn75E/tlpsd1l3eBqYyWcWboYor9r5ZBi3BKnUJM=; b=bxjgROKMmRGbHcQCNAGyH7yvxH47FgzUhWIILp30cGRKPFaFA8/3ikbbh+K/fiqfWQ vM4uC3+pYD0b+ScinnNxV18eB9AIv1F52OtExst8llURMRYng2o1ia7Yv3HiTtUPwMsn pA+HJ1OMmU8PqQLjC2Ko57vbkb9Hi0M8TKxuLirNvbs6DeIykkm90JP2Q6bdVlfq366S 6ShZXV/9Tgal2bbxTABwCX6ilRTChEIZTVJE8iEsWYJy2Yk5rJBMESfDeWo+DNJYbOOX yNLmDbmdFbTVr76tc4M49Xwnsv4HHWAb3vkVajsUJg8/hOu7ye2DOPKpchEaGKIcnTA3 BJQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=wr9Yn75E/tlpsd1l3eBqYyWcWboYor9r5ZBi3BKnUJM=; b=PfmDRwH1wDL2MYkipj7Nms1oGFunKrxOsi29aA0oWukne6+Kv76L3+i7bH+vZPE4bV TmYhlHAnfkuKQiX6M7yOST+7YBLSQBJpIQh06jkmDIhn3CNBxI28NoRHdHtO2ggVtetc uwaXSbLfVS9w/2iV6OY5hOnZy0kas6mxDNkI3jZ822OO/O/63cZ+/D6wKFKVPQ3khABp wHx3f9g6WZIo+/94MroC5LQV/PQ/75rWlyIF4gFJ7ezd2WLbZfr2N4vwHNGcHKzkMSa4 HCP1AIbfGvGDesDZ/RflIeE0W1Hm0KlnUnRjac8STYv2qotH/AP9hpX7FfUUD+BP4qYl QoyQ== X-Gm-Message-State: AO0yUKU8uqHrmHCsC7zq9VyYrQXPlS1h1QN3R00C6tUAdY8J+8RJEVD0 I+a4+bjHZgbiNnYohTrdPD0= X-Google-Smtp-Source: AK7set/FlechXxfb8i75gHmZnasZo2iiAx1e3MZtKgIDOgE+rXBsCwyU8Pkf8J7yaXJm/zFvVCn8dg== X-Received: by 2002:aa7:91d2:0:b0:581:1c:82a3 with SMTP id z18-20020aa791d2000000b00581001c82a3mr14509817pfa.1.1675607934293; Sun, 05 Feb 2023 06:38:54 -0800 (PST) Received: from localhost ([2400:8902::f03c:93ff:fe27:642a]) by smtp.gmail.com with ESMTPSA id h6-20020a056a00218600b0058772c6b228sm5207840pfi.116.2023.02.05.06.38.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 05 Feb 2023 06:38:53 -0800 (PST) Date: Sun, 5 Feb 2023 14:38:44 +0000 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: "Huang, Ying" Cc: linux-mm@kvack.org, akpm@linux-foundation.org, Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin , Minchan Kim Subject: Re: [BISECTED] first bad commit is c203c6d5b3f0597 ("migrate_pages: batch _unmap and _move") Message-ID: References: <87h6w5othj.fsf@yhuang6-desk2.ccr.corp.intel.com> <87wn4zmzd1.fsf@yhuang6-desk2.ccr.corp.intel.com> <874js2n65l.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <874js2n65l.fsf@yhuang6-desk2.ccr.corp.intel.com> X-Rspamd-Queue-Id: 9CE8340011 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: e39ducd47mf7xgn7wttp4y4ifdni3kqr X-HE-Tag: 1675607935-453255 X-HE-Meta: U2FsdGVkX19lVHI5NzW6FZ0pDbv5wSaqOH9cwTa0eBJxdEbPa915pPDInRwmzSuGVd1pZ88OCInc3eDGBm4M3z+ZdiifMpHeFPCkKdbxpSj6CL7M4IikBjzAFOs451/EfcHddAT9k0MGHj+qvECt0bAu8SleO+G6lVXnoFlDLtt2Ao8qKRdsVMUk2lOr0mgDJ+rBq/MDBKtVByXoWxVJSnxasjDhZSEUiTrXCldJDypX5/38vbuMKS3rHoJi/xJbPI76ei778x7WThAUmM+/tCZHO6PZWNylf9UrOlrQa7d6mQKJnlVWSaXEBNYCGc9Lc8i5x7iNJCll7MbarohKqDxfd3qLbdQvczyhQUG0+VAGWky51Cdk3jRtvII/JMGtXiiVSzFSMVRXvDD9X5lxvgugHKGgHHj07A/UY2nNMaVWme7BawXK+XFUEvEbc44/omAIDigMKK27VjFqVe+435wnSoBNuj9cKWabyhQ1cff0d+IQX1Op8IdqSfzKljBXYgFvX7lb/HDZREnxEhd1LGOw1Mo7fwUX+Sez0sl7kK6z74oBNvTOyv9a3492OQA2XdSma4ULWOSBM1MD+rHvdjsVIW0PTYIZdfjeGOH4CPBKd4ZDuk5uF9QWtkhhlrYZNNlrvj2fLkgqGxkwKxdUTmW5DfNfXUUM8513sEzBUtJTUJVzrW6jUPOEH1zK9kFbKUxP1+C3jRp39w3eJttaexcMluMIvkDHubWQxv2eXohKZJ4B5ekZ3TQcmTZJGmTcdfK2czhJNx0c4GUxqxpgwElzs+9ocRUCL6z/qI32qGQACA0xoNrCAqVOxxLo6fICv3K6evmAv97RcCG4rQGuXGyC8gyF4NYSrNpcvRcQu384GKCk2MM6AOnyNlzwlGpmAW5XllqvOLuIbPMcWvMZn4hZlJxeeQJWm+sqQ9dUzLpe3I8JisssJOR2ogKPte3hBkKoYdY6J76cml9qszs 9KyYDJsf eF7p0+HaHmqGSsEq/YrDclI8dFZZSZestPL7/h5QpoO6cFkgjuxk71cqVmbPWFmFtj2B9gOQbdPKEi8KXXJjDHX8VDba1pSvkjnm6W1sWrxUgFJXaEVy1F+WalPYsUgw6vt2ZhKv8C5zP/aRcS8W8NbmfA3A7pTpZHNuCCaMIMPeu5In40BUve9qSdA+OnxYEzbVCKT0mUi6h+hH+v4JVaz2tM12DspaZyzSi87XRVITibKoc0vriTEPrcPimVo2p2jxiRMYaqgZo6hNCeZZSn5aiixJ7kNnchsEzof52TGZzRA3LGjHSEhGJDYq0bO6ta/y82OCxMjbBwdP51EyCobqdCNAWwgmly+od/3vtlI8kbDLJxG9x8yCSphl18FNGgw43yZaEbmED7YPrbJDLinrHMWyNsrwVzfCHPOrbQ54NZB49e8UGGdd8cSAYJqPvZDfPIskKATF2PQ33iGFJFte32ey1A/LM4D/yo8OYJ7Ei1H3+JrNzuzcOZ39tgR8MXlz3grS84WPELaGW77xv+QP6X1EgAsY7zU1SJtcOHm4gjU0ey97Qdr051vNU4VdCB7gSyNr1/ayhpdFzysxh/JYyvYvLW7zMUzwxXNOCUxVyEd5MrkpgFOuJagXkmVvxXmT56SZA8wyUDD4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Feb 03, 2023 at 11:02:46PM +0800, Huang, Ying wrote: > Hyeonggon Yoo <42.hyeyoo@gmail.com> writes: > > > On Fri, Feb 03, 2023 at 07:17:14AM +0800, Huang, Ying wrote: > >> "Huang, Ying" writes: > >> > >> > Hi, Hyeonggon, > >> > > >> > Hyeonggon Yoo <42.hyeyoo@gmail.com> writes: > >> > > >> >> On Wed, Feb 01, 2023 at 01:09:10AM +0900, Hyeonggon Yoo wrote: > >> >>> I've observed random list_del corruption on mm-unstable, > >> >>> where HEAD is commit d69862e693c069f4 > >> >>> ("mm/migrate: convert putback_movable_pages() to use folios"). > >> >>> > >> >>> The issue can be easily reproduced by stressing MM multiple times: > >> >>> stress-ng --bigheap 0 --timeout 300 > >> >>> > >> >>> The compiler is gcc 12.2.1 and config, dmesg are included as attachment. > >> >>> I will try to bisect but can't promise quick resolution :) > >> >> > >> >> > >> >> The first bad commits appears to be: > >> >> c203c6d5b3f0597 ("migrate_pages: batch _unmap and _move") > >> >> > >> >> the first bad commit _probably_ be earlier, but this is quite > >> >> easy to reproduce so at this point I think above is the real bad commit. > >> > > >> > Thank you very much for reporting the bug. I'm in travel now but I will > >> > try to find some time to reproduce and debug it. > >> > >> Still haven't reproduced the issue. But after reviewing the code, I > >> found a bug in the code, which may cause list corruption. Can you try > >> the debug patch below? > > > > Unfortunately my home server seems to be broken again :( > > That means I only have access to VMs and not a real machine now. > > > > FYI it was not reproduced on KVM but reproduced on real machine. > > Could you try checking on your machine with the config I attached? [1] > > Thank you very much for testing! > > > Sorry to bother your travel! > > Never mind. Your report helps me very much! > > > [1] https://marc.info/?l=linux-mm&m=167518135116956 > > I have reproduced the bug successfully! And I can reproduce the bug > with the previous debug patch too, although the reproduction rate isn't > high. > > And in my test, the following patch can fix the bug. > > It appears that zswap code will touch dst->lru during moving page. After setting swap space I was able to reproduce even on VM. > -------------------------8<---------------------------------- > From b2e3f4aab16d8af0033286fde669b46e7467c7ec Mon Sep 17 00:00:00 2001 > From: Huang Ying > Date: Fri, 3 Feb 2023 22:03:24 +0800 > Subject: [PATCH] dbg,migrate_pages: restore destination folio state before > move > > --- > mm/migrate.c | 15 ++++++++------- > 1 file changed, 8 insertions(+), 7 deletions(-) This fixes the bug on my test: Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Thanks for such a quick fix! > > diff --git a/mm/migrate.c b/mm/migrate.c > index 143d96775b4d..fa7212330cb6 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1225,13 +1225,19 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, > int page_was_mapped = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__PageMovable(&src->page); > + struct list_head *prev; > > __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > + prev = dst->lru.prev; > + list_del(&dst->lru); BTW may be silly questions, - How can zswap touch dst->lru during moving page, is there no lock that prevents this to happen? - Does this race (?) happen only during moving page? (I mean, why is it safe to perform list_del()/list_add() before and after moving page?) > > rc = move_to_new_folio(dst, src, mode); > > - if (rc != -EAGAIN) > - list_del(&dst->lru); > + if (rc == -EAGAIN) { > + list_add(&dst->lru, prev); > + __migrate_folio_record(dst, page_was_mapped, anon_vma); > + return rc; > + } > > > if (unlikely(!is_lru)) > goto out_unlock_both; > @@ -1251,11 +1257,6 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, > lru_add_drain(); > } > > - if (rc == -EAGAIN) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > - return rc; > - } > - > if (page_was_mapped) > remove_migration_ptes(src, > rc == MIGRATEPAGE_SUCCESS ? dst : src, false); > -- > 2.35.1