From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A24C54E58 for ; Fri, 15 Mar 2024 11:03:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 984198011D; Fri, 15 Mar 2024 07:03:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 93454800B4; Fri, 15 Mar 2024 07:03:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D54A8011D; Fri, 15 Mar 2024 07:03:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6BB45800B4 for ; Fri, 15 Mar 2024 07:03:26 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 28862A1398 for ; Fri, 15 Mar 2024 11:03:26 +0000 (UTC) X-FDA: 81898987212.07.5C4CA25 Received: from mail-lj1-f176.google.com (mail-lj1-f176.google.com [209.85.208.176]) by imf17.hostedemail.com (Postfix) with ESMTP id 6B9444000D for ; Fri, 15 Mar 2024 11:03:23 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dMTB8CCa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710500603; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YWruk/k7WtclqkhSotMzYwMbwCi19ShEXbXynD4hB6E=; b=dftLpv/rLk7JRS3cvmh7+r+VjmiPdlyaZtQwcsnRAcPx9CLOwqkIQ3rNPmlov3st4CwA+C nXhOSJyB3FCCOLiyaGT7QzCmwW5MG5U/CTYUzWdF98IainCmnvgdAnblbcz4okCaV7JcJF dqy44sLeeeEVJDKXuLEUPv9QgejBwDQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dMTB8CCa; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf17.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.208.176 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710500603; a=rsa-sha256; cv=none; b=6ADk8q997/Ht6xb1gmsrJn2gwviDcvADzbNmVqiOOkp9rPPeiSKOhsotXBza8XvxQZoXQB whO6dPLzqGQSquENe7UPNUrj/I7wnarkpWtj/4qIx2S9//nzKBmISpkLsIT9hBTT7pCTw2 CteTqTAaqYm/fFKDuE8QAl2YuM4IPmA= Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2d28051376eso26599961fa.0 for ; Fri, 15 Mar 2024 04:03:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1710500601; x=1711105401; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=YWruk/k7WtclqkhSotMzYwMbwCi19ShEXbXynD4hB6E=; b=dMTB8CCavn7es77zA7BoqzZVT4Q3PkZhW9FGY5wrzYmg2f6XNTdM8F/Xl5pcLMb1UH JB26+gepaJez34ruf10FbJrkcCowEYs2B+Ih7Skv/Je+WsKfGVpPMaeL0Aa5yIEEG5hi eMUSirEZL0XOvlYkr6rstY/Zj7m8/8tT8WBhuqegkwDWbVgel8sGnbhe2YCP0bOtDsZf Y3HDSaGUIVq5HOZ8GhpiM7yGSX8y3BwmQ+SmeL3nqVmJxDQio4lQEY13YMwoqDv4Bm/2 Y3j1yeqcDlDAQFx5wq5SrgygZuKrCWRZXZ5uW3VQDIK70wbYhrq1LJyUI1f06ZTTpA2Y Nkrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710500601; x=1711105401; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YWruk/k7WtclqkhSotMzYwMbwCi19ShEXbXynD4hB6E=; b=CzOYG10ZFK/gf8BLP9Fr+xZA4K/b8fR6B3PCh5bEamxNf6OpLMmmhP6xWIiVrWaPvO Cs2nxGt2V1mAr9i5F9pQkvIExOFzmzS43ajZ/hemePLv3Cv/HnbdsR5O12LucTwoLQ+/ 0ojD4KyVQsw3iEmXOkihTxbgY/UEdB5gBrsU1tRBB6U1rZ/ovHhuD7Qt46XHD3ssRoex FyatwggG9AaXUeGRz/B4bYCz29mCssy6xqFTV43o+eJdYDPIl4QLKo5eGwyp6+V9V0Oc ib18EQd77dWNyGG22GCqSme+kg5yrTnLpv1F8SnH+LYPtU8ibPCXmJKSkurICRx5po7B 2eTg== X-Forwarded-Encrypted: i=1; AJvYcCXivBWi4qSf/DsuoGEPKtkRnC6cRYU36eSgEdj/1UhGvVvX/y0RZIJAXIrDuQo3a/tRSFlsaKdt5CeL3MlK/gHvStc= X-Gm-Message-State: AOJu0Yy9knuQXATzMhzp3ppd3j3CBoY2+Zhl6XsfbVLn+0+vzfTh06fn 28UKGG96CM892EsZ/QKpTKEGeHNnBr1l8M+bcubxskKJYAGd9su3EdDJaQnYXufTzE4fFlBC0mw soo5pS9nrgASHpSiZjurMx+HSvo8= X-Google-Smtp-Source: AGHT+IHnEmXKh/25GaGXOP390WghXc8EBxlur0/Likjim+XiC8g7iQwzJhZtpvdzi7LgEnJje8V5yegfJGJkexcDn/Q= X-Received: by 2002:a19:3814:0:b0:513:c4b8:388e with SMTP id f20-20020a193814000000b00513c4b8388emr2151922lfa.56.1710500601039; Fri, 15 Mar 2024 04:03:21 -0700 (PDT) MIME-Version: 1.0 References: <20240314083921.1146937-1-zhaoyang.huang@unisoc.com> In-Reply-To: <20240314083921.1146937-1-zhaoyang.huang@unisoc.com> From: Zhaoyang Huang Date: Fri, 15 Mar 2024 19:03:09 +0800 Message-ID: Subject: Re: [PATCH] mm: fix a race scenario in folio_isolate_lru To: "zhaoyang.huang" Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, steve.kang@unisoc.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 6B9444000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: cmha5n69ezxwm5h7d4679jehapzsc8jq X-HE-Tag: 1710500603-27049 X-HE-Meta: U2FsdGVkX1/mZpkQ9umEOorXZNfe0+wJkoH39utV3Gq+bUlQLx2/ILxmjcGrAA0jF/OrvdBwazOFS3KDeR6o1Z5Vtv37R6FD71f29tvaNK/PMvO3YLR4yR8zwJTTh9D8mvHwssEYpOSRVaTEl7InCAvunPoxoN42iOXs3q74jSInfZqrMo4KR8REtVjEfpMMu9OvbgVN3lKDcRjeCIYXDo4kWrr1fvmCjRciQzBBh0TyIQlM5o4U9zj+7/1pew2XbDUXrbV/IWtS7H2dXZXDREx+/JN3F+9AV6cq70hAT3A1nx04WSuCicDd8iXEYDiesoUmMQHhb0IUePhZm598YVaqEjEkOXBo3mZomDAkfzDF4lQaJ8bkPbHoCH0leILnU3rP6QxdlxJvDfDrHUITPQxRLkdZ2y8ImxumuOYoFBSJH2Wf0zOVcevyc6paoZuHKOyVf3lUck6pub5XLVf5bgM6udb45QOjfjZ+V/6mXszMqNDVwj95E/qQGhZSQMQmcGllXlY8YAySzWBWcL3sm9igeEW6X7gVCanOmjRuZ4+GJNdbnTnNiPfRF6cYWL+TT6Zf8eGfVqP1n26iG1q2Uqnn2guhqUddd4FaD+gG7kANkh7iQeG5Q/ImC80ZDwGvGvvSBk031GkS8LPi/JU61lIpV9EolXauatIene+cxRy77gJjcCTInXl9Vg7PT7P39ggeY+PZKZuWLaqzSHBQK4Rw/0jxMTsA0EHV0dTWPy6DZEwbRFRrd+lbsDt6zuTzmAfy125YPZc2nGUFytnve4lRAeU+iXnoBElI07gPZywVqwA8otKB7HU1ikevdWdeSMg9LOQRpNSEiNzLYAi+K6K2UDNVOYmwoVH2aAiPxObsnSWEzeN8qYr/k1c9K8o+epuhgUUfr5xYpFgTrm8kl8y3avu9B/2YEVruY4O1MZkH0lYp/euR/HLS2ha/uprZySQPkjC41Fia1OtiimS wBlQIyGM B0u/oJmmxwZGTaSzmHwGGJS+J3w32U+NgVV8LH1tEN8XROWNua86VlSx2D6KQ0LSXEJsnWNyGrRVFsE0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000167, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 14, 2024 at 4:39=E2=80=AFPM zhaoyang.huang wrote: > > From: Zhaoyang Huang > > Panic[1] reported which is caused by lruvec->list break. Fix the race > between folio_isolate_lru and release_pages. > > race condition: > release_pages could meet a non-refered folio which escaped from being > deleted from LRU but add to another list_head > #0 folio_isolate_lru #1 release_pages > if (folio_test_clear_lru()) > if (folio_put_testzero()) > if (!folio_test_lru()) > > folio_get(folio) > list_add(&folio->lru,) > list_del(&folio->lru,) > > fix action 1: > have folio_get prior to folio_test_clear_lru is not enough as there > could be concurrent folio_put(filemap_remove_folios) to make > release_pages pass refcnt check and failed in delete from LRU > > #0 folio_isolate_lru #1 release_pages > folio_get(folio) > if (folio_test_clear_lru()) > if (folio_put_testzero()) > if (!folio_test_lru()) > > list_add(&folio->lru,) > list_del(&folio->lru,) correct the timing description of fix action 1, that is moving folio_get prior to folio_test_clear_lru, which can't guarantee it happens before folio_put_testzero of release_pages #0 folio_isolate_lru #1 release_pages BUG_ON(!folio_refcnt) if (folio_put_testzero()) folio_get(folio) if (folio_test_clear_lru()) if (!folio_test_lru()) list_add(&folio->lru,) list_del(&folio->lru,) > > fix action 2: > folio_test_clear_lru should be considered as part of critical section of > lruvec which require be within lruvec->lock. > > #0 folio_isolate_lru #1 release_pages > spin_lock(lruvec->lock) > folio_get(folio) > if (folio_test_clear_lru()) > list_del(&folio->lru,) > > spin_unlock(lruvec->lock) spin_lock(lruvec->lock) > if (folio_put_testzero()) > if (!folio_test_lru()) > list_add(&folio->lru,) > > [1] > [ 37.562326] pc : __list_del_entry_valid_or_report+0xec/0xf0 > [ 37.562344] lr : __list_del_entry_valid_or_report+0xec/0xf0 > [ 37.562351] sp : ffffffc085953990 > [ 37.562355] x29: ffffffc0859539d0 x28: ffffffc082144000 x27: 000000000= 000000f > [ 37.562367] x26: 000000000000000d x25: 000000000000000d x24: 00000000f= fffffff > [ 37.562377] x23: ffffffc085953a08 x22: ffffff8080389000 x21: ffffff808= 0389000 > [ 37.562388] x20: fffffffe05c54180 x19: ffffffc085953b30 x18: ffffffc08= 5989098 > [ 37.562399] x17: 20747562202c3838 x16: ffffffffffffffff x15: 000000000= 0000004 > [ 37.562409] x14: ffffff8176980000 x13: 0000000000007fff x12: 000000000= 0000003 > [ 37.562420] x11: 00000000ffff7fff x10: ffffffc0820f51c4 x9 : 53b71233d= 5d50e00 > [ 37.562431] x8 : 53b71233d5d50e00 x7 : ffffffc081161ff0 x6 : 000000000= 0000000 > [ 37.562441] x5 : 0000000000000001 x4 : 0000000000000001 x3 : 000000000= 0000000 > [ 37.562451] x2 : ffffff817f2c4178 x1 : ffffff817f2b71c8 x0 : 000000000= 000006d > [ 37.562461] Call trace: > [ 37.562465] __list_del_entry_valid_or_report+0xec/0xf0 > [ 37.562472] release_pages+0x410/0x4c0 > [ 37.562482] __folio_batch_release+0x34/0x4c > [ 37.562490] truncate_inode_pages_range+0x368/0x63c > [ 37.562497] truncate_inode_pages+0x14/0x24 > [ 37.562504] blkdev_flush_mapping+0x60/0x120 > [ 37.562513] blkdev_put+0x114/0x298 > [ 37.562520] blkdev_release+0x28/0x40 > [ 37.562526] __fput+0xf8/0x2a8 > [ 37.562533] ____fput+0x10/0x20 > [ 37.562539] task_work_run+0xc4/0xec > [ 37.562546] do_exit+0x32c/0xa3c > [ 37.562554] do_group_exit+0x98/0x9c > [ 37.562561] __arm64_sys_exit_group+0x18/0x1c > [ 37.562568] invoke_syscall+0x58/0x114 > [ 37.562575] el0_svc_common+0xac/0xe0 > [ 37.562582] do_el0_svc+0x1c/0x28 > [ 37.562588] el0_svc+0x50/0xe4 > [ 37.562593] el0t_64_sync_handler+0x68/0xbc > [ 37.562599] el0t_64_sync+0x1a8/0x1ac > > Signed-off-by: Zhaoyang Huang > --- > mm/swap.c | 25 +++++++++++++++++-------- > mm/vmscan.c | 25 +++++++++++++++++++------ > 2 files changed, 36 insertions(+), 14 deletions(-) > > diff --git a/mm/swap.c b/mm/swap.c > index cd8f0150ba3a..287cf7379927 100644 > --- a/mm/swap.c > +++ b/mm/swap.c > @@ -968,6 +968,7 @@ void release_pages(release_pages_arg arg, int nr) > > for (i =3D 0; i < nr; i++) { > struct folio *folio; > + struct lruvec *prev_lruvec; > > /* Turn any of the argument types into a folio */ > folio =3D page_folio(encoded_page_ptr(encoded[i])); > @@ -996,9 +997,24 @@ void release_pages(release_pages_arg arg, int nr) > free_zone_device_page(&folio->page); > continue; > } > + /* > + * lruvec->lock need to be prior to folio_put_testzero to > + * prevent race with folio_isolate_lru > + */ > + prev_lruvec =3D lruvec; > + lruvec =3D folio_lruvec_relock_irqsave(folio, lruvec, > + &flags); > + > + if (prev_lruvec !=3D lruvec) > + lock_batch =3D 0; > > - if (!folio_put_testzero(folio)) > + if (!folio_put_testzero(folio)) { > + if (lruvec) { > + unlock_page_lruvec_irqrestore(lruvec, fla= gs); > + lruvec =3D NULL; > + } > continue; > + } > > if (folio_test_large(folio)) { > if (lruvec) { > @@ -1010,13 +1026,6 @@ void release_pages(release_pages_arg arg, int nr) > } > > if (folio_test_lru(folio)) { > - struct lruvec *prev_lruvec =3D lruvec; > - > - lruvec =3D folio_lruvec_relock_irqsave(folio, lru= vec, > - &= flags); > - if (prev_lruvec !=3D lruvec) > - lock_batch =3D 0; > - > lruvec_del_folio(lruvec, folio); > __folio_clear_lru_flags(folio); > } > diff --git a/mm/vmscan.c b/mm/vmscan.c > index 4255619a1a31..13a4a716c67a 100644 > --- a/mm/vmscan.c > +++ b/mm/vmscan.c > @@ -1721,18 +1721,31 @@ static unsigned long isolate_lru_folios(unsigned = long nr_to_scan, > bool folio_isolate_lru(struct folio *folio) > { > bool ret =3D false; > + struct lruvec *lruvec; > > VM_BUG_ON_FOLIO(!folio_ref_count(folio), folio); > > - if (folio_test_clear_lru(folio)) { > - struct lruvec *lruvec; > + /* > + * The folio_get needs to be prior to clear lru for list integrit= y. > + * Otherwise: > + * #0 folio_isolate_lru #1 release_pages > + * if (folio_test_clear_lru()) > + * if (folio_put_testzero()) > + * if (!folio_test_lru()) > + * > + * folio_get(folio) > + * list_add(&folio->lru,) > + * list_del(&folio->lru,) > + */ > + lruvec =3D folio_lruvec_lock_irq(folio); > + folio_get(folio); > > - folio_get(folio); > - lruvec =3D folio_lruvec_lock_irq(folio); > + if (folio_test_clear_lru(folio)) { > lruvec_del_folio(lruvec, folio); > - unlock_page_lruvec_irq(lruvec); > ret =3D true; > - } > + } else > + folio_put(folio); > + unlock_page_lruvec_irq(lruvec); > > return ret; > } > -- > 2.25.1 >