From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D07FCD1284 for ; Tue, 2 Apr 2024 10:17:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99C396B0083; Tue, 2 Apr 2024 06:17:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94C816B0095; Tue, 2 Apr 2024 06:17:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ECBD6B0099; Tue, 2 Apr 2024 06:17:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5EAE76B0083 for ; Tue, 2 Apr 2024 06:17:01 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 14EF040B35 for ; Tue, 2 Apr 2024 10:17:01 +0000 (UTC) X-FDA: 81964188642.18.119E07E Received: from mail-vs1-f48.google.com (mail-vs1-f48.google.com [209.85.217.48]) by imf30.hostedemail.com (Postfix) with ESMTP id 4AFCB8000E for ; Tue, 2 Apr 2024 10:16:58 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=d9W7JaOz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712053018; a=rsa-sha256; cv=none; b=CkfD7n1Oliai55PqpYggbwLUAGa8Qz4Rby9zvzaQ2PLGJhYMRMD+JPqjIbaYCSskHHc7xU GHyT1Jet9yKs3G4IgHuPfE+eHhyDhiGA0pwWQ76flcz0awEJnR4I5RHgyhJQP+c5E1TCat hpm8VHu4XNKIVS8ES3PqV+a1276q2YY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=d9W7JaOz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.48 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712053018; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CGZBV8K4LacC6tYCXDam0yYUxzCXbXCExJryqvcvGOY=; b=axM/cSu+qRoz1qETBDGUIu/IGDC7DhMcc2NdEnibqFXDa++4GdWVzVo71Gkcx3N5mwboxN nvPstbkOkWdDhzfWNVfUkYNK5VsZelVjdyrMrQA6kGlNvlr//2ihEovTF5Hf4pe9OlL5qb 3zqTTTmkJiyRWOEDtgP45gniyvaNkfw= Received: by mail-vs1-f48.google.com with SMTP id ada2fe7eead31-475ffc62cbaso1245355137.1 for ; Tue, 02 Apr 2024 03:16:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712053017; x=1712657817; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=CGZBV8K4LacC6tYCXDam0yYUxzCXbXCExJryqvcvGOY=; b=d9W7JaOzOra8ehWgQ8yFtJY0X4mYwZSuEOb9iCMvs7DsrpC+fJrSvGXDMv1d40/Vq0 NJNf8q2B1zjAxigijD7jXK3W3KIJCDxvk3A14gDXN16KRpQsnUEzyblvqraIUSIhHoDG 0krNypvUg2dG8oJZl3kXeV9wlxOyOL63C5nE0ZafdKblePveGeAOiUkUsACWDhlAg1zj aT7X1MaW/IL9vXE6uMukh9FD4GSt8uNoCg9enDgz8s1/3zsFGC5mXwXYdoIj69cjfzCL Eyl4xrpZS/QBQnSDXtFuu7nztyNA4neichTCrPyR32iqcP6eaU6rOc4gjfiv4pv+aQAJ +FgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712053017; x=1712657817; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CGZBV8K4LacC6tYCXDam0yYUxzCXbXCExJryqvcvGOY=; b=cy/uss1VhNSoQcOw5nh8S72NztC40X38IRp6ygA8QK1FMZK/vaOAFzz3PM5P0LxsnP dFIojZXA7kLj/wU9r2xdNk+FPiy2G/YeujBeLinRgaqjSHJMVFbX4JDI3XlHSXewtoC5 FvLP8NGV54S2e826vMWytwrJaIRA8zDDncm1GndbgdQ6cKIpxEF/1uoopyD0uYHLCjl8 +S6FGq4DfkDOp/ndCgbsDROjRK1xYOcFcg+DVrFuTbbgvtmH5oG9awf/mlAruitJddTZ KglJeUnlh6gN0cFJ2CeavroW4JI63bOX7IJ3OHJ+vrx2+peM2PVz6tHCiY734hrP5x5G 34ZQ== X-Forwarded-Encrypted: i=1; AJvYcCVlEhYIftHCMjTwbURY7k7h10MftiZFNlBOpp9NQl0QE/UIUpzeqj36+Om4uxRnlKQh6BqUosHmzuaqYAPmAqDXyMo= X-Gm-Message-State: AOJu0YyQlz/s3RvQGteuWzOOoFrnUkwwm0A2XDO7fca85oJOmeIDt2iB Ra3TwVsHta4/itX6cbZqZvmsfQK3zBEIMX91KUs1SHdwhCIt4EHV0R53CSOKo8Xa3mUuYZYO2r/ vAwshpFbducCuTnmoMRwy9imjCno= X-Google-Smtp-Source: AGHT+IEPai0nVOwJBFlwuKTlSMx6peY/NctoO8iTDzVRmTQScCClgG0SQFlJ0tamm//DJ3OOLxHOYB+MjxmgoQAmvSE= X-Received: by 2002:a05:6102:3bd7:b0:478:8e59:591c with SMTP id a23-20020a0561023bd700b004788e59591cmr1119050vsv.17.1712053015790; Tue, 02 Apr 2024 03:16:55 -0700 (PDT) MIME-Version: 1.0 References: <20240327144537.4165578-1-ryan.roberts@arm.com> <20240327144537.4165578-7-ryan.roberts@arm.com> In-Reply-To: <20240327144537.4165578-7-ryan.roberts@arm.com> From: Barry Song <21cnbao@gmail.com> Date: Tue, 2 Apr 2024 23:16:44 +1300 Message-ID: Subject: Re: [PATCH v5 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD To: Ryan Roberts Cc: Andrew Morton , David Hildenbrand , Matthew Wilcox , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang , Chris Li , Lance Yang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Barry Song Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 4AFCB8000E X-Stat-Signature: g4acduqbxyeoxofas76hjr5zs3kdda8x X-HE-Tag: 1712053018-935278 X-HE-Meta: U2FsdGVkX1+1x287qz7awrot+Y1gSQPgpDn8n2s/844+BVXhKwIkwdBbVHDDAJXqczn6+UXbsei2JkPzss06JMhdLtNdRX1eieVP3CBlNsCPq4DZKV94vpvdMZNQWtkOCrz8JB5bzsih5fRIL3OszIW6Lyr0d0bdKqL5ci/ZpZAmq8iN/B67SgW6cMB0I97ku2MPH6GgwjVnfqPm98YbaKxZYywm8rAoKa13gOni2CFuWyhv0H5lm0SB9OId8p/w3kDDApZdgYbZkLz06qrc4QClYBQqa9ThlwPzHFC4RCfrHDjlQYp23JLSQvo7kxgYA0SP71HwQmd5YWcRb2Ey2XZC6fPwruczkfaPkMLo8pF2BCsvBEB7GeVYJcnEwWnQutxi4A21H9PfcKFYggDe+hoqmYePcXXbIhk0W4nN5AeMYMNCdNRXvhpaE13AjnZISAlI+i1ILO/DoDo1H7w4GgBBUmKOSV8HfC5EESo1Zi4D/WVm1Jtt35ixBed2BECCZM9/CH19l4MzwrQNEZ+FVeVNeSfikLdqmfISeSyFDCs1B0DGKbueFbQVcCNHb5CPeNbvQLCH/UUxXZB5LC3xJbl3eSRc/s4htBx2/YggMavMKtsAEYRG8cRoE1xRTW3b7VTe74CSSfLW7j6r8DlmH65e+vdJja+meDQxCBh82P1IjBn7K5gABKoduV6SZdm83wb0y9Jyg6IKuCRUP+FAmvD/DnUQyUAc6J8/lSUfCVgYqL1aWFxQR832szhtzSAtXuW2ts7DFtfE0VLoULM97N23obakMFjMbpjCOBGVllqWE2+ENG5p16lH0PgmjcL2tCO4JtaYDANQj0Ojla1gsmkLjVpl5A6ae4qW3VaynLobGnTrSkdjcDeyywvnuNCxA5mQ7XgGZffQepv9Ycv7LJzVB9pXVviJiDUS9g/vzZZZp65oYro/JaBmW2iRTWc2aVz2c+VNp/nL4uBxw5h 2zcX5Ayz oeJTV8DnRPXEYZzIZFmJO0r5n+3FcihAWy1Xu/v+qVdWC6hWTIx1qC/apB62IhF35UIwt67xS1oyEQhSfPCr2oW0gouY5pQgvJVCRyTuq0eLr9B+LEMzAZBe66Bj4KRwn6TxUm/GlQaFzRYYe+sGI+2ns7KunDCizwFp2h2reRZC6oV64+CFeW2CdxplDhcoA7mNupsdp4860uGoUXzx0UYrO9+/alpuFQ2bzNZekVWo+X5W3hLSjKnqs/7CV2SPP6RkQJmGJwdDOdpYagQdfb+Tjmr4EIDo4PXhOI8AWkwn0+tsvpJSMZ+qCuzm4cU2VE/MDrnufgwv+P71dox1DCmPd1L9JopYQg1xvjFzDPD0BAILcpzkOr85aVnjNr6TFheiVVNSKI5Bvpz2nuxtvhdI5fkUhk0fldEUyXtHfFyOdTV1+pldNLeSIzMznlDHT5YSRFkii8zqCzPCBxmz50cxuAaSyXYJpkpi3opOXnD075kWMMQqzGpYMf5u7NmkWjZQ61qVaMX/Dumg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 28, 2024 at 3:46=E2=80=AFAM Ryan Roberts = wrote: > > Rework madvise_cold_or_pageout_pte_range() to avoid splitting any large > folio that is fully and contiguously mapped in the pageout/cold vm > range. This change means that large folios will be maintained all the > way to swap storage. This both improves performance during swap-out, by > eliding the cost of splitting the folio, and sets us up nicely for > maintaining the large folio when it is swapped back in (to be covered in > a separate series). > > Folios that are not fully mapped in the target range are still split, > but note that behavior is changed so that if the split fails for any > reason (folio locked, shared, etc) we now leave it as is and move to the > next pte in the range and continue work on the proceeding folios. > Previously any failure of this sort would cause the entire operation to > give up and no folios mapped at higher addresses were paged out or made > cold. Given large folios are becoming more common, this old behavior > would have likely lead to wasted opportunities. > > While we are at it, change the code that clears young from the ptes to > use ptep_test_and_clear_young(), via the new mkold_ptes() batch helper > function. This is more efficent than get_and_clear/modify/set, > especially for contpte mappings on arm64, where the old approach would > require unfolding/refolding and the new approach can be done in place. > > Reviewed-by: Barry Song > Signed-off-by: Ryan Roberts > --- Hi Ryan, I'm not entirely certain if this issue is related to this patch, but I've encountered the KNIC twice while using the latest mm-unstable kernel. Each time I attem= pted to debug it, the issue vanished. I'm posting here to see if you have any ideas on it :-) [ 50.444066] [ 50.444495] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [ 50.444954] WARNING: bad unlock balance detected! [ 50.445443] 6.9.0-rc2-00257-g2d9f63c285db #128 Not tainted [ 50.446233] ------------------------------------- [ 50.446684] singlethread/102 is trying to release lock (ptlock_ptr(ptdesc)) at: [ 50.447635] [] madvise_cold_or_pageout_pte_range+0x80c/0xea0 [ 50.449066] but there are no more locks to release! [ 50.449535] [ 50.449535] other info that might help us debug this: [ 50.450140] 1 lock held by singlethread/102: [ 50.450688] #0: ffff0000c001f208 (&mm->mmap_lock){++++}-{4:4}, at: do_madvise.part.0+0x178/0x518 [ 50.452321] [ 50.452321] stack backtrace: [ 50.452959] CPU: 3 PID: 102 Comm: singlethread Not tainted 6.9.0-rc2-00257-g2d9f63c285db #128 [ 50.453812] Hardware name: linux,dummy-virt (DT) [ 50.454373] Call trace: [ 50.454755] dump_backtrace+0x9c/0x100 [ 50.455246] show_stack+0x20/0x38 [ 50.455667] dump_stack_lvl+0xec/0x150 [ 50.456111] dump_stack+0x18/0x28 [ 50.456533] print_unlock_imbalance_bug+0x130/0x148 [ 50.457014] lock_release+0x2e0/0x360 [ 50.457487] _raw_spin_unlock+0x2c/0x78 [ 50.457997] madvise_cold_or_pageout_pte_range+0x80c/0xea0 [ 50.458635] walk_pgd_range+0x388/0x7d8 [ 50.459168] __walk_page_range+0x1e0/0x1f0 [ 50.459682] walk_page_range+0x1f0/0x2c8 [ 50.460225] madvise_pageout+0xf8/0x280 [ 50.460711] madvise_vma_behavior+0x310/0x9b8 [ 50.461169] madvise_walk_vmas+0xc0/0x128 [ 50.461605] do_madvise.part.0+0xf8/0x518 [ 50.462041] __arm64_sys_madvise+0x68/0x88 [ 50.462529] invoke_syscall+0x50/0x128 [ 50.463001] el0_svc_common.constprop.0+0x48/0xf8 [ 50.463508] do_el0_svc+0x28/0x40 [ 50.464004] el0_svc+0x50/0x150 [ 50.464492] el0t_64_sync_handler+0x13c/0x158 [ 50.465021] el0t_64_sync+0x1a4/0x1a8 [ 50.466959] ------------[ cut here ]------------ [ 50.467451] WARNING: CPU: 3 PID: 102 at kernel/rcu/tree_plugin.h:431 __rcu_read_unlock+0x74/0x218 [ 50.468160] Modules linked in: [ 50.468803] CPU: 3 PID: 102 Comm: singlethread Not tainted 6.9.0-rc2-00257-g2d9f63c285db #128 [ 50.469658] Hardware name: linux,dummy-virt (DT) [ 50.470293] pstate: a3400005 (NzCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE= =3D--) [ 50.470991] pc : __rcu_read_unlock+0x74/0x218 [ 50.471594] lr : madvise_cold_or_pageout_pte_range+0x828/0xea0 [ 50.472236] sp : ffff800080abb7e0 [ 50.472622] pmr_save: 000000e0 [ 50.473010] x29: ffff800080abb7e0 x28: 0000ffffa467a000 x27: fffffdffc31= 28c00 [ 50.474006] x26: 0010000000000001 x25: 000000000000001b x24: ffff0000c32= d73d0 [ 50.474971] x23: 0060000104a3afc3 x22: ffff0000c2492840 x21: 04000000000= 00001 [ 50.475943] x20: ff77fffffffffbff x19: ffff0000c3230000 x18: fffffffffff= fffff [ 50.477286] x17: 672d37353230302d x16: 3263722d302e392e x15: ffff800100a= bb227 [ 50.478373] x14: 0000000000000001 x13: 38613178302f3461 x12: 3178302b636= e7973 [ 50.479354] x11: fffffffffffe0000 x10: ffffb03159697d08 x9 : ffffb03155f= e3048 [ 50.480265] x8 : 00000000ffffefff x7 : ffffb03159697d08 x6 : 00000000000= 00000 [ 50.481154] x5 : 0000000000000001 x4 : ffff800080abbfe0 x3 : 00000000000= 00000 [ 50.482035] x2 : ffff4fd055074000 x1 : 00000000ffffffff x0 : 000000003ff= fffff [ 50.483163] Call trace: [ 50.483599] __rcu_read_unlock+0x74/0x218 [ 50.484152] madvise_cold_or_pageout_pte_range+0x828/0xea0 [ 50.484780] walk_pgd_range+0x388/0x7d8 [ 50.485328] __walk_page_range+0x1e0/0x1f0 [ 50.485725] walk_page_range+0x1f0/0x2c8 [ 50.486117] madvise_pageout+0xf8/0x280 [ 50.486547] madvise_vma_behavior+0x310/0x9b8 [ 50.486975] madvise_walk_vmas+0xc0/0x128 [ 50.487403] do_madvise.part.0+0xf8/0x518 [ 50.487845] __arm64_sys_madvise+0x68/0x88 [ 50.488374] invoke_syscall+0x50/0x128 [ 50.488946] el0_svc_common.constprop.0+0x48/0xf8 [ 50.489732] do_el0_svc+0x28/0x40 [ 50.490210] el0_svc+0x50/0x150 [ 50.490674] el0t_64_sync_handler+0x13c/0x158 [ 50.491257] el0t_64_sync+0x1a4/0x1a8 [ 50.491793] irq event stamp: 3087 [ 50.492243] hardirqs last enabled at (3087): [] _raw_spin_unlock_irq+0x38/0x90 [ 50.492917] hardirqs last disabled at (3086): [] _raw_spin_lock_irq+0x9c/0xc0 [ 50.493742] softirqs last enabled at (2470): [] __do_softirq+0x534/0x588 [ 50.494567] softirqs last disabled at (2461): [] ____do_softirq+0x18/0x30 [ 50.495328] ---[ end trace 0000000000000000 ]--- [ 50.497110] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:1578 [ 50.497544] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 102, name: singlethread [ 50.497652] preempt_count: ffffffff, expected: 0 [ 50.497728] RCU nest depth: -1, expected: 0 [ 50.497851] INFO: lockdep is turned off. [ 50.498023] CPU: 3 PID: 102 Comm: singlethread Tainted: G W 6.9.0-rc2-00257-g2d9f63c285db #128 [ 50.498166] Hardware name: linux,dummy-virt (DT) [ 50.498221] Call trace: [ 50.498260] dump_backtrace+0x9c/0x100 [ 50.498378] show_stack+0x20/0x38 [ 50.498487] dump_stack_lvl+0xec/0x150 [ 50.498574] dump_stack+0x18/0x28 [ 50.498659] __might_resched+0x158/0x278 [ 50.498741] __might_sleep+0x50/0xa0 [ 50.498849] down_write+0x30/0x1a8 [ 50.498950] split_huge_page_to_list_to_order+0x3c8/0x1130 [ 50.499052] madvise_cold_or_pageout_pte_range+0x84c/0xea0 [ 50.499138] walk_pgd_range+0x388/0x7d8 [ 50.499224] __walk_page_range+0x1e0/0x1f0 [ 50.499334] walk_page_range+0x1f0/0x2c8 [ 50.499458] madvise_pageout+0xf8/0x280 [ 50.499554] madvise_vma_behavior+0x310/0x9b8 [ 50.499657] madvise_walk_vmas+0xc0/0x128 [ 50.499739] do_madvise.part.0+0xf8/0x518 [ 50.499851] __arm64_sys_madvise+0x68/0x88 [ 50.499953] invoke_syscall+0x50/0x128 [ 50.500037] el0_svc_common.constprop.0+0x48/0xf8 [ 50.500121] do_el0_svc+0x28/0x40 [ 50.500203] el0_svc+0x50/0x150 [ 50.500322] el0t_64_sync_handler+0x13c/0x158 [ 50.500422] el0t_64_sync+0x1a4/0x1a8 [ 50.501378] BUG: scheduling while atomic: singlethread/102/0x00000000 [ 50.517641] INFO: lockdep is turned off. [ 50.518206] Modules linked in: [ 50.521135] CPU: 2 PID: 102 Comm: singlethread Tainted: G W 6.9.0-rc2-00257-g2d9f63c285db #128 [ 50.522026] Hardware name: linux,dummy-virt (DT) [ 50.522623] Call trace: [ 50.522993] dump_backtrace+0x9c/0x100 [ 50.523527] show_stack+0x20/0x38 [ 50.523950] dump_stack_lvl+0xec/0x150 [ 50.524405] dump_stack+0x18/0x28 [ 50.524849] __schedule_bug+0x80/0xe0 [ 50.525309] __schedule+0xb1c/0xc00 [ 50.525750] schedule+0x58/0x170 [ 50.526227] schedule_preempt_disabled+0x2c/0x50 [ 50.526762] rwsem_down_write_slowpath+0x1ac/0x718 [ 50.527342] down_write+0xf8/0x1a8 [ 50.527857] split_huge_page_to_list_to_order+0x3c8/0x1130 [ 50.528437] madvise_cold_or_pageout_pte_range+0x84c/0xea0 [ 50.529012] walk_pgd_range+0x388/0x7d8 [ 50.529442] __walk_page_range+0x1e0/0x1f0 [ 50.529896] walk_page_range+0x1f0/0x2c8 [ 50.530342] madvise_pageout+0xf8/0x280 [ 50.530878] madvise_vma_behavior+0x310/0x9b8 [ 50.531395] madvise_walk_vmas+0xc0/0x128 [ 50.531849] do_madvise.part.0+0xf8/0x518 [ 50.532330] __arm64_sys_madvise+0x68/0x88 [ 50.532829] invoke_syscall+0x50/0x128 [ 50.533374] el0_svc_common.constprop.0+0x48/0xf8 [ 50.533992] do_el0_svc+0x28/0x40 [ 50.534498] el0_svc+0x50/0x150 [ 50.535029] el0t_64_sync_handler+0x13c/0x158 [ 50.535588] el0t_64_sync+0x1a4/0x1a8 > include/linux/pgtable.h | 30 ++++++++++++++ > mm/internal.h | 12 +++++- > mm/madvise.c | 88 ++++++++++++++++++++++++----------------- > mm/memory.c | 4 +- > 4 files changed, 93 insertions(+), 41 deletions(-) > > diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h > index 8185939df1e8..391f56a1b188 100644 > --- a/include/linux/pgtable.h > +++ b/include/linux/pgtable.h > @@ -361,6 +361,36 @@ static inline int ptep_test_and_clear_young(struct v= m_area_struct *vma, > } > #endif > > +#ifndef mkold_ptes > +/** > + * mkold_ptes - Mark PTEs that map consecutive pages of the same folio a= s old. > + * @vma: VMA the pages are mapped into. > + * @addr: Address the first page is mapped at. > + * @ptep: Page table pointer for the first entry. > + * @nr: Number of entries to mark old. > + * > + * May be overridden by the architecture; otherwise, implemented as a si= mple > + * loop over ptep_test_and_clear_young(). > + * > + * Note that PTE bits in the PTE range besides the PFN can differ. For e= xample, > + * some PTEs might be write-protected. > + * > + * Context: The caller holds the page table lock. The PTEs map consecut= ive > + * pages that belong to the same folio. The PTEs are all in the same PM= D. > + */ > +static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long = addr, > + pte_t *ptep, unsigned int nr) > +{ > + for (;;) { > + ptep_test_and_clear_young(vma, addr, ptep); > + if (--nr =3D=3D 0) > + break; > + ptep++; > + addr +=3D PAGE_SIZE; > + } > +} > +#endif > + > #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG > #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONL= EAF_PMD_YOUNG) > static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, > diff --git a/mm/internal.h b/mm/internal.h > index eadb79c3a357..efee8e4cd2af 100644 > --- a/mm/internal.h > +++ b/mm/internal.h > @@ -130,6 +130,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t p= te, fpb_t flags) > * @flags: Flags to modify the PTE batch semantics. > * @any_writable: Optional pointer to indicate whether any entry except = the > * first one is writable. > + * @any_young: Optional pointer to indicate whether any entry except the > + * first one is young. > * > * Detect a PTE batch: consecutive (present) PTEs that map consecutive > * pages of the same large folio. > @@ -145,16 +147,18 @@ static inline pte_t __pte_batch_clear_ignored(pte_t= pte, fpb_t flags) > */ > static inline int folio_pte_batch(struct folio *folio, unsigned long add= r, > pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags, > - bool *any_writable) > + bool *any_writable, bool *any_young) > { > unsigned long folio_end_pfn =3D folio_pfn(folio) + folio_nr_pages= (folio); > const pte_t *end_ptep =3D start_ptep + max_nr; > pte_t expected_pte, *ptep; > - bool writable; > + bool writable, young; > int nr; > > if (any_writable) > *any_writable =3D false; > + if (any_young) > + *any_young =3D false; > > VM_WARN_ON_FOLIO(!pte_present(pte), folio); > VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); > @@ -168,6 +172,8 @@ static inline int folio_pte_batch(struct folio *folio= , unsigned long addr, > pte =3D ptep_get(ptep); > if (any_writable) > writable =3D !!pte_write(pte); > + if (any_young) > + young =3D !!pte_young(pte); > pte =3D __pte_batch_clear_ignored(pte, flags); > > if (!pte_same(pte, expected_pte)) > @@ -183,6 +189,8 @@ static inline int folio_pte_batch(struct folio *folio= , unsigned long addr, > > if (any_writable) > *any_writable |=3D writable; > + if (any_young) > + *any_young |=3D young; > > nr =3D pte_batch_hint(ptep, pte); > expected_pte =3D pte_advance_pfn(expected_pte, nr); > diff --git a/mm/madvise.c b/mm/madvise.c > index 070bedb4996e..bd00b83e7c50 100644 > --- a/mm/madvise.c > +++ b/mm/madvise.c > @@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *p= md, > LIST_HEAD(folio_list); > bool pageout_anon_only_filter; > unsigned int batch_count =3D 0; > + int nr; > > if (fatal_signal_pending(current)) > return -EINTR; > @@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *p= md, > return 0; > flush_tlb_batched_pending(mm); > arch_enter_lazy_mmu_mode(); > - for (; addr < end; pte++, addr +=3D PAGE_SIZE) { > + for (; addr < end; pte +=3D nr, addr +=3D nr * PAGE_SIZE) { > + nr =3D 1; > ptent =3D ptep_get(pte); > > if (++batch_count =3D=3D SWAP_CLUSTER_MAX) { > @@ -447,55 +449,67 @@ static int madvise_cold_or_pageout_pte_range(pmd_t = *pmd, > continue; > > /* > - * Creating a THP page is expensive so split it only if w= e > - * are sure it's worth. Split it if we are only owner. > + * If we encounter a large folio, only split it if it is = not > + * fully mapped within the range we are operating on. Oth= erwise > + * leave it as is so that it can be swapped out whole. If= we > + * fail to split a folio, leave it in place and advance t= o the > + * next pte in the range. > */ > if (folio_test_large(folio)) { > - int err; > - > - if (folio_likely_mapped_shared(folio)) > - break; > - if (pageout_anon_only_filter && !folio_test_anon(= folio)) > - break; > - if (!folio_trylock(folio)) > - break; > - folio_get(folio); > - arch_leave_lazy_mmu_mode(); > - pte_unmap_unlock(start_pte, ptl); > - start_pte =3D NULL; > - err =3D split_folio(folio); > - folio_unlock(folio); > - folio_put(folio); > - if (err) > - break; > - start_pte =3D pte =3D > - pte_offset_map_lock(mm, pmd, addr, &ptl); > - if (!start_pte) > - break; > - arch_enter_lazy_mmu_mode(); > - pte--; > - addr -=3D PAGE_SIZE; > - continue; > + const fpb_t fpb_flags =3D FPB_IGNORE_DIRTY | > + FPB_IGNORE_SOFT_DIRTY; > + int max_nr =3D (end - addr) / PAGE_SIZE; > + bool any_young; > + > + nr =3D folio_pte_batch(folio, addr, pte, ptent, m= ax_nr, > + fpb_flags, NULL, &any_young)= ; > + if (any_young) > + ptent =3D pte_mkyoung(ptent); > + > + if (nr < folio_nr_pages(folio)) { > + int err; > + > + if (folio_likely_mapped_shared(folio)) > + continue; > + if (pageout_anon_only_filter && !folio_te= st_anon(folio)) > + continue; > + if (!folio_trylock(folio)) > + continue; > + folio_get(folio); > + arch_leave_lazy_mmu_mode(); > + pte_unmap_unlock(start_pte, ptl); > + start_pte =3D NULL; > + err =3D split_folio(folio); > + folio_unlock(folio); > + folio_put(folio); > + if (err) > + continue; > + start_pte =3D pte =3D > + pte_offset_map_lock(mm, pmd, addr= , &ptl); > + if (!start_pte) > + break; > + arch_enter_lazy_mmu_mode(); > + nr =3D 0; > + continue; > + } > } > > /* > * Do not interfere with other mappings of this folio and > - * non-LRU folio. > + * non-LRU folio. If we have a large folio at this point,= we > + * know it is fully mapped so if its mapcount is the same= as its > + * number of pages, it must be exclusive. > */ > - if (!folio_test_lru(folio) || folio_mapcount(folio) !=3D = 1) > + if (!folio_test_lru(folio) || > + folio_mapcount(folio) !=3D folio_nr_pages(folio)) > continue; > > if (pageout_anon_only_filter && !folio_test_anon(folio)) > continue; > > - VM_BUG_ON_FOLIO(folio_test_large(folio), folio); > - > if (!pageout && pte_young(ptent)) { > - ptent =3D ptep_get_and_clear_full(mm, addr, pte, > - tlb->fullmm); > - ptent =3D pte_mkold(ptent); > - set_pte_at(mm, addr, pte, ptent); > - tlb_remove_tlb_entry(tlb, pte, addr); > + mkold_ptes(vma, addr, pte, nr); > + tlb_remove_tlb_entries(tlb, pte, nr, addr); > } > > /* > diff --git a/mm/memory.c b/mm/memory.c > index 9d844582ba38..b5b48f4cf2af 100644 > --- a/mm/memory.c > +++ b/mm/memory.c > @@ -989,7 +989,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, str= uct vm_area_struct *src_vma > flags |=3D FPB_IGNORE_SOFT_DIRTY; > > nr =3D folio_pte_batch(folio, addr, src_pte, pte, max_nr,= flags, > - &any_writable); > + &any_writable, NULL); > folio_ref_add(folio, nr); > if (folio_test_anon(folio)) { > if (unlikely(folio_try_dup_anon_rmap_ptes(folio, = page, > @@ -1553,7 +1553,7 @@ static inline int zap_present_ptes(struct mmu_gathe= r *tlb, > */ > if (unlikely(folio_test_large(folio) && max_nr !=3D 1)) { > nr =3D folio_pte_batch(folio, addr, pte, ptent, max_nr, f= pb_flags, > - NULL); > + NULL, NULL); > > zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent,= nr, > addr, details, rss, force_flush, > -- > 2.25.1 >