From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D87AFD49771 for ; Sun, 1 Dec 2024 15:39:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 612746B0093; Sun, 1 Dec 2024 10:39:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C0EB6B0095; Sun, 1 Dec 2024 10:39:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EC6D6B0096; Sun, 1 Dec 2024 10:39:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 21F3B6B0093 for ; Sun, 1 Dec 2024 10:39:27 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9E847AF51E for ; Sun, 1 Dec 2024 15:39:26 +0000 (UTC) X-FDA: 82846799196.30.814715C Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2060.outbound.protection.outlook.com [40.107.220.60]) by imf19.hostedemail.com (Postfix) with ESMTP id 66BD51A09D5 for ; Sun, 1 Dec 2024 15:39:11 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=V0p5WyhP; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf19.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.60 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1733067553; a=rsa-sha256; cv=pass; b=7JbXJHPNbESdq6k1iCsMFtdJGR3sfN3uS2sPfsdZ3M+miOK7oZkvU0eFS/4v8MMyYRtykF mku2OKmdT9bxp0G4ENKpSyo2dlh0IEsQO5PMEcRDMCB+EVo1n0DrzBAgTMzqT8mItS6CVp 5ZtN/b53syrpq1NPP5krTCx6P5yMHb4= ARC-Authentication-Results: i=2; imf19.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=V0p5WyhP; dmarc=pass (policy=quarantine) header.from=amd.com; spf=pass (imf19.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.220.60 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733067553; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Se4i7oWsjWVI2dcozt7u1pG3jASFmxn3pMpuXEaD3CM=; b=xW47wNSy7B1x28UphxXf5zaQYRjPmFsFZz+dCI37nyqsANRMvtpeaQBbS7wyOfHxErFhWX giKzrR3fhVkq3u6nAZpEaH2n49cMaM2mEjnOsv0t8rjzXPD4QA0RkYtjAkz8v/cLnbwhno lJ7Qw1Ipv4RLMrPzvsjfCjxMPQyapUc= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Eq/wLDx6ugTfmZshqkqmyVHV9T2lCfTgmz4ibIvVB4gSJE5A10DAMqs6zhixk+gJLWZPNDbNXXDoGj+ia2TbSNcdKJgIqKwHrS5Zw9N7fB+eB24X/qC4O/Rgj0rdcDuoSqqW1JV+oXRy3W+C4+oC0DcLP3w2Bo0BL06wDbWP3gq4MGDb7ZqxKJxnuvuTSG9WB3Y+RueqQSER1hgpca3DmOYEY6xB7X5mBDepbxPx4PCy9pKdap45QJrjNrlC26W7ng9hZ2KYF81pvoNlMsKhZMrcsN8D7BnWy+hD/CUAnQrgOPgElcMhDo9xtR4jSe3B2/g084F3gjVW1jdI318j0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Se4i7oWsjWVI2dcozt7u1pG3jASFmxn3pMpuXEaD3CM=; b=ROBN0q8trI19gWNd1HweayjKHLg2EMe6BtKSR6IfYshmhvvx3tZ+3T5T1Cp4Tr2RZE+IX8X/e5miAOX5isVWgw7ove2Oi5M06tHjB9a/Y/pctfksUMSnt1DDmkSh1Gqkysi3dsowQGu8mLWTvUR/stzJqckuBMNoM9jOVFU0vwKccQsvc9ka/GEXqKRpagHdbbdbgD19OT5d9du8hK0iOj1hstODUyuQBKZq6hR1a+r0kMs8fc75RhhmQ73aMy5vgRGvyKhDWdRIgjVCPrW0358dXGHmCBHD+Jb3B0gK9RaPK3BSVqt5F37P0ne6r/tl8QIslr9whHt1gPXmXLmNUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Se4i7oWsjWVI2dcozt7u1pG3jASFmxn3pMpuXEaD3CM=; b=V0p5WyhPr85VBWbuRnbHwhW7ndb+B323uVjVWHtw+8pziiNpmyd89OX8pSYARC1pAK0BvWZ2TgPjG04KQsY8+ZFqUmQSMmozB2rQoFDsyh0dJXq8PSOuiXvwpa8HxGYN7rqq2EGGHjEgfvOyOHi86rrfFeJkd91K33aXZOCPZDw= Received: from BY3PR04CA0015.namprd04.prod.outlook.com (2603:10b6:a03:217::20) by CY8PR12MB7266.namprd12.prod.outlook.com (2603:10b6:930:56::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.17; Sun, 1 Dec 2024 15:39:17 +0000 Received: from CO1PEPF000044F7.namprd21.prod.outlook.com (2603:10b6:a03:217:cafe::7f) by BY3PR04CA0015.outlook.office365.com (2603:10b6:a03:217::20) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044F7.mail.protection.outlook.com (10.167.241.197) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:39:16 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:39:09 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T Subject: [RFC PATCH V0 04/10] mm/migration: Migrate accessed folios to toptier node Date: Sun, 1 Dec 2024 15:38:12 +0000 Message-ID: <20241201153818.2633616-5-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044F7:EE_|CY8PR12MB7266:EE_ X-MS-Office365-Filtering-Correlation-Id: 0e3f6520-b0f3-4df4-344d-08dd121e50ad X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|1800799024|36860700013|7416014|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?/lyhM3aHDzOb6iHkc+rtO3hrTk0/24KGdoW3LbpckxcQLrTD3s1KO4VWcdat?= =?us-ascii?Q?D7oAHBkHxPQpWtaHjE4Je62rTpRGoxbJmqqw+Z+Qn/dEZV4R1eDq9G4RBQfh?= =?us-ascii?Q?yFPAS0dAB+/g/vmxaQnsB7sAZmMvRvvXzq9EsuzOPb+9585h64m3skR5E8nr?= =?us-ascii?Q?W2M/YCf2pYOlvGO3j3gPS4Ee3WAoJ55jHIks4TnSBPV9TnOd5PDLnvvIIixd?= =?us-ascii?Q?mZ8PN5hilKMqXSYJ2uECIG/B23MFRWGzfZexN2TsOJNAGzS4KCbudkTqQeIe?= =?us-ascii?Q?2jHlLgTAG/TnitgNtys9UfFTZ1Xy+gDFQeafXOIx3Xbrze0/jp42qEJiBem/?= =?us-ascii?Q?ddahQRAWlHivYAtatoOxjhiqUqYxxbbBeN7vjDUz/9Ocg/D5beXSyMRpetqn?= =?us-ascii?Q?qmk2qMmb8AASg7Bvlb5ZMe3B4NP3n/lmjD98ujS9UHWcO/zWr5xIX+Mw9xU2?= =?us-ascii?Q?C0WLFb7ihzz31Pnc02f/zBh2O2Ur4aHWhLfaWNGfpyayc3qAlX/zo3WSLyw8?= =?us-ascii?Q?N3o8VBpQIeEyK258TP/dr3AJRd9bJowRvdT20cEh2BWeK0L/VkybKxu1eLKi?= =?us-ascii?Q?kzRt6SxPv2UAdQAdjUzmOBrD+4How8XpbI850kWIl1jcDSRO3Gpp4ti+Xi7c?= =?us-ascii?Q?Df/RhxSXY45cZ6Vjn1hTbeJEBWZi1rovq6AsDQjePI2tlOmfkSvlfYYFU9J/?= =?us-ascii?Q?qNiINhP6W42d9cpJOdpabmuj1073C9fZ0Ea/8v6sdJ5/xZXLLyYoUbh3obd8?= =?us-ascii?Q?OPbBYaXfEcVbYVtgcK+OfkBsIJrD+UAmlGRUHUTsvhGUutNDLalxkfQTvoGV?= =?us-ascii?Q?XG1HR71jwW1NF1mfF89Ls+bh9BkN9DIwxTyVW1Pe/iaLsnFPjuciZUM/xALm?= =?us-ascii?Q?4li1SwjKuXFjfqRwTezk6Le1AL5y0n0X2gQytxSIom1nKqefP3dLtgJQKKMC?= =?us-ascii?Q?3a9Fl7kraQACqjbl+AAxbQcdeMmzjyufa8DITjdubgmAFbLi1L1z22FiRpHF?= =?us-ascii?Q?CNxQbySm6QOdh41ojo3ONuXkLgDRs5WrdoPnsdYQjGvDfbVIarg3OlvOYKAb?= =?us-ascii?Q?kXfhUuADShC3qKaEgdLtUp/GBjCcmUkbFV6vphdCbX0tpJI+qgZrglukJ7Iv?= =?us-ascii?Q?2juL6BYRHXrLEdZrriuFqbyEE0dexUzRYDT5lFZmclMxlRlpqfabZYop6vO4?= =?us-ascii?Q?K9B5SYXNacu8BffV9HAwON9r9PLt3Wy8eeFt3UC5uw0XxdDzy1toMXAD1ghH?= =?us-ascii?Q?fBCpTlLc/zU9MXD4w6bTRO+phQNu68zslnyuEmRX0/3IX7aNYVXp+aCF5aUH?= =?us-ascii?Q?aQFd/chAJJ1PjQmS5Ug/O+ze1SXAm7Xl0pbkslvFcqKt9E5geqAw45nD3f97?= =?us-ascii?Q?ezbyVKkBpeM0pr1sAHxA9Bjmws9uQalPSuAA2uOJNoVLH3Xolv9co73qLfuj?= =?us-ascii?Q?Xa5LhNcy6ZvF1FsbPdhNofu9y7z+psl8oqdJU/+BxUdZeyS63VtY3Q=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(1800799024)(36860700013)(7416014)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:16.6763 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0e3f6520-b0f3-4df4-344d-08dd121e50ad X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044F7.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7266 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 66BD51A09D5 X-Stat-Signature: 3zmm6q6x9kwbt3664st5hr9xjyj19s9p X-Rspam-User: X-HE-Tag: 1733067551-612816 X-HE-Meta: U2FsdGVkX18xWf7JjjnKRsUseSjukM59vIwoSe6w+/vWOD19BoZ7zRIHgAf47Y6XxQLApQCPsqLE01sG51DVk9q3k9uC8oWsOCCzeabIgWyTSU+BdDQDmHxDnXXaEHuAX9Dx1/dnFuW8r2HFV/wkt72QEfW1UUSm/1WoScPVuiBh0TeFyTSbO0ZMIf8bNi1u+3UzxG5LBWy8lyX9d2usPvvk3NkoK3HNf0Ks3uLfHTp80hhCfUvtRFYNldLr2e2jkvT3XblsrwelpECmaK0tmbuGV4jT0G5h8EjluCMoRZS9pVTecKt5dVW6QG5wJ7vV+Q+CW207ypXlKY8ljb4eDnciBx2b2R7rD8ZbUf/i4iTnJtOuXAia3jQFuiGYw1h1E4Jyv6SsiNGn8z8XpG25wzCBy0v2chKJFDyPbhBqPMU3F7XbZesf9JS9vKYNoN5cR2QoYWo7zUYlXXkBrfkbdOBZXsw0AZmzoL8CYOD06Yf6z+JeRAiwtc6oMjpiWwARN6FCktlMMmJnLhtC8Fn9Sq1YiPBiUNlJwhrhvbu+sV8km8I3fgbjT/UiNLN+pi/DZ/bljPh+JWSBTFs1mtTXuLH88bdR6MJp+K04jk9QxcbYfty/k5Ap1eWNj/O9DXj6N+ZAyKLROJESDm7MZOV5DN5IYt4T+rrdV9czEJeX9fKYYUpmSbgwnAdMHg8A1g7Lo4GCC9UVsoCvXBmGcePZ2wowQ6+GdDDmcZo5qMQl+kZ680HYpufTa0yl3Hcksuqdui3s7J+AaY1yNMif5EGIlak418rFOFOKyj99tt4dyxkleh3axhgpqiAWF1Zw/AsvMKBkrl/QXvDCVvawC4oS67jjLYr+3h/TcY6RHn1xMn94WCox/q/dR4ot+h3k2yfRpcB+bRH0jS1MXhPbi7jU2MPgPVK1fRVpvpNt5sFhLjdV4OfPierNzRqfU/KG3tZl4ZK0ZZ+RHqyVxpx+0Gc 9Ci+jPSG Rg+5uabkcOrIDJiKqeUNQbNPa2Ffoi82NOR57+mLxTN2V6RoOExGjA6Gd5GeJ8Ot1/GK1HrM0myaLPPo9LScKjcXDeDAre3l7IljcBVoux47vurFi+OC1D9nmK3kWR0imwSjwoBGKrQEidpOUT6xfcHNjzHVb7nXN/xRYSLog6qc1d4UeutnnjHZi5eeQuzgFk7d3jqeO5/4oL2KKtDFlNt+DNm3NQGwSMh+WzNI2tqiUnJ4bfbhxvs/IjKv7Ti91xvSUxmyZudN77OlBfUXyyyW+CfWtGNR4XTy+GGe1vaGzhzsAmuiW7WovWk82Dhs/CZph19jyAlXLVVl6UhPyag3a9TOzPOBjsn7C1di5CfbwkiE78sd3sm6nQE+SoVemwYdgKpSMIowlRuwasdHNoswLKH0e5epfxL+xOjg2Ri+197e2zxmIa8UZnvlVn13siOej4Ls+hd0CayUdyKhQlCTdga7pmlZZRqiYZzWIoLEecRchjSptaIryPWW/QTnRGLqE9U4jnWcXvYm3so84cRauF6yqAFNXjSZUTgROatF19a9bGfo2hdPOC5IeyCBkh6mAizSTWptAG7jl7C85MbfuL6CooVSXlL/XMHBSN10y4jfthZN4Lz0mAVyrHM0sTn/FnWPM8iR6Hh581yL7qXDLCSEVXyP+vUD1kponU+dMMLpFn3iUkaNb+HFTDDHORhnM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For each recently accessed slowtier folio in the migration list: - Isolate LRU pages - Migrate to a regular node. The rationale behind whole migration is to speedup the access to recently accessed pages. Limitation: PTE A bit scanning approach lacks information about exact destination node to migrate to. Reason: PROT_NONE hint fault based scanning is done in a process context. Here when the fault occurs, source CPU of the fault associated task is known. Time of page access is also accurate. With the lack of above information, migration is done to node 0 by default. Signed-off-by: Raghavendra K T --- mm/kmmscand.c | 178 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) TBD: Before calling migrate_misplaced_folio, we need to hold a PTL lock. But since we are not coming from fault path, it is tricky. We need to fix this before the final patch. diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 0496359d07f5..3b4453b053f4 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -36,7 +37,15 @@ static unsigned long kmmscand_mms_to_scan __read_mostly = KMMSCAND_MMS_TO_SCAN; volatile bool kmmscand_scan_enabled = true; static bool need_wakeup; +/* mm of the migrating folio entry */ +static struct mm_struct *kmmscand_cur_migrate_mm; + +/* Migration list is manipulated underneath because of mm_exit */ +static bool kmmscand_migration_list_dirty; + static unsigned long kmmscand_sleep_expire; +#define KMMSCAND_DEFAULT_TARGET_NODE (0) +static int kmmscand_target_node = KMMSCAND_DEFAULT_TARGET_NODE; static DEFINE_SPINLOCK(kmmscand_mm_lock); static DEFINE_SPINLOCK(kmmscand_migrate_lock); @@ -115,6 +124,107 @@ static bool kmmscand_eligible_srcnid(int nid) return false; } +/* + * Do not know what info to pass in the future to make + * decision on taget node. Keep it void * now. + */ +static int kmmscand_get_target_node(void *data) +{ + return kmmscand_target_node; +} + +static int kmmscand_migrate_misplaced_folio_prepare(struct folio *folio, + struct vm_area_struct *vma, int node) +{ + if (folio_is_file_lru(folio)) { + /* + * Do not migrate file folios that are mapped in multiple + * processes with execute permissions as they are probably + * shared libraries. + * + * See folio_likely_mapped_shared() on possible imprecision + * when we cannot easily detect if a folio is shared. + */ + if (vma && (vma->vm_flags & VM_EXEC) && + folio_likely_mapped_shared(folio)) + return -EACCES; + /* + * Do not migrate dirty folios as not all filesystems can move + * dirty folios in MIGRATE_ASYNC mode which is a waste of + * cycles. + */ + if (folio_test_dirty(folio)) + return -EAGAIN; + } + + if (!folio_isolate_lru(folio)) + return -EAGAIN; + + return 0; +} + +enum kmmscand_migration_err { + KMMSCAND_NULL_MM = 1, + KMMSCAND_INVALID_FOLIO, + KMMSCAND_INVALID_VMA, + KMMSCAND_INELIGIBLE_SRC_NODE, + KMMSCAND_SAME_SRC_DEST_NODE, + KMMSCAND_LRU_ISOLATION_ERR, +}; + +static int kmmscand_promote_folio(struct kmmscand_migrate_info *info) +{ + unsigned long pfn; + struct page *page; + struct folio *folio; + struct vm_area_struct *vma; + int ret; + + int srcnid, destnid; + + if (info->mm == NULL) + return KMMSCAND_NULL_MM; + + folio = info->folio; + + /* Check again if the folio is really valid now */ + if (folio) { + pfn = folio_pfn(folio); + page = pfn_to_online_page(pfn); + } + + if (!page || !folio || !folio_test_lru(folio) || + folio_is_zone_device(folio) || !folio_mapped(folio)) + return KMMSCAND_INVALID_FOLIO; + + vma = info->vma; + + /* XXX: Need to validate vma here?. vma_lookup() results in 2x regression */ + if (!vma) + return KMMSCAND_INVALID_VMA; + + srcnid = folio_nid(folio); + + /* Do not try to promote pages from regular nodes */ + if (!kmmscand_eligible_srcnid(srcnid)) + return KMMSCAND_INELIGIBLE_SRC_NODE; + + destnid = kmmscand_get_target_node(NULL); + + if (srcnid == destnid) + return KMMSCAND_SAME_SRC_DEST_NODE; + + folio_get(folio); + ret = kmmscand_migrate_misplaced_folio_prepare(folio, vma, destnid); + if (ret) { + folio_put(folio); + return KMMSCAND_LRU_ISOLATION_ERR; + } + folio_put(folio); + + return migrate_misplaced_folio(folio, vma, destnid); +} + static bool folio_idle_clear_pte_refs_one(struct folio *folio, struct vm_area_struct *vma, unsigned long addr, @@ -266,8 +376,74 @@ static void kmmscand_collect_mm_slot(struct kmmscand_mm_slot *mm_slot) } } +static void kmmscand_cleanup_migration_list(struct mm_struct *mm) +{ + struct kmmscand_migrate_info *info, *tmp; + +start_again: + spin_lock(&kmmscand_migrate_lock); + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + + if (mm == READ_ONCE(kmmscand_cur_migrate_mm)) { + /* A folio in this mm is being migrated. wait */ + WRITE_ONCE(kmmscand_migration_list_dirty, true); + spin_unlock(&kmmscand_migrate_lock); + goto start_again; + } + + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (info && (info->mm == mm)) { + info->mm = NULL; + WRITE_ONCE(kmmscand_migration_list_dirty, true); + } + } + } + spin_unlock(&kmmscand_migrate_lock); +} + static void kmmscand_migrate_folio(void) { + int ret = 0; + struct kmmscand_migrate_info *info, *tmp; + + spin_lock(&kmmscand_migrate_lock); + + if (!list_empty(&kmmscand_migrate_list.migrate_head)) { + list_for_each_entry_safe(info, tmp, &kmmscand_migrate_list.migrate_head, + migrate_node) { + if (READ_ONCE(kmmscand_migration_list_dirty)) { + kmmscand_migration_list_dirty = false; + list_del(&info->migrate_node); + /* + * Do not try to migrate this entry because mm might have + * vanished underneath. + */ + kfree(info); + spin_unlock(&kmmscand_migrate_lock); + goto dirty_list_handled; + } + + list_del(&info->migrate_node); + /* Note down the mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, info->mm); + spin_unlock(&kmmscand_migrate_lock); + + if (info->mm) + ret = kmmscand_promote_folio(info); + + kfree(info); + + spin_lock(&kmmscand_migrate_lock); + /* Reset mm of folio entry we are migrating */ + WRITE_ONCE(kmmscand_cur_migrate_mm, NULL); + spin_unlock(&kmmscand_migrate_lock); +dirty_list_handled: + //cond_resched(); + spin_lock(&kmmscand_migrate_lock); + } + } + spin_unlock(&kmmscand_migrate_lock); } static unsigned long kmmscand_scan_mm_slot(void) @@ -450,6 +626,8 @@ void __kmmscand_exit(struct mm_struct *mm) spin_unlock(&kmmscand_mm_lock); + kmmscand_cleanup_migration_list(mm); + if (free) { mm_slot_free(kmmscand_slot_cache, mm_slot); mmdrop(mm); -- 2.39.3