From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E471D49771 for ; Sun, 1 Dec 2024 15:39:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F2B26B0089; Sun, 1 Dec 2024 10:39:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A1896B008A; Sun, 1 Dec 2024 10:39:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 01B246B008C; Sun, 1 Dec 2024 10:39:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D0EA46B0089 for ; Sun, 1 Dec 2024 10:39:15 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 825521410FD for ; Sun, 1 Dec 2024 15:39:15 +0000 (UTC) X-FDA: 82846798944.02.9313210 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2042.outbound.protection.outlook.com [40.107.100.42]) by imf04.hostedemail.com (Postfix) with ESMTP id 6EBED40A18 for ; Sun, 1 Dec 2024 15:39:00 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=pQ+doSH8; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf04.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.100.42 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733067541; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=THqpOYzVxsi2M9Jxr93ZEPARXIH0jqWyq/8EdCQRhh4=; b=GS5Fxn74GaDsyKh7J4vvkbr7NPxlgFTjWIYNMKWNoksgf0+yGEiCVY5fOwkyzIMC+UYdCW xHwwbYTmM/iBWYyNBx6bht4U4cR2ZDWGH1q5zLmALiV9IPNu2ui3zwQ/Y79OJQ7QF8lxG3 sfy0Jn81HFAcvjWWA99FvovXYZGHeV0= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1733067541; a=rsa-sha256; cv=pass; b=bWsb4xrEfudvcqystVGeDAO0uBWA2KmfyRpi3UQLX05UEOMHUxra1zYHZrTbPg7Us1TeHO e1P1V+tKMHhsCotE7ftjmCxMWD7V5BmJ+n6miM/xo8yTUzlXIsdqalpdXoAfvdgp7TYdob FtoHbhxy2zFAi3uCUqY6iE5Nhb6IyA0= ARC-Authentication-Results: i=2; imf04.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=pQ+doSH8; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf04.hostedemail.com: domain of Raghavendra.KodsaraThimmappa@amd.com designates 40.107.100.42 as permitted sender) smtp.mailfrom=Raghavendra.KodsaraThimmappa@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=gb5u9F5hK/odT+SAs6kUNC1rDMGYSVBJtXm9Zwtxma3v1WuoAq4QsPfn+p+XZnteauxk3okHDM6NrVvQP3g0pEfzbu/mJLAdVGJ5lFVy0m1TIW+GNpqeuRSfeG6JKbtfcBSnLgPAnI0K7xUOg5pxArqWjvIzS/5UiS4ruQVEo3wSvmwBhFsMFKNMqtKiAEDBl4kzkU0aH2+fnvIzAk8SdkbEaIuDFzDiAKw5lKQsHsS1IUPwCc5SOtoQ3OzKjDpzw0vjEMA0aP7irmAtQGayXEJt8nNEPth+UDz2eCirBsaPNqFoxkZCpYflYByhvmF6qm+SYjVOd/xS5zIxbsO2gw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=THqpOYzVxsi2M9Jxr93ZEPARXIH0jqWyq/8EdCQRhh4=; b=tgvggDPG7uUQaZBA64PfPyYFzUMI0hhuuEFZk7Fcyo04dQK4FPdo/vTuwo3GqiXlRGvXkR7aqayoROHdwGDaTseXqnBqWJONB37s5D43UcnOCklqCp+0GGc9EDEhCe7ye7h9OSHN/eH/9kdtRu4Wl6IzSLjYLFvPRmMQohexY92l93gU6ZSeuiXVgHyCrq4Y/TtBdPTJGsT+Fo7SvlYCPwjoDgHGJFmBvhPtNP4wyWc6yVzcTYXcxlBKlBZpTFJK9P8mTCkCaG++IreB33SwKKp47rKuV19sBPWtmzF9iADg7bXXrK4/UoqZYNnA6OdU0odDrW0oW69rBHYeNBOESA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=kvack.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=THqpOYzVxsi2M9Jxr93ZEPARXIH0jqWyq/8EdCQRhh4=; b=pQ+doSH864GJ+c/Ch66pVLvlp6uxuP4vahkZOt56OK8yzVEnUFUDIiYYSrV/nZuqJ/jNpCSDlQNHTrC18qnf7pQjRzObnjZl/xH/sBOThHf6DFMyp09p0LctaRLmBCpsbcOXzXu7tNxmhwNHgkNtHSgOS+xnWEIGCuJolONauVE= Received: from SJ0PR03CA0203.namprd03.prod.outlook.com (2603:10b6:a03:2ef::28) by DS0PR12MB9445.namprd12.prod.outlook.com (2603:10b6:8:1a1::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8207.15; Sun, 1 Dec 2024 15:39:04 +0000 Received: from CO1PEPF000044FB.namprd21.prod.outlook.com (2603:10b6:a03:2ef:cafe::cc) by SJ0PR03CA0203.outlook.office365.com (2603:10b6:a03:2ef::28) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.8207.16 via Frontend Transport; Sun, 1 Dec 2024 15:39:04 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1PEPF000044FB.mail.protection.outlook.com (10.167.241.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.8251.1 via Frontend Transport; Sun, 1 Dec 2024 15:39:03 +0000 Received: from tunga.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Sun, 1 Dec 2024 09:38:56 -0600 From: Raghavendra K T To: , , , , , , , , , , , , , , CC: , , , , , , , , , , Raghavendra K T , Subject: [RFC PATCH V0 02/10] mm: Maintain mm_struct list in the system Date: Sun, 1 Dec 2024 15:38:10 +0000 Message-ID: <20241201153818.2633616-3-raghavendra.kt@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241201153818.2633616-1-raghavendra.kt@amd.com> References: <20241201153818.2633616-1-raghavendra.kt@amd.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1PEPF000044FB:EE_|DS0PR12MB9445:EE_ X-MS-Office365-Filtering-Correlation-Id: 05cc5bb9-b9b7-4986-fe6b-08dd121e48b7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|82310400026|7416014|36860700013|376014|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?+xxasN5tC1YEgwOqWe7D9mzO3hoMyzCCl7+OGOoIqirHPX/+SMVbHqjhRTSf?= =?us-ascii?Q?sTuds4iwOoNBiDOaAA+1nwyaCLXuBGehRfeYhleaeM8rXJQtPYm2G6ntplVo?= =?us-ascii?Q?8A35QWIC+nuf9KYRTekb29yor9cdQQnE4g08I8UNxQ1TASQiRabUTuchECm5?= =?us-ascii?Q?SbNy8zEjFd743pUomJcFOJGQgMxQwi4zf8FR3nRf8uEJmF24brSwsMe2dPkC?= =?us-ascii?Q?JEwUIusuB3+V8Xb20fTT1N1Rp+dbH1fYO6Eh7RwD6Mwgj5EwnoZyeRd8RFv5?= =?us-ascii?Q?fhUOekLcLjTB6KMzooeD/BRa71/8nBBmBtz3TCR6/7oE0+MCGO8yh+6oXS63?= =?us-ascii?Q?9BcVEtPJSScTVGWJWTBlO6/JXOoPSQ0dz/L1SZBQ/Vuk4ndiWkFJ2x9JLKtr?= =?us-ascii?Q?UqKKRMompfQpSXX1Ji1pagoB1gwbqNxJ6BxraZ9oe4pdSgVQxBVWTYZ+SsC8?= =?us-ascii?Q?rf6MA4V638/8l04//QVFEaBzs0fbWEC6Ufn+ds005dFKvrGM76+38dSicb9j?= =?us-ascii?Q?LWWy8hFpJa0s3fiqFT/+ljlJttFRoVQg+5ew+204ViYbD/TEolWcjgvAOhz5?= =?us-ascii?Q?QecOQrYqi5L6cCRhvudR9OvklHUjpyL12V6RoMNmGMb8kN5W/bGvVAS0j+sV?= =?us-ascii?Q?+fkbr/nfJn3l24NR3yxyspleymd8+PZP4tuP95ICXcs8bv/Xtd547gnU2roR?= =?us-ascii?Q?kSRsUdxuRfB6sdwCDUg3NkRG9ti3QPA+7ZYX93NVzswdg9HUUqYjZ89noQ5j?= =?us-ascii?Q?o4j65UTly11UMbyiiIpKzWkz41vzNQ2jAK2CHmQ+ZP43EitJUzKGLObphusp?= =?us-ascii?Q?CiDwBXHSd5sBG7NHxiZd6nmXk6GbRFkyREGvR7mFzBaWUM3zAT44tNka9Eou?= =?us-ascii?Q?naYYIBJvLQMXh+pXp9w67SFq1R02gJP6d8IcXtR12/JmvjYBkSJF4ZzUKWOx?= =?us-ascii?Q?YRzyZOh7wPOSWvojLbhPtC4rehlYrIaPvhrcUfBYLyLHHpgxr+dOdmCkg3vn?= =?us-ascii?Q?Pu6GkHbM56FegXgAuqmCrRwHFil5X07ELERSMqqUgR2QB9YYeB2/VgRYc0H7?= =?us-ascii?Q?g5tLzBnhY//YIoXh997aylcAh/7pYvNrE1ElQnW2gkTTFQQD6DJwkbdhijFe?= =?us-ascii?Q?ewmPDQfrwEBQI2EOJPLVh3nun741yLQ8IlAca7XqeGn77GtJ/j8OQ4Difgm6?= =?us-ascii?Q?YFozVb1JwyRKoJyonOZPzR6QeXGLAJZvn/XtIsoysxf4MfG4eRmw0Vwv1a0v?= =?us-ascii?Q?t1OQob5QuHR3HKPRynASg8BWiosWsAuLEC6vvGozuOIWMVHsUQrGGqnf1A1F?= =?us-ascii?Q?kQr4M4l4zvvadqR1F5hHNpfl7vb++MbjOzsenlOaBN5OIxRw0FPgCVV5WMIO?= =?us-ascii?Q?4eTahIK7wsUVoM/d3R2cJ877VGr1aejbc4rqH9PF/qIs03UdgJ7EIDkPkXko?= =?us-ascii?Q?HFzRtg74gyZixCNepykqk+vf+B+HfR0TlDFKMoJVhMOiFtLw1nf2Kw=3D=3D?= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(1800799024)(82310400026)(7416014)(36860700013)(376014)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2024 15:39:03.3252 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 05cc5bb9-b9b7-4986-fe6b-08dd121e48b7 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1PEPF000044FB.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB9445 X-Rspamd-Queue-Id: 6EBED40A18 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: mm8twh37rxdeh611dyorn8du1amytqw4 X-HE-Tag: 1733067540-51822 X-HE-Meta: U2FsdGVkX19Pf2kmB1oZF85MlrS9uKwMLgfk7AoleP8f+P0kk2lXRYnNfcJOLBwPyq/ye/XKDnr0ii/nO4kJIFuQHxJg3NzubpG+rARm3I74tz8XQaKQyijBtbtsv/k9U/oKOZe0aebgm4ubukPUAY1JkNSAYFOf4r7/nIDcZZ/WR5xdIzy/fUVe2Hlg/2uF7jU7EAGjUV7g3BJaPS6C5VF+WJSYxMZKrmuTKMr8FiA9zvuh04tcgai9On43H9SZuqi+1VTdOlIu2uuF2gcNs8KFyfk6NOUPZw5agiW2XSPndm5r2sumOrZfNlzNsVTCOQvPoRsiMsvQKn/iQXG4bZQEC/jMHllagqmPfL/B81ciRvRGEs2HCCn0azzpJ38nSlKR4bykqCbVFVOVJaD4Of+VulNjcsLBKSK4hZT8vdu46aaH+/uvEuTwOmc/D3s/y1ne3rERS6cJDZERl1v58rnhk9Am7vdcUVA2boVZlMAB/AnlVS+J1Vw8h9kJppfvHXG4i632RHU7PqlhLHiXNAlhsdROgUuevJtehZsed1Mn/lnFOrBAdgMQnapKJHDirxUEcOpT2LOh6QS1pNY+t1icWtCN4slw3BoSvyFC6MEByPyunMCIyHmowOy6T9mAov/48jyh4fh1cniUvXywBSS4yQLos+4jlMkpXdXHmAdWRqmak6eHTcyNDtDLkuOg25zywj5oz2XS1NmThsZPTThwfmyIb7kss9e7ZqXMalEd3eECeUb+SEWnCr+W+Uo6AzaprQzONilIBLfPtDCUBU/NFSucUgPbouxn2dPIGSo/z96x9gLSi1G4u5B2zuK3pjWbUCiZNZMDW9q+fpYT3+qgitU1lNfEqLElvS8heIe/cGbIU5uXmL90FqvyGuDgl43LK/AYzKaTPYW5bCQXiuBaj9Gb1Jc7P50xIjE0Mr+0Yt/ScyS5K/kA/mfrBqct0Pcs9Cgx9xulT76g6zI y90KrqYD oACraxnsFcdJ3eqMEm3bglL+5+JxyOJ4iOKFrPBPRHt9odA4xMs8Hz6CroIN766EYo75+OaLeB1EyofFGknOYrzJuxV5dFHVD8VJAPYqOBFnymCaI4EqtEvvDMjkPIlKxzC7GbAX+cHd6cQTP2cLWbsxC8kuenBilfzTf8C18d5bz48zo8pOhK1vWbCmDLeKitbQNn0NEEnjApCLDAgRJzbd1EzbYyzmna2MIIMLt+L+/5O3ni06nHXBBFeaLkfQm45IGiklkzKmAH497gUcQtezQtVlmR9O2srrkp9Dx1QK9ODC24ylYk3AnRiWJMbnyt77X1iLLCcvuRtXQZTKH92OSaInSS8EGZ4gOr+zQC8AVH17ffLvQYEpjXrvz+zp61VJUzUpDGcVslHWMF7qk4VR2PDhiFN5JkD08v6Yym2cu29fCtfggQEWcPCHqVI5CzafMla+9rTZ9zgmLjMF8c3iDfgjInapziXDTKQA6AmX5Bm32C+TseYroSCzWLVtDT/dmRoQxRwsqHtl9flgIPbJEZgR24e+uqCILQLt+PhtsNTXXa6gteOytK0GxUtwFlvAwQ5R5zjoaE+euQnWT+O54sc/IcXILgD46b++0twt/XgCHg4CselmXsXKO8EbINw96TAW/Dk2gsOaq4g09R9+W3s4gLCBLa8EJ+4VM8fcOJyeSn7rXbFYeJp60MxZx/+LlxB+2lrQpf/tiv4uRCVRvt8qwYFJ4spBaACERX3hp7Ed3bSRlVOVnqphF/aU26cbb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a hook in the fork and exec path to link mm_struct. Reuse the mm_slot infrastructure to aid insert and lookup of mm_struct. CC: linux-fsdevel@vger.kernel.org Suggested-by: Bharata B Rao Signed-off-by: Raghavendra K T --- fs/exec.c | 4 ++ include/linux/kmmscand.h | 30 ++++++++++++++ kernel/fork.c | 4 ++ mm/kmmscand.c | 86 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 include/linux/kmmscand.h diff --git a/fs/exec.c b/fs/exec.c index 98cb7ba9983c..bd72107b2ab1 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -68,6 +68,7 @@ #include #include #include +#include #include #include @@ -274,6 +275,8 @@ static int __bprm_mm_init(struct linux_binprm *bprm) if (err) goto err_ksm; + kmmscand_execve(mm); + /* * Place the stack at the largest stack address the architecture * supports. Later, we'll move this to an appropriate place. We don't @@ -296,6 +299,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) return 0; err: ksm_exit(mm); + kmmscand_exit(mm); err_ksm: mmap_write_unlock(mm); err_free: diff --git a/include/linux/kmmscand.h b/include/linux/kmmscand.h new file mode 100644 index 000000000000..b120c65ee8c6 --- /dev/null +++ b/include/linux/kmmscand.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_KMMSCAND_H_ +#define _LINUX_KMMSCAND_H_ + +#ifdef CONFIG_KMMSCAND +extern void __kmmscand_enter(struct mm_struct *mm); +extern void __kmmscand_exit(struct mm_struct *mm); + +static inline void kmmscand_execve(struct mm_struct *mm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) +{ + __kmmscand_enter(mm); +} + +static inline void kmmscand_exit(struct mm_struct *mm) +{ + __kmmscand_exit(mm); +} +#else /* !CONFIG_KMMSCAND */ +static inline void __kmmscand_enter(struct mm_struct *mm) {} +static inline void __kmmscand_exit(struct mm_struct *mm) {} +static inline void kmmscand_execve(struct mm_struct *mm) {} +static inline void kmmscand_fork(struct mm_struct *mm, struct mm_struct *oldmm) {} +static inline void kmmscand_exit(struct mm_struct *mm) {} +#endif +#endif /* _LINUX_KMMSCAND_H_ */ diff --git a/kernel/fork.c b/kernel/fork.c index 1450b461d196..812b0032592e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -85,6 +85,7 @@ #include #include #include +#include #include #include #include @@ -659,6 +660,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, mm->exec_vm = oldmm->exec_vm; mm->stack_vm = oldmm->stack_vm; + kmmscand_fork(mm, oldmm); + /* Use __mt_dup() to efficiently build an identical maple tree. */ retval = __mt_dup(&oldmm->mm_mt, &mm->mm_mt, GFP_KERNEL); if (unlikely(retval)) @@ -1350,6 +1353,7 @@ static inline void __mmput(struct mm_struct *mm) exit_aio(mm); ksm_exit(mm); khugepaged_exit(mm); /* must run before exit_mmap */ + kmmscand_exit(mm); exit_mmap(mm); mm_put_huge_zero_folio(mm); set_mm_exe_file(mm, NULL); diff --git a/mm/kmmscand.c b/mm/kmmscand.c index 23cf5638fe10..957128d4e425 100644 --- a/mm/kmmscand.c +++ b/mm/kmmscand.c @@ -7,13 +7,14 @@ #include #include #include +#include #include #include #include #include #include "internal.h" - +#include "mm_slot.h" static struct task_struct *kmmscand_thread __read_mostly; static DEFINE_MUTEX(kmmscand_mutex); @@ -30,10 +31,21 @@ static bool need_wakeup; static unsigned long kmmscand_sleep_expire; +static DEFINE_SPINLOCK(kmmscand_mm_lock); static DECLARE_WAIT_QUEUE_HEAD(kmmscand_wait); +#define KMMSCAND_SLOT_HASH_BITS 10 +static DEFINE_READ_MOSTLY_HASHTABLE(kmmscand_slots_hash, KMMSCAND_SLOT_HASH_BITS); + +static struct kmem_cache *kmmscand_slot_cache __read_mostly; + +struct kmmscand_mm_slot { + struct mm_slot slot; +}; + struct kmmscand_scan { struct list_head mm_head; + struct kmmscand_mm_slot *mm_slot; }; struct kmmscand_scan kmmscand_scan = { @@ -76,6 +88,11 @@ static void kmmscand_migrate_folio(void) { } +static inline int kmmscand_test_exit(struct mm_struct *mm) +{ + return atomic_read(&mm->mm_users) == 0; +} + static unsigned long kmmscand_scan_mm_slot(void) { /* placeholder for scanning */ @@ -123,6 +140,65 @@ static int kmmscand(void *none) return 0; } +static inline void kmmscand_destroy(void) +{ + kmem_cache_destroy(kmmscand_slot_cache); +} + +void __kmmscand_enter(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *kmmscand_slot; + struct mm_slot *slot; + int wakeup; + + /* __kmmscand_exit() must not run from under us */ + VM_BUG_ON_MM(kmmscand_test_exit(mm), mm); + + kmmscand_slot = mm_slot_alloc(kmmscand_slot_cache); + + if (!kmmscand_slot) + return; + + slot = &kmmscand_slot->slot; + + spin_lock(&kmmscand_mm_lock); + mm_slot_insert(kmmscand_slots_hash, mm, slot); + + wakeup = list_empty(&kmmscand_scan.mm_head); + list_add_tail(&slot->mm_node, &kmmscand_scan.mm_head); + spin_unlock(&kmmscand_mm_lock); + + mmgrab(mm); + if (wakeup) + wake_up_interruptible(&kmmscand_wait); +} + +void __kmmscand_exit(struct mm_struct *mm) +{ + struct kmmscand_mm_slot *mm_slot; + struct mm_slot *slot; + int free = 0; + + spin_lock(&kmmscand_mm_lock); + slot = mm_slot_lookup(kmmscand_slots_hash, mm); + mm_slot = mm_slot_entry(slot, struct kmmscand_mm_slot, slot); + if (mm_slot && kmmscand_scan.mm_slot != mm_slot) { + hash_del(&slot->hash); + list_del(&slot->mm_node); + free = 1; + } + + spin_unlock(&kmmscand_mm_lock); + + if (free) { + mm_slot_free(kmmscand_slot_cache, mm_slot); + mmdrop(mm); + } else if (mm_slot) { + mmap_write_lock(mm); + mmap_write_unlock(mm); + } +} + static int start_kmmscand(void) { int err = 0; @@ -168,6 +244,13 @@ static int __init kmmscand_init(void) { int err; + kmmscand_slot_cache = KMEM_CACHE(kmmscand_mm_slot, 0); + + if (!kmmscand_slot_cache) { + pr_err("kmmscand: kmem_cache error"); + return -ENOMEM; + } + err = start_kmmscand(); if (err) goto err_kmmscand; @@ -176,6 +259,7 @@ static int __init kmmscand_init(void) err_kmmscand: stop_kmmscand(); + kmmscand_destroy(); return err; } -- 2.39.3