From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D68B2C71132 for ; Fri, 18 Aug 2023 02:29:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB6B1940050; Thu, 17 Aug 2023 22:29:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E67A1940009; Thu, 17 Aug 2023 22:29:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE007940050; Thu, 17 Aug 2023 22:29:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BEBE9940009 for ; Thu, 17 Aug 2023 22:29:29 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9560A120543 for ; Fri, 18 Aug 2023 02:29:29 +0000 (UTC) X-FDA: 81135644058.08.79043A3 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2081.outbound.protection.outlook.com [40.107.243.81]) by imf30.hostedemail.com (Postfix) with ESMTP id 927C28000D for ; Fri, 18 Aug 2023 02:29:26 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="AikiJu7/"; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.243.81 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692325766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q/BMvWwHFo52FTv/uJJvXh7Eh1swdBtPYZHlmZnOuM4=; b=fWhWVLS1P4CgoFChpYMAW8lZP2MdhUxEKa+1wW4r5TlawgsshW6OG2i+BZLbwtHVbuBRxS iCgn+Dptjj1FWwdWgZMe5XgNGo1hT9mHsmgRG/XYdLLlA7bnRoYFEYSxyi+9+aQNnMtFFB zfObxpATk3CAGb65HcNsyE/0WUjqHbA= ARC-Authentication-Results: i=2; imf30.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="AikiJu7/"; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf30.hostedemail.com: domain of jhubbard@nvidia.com designates 40.107.243.81 as permitted sender) smtp.mailfrom=jhubbard@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1692325766; a=rsa-sha256; cv=pass; b=3RQJIAzk2Su0kbRJqk5qW0MO2F78FKnXRFox848DrVyQGPFCwB6PCF1vxsjsjklHhrt24r 5aHUC9klZCMQ3y2cwltlCp6S2yVkMGSSzQaI1aLV7RlESwXvwwHuw2nxPmiT8RcayQtHd3 1N30OoY9n8+rdFNG8ey9Y7B55R4tMhM= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YFaleqMee5RWpuHOyOA28g77dZztQmCK4J0uNuDymESnR6yRu6qDeSc8oXXwq9uQaAhZVvsZSo1pYgyoGXpjE2bfpBtZQXUoQ9spbQHa6ZVt09EfR5BUkTL6n/4lZo7hOl+RLmUy7FRXI0+yy7hoJJt0IaD/sCw5tDG1j5uJx264xG+gkNbwjoWX7Ba0kmmNWfJSqFfSqTmjMERkS7PANCoZT2ZsCEVf6OS7SwVokIHjfzVIxCV6kmoOb53zFqy4964j3UPu0HrXUPJbSAXcSS+wJJV6YFNLQYNUCRXYyVQ3KT25DB5uEiORL05aeVUPDJgBQiEKFINsyYfn8i6fig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Q/BMvWwHFo52FTv/uJJvXh7Eh1swdBtPYZHlmZnOuM4=; b=coOsfM2+bimn/ILnJVE6qnVS8pE2ENvqH4tEwqmxGdOIX1ZQ9ZIhiTRsh/rz62HC6zuMIVpHT/4N/GWjgB6DO1XBF7m2gu9mVrMKmlYg8bKgXh3ono5orCAoTWl3RpSM7wPUHAIHwHWidfIsnPjgzBp3sWIULKKi17IOCDEcpxyXlnS36oOVzxo7AeW1AvJcyED6XBrK7e6XD2B45zESscrNw+Io/7i8TE9+rRZKVgv59ronw818yjWUGGSiJe+feXSGCj/gbCsGEuhfHCokdqtZFg9qN1TFbolGjgJpxRDD+xd6Lw9sBBIyCOwjDwcQ05c+gvFkkKLNgw8bleZlcA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Q/BMvWwHFo52FTv/uJJvXh7Eh1swdBtPYZHlmZnOuM4=; b=AikiJu7/RCu1b9cBhbo+szUrQnGQg/WlCCmMVeZpYkLEiqqU8i5Ocupftloej09CQAfejHa3xbt/o1gwb6S/fC0sEryh/SwbmtqltLj3JSW6scVn2pg2p5Qc/DSoM48oAFPJ18qM+rU5l5QqxxpcSK0J0eI5CdpI9bVVgksw8uiq1/j1Ev+vh4d0Ge8YG8bwC/7MrB0cpf4F/wrVvFFSFSMjBJN0724EKEFYZYuzuawJ5laBqmWgNeUEwr8DZ4po1T9l8huBI8MkDtbvTt3qK22GMV2DYCifzoCFZZvBGIEXKUIkwj/y57GWYRLEFAdk8Swqh5QqifRK5ifRYspcTA== Received: from SN4PR0501CA0072.namprd05.prod.outlook.com (2603:10b6:803:41::49) by DS7PR12MB6168.namprd12.prod.outlook.com (2603:10b6:8:97::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.20; Fri, 18 Aug 2023 02:29:24 +0000 Received: from SN1PEPF00026368.namprd02.prod.outlook.com (2603:10b6:803:41:cafe::ba) by SN4PR0501CA0072.outlook.office365.com (2603:10b6:803:41::49) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.15 via Frontend Transport; Fri, 18 Aug 2023 02:29:24 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SN1PEPF00026368.mail.protection.outlook.com (10.167.241.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.14 via Frontend Transport; Fri, 18 Aug 2023 02:29:23 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.5; Thu, 17 Aug 2023 19:29:13 -0700 Received: from [10.110.48.28] (10.126.230.37) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.37; Thu, 17 Aug 2023 19:29:13 -0700 Message-ID: <4cb536f6-2609-4e3e-b996-4a613c9844ad@nvidia.com> Date: Thu, 17 Aug 2023 19:29:12 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v2 0/5] Reduce NUMA balance caused TLB-shootdowns in a VM Content-Language: en-US To: Yan Zhao , David Hildenbrand CC: , , , , , , , , , , , "Mel Gorman" , References: <846e9117-1f79-a5e0-1b14-3dba91ab8033@redhat.com> <4271b91c-90b7-4b48-b761-b4535b2ae9b7@nvidia.com> <37325c27-223d-400d-bd86-34bdbfb92a5f@nvidia.com> <5c9e52ab-d46a-c939-b48f-744b9875ce95@redhat.com> From: John Hubbard In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.126.230.37] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SN1PEPF00026368:EE_|DS7PR12MB6168:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d4058d0-11c6-4020-408c-08db9f92efcd X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: K7dff5zU5xFUZl79D4iUyi7kJh840sBtEJBG1BI2Tamfl+tY+zB6QOANR/O9BKS+HoVf4HqiYtXgj04+SSNc5JhAeyyDOg4fdvRUl2VxTj07rDxGZSR2nywjZAZ0MfYcPdUzBKN8Y8U8G32s4weJvFG/elV/IAN2fX86HAvc3dsuzgenKSmr7QSBNrWsJYYxDF85vX+C75NUTTLcnj8DdjEDVcPtlNVTtvTCNwR7kiexjNKI3BofxOuTQpWgxci2DeDUnLrqbFnFXV/GoD6yNRgNBD6uZvX3ZLGN0buIFiJ15BnnrCUIQvZxw2z5vj0swPaR1h/VMVL8o/lvRNOz07TDufWiGl+XDChuo6c3NZYQ1XoG8YIC5riU1GLrBnUSF3pH/oKHhjWzp/mhz2jHCaDQVz1AeCa7q8oR76eMFwlcM93v3+618+BekJA38mrrKMtZabeBrcq5qoyiGy0T3d98yM449/Qr2vZDihW/EkPevbFMJfmDst8HJV0KY0jMJ2zKHr9+2t7Vxb7B3N8x39aW0pbKCr3AzfDgoo9mhF/JXrwkaX+E8Nzkb7ihO5h4ISnxR/vcbl1FsDWWPrX/F4JSbsyWPV0ar6Am/tgvoYdp+/9HJM0OIjrdVrXfiSCp15qgaitAy8B1Rrl1PyObF6UbDQu1Nw3mSlP0+k3TKgtWak+EXigpi/hmFx+ZI2+wUxueqWVA9G5XLWAcFemtFy0WrxbnFciDqer5Zkf7vz4v5ZVdD6Pa1Y8VtyzE/mlLeBtIz8ywFmYsmwYC4Jf8TQ== X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(376002)(346002)(136003)(396003)(39860400002)(451199024)(82310400011)(186009)(1800799009)(46966006)(40470700004)(36840700001)(356005)(40460700003)(2616005)(426003)(336012)(26005)(16526019)(53546011)(47076005)(36860700001)(83380400001)(5660300002)(8936002)(4326008)(8676002)(2906002)(7416002)(4744005)(478600001)(41300700001)(70586007)(110136005)(16576012)(54906003)(316002)(70206006)(82740400003)(36756003)(40480700001)(31696002)(86362001)(7636003)(31686004)(43740500002);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Aug 2023 02:29:23.8898 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4d4058d0-11c6-4020-408c-08db9f92efcd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SN1PEPF00026368.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6168 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 927C28000D X-Stat-Signature: r9thzfz9fdd98mhgu975gcx1iw59tytc X-HE-Tag: 1692325766-321703 X-HE-Meta: U2FsdGVkX1/xSiv78MA9xXMOyfNXSQ8DmvVTe2jpS3mAnYrwjQMWOMmuuORRAdwl2a40hfnwDR7OrQhXsadgUg+/fhGsxWdte8VMeMmIGt2TNE0n5uKpYvaCiJngWhtpf1TLaAPyG5D44t4LxCBBDbBpQDOJTrWmI9pw54nw0h+gIm3Tbo84VPX0sMGM56x5QirpvXmvBVGGmj+ASyEdadcZdYKBktsDdwSsuPoe60f7NG3uQImTuiWd2gqRJey/wnVtRSLl+L9+lszc4GVesF6zVJrlNQXTPIPQzLvRS2gJl56anMahXvNcGCEkBtBA//s35BSoIGu2e2KrtAi6O2bnERs6aBIERDE0yEhdScKhq4/oBOjzcdk3desRBvbofIbQkJ8jADuvTcLg91jSHnYtGEJjCMeSTPA1eZdEQa7CFxttYuyaP+AKk4vtNMHQCY6PTIZ2RKc827verIFkHMaVziPTCbKrePsBbps8jMuFUEh+fY5drmhDeSnOfuq6QaFJIDAqVsW/UZ7iFY3jBfNL4/dTf1XEli7T/ZQt8vCPEfKRcBYWLFaKnE7cbtt/0s0Jux78EOMlTmy1l1ZyeoZarzrbWIkIhNXtwGKXvIfRZwF9KHFZ5XBsLiA3WOzTUQFT3EgZ81ot8OibIr2bP+FnJRRpByKdUo33+39FkIQnEN1iQlapFjUSHlStG6llfDbsFOIcC3VfdTbpOZmAW4O0N0WVIbS0N4oYggxc72Z6mG4Ms36wNzyke7ZRl9g5VAXGGDqz0ODNWfdPLC4DGIIrBKxrrCe2jDinqrm1t+8GkFrPvYJdF3cWh6iQrgfEl/QiNdfJtY08Yowi/k96+ofuBcPVpwIXwXusEJ6sBGl4Sda3jPUIYGmapcimjlUN7b2OfwttwN84feZk6JSCQmpVLUQ/MTZm6ab1KTWDAh6q7XKrAdK6PsyDUlGWvUUDsUvKkMPaCCX4aCTp7h8 i2ZKUh1W c8mOT9tY+7BjqyWsFH4OmJ92rYceI/6DYtIw09dyP7/O2TZRR5t3pHAp7h5uhCe7mDN+xg64feXZykPugBAnbJUlbxo9F8LxGl+EfzdhlpjGdpJkjTI37KESRvlWaHLBPJnJsW9CXtLRiTXLsSOQTi/fgj4v3RFdp7AGLEzHwT/D1HDiH4NPabverR25C6KpqN6A4x4qi+jhln5Qc+Xieh0cQ67/gHiW2J6LVsXIFC8MonYGoKvkOFxP9DE3bAtjDnzO0RboHitBtvB/YQEJhQdzNBa7zDH58dwE33mPXQfRD6qFJa+AbgjcY/6oJurQmn4wrID0PdYckpiUfxIzd91kmqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 8/17/23 17:13, Yan Zhao wrote: ... > But consider for GPUs case as what John mentioned, since the memory is > not even pinned, maybe they still need flag VM_NO_NUMA_BALANCING ? > For VMs, we hint VM_NO_NUMA_BALANCING for passthrough devices supporting > IO page fault (so no need to pin), and VM_MAYLONGTERMDMA to avoid misplace > and migration. > > Is that good? > Or do you think just a per-mm flag like MMF_NO_NUMA is good enough for > now? > So far, a per-mm setting seems like it would suffice. However, it is also true that new hardware is getting really creative and large, to the point that it's not inconceivable that a process might actually want to let NUMA balancing run in part of its mm, but turn it off to allow fault-able device access to another part of the mm. We aren't seeing that yet, but on the other hand, that may be simply because there is no practical way to set that up and see how well it works. thanks, -- John Hubbard NVIDIA