From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A3CEC38A2D for ; Thu, 27 Oct 2022 07:51:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30B5B8E0002; Thu, 27 Oct 2022 03:51:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 295E98E0001; Thu, 27 Oct 2022 03:51:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C0478E0002; Thu, 27 Oct 2022 03:51:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EADCC8E0001 for ; Thu, 27 Oct 2022 03:51:25 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B142A1A1214 for ; Thu, 27 Oct 2022 07:51:25 +0000 (UTC) X-FDA: 80065959330.19.7AE792E Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf07.hostedemail.com (Postfix) with ESMTP id DE4BE4001A for ; Thu, 27 Oct 2022 07:51:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666857085; x=1698393085; h=date:from:to:cc:subject:message-id:references: in-reply-to:mime-version; bh=no6JZxzuKIXDL1nI+sQ6MVtarCowvwN2y8ViNLYssKs=; b=WrdofcfYCsx1h89vKzGu6mLCuNJMIgr4gaVgxQbP/GJkHsjIjJXnCNlD U2iMxCFYQn1u770XldCb3OelJtVpy0yPcC4xVUIc8pOuLsM04dY7ynmIV wT1ieV2OKKF4i3rhWIW3cOIBqgm57uEQ0YSIxUsegmgFK3f7+coOBkxzw bcfxlJy582Wrdc16JRfafnCZR4dNUP0KF0/B+IUclFhzGNvZQLayqoPjX 0eksl4ZuZfRX7n7mn+OET4xGkTeaeiY9ZisN2EjoLS1Z3kaAB4I1gxvsb +4g8Pehm7JjHHiuAje074ELVrgG72At+7vt3u7cmwlqeypdmlx8JXZnY5 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10512"; a="309853202" X-IronPort-AV: E=Sophos;i="5.95,217,1661842800"; d="scan'208";a="309853202" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Oct 2022 00:51:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10512"; a="695703984" X-IronPort-AV: E=Sophos;i="5.95,217,1661842800"; d="scan'208";a="695703984" Received: from orsmsx602.amr.corp.intel.com ([10.22.229.15]) by fmsmga008.fm.intel.com with ESMTP; 27 Oct 2022 00:51:22 -0700 Received: from orsmsx610.amr.corp.intel.com (10.22.229.23) by ORSMSX602.amr.corp.intel.com (10.22.229.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31; Thu, 27 Oct 2022 00:51:21 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx610.amr.corp.intel.com (10.22.229.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.31 via Frontend Transport; Thu, 27 Oct 2022 00:51:21 -0700 Received: from NAM02-SN1-obe.outbound.protection.outlook.com (104.47.57.43) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2375.31; Thu, 27 Oct 2022 00:51:21 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XDZj8gkI2ISOni9a6zRe3GTL9ykSaUq0I3Slz/abtJq3FjBxL3OTBJKIR/ocb+FItb6Dm/CQ91zqeUrdV021haWfT/7tgHZr73oCYciv6bDr91XPJiK8UE4I2KcphsuIlmgnXHmfalDHE6FfNDbjgHhupXl0uDDgUz9ATEy2JgqmYDa8x7WCzY337XiTrSb4nk7A29sFodLQwYzmyXpZ2gndKcaLJIYD50Y95LS7dBgDEI4W8vnRdHW6sVUpKKYH+9TWaYZKpurMXiAJvRzaqR55hG58AKQZ7a9R98lrWKaeFEpxn9iaGYgX4jxYGGzvJA0C3nzaiLCyiH0BeZXsWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=99DWsW4vjoliw0hPPa1d5bSibfaygoEWG5D48+ldTBo=; b=IA8CkpEmk+gYSMviSPkToxoUgir20SZZdMJs26yAQTMdYV0hW3DdYqwn/f4uiHvx2nBm/QpbPsnKUTpj0Nev/oU3kEUVEcjU37v3BLETrCmM0FG1faQHKwvxYt1I6ht9kiSLXrzZQHPpZGepx20cSinIff/yzQ1oy2cKn8NbVAqHiseCBf50hwc+hQuCRdtMvw8dr7MNvRJXdbO4MxS3GdOYk/iQks4lDwt2e9L43SI5/Vak8wVqth1fDdngD2CmhPoyDEq75s7lpWdDkdLTIji0vskPa4crmWxY4puJRhFJexS6v9N695nx9+L9OId+xQte+9kWLE93fjgy6AQPNQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from MN0PR11MB6304.namprd11.prod.outlook.com (2603:10b6:208:3c0::7) by SJ0PR11MB5119.namprd11.prod.outlook.com (2603:10b6:a03:2d6::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5746.21; Thu, 27 Oct 2022 07:51:17 +0000 Received: from MN0PR11MB6304.namprd11.prod.outlook.com ([fe80::3f8d:1406:50e2:7bcc]) by MN0PR11MB6304.namprd11.prod.outlook.com ([fe80::3f8d:1406:50e2:7bcc%3]) with mapi id 15.20.5723.033; Thu, 27 Oct 2022 07:51:17 +0000 Date: Thu, 27 Oct 2022 15:51:07 +0800 From: Feng Tang To: "Huang, Ying" CC: Yang Shi , "Hocko, Michal" , "Aneesh Kumar K V" , Andrew Morton , Johannes Weiner , Tejun Heo , Zefan Li , Waiman Long , "linux-mm@kvack.org" , "cgroups@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Hansen, Dave" , "Chen, Tim C" , "Yin, Fengwei" Subject: Re: [PATCH] mm/vmscan: respect cpuset policy during page demotion Message-ID: References: <20221026074343.6517-1-feng.tang@intel.com> <87k04lk8vr.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <87k04lk8vr.fsf@yhuang6-desk2.ccr.corp.intel.com> X-ClientProxiedBy: SG2PR02CA0089.apcprd02.prod.outlook.com (2603:1096:4:90::29) To MN0PR11MB6304.namprd11.prod.outlook.com (2603:10b6:208:3c0::7) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: MN0PR11MB6304:EE_|SJ0PR11MB5119:EE_ X-MS-Office365-Filtering-Correlation-Id: 900b2a72-20e9-434f-7372-08dab7f0078d X-LD-Processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iuRosUvRHUvxlM+OrY81Vkssi+dJHXC2iE+L9In4NeYu4wo+XYoDwqMu7zNhVXl9tiTeUFjh3GoQsgrdWsyfIly7wdIM9UBSeqrLKMYn/uOrMpx4UdvNVj06ytgc9GXAS0Kdig8w72O/B3Tef2+DQFCjNyVjRkxXlqqt+SDcXm0c6CTTAUKuCWOcpEupAYmVZwQqPcsM+ueflpJZSumAGz2Rl7XbpEylhauOXzA3CFn9Poys/b32uv/PvUWzBdS1asOMOY0aND1nmljqYjYtPao2eN9izMmFesfsOF+KscSJEkStpNV2cH3qk3mvgl2pVhtbVMSn1C6z5wn7YPgFDn8Hswx9MAnal0KqGqrq3mD/bAN2GRz6LgoR0cgs8OAG6DAcFbuPCTi9pRZrjflmn6h6u0EGVC1xCQqrp3Icj39s16txkq1gBuKxQ+OEVPreWrArjMzgXo0uW7qPP6cbaSICLTQ+Db4aaU3RPeKnbkrm3/2L5i6iQhxv5lz77Ykf+DuioV+GkyGPHBARm1Zind6AoWzu2kyf4O/IGyOPz7UNFAjTUAE/DrzXKZKvK9X60wj7bhiK7HECb/LimBqObapfFbQxfEkiC0W5KGL68slMOfJEpyoJTXfXlX3blEOAnLkG+oFz49p1Ds+wjRytzx50EsJEKHFKu9rJIbhHBQQFh3ztBCVB3g7G+OE81TL0/LRcYGyBqnj+jwtR4iVx4w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN0PR11MB6304.namprd11.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(7916004)(346002)(136003)(39860400002)(396003)(366004)(376002)(451199015)(6512007)(66556008)(41300700001)(82960400001)(8676002)(6636002)(66476007)(54906003)(316002)(6862004)(66946007)(44832011)(5660300002)(38100700002)(4326008)(2906002)(6666004)(7416002)(6486002)(107886003)(478600001)(186003)(53546011)(8936002)(26005)(6506007)(9686003)(33716001)(86362001);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Gd0HFP2NywL4LYdwg92Dj5sADTmgd5BbuJqHN6egjFdLt6sH6AuoPeVOXbvA?= =?us-ascii?Q?LlzB4V6QVcUOqTAeKKsIddd6DKNVrQo6nnLnAGcsIEFoAEsmwYKss+BG6/KS?= =?us-ascii?Q?6t48v6UFwExs/VbvIMAk90LKCI028fVhh5FllwWaccMOAQRxvv8Shuezvbcf?= =?us-ascii?Q?zJ62ccoF5Flv0hhRDz0znS8C3/Mxq8m9OPChNKxOMEKVHZl9eL1ADuqqhd+Z?= =?us-ascii?Q?gq01QeoYeAN7FHbxLL04xm5pJ6EOJMgWchB3eTwi2qq9sIggNVaak7Kfnjua?= =?us-ascii?Q?rzMGc7k1ZRb5AmO3z4LJtuGOIdSkmm2P4eCMCS3ayPE1MLgwAHfNXrIIYv6d?= =?us-ascii?Q?kPxFXBSgPD7QAkF4gEBRT7P6SXJ1EwOTln6jfqtSj3ICwbYZpLhO9x3PMjx3?= =?us-ascii?Q?swKtv1QqMtmyCYtPDkHe+VEH26xvuZzfj8LU9dgvKjpn3Xp7O6H8j9wt3578?= =?us-ascii?Q?COEmW1MmlzfRWNNV0kL8SteZJEc40ymdrjTlwWlrjz+OseGrfPa91ArewbUl?= =?us-ascii?Q?aSp9cO70YAcFc/jEVDqGs/YCmDG0FGmrJYWtqaY/7tTMpwtcS/IuD66x31uY?= =?us-ascii?Q?Z3Er/WB9uWMa4sKj+lMa1tDkNpwP90ECEVwl3vY8q1Tn5PDWSN+xXWvVyM29?= =?us-ascii?Q?J3IRpNN5T1g7dm9yegJhNk09JDub0imlYjS6fKi75n9OX2T5Yk8Ixq+YZZm2?= =?us-ascii?Q?6nP2J34DVptYeuEoiy09mNbF8iWN/+TP8uCdj21Yn50QN82ghxcJ0BahrXTc?= =?us-ascii?Q?QrWBtl/1sHiYXzc7UbnnIGnS8ia6qw9Ezo/6CnOEam7N/j+8Brq17ZehjUlH?= =?us-ascii?Q?MGxhVVw0BRWQrfPyfV36/muIFn99DajTdf30wbnNh6dpks59XlNB7yylanVj?= =?us-ascii?Q?md6wKtVDIUKRC/e1Yp3UOeOMqT3VALvHIsc5JZfs9H6sVQBmPqVzvCgaC6we?= =?us-ascii?Q?BZ8JcCSbEyBHxPcrRFkWDgx+l7Tj5m7wSJQu1Z8MrCTrPwW7TgzkRmahxR7T?= =?us-ascii?Q?HTR0xUskp9BhK9UypC5D8WwnrpleBkcukKA3SnxwQIGVmZPpjqWZComJhL+8?= =?us-ascii?Q?UcPBBoJuBtU367lvZe8AU+Z5hITHw9CeobT2Y7E43JZZq4gVFDWf1vkmPKJP?= =?us-ascii?Q?EWWzjA7/JQAiXDDgk+JLYvAqN6o6AdgmkbFR8rp4MWKjkShEvZwx1lOM279Z?= =?us-ascii?Q?y0EGc9Vi3jIa6Elib2EhB3KN7S8eaoQRMAfMQfNiXo7trvZKKpS73unCo5v1?= =?us-ascii?Q?sTrDnV8Fk1NTjkzU/rGJ4ghU3qMY5gqZ7MuSHBW5Ch+VvPVG6JDnaiB+kHDZ?= =?us-ascii?Q?Lc0JGF8JbFk2tCsOruB3e/cBhCmn02c9wX2VCXlZAO4H6e16L9GH5F/Ft7Ph?= =?us-ascii?Q?f/C3vISaZirWIouMrMNjQ6U9mceBCVtWnRtKGf0dKQT5FIg/RK28Tr6Y5y96?= =?us-ascii?Q?Ol1DflUKu3qM6vw5K2Wo9RV+TUazhCvBxhsezwFIozOCFYJNHCHQOAShOnT7?= =?us-ascii?Q?alS32xAsFRLnYa8r8Z6r1VJfVQCkOP1Eon3CjrDVT/lVdMuuDE5QTbX6oDYY?= =?us-ascii?Q?L2VUOPpma9m2oDvl7ddGr65JQDWMKCFonUbVms5j?= X-MS-Exchange-CrossTenant-Network-Message-Id: 900b2a72-20e9-434f-7372-08dab7f0078d X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2022 07:51:17.4945 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: M5JAQUkewo7tRZMU+Nmy7dZ4SUgSrwSNd7+SMOY8D9oDBTT74ttyA7uphI9Mwwv7rl1gkaUePZqwXFWiFzwf7w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5119 X-OriginatorOrg: intel.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666857085; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=99DWsW4vjoliw0hPPa1d5bSibfaygoEWG5D48+ldTBo=; b=7eyVRdNChxtUoo027rGluqBditfLNm++JTkEO36YlIgg1s5Y+whvwTgfaJtTOD3VHIRgQF FPxNZyNOamLiXG1Gh7FomH25RJRBz0W9bwHiaBqYYYpN7LmMxwHL6+gXYgrJGN8Vzn15wc GWD9Ix4/7RNwOhNEptyTi35GFT9sfwY= ARC-Authentication-Results: i=2; imf07.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=WrdofcfY; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}"); spf=pass (imf07.hostedemail.com: domain of feng.tang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1666857085; a=rsa-sha256; cv=fail; b=duk0ESjwC79lMyAoail19eVxbzMgDEIRcgJx88Rc8uzIpXVSXZvZvA15SPN0aJpjuMOafR MLBcEy+IpjmkgOoTJYLvRXSxue7nzwUjfxqSA1wbe0cwp36GoFASHvQ7J+i/c7J3WtvvUO JBfFgvJphXzQWqgaqqg4uaFWUyK3olo= Authentication-Results: imf07.hostedemail.com; dkim=none ("invalid DKIM record") header.d=intel.com header.s=Intel header.b=WrdofcfY; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}"); spf=pass (imf07.hostedemail.com: domain of feng.tang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=feng.tang@intel.com; dmarc=pass (policy=none) header.from=intel.com X-Stat-Signature: wmx3i5h9kkis4wubzkb3dtd6uihx965a X-Rspamd-Queue-Id: DE4BE4001A X-Rspamd-Server: rspam07 X-Rspam-User: X-HE-Tag: 1666857084-523269 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Oct 27, 2022 at 03:45:12PM +0800, Huang, Ying wrote: > Feng Tang writes: > > > On Thu, Oct 27, 2022 at 01:57:52AM +0800, Yang Shi wrote: > >> On Wed, Oct 26, 2022 at 8:59 AM Michal Hocko wrote: > > [...] > >> > > > This all can get quite expensive so the primary question is, does the > >> > > > existing behavior generates any real issues or is this more of an > >> > > > correctness exercise? I mean it certainly is not great to demote to an > >> > > > incompatible numa node but are there any reasonable configurations when > >> > > > the demotion target node is explicitly excluded from memory > >> > > > policy/cpuset? > >> > > > >> > > We haven't got customer report on this, but there are quite some customers > >> > > use cpuset to bind some specific memory nodes to a docker (You've helped > >> > > us solve a OOM issue in such cases), so I think it's practical to respect > >> > > the cpuset semantics as much as we can. > >> > > >> > Yes, it is definitely better to respect cpusets and all local memory > >> > policies. There is no dispute there. The thing is whether this is really > >> > worth it. How often would cpusets (or policies in general) go actively > >> > against demotion nodes (i.e. exclude those nodes from their allowes node > >> > mask)? > >> > > >> > I can imagine workloads which wouldn't like to get their memory demoted > >> > for some reason but wouldn't it be more practical to tell that > >> > explicitly (e.g. via prctl) rather than configuring cpusets/memory > >> > policies explicitly? > >> > > >> > > Your concern about the expensive cost makes sense! Some raw ideas are: > >> > > * if the shrink_folio_list is called by kswapd, the folios come from > >> > > the same per-memcg lruvec, so only one check is enough > >> > > * if not from kswapd, like called form madvise or DAMON code, we can > >> > > save a memcg cache, and if the next folio's memcg is same as the > >> > > cache, we reuse its result. And due to the locality, the real > >> > > check is rarely performed. > >> > > >> > memcg is not the expensive part of the thing. You need to get from page > >> > -> all vmas::vm_policy -> mm -> task::mempolicy > >> > >> Yeah, on the same page with Michal. Figuring out mempolicy from page > >> seems quite expensive and the correctness can't be guranteed since the > >> mempolicy could be set per-thread and the mm->task depends on > >> CONFIG_MEMCG so it doesn't work for !CONFIG_MEMCG. > > > > Yes, you are right. Our "working" psudo code for mem policy looks like > > what Michal mentioned, and it can't work for all cases, but try to > > enforce it whenever possible: > > > > static bool __check_mpol_demotion(struct folio *folio, struct vm_area_struct *vma, > > unsigned long addr, void *arg) > > { > > bool *skip_demotion = arg; > > struct mempolicy *mpol; > > int nid, dnid; > > bool ret = true; > > > > mpol = __get_vma_policy(vma, addr); > > if (!mpol) { > > struct task_struct *task; > > task = NULL; > > > if (vma->vm_mm) > > task = vma->vm_mm->owner; > > > > if (task) { > > mpol = get_task_policy(task); > > if (mpol) > > mpol_get(mpol); > > } > > } > > > > if (!mpol) > > return ret; > > > > if (mpol->mode != MPOL_BIND) > > goto put_exit; > > > > nid = folio_nid(folio); > > dnid = next_demotion_node(nid); > > if (!node_isset(dnid, mpol->nodes)) { > > *skip_demotion = true; > > ret = false; > > } > > I think that you need to get a node mask instead. Even if > !node_isset(dnid, mpol->nodes), you may demote to other node in the node > mask. Yes, you are right. This code was written/tested about 2 months ago, before Aneesh's memory tiering interface patchset. It was listed to demonstrate idea of solution. Thanks, Feng > Best Regards, > Huang, Ying