From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72E57C4828F for ; Fri, 2 Feb 2024 17:03:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1E2A6B007D; Fri, 2 Feb 2024 12:03:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EA7126B007E; Fri, 2 Feb 2024 12:03:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D206C6B0080; Fri, 2 Feb 2024 12:03:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BE0386B007D for ; Fri, 2 Feb 2024 12:03:04 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8A3AB1A0778 for ; Fri, 2 Feb 2024 17:03:04 +0000 (UTC) X-FDA: 81747483888.30.556A12A Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf23.hostedemail.com (Postfix) with ESMTP id 6DB6414001D for ; Fri, 2 Feb 2024 17:03:02 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JKy7EIG6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of gourry.memverge@gmail.com designates 209.85.210.193 as permitted sender) smtp.mailfrom=gourry.memverge@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706893382; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HhrTG5AYewh5phNKX1uPxhcy8QhnMld2uJe/mWMdTe0=; b=2aufUWVZyANXcXwIR4OqL+vjyWEr1W7CU6Zcu+OWKCUApyzs9pXRkXwRA7L0qSFMzCxgD6 /ZVKxVkdgGVoWhCd/x95bAR96hZj+JYXo5UUZVlY0NJjjygMJuhsbueZXXfGAweq6e2TFJ YEnqh1/HvGhnGgSOIQVw2vCQ23IVNSg= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=JKy7EIG6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf23.hostedemail.com: domain of gourry.memverge@gmail.com designates 209.85.210.193 as permitted sender) smtp.mailfrom=gourry.memverge@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706893382; a=rsa-sha256; cv=none; b=l9oftCnUo4ZIWnrqdgjPW2K1nORMj6RiHidiaIBoHEFxkAJo+rxGNE2aq7qyEN+0z1Sqlm eZlU6ulfmqYR08V5btxOJovyMLNlJ+11/n+s4pGTKjTCMkuF6IrN21T9/tpiXtc5Vj3TeQ OhDtnbdIXzr1gZuFF+ZSFyeK0HCZqO8= Received: by mail-pf1-f193.google.com with SMTP id d2e1a72fcca58-6dddf4fc85dso1839915b3a.0 for ; Fri, 02 Feb 2024 09:03:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706893381; x=1707498181; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HhrTG5AYewh5phNKX1uPxhcy8QhnMld2uJe/mWMdTe0=; b=JKy7EIG6wIac1SyGSLknegbUxXXtgiuvQZ78xdGfNBrwzXuSdf4aCrhecfXitJCYoa YS5gI3OXdsYHzPXNTQlM36nJblZhbAld8KOBfKcQC55SNXp7xKriKLgpVJaPyNq+i1n5 DcFQdDYGHKG4yRZ/maArszNNOScgdz1UB5FyDWZsAOy8oGHCby8p1W4LIo4I1+9/Lz1k LfhFdyqkbLhTEXp0kBxx9D4pnSdDkNsjcsnybSfs52jQaqxY/hkUvjUPivCPcjfY8U1f tbSeinBUGnUG6qUWMIQE6Xp3DPJLBcwPWmGRwO96SWC2nWVBOC7qX/lnMiglc+5wSdOg vRtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706893381; x=1707498181; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HhrTG5AYewh5phNKX1uPxhcy8QhnMld2uJe/mWMdTe0=; b=OhM+n3uAVkmY+pcc12/ZkXDDk2tP57QxcSQT3l4yA1NHRKTw5z+dEmRqvfuIlROjum Agaxg7NzCPx9eG6a1CVnu4++13tVESzyTUz5ZUYWMEyDvthSJ4SNJXZDx+d/Iyj9w3+Q NTnpCzhLtWWw8G55JbV6ull7vjByRa7YCGl2rBnLPJ+MGYgGVwsvrYn+JWEEfcrhXHkO KrDadiCl6gM8VJ8dcj8qycYYRRWFNXEe4/2QkqYT/3TA7ImSkp3opF8NpjkXJJnIAfUC eknt8JBWdKm2a1xgYwlVnGCODSd2MzWuxHmn0JzRDJpREsvAgvb9jGAo2nSo+OX7TvJe sYOg== X-Gm-Message-State: AOJu0YwiFm908wzQeJh8tr6OUn2pK6aWbv3pjQPRk8aiGeElFfjHjnLk cZ7S/OdZhMQP/z0NY0tS94UwkmyfFJD2wbAbzxYT3mK7sEQQqb4N1YEpvuF3AA== X-Google-Smtp-Source: AGHT+IFogIopQqwUpwpEFsFzdYwSc8X+34bpQwF6wcEJlxSEsQ2LoOTeQtLBa1R1mPNFivEZAibTKQ== X-Received: by 2002:aa7:8698:0:b0:6dd:c3b1:797e with SMTP id d24-20020aa78698000000b006ddc3b1797emr2585012pfo.19.1706893381133; Fri, 02 Feb 2024 09:03:01 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCVt1bpVi7/1usRCD00UfVroAEBs2kkA7ivPuJBFs7e4G+50XRtr/OkpAzFSMfFRAHdUrmWFaRgbmluqN9+kP6pYoISXSBVGj3gGym0kGZAPJqyC9zCnaxeKxoVqIg+YsKhcBnckhYr0jqT+88HBKWVVvKol1k9qKs6zfze27eMF8LMwoCCImKMQ1QboF2LNyUiQlGDhILX78kacwGH3CEFwJF4MjCCPH0WZzUCY/EqU11rSp+7VSCcPyhqnACvJLqIenX7ox5uk6g+0BRmKDyW+u0Kc96vapdvPzWcSfp5uXlV8r4+b1LugH1OLSTzxHTdLvSuzkh+bhPyPGuGqqkRp6Jq/WmFwjFmTpaVnvynSrPgg6EXt8aJXZqPdvgU1hCvlfMTDAyWGPgEprUT0uA7CjpAMGLx9aO2UBB/uFO4lpMVdoIkgg+FhMnZNXJxNugpKs6x6pYKxEzulQq0Qc4VtjlWtT3Ml2/G1MpSSw2VARz4d2EBU7mh4um+v+jw+7cNDhrTAWNgBx7r4cq/lD29zYteZKCs+AhvgyOw94L0HxtBf8Mh8WR9T8cZvnNh0uxpT4ozEnaaTYxN/bWhlS0dY1lsdRGSWlqopk5eik5zKkpWz/fv0D5jTdAaQ5ARjZhTX69CwPcQUtgzfBrZ0y9Ic17GPXYKjlKblDPYcsHw= Received: from fedora.mshome.net (pool-173-79-56-208.washdc.fios.verizon.net. [173.79.56.208]) by smtp.gmail.com with ESMTPSA id z22-20020aa785d6000000b006ddddc7701fsm1866578pfn.4.2024.02.02.09.02.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 09:03:00 -0800 (PST) From: Gregory Price X-Google-Original-From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-api@vger.kernel.org, corbet@lwn.net, akpm@linux-foundation.org, gregory.price@memverge.com, honggyu.kim@sk.com, rakie.kim@sk.com, hyeongtak.ji@sk.com, mhocko@kernel.org, ying.huang@intel.com, vtavarespetr@micron.com, jgroves@micron.com, ravis.opensrc@micron.com, sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com, seungjun.ha@samsung.com, hannes@cmpxchg.org, dan.j.williams@intel.com Subject: [PATCH v5 4/4] mm/mempolicy: protect task interleave functions with tsk->mems_allowed_seq Date: Fri, 2 Feb 2024 12:02:38 -0500 Message-Id: <20240202170238.90004-5-gregory.price@memverge.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20240202170238.90004-1-gregory.price@memverge.com> References: <20240202170238.90004-1-gregory.price@memverge.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: t88omebkxczey9p1dbnhp1nphz4z94i4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 6DB6414001D X-HE-Tag: 1706893382-514623 X-HE-Meta: U2FsdGVkX1+JRcmX2F4GR1jVU3eqz4eZM5LUix7S+JNMr6BYISFmIUE7Jsu/UySlJV5Uhpv0eqjmorrsb+UktmrQ3DFDTK64LHxEQzWZzFLlMbf5ksphsQoe5lKZ1hB3fe1Ck+o+dDDqMYst8a1b22bSYDyQlQK16noQ2zfPFozy93HOi94RDH1rKbnT2Vvi4VH/BT5lyPo42VInG7I7tF+IcHrP/ioYRhppyj3VBxN33qJVHr8lwvoujWtkxVYeptPzJXtkjaD6tNg9W0Hmz/38tD9x8Y9kLfUvooX9JFTXzmLX7eEQhw9NqBMX9/LZQDRRLZJWUwbhUWw3e7AbVJY337Ba9wLxILW4rsyRjTiEDx+zTdVg8v3tzsCl0aCSV3O4Vgf7Q/P54mBVzZzY8exrJJxFeJkTJIOtTrqWSyO3ClFyW8p6bfAXol6JLJCFDI09wk1J8Y5LMOoZ72p9UCP0XI42yiLdEkLBvy373pIYKqiBBPm/XHhhwNvcmsVoukXs+LZD5Ljs550vyk9t5nTvQKDZQGkbEgcdyOTUGozNCyFHW2TSXRDMecfF+BKyqWbh2RnOq0g9TkxtP8bSVTvQf/bsoi+JQ0SOMxTy0BHVaTKLZHGjSh+aQDJhuve8yc09xzrtVOOYvA5UGSPZW0ybPQ5QK9dK23EDQTuqCPZ2aolgguwJWigy3pFUCVKW4+QZhfIE5R7vfa8jsyfHkJbdgsyCQj9jo9WeJ0R1AEu0djL3KZas4jvavZ6gcsJxrbwCDmKWRt2OdwF4TakY1MmpoRCcw9vUdoF0bWbEbHUH9Si+4RFvwtKJ3VAdX6t2PndLPlYq1hLwii/qDO+zJO7W91ddQ02CKT7uCyLOBDbUGJDIgTuEjkSEbE+FdaDZ/Iw8fsvDsY7aA5F01GnlVS7HZe0QfXksiOR4HLDzhZxu9zE3IHft5cUAvszDTl55gx378/Zm6hqzxa7TWcY ZF1M7/eA BOMfPmSkVeeUWPWEMNaYP1R0XbpxLJ5NMB82eCoffHKqJPC+MWGt/WCfhb0Bul6zwREi1snFncJ0ku/FuF4aAm1nD/uBtaCQEDveEOSL+DgqS3k77NNVYx7mbklXpBYuNydUx7Y+WKHDrFI7E+X3R/S0Da4vnyfL/oomGYulLVlPy2MEkMxq19Tm7qIRCOBhY14T0rNgoAcC8CMEgxNKuu7PHiExaSopBsaCaAk03nVLsJdpG6jsCD+RgNH925GZidqWDIcFHZLlTgJqPUzO9EN9qNcD6WzN5QPKrLKKqYTbHPWJAG8ByQKjYzNhxYUrmP2uf4OhNz4vnJOv7U/3l7ai5fP0DdGD0y2eGwOlRQFRReY0ejARH0vPmdO2AyHMAXJ+tEZ6bRrDU6KYjTr3FZy2uaI61FSTPVKTLDE64FA4yrxj5AyjrisdcjZszaMtETDB50EWvm0B+BUvjkZXv9AKHfHFmY0/NCGiQgw9HQWqiuvG096CXuV0BMYxNk/cwf/8vJ8NV5iiq1mp5/g7+JDL8Bn11gJHp4xsZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In the event of rebind, pol->nodemask can change at the same time as an allocation occurs. We can detect this with tsk->mems_allowed_seq and prevent a miscount or an allocation failure from occurring. The same thing happens in the allocators to detect failure, but this can prevent spurious failures in a much smaller critical section. Suggested-by: "Huang, Ying" Signed-off-by: Gregory Price --- mm/mempolicy.c | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d8cc3a577986..ed0d5d2d456a 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1878,11 +1878,17 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) static unsigned int weighted_interleave_nodes(struct mempolicy *policy) { - unsigned int node = current->il_prev; - - if (!current->il_weight || !node_isset(node, policy->nodes)) { + unsigned int node; + unsigned int cpuset_mems_cookie; + +retry: + /* to prevent miscount use tsk->mems_allowed_seq to detect rebind */ + cpuset_mems_cookie = read_mems_allowed_begin(); + node = current->il_prev; + if (!node || !node_isset(node, policy->nodes)) { node = next_node_in(node, policy->nodes); - /* can only happen if nodemask is being rebound */ + if (read_mems_allowed_retry(cpuset_mems_cookie)) + goto retry; if (node == MAX_NUMNODES) return node; current->il_prev = node; @@ -1896,8 +1902,14 @@ static unsigned int weighted_interleave_nodes(struct mempolicy *policy) static unsigned int interleave_nodes(struct mempolicy *policy) { unsigned int nid; + unsigned int cpuset_mems_cookie; + + /* to prevent miscount, use tsk->mems_allowed_seq to detect rebind */ + do { + cpuset_mems_cookie = read_mems_allowed_begin(); + nid = next_node_in(current->il_prev, policy->nodes); + } while (read_mems_allowed_retry(cpuset_mems_cookie)); - nid = next_node_in(current->il_prev, policy->nodes); if (nid < MAX_NUMNODES) current->il_prev = nid; return nid; @@ -2374,6 +2386,7 @@ static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp, struct page **page_array) { struct task_struct *me = current; + unsigned int cpuset_mems_cookie; unsigned long total_allocated = 0; unsigned long nr_allocated = 0; unsigned long rounds; @@ -2391,7 +2404,13 @@ static unsigned long alloc_pages_bulk_array_weighted_interleave(gfp_t gfp, if (!nr_pages) return 0; - nnodes = read_once_policy_nodemask(pol, &nodes); + /* read the nodes onto the stack, retry if done during rebind */ + do { + cpuset_mems_cookie = read_mems_allowed_begin(); + nnodes = read_once_policy_nodemask(pol, &nodes); + } while (read_mems_allowed_retry(cpuset_mems_cookie)); + + /* if the nodemask has become invalid, we cannot do anything */ if (!nnodes) return 0; -- 2.39.1