From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BFB4C433FE for ; Thu, 10 Dec 2020 00:43:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DED6122D5B for ; Thu, 10 Dec 2020 00:43:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DED6122D5B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 528966B0070; Wed, 9 Dec 2020 19:43:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F07F6B0071; Wed, 9 Dec 2020 19:43:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 215176B0072; Wed, 9 Dec 2020 19:43:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0005.hostedemail.com [216.40.44.5]) by kanga.kvack.org (Postfix) with ESMTP id F0A4C6B0070 for ; Wed, 9 Dec 2020 19:43:47 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C24828249980 for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) X-FDA: 77575524894.19.men92_0f01d21273f4 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 9DCD51AD31E for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) X-HE-Tag: men92_0f01d21273f4 X-Filterd-Recvd-Size: 7266 Received: from mail-qk1-f193.google.com (mail-qk1-f193.google.com [209.85.222.193]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Thu, 10 Dec 2020 00:43:47 +0000 (UTC) Received: by mail-qk1-f193.google.com with SMTP id y18so3284208qki.11 for ; Wed, 09 Dec 2020 16:43:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=sUpjC2JGVFFWHT2eOX4v5Pd/VbxmqCdGzRg2mz4PxH0=; b=cZikR+BeHtWn6M7s+IlCMr3/UuAQ5Ar0APOR0SvjzWfOTd5DE7tvyvarLLSgwkqSis XbrAoIWDR3Fac/+0JXDiXLCy/u95KDigXA3b1hdYsBMGjeokv5Qyw/h9fa+CoWJKfG43 WnF5pg8pyyJzwJLzfwD2HMuKz5i0JvcZnMhyDndY81imB9SVwQR5xQyN02diPodndXsY S/CXCR9ezkPpyFFpMRxlr1DMUKt2qrw1tM4Z0lAtiNR8lLorsKDZF3zx94i2YJtL+LXm LAkbOfFiz5A/XiffhzSLpvBTddVDN+cv2HALO5EazslsZiJ/W3+omgm76d+odBso4xKN J2AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sUpjC2JGVFFWHT2eOX4v5Pd/VbxmqCdGzRg2mz4PxH0=; b=c9IrzhaGmGh0KgDCfVwhm3BKJeEPRtOPzp6KBOrDhpiKWaUqhmnAESGtb+qQZx3oQ6 pGj9+QkEMIqzFtDoA7yxKe4X6P4tPJye1C8Zz0XPmFRSY/3dafVuvyvxTCfMtwElsT9I DQDnrkvvBEB6lKgN8XLP6ffsYORePAuIpHhg9b9IrbKIA4jTFDalBSIea3koILoeQpNT tWEdV4HXefz6Qx5ikt95oH45ge1wje+/3DL2H+lFoTeao91z1oWEr8VcDUSGwj0U1hvb liYAOO9LgZxX/1hJovslcMpB+3UTWz+gbUduzpKyS+fKqHT8I6ozKygo18A9DhnREk3I UlmQ== X-Gm-Message-State: AOAM531q/bCTgFua/TTYuTf5mYLLDsl9lOm4Mm6hqir3z/YB+Bses9xg jAlJjknpBJA3KHYjY1CyZL8d1w== X-Google-Smtp-Source: ABdhPJxpG6COXfUw6XI/ibVZKCss5u6FLno1P2o5X8L0HXLNfwnptSeK3jVyPLbr4+90apRx6fDu0A== X-Received: by 2002:a37:4d58:: with SMTP id a85mr6051943qkb.324.1607561026529; Wed, 09 Dec 2020 16:43:46 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id y1sm2538745qky.63.2020.12.09.16.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 16:43:45 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, vbabka@suse.cz, mhocko@suse.com, david@redhat.com, osalvador@suse.de, dan.j.williams@intel.com, sashal@kernel.org, tyhicks@linux.microsoft.com, iamjoonsoo.kim@lge.com, mike.kravetz@oracle.com, rostedt@goodmis.org, mingo@redhat.com, jgg@ziepe.ca, peterz@infradead.org, mgorman@suse.de, willy@infradead.org, rientjes@google.com, jhubbard@nvidia.com, linux-doc@vger.kernel.org Subject: [PATCH v2 5/8] mm: apply per-task gfp constraints in fast path Date: Wed, 9 Dec 2020 19:43:32 -0500 Message-Id: <20201210004335.64634-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20201210004335.64634-1-pasha.tatashin@soleen.com> References: <20201210004335.64634-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Function current_gfp_context() is called after fast path. However, soon w= e will add more constraints which will also limit zones based on context. Move this call into fast path, and apply the correct constraints for all allocations. Also update .reclaim_idx based on value returned by current_gfp_context() because it soon will modify the allowed zones. Note: With this patch we will do one extra current->flags load during fast path= , but we already load current->flags in fast-path: __alloc_pages_nodemask() prepare_alloc_pages() current_alloc_flags(gfp_mask, *alloc_flags); Later, when we add the zone constrain logic to current_gfp_context() we will be able to remove current->flags load from current_alloc_flags, and therefore return fast-path to the current performance level. Suggested-by: Michal Hocko Signed-off-by: Pavel Tatashin --- mm/page_alloc.c | 15 ++++++++------- mm/vmscan.c | 10 ++++++---- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2dea5600f308..24c99b3b12af 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4932,6 +4932,13 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned in= t order, int preferred_nid, } =20 gfp_mask &=3D gfp_allowed_mask; + /* + * Apply scoped allocation constraints. This is mainly about GFP_NOFS + * resp. GFP_NOIO which has to be inherited for all allocation requests + * from a particular context which has been marked by + * memalloc_no{fs,io}_{save,restore}. + */ + gfp_mask =3D current_gfp_context(gfp_mask); alloc_mask =3D gfp_mask; if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac,= &alloc_mask, &alloc_flags)) return NULL; @@ -4947,13 +4954,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned in= t order, int preferred_nid, if (likely(page)) goto out; =20 - /* - * Apply scoped allocation constraints. This is mainly about GFP_NOFS - * resp. GFP_NOIO which has to be inherited for all allocation requests - * from a particular context which has been marked by - * memalloc_no{fs,io}_{save,restore}. - */ - alloc_mask =3D current_gfp_context(gfp_mask); + alloc_mask =3D gfp_mask; ac.spread_dirty_pages =3D false; =20 /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 7b4e31eac2cf..f51581e33fe6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3233,11 +3233,12 @@ static bool throttle_direct_reclaim(gfp_t gfp_mas= k, struct zonelist *zonelist, unsigned long try_to_free_pages(struct zonelist *zonelist, int order, gfp_t gfp_mask, nodemask_t *nodemask) { + gfp_t current_gfp_mask =3D current_gfp_context(gfp_mask); unsigned long nr_reclaimed; struct scan_control sc =3D { .nr_to_reclaim =3D SWAP_CLUSTER_MAX, - .gfp_mask =3D current_gfp_context(gfp_mask), - .reclaim_idx =3D gfp_zone(gfp_mask), + .gfp_mask =3D current_gfp_mask, + .reclaim_idx =3D gfp_zone(current_gfp_mask), .order =3D order, .nodemask =3D nodemask, .priority =3D DEF_PRIORITY, @@ -4157,17 +4158,18 @@ static int __node_reclaim(struct pglist_data *pgd= at, gfp_t gfp_mask, unsigned in { /* Minimum pages needed in order to stay on node */ const unsigned long nr_pages =3D 1 << order; + gfp_t current_gfp_mask =3D current_gfp_context(gfp_mask); struct task_struct *p =3D current; unsigned int noreclaim_flag; struct scan_control sc =3D { .nr_to_reclaim =3D max(nr_pages, SWAP_CLUSTER_MAX), - .gfp_mask =3D current_gfp_context(gfp_mask), + .gfp_mask =3D current_gfp_mask, .order =3D order, .priority =3D NODE_RECLAIM_PRIORITY, .may_writepage =3D !!(node_reclaim_mode & RECLAIM_WRITE), .may_unmap =3D !!(node_reclaim_mode & RECLAIM_UNMAP), .may_swap =3D 1, - .reclaim_idx =3D gfp_zone(gfp_mask), + .reclaim_idx =3D gfp_zone(current_gfp_mask), }; =20 trace_mm_vmscan_node_reclaim_begin(pgdat->node_id, order, --=20 2.25.1