From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 632D7C36018 for ; Wed, 2 Apr 2025 19:50:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C11EC280003; Wed, 2 Apr 2025 15:50:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBFBE280001; Wed, 2 Apr 2025 15:50:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6035280003; Wed, 2 Apr 2025 15:50:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 84FAD280001 for ; Wed, 2 Apr 2025 15:50:47 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C8768ABFCD for ; Wed, 2 Apr 2025 19:50:47 +0000 (UTC) X-FDA: 83290146534.06.FE4DB8A Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) by imf12.hostedemail.com (Postfix) with ESMTP id A928740009 for ; Wed, 2 Apr 2025 19:50:45 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=CbHjg4o6; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.178 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743623446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FfMrrYY13YZzcIjXFNy74fRKR0QYf/adnBC1OdclTvQ=; b=LhiFSN/KsO0FzbKcyxEI94Xsv7XZYiOHI68ikOHdo2lzQSzMbueOyu3/mZLbiV82j4MsB3 /emCDTK38893S7Br7uAPsjwJW8Y7vrthv6PYYwHQJcQtLIRFIIKsD3FiPDPsA5F9IfbyqD IuJ5sxDE67fEVEvLuBXBFgFRaPxiG14= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=CbHjg4o6; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf12.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.160.178 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743623446; a=rsa-sha256; cv=none; b=bWq8KbuSbl0wxbbkPXD5CR2lV8ou82RJurKNBZN2tT0iSthlKtgbllTP7AOQsBX/bJiQ6R rKnsAe7SSQrW/q3vGlySHlgh5HwFrJwBWaM/zaeHMd7W5WvOZ4nuzogspEi9FqCW7Sav0j RT3TJ2luiQJsBXTpTw0ypxR01Qiv3yg= Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-4769aef457bso1062821cf.2 for ; Wed, 02 Apr 2025 12:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1743623444; x=1744228244; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=FfMrrYY13YZzcIjXFNy74fRKR0QYf/adnBC1OdclTvQ=; b=CbHjg4o6dgKf4WY7GirYn2HwKjo+RqAeneNHLKr+2UhA+D3Ewz8HPHX1xL8jBkKWRh NcIHqyfqrBU0ExXI9seAIrPldhR9/3owc5wq+axg42fcl6C9wea+sPge471m5sgM3gbI HdlWdsBraIDYmscOxJYD3V5J2cxtKXXrhwe4xuGxLXxfxL9a49FP8mWimJO2VX0l4Q/P KuKx8LjmE84YGSWR7tYDVw45RtkrPL7usQD6Usm0QRoV08AOtVUz3H+svJhkPI3i5Oad ADBF3qf928RHY19PEf+qhCv6D8XvswgL6nrJlKg+jkWdj3oAGWq5jhWrwJJARgxnh0F6 EmJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743623444; x=1744228244; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=FfMrrYY13YZzcIjXFNy74fRKR0QYf/adnBC1OdclTvQ=; b=F8hI8kT1r+jrjofEP8S4Gb7nVt0cs2xyA31qemRgoOXygyydCITf+p2z6/Lf6yAMC6 nZ97R3wR/du7djkdAJfy/RlwTIYhkJ5EIkkmkEWCDqQTVtPHipYabdH0aSMOMQPvcYF/ hi1ztRUEn+EkClCqSO4zeY4wXbzs64mhPq8YCmJXNsiUnVEaqfuUGEBDReY9ZiiY3GZC VG3B/6PfaWXXruOdSqSK3iXYziJOj57bNoU/fccMELDoF5+gZmsRjqc13C1HqaYBPZrM pbmS8ZFyTWyy7iP+17Er6soUIKRfkSOfvyzTL9DWPgYPXStt3PJ3/odi6ZuFp2vSCE1N 0qkw== X-Forwarded-Encrypted: i=1; AJvYcCVu1EH03ajgV8CunjKJmSfdRbThYIARuk72MRc2qgjjPNzLuJQ/0o8oW0t6qwO3+6QXhSsqobigRw==@kvack.org X-Gm-Message-State: AOJu0YxsoohhB+vVGxBZzzFtL+hI5uXQTjnQJf7reFc6kc246osexOZ5 Nkz70qa1W4uwMRrRSu9/oRBYJjJYFOQAoEYcxGFeaevKniYsqxpnk9tZ/xJAAfQ= X-Gm-Gg: ASbGnctVy0jf/AylaQTC/+CG3DTRACFmUrRLDD76ghf04t2WkieD+c50IeOLHoD0lr9 VqiU3DqEc1CQO6dEtmHH05N1JeFKKUDSaFpLYdKs6S7cwVqRBSt5hGziCNKWPisLO9ifmmOjEfb Vkinwcnox8fzgoni0qarWvzGeURfXB0sQo4LJdTh7l/GWUS8oQuF4g1uEs4GJHduG0ItlLKLnjR eagTQMeH48GSwkoqkLjqT3rvPHACnqNYHv1P+3/fTg2pAEEk386SG0O4RC9vG+puLNtjFU803WB lbIEqDaPiXqp0N0A3MuUEUzl/lo9VFbBxJc4NhJOJPU= X-Google-Smtp-Source: AGHT+IHn/2owON+aw3mMItcmXsuTXk528yVv5D09IgwdkjIRjFqz9OSMXPbppYkHulZkuWMXfVHI8Q== X-Received: by 2002:a05:622a:252:b0:477:419a:a3bc with SMTP id d75a77b69052e-477e4b93c7amr286709531cf.27.1743623444554; Wed, 02 Apr 2025 12:50:44 -0700 (PDT) Received: from localhost ([2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with UTF8SMTPSA id d75a77b69052e-47782a1033csm83134521cf.16.2025.04.02.12.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Apr 2025 12:50:43 -0700 (PDT) Date: Wed, 2 Apr 2025 15:50:42 -0400 From: Johannes Weiner To: kernel test robot Cc: oe-lkp@lists.linux.dev, lkp@intel.com, Andrew Morton , Vlastimil Babka , Brendan Jackman , linux-mm@kvack.org Subject: Re: [linux-next:master] [mm] c2f6ea38fc: vm-scalability.throughput 56.4% regression Message-ID: <20250402195042.GC198651@cmpxchg.org> References: <202503271547.fc08b188-lkp@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <202503271547.fc08b188-lkp@intel.com> X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A928740009 X-Stat-Signature: mrgmui85xynmenynt8g9m5zm3qmywaxd X-Rspam-User: X-HE-Tag: 1743623445-512410 X-HE-Meta: U2FsdGVkX1/8Ch1zLJ9ni7m27CqfAkW/AcqtHeMKmq8WLbVrUuMh7SqNEqwhJtpjLP3k7OLsMjNeka1WUqla2mFehF6a/sUwMRCnPdbIJvxQv1EFw2nkw+10jMPKCKoBazO4JZc/R1z0M0XmuZozrVoOs2xE25rxreOOAgdeB843ACxExzsEKmU/g4WzmBLST3/JByUjx2SnJ38uW83qwh+D/9V5tOEhwbgCc0f7tWA9iJ2jT43ni3NA0DIGYr+dXCbfUvxOGcCLnMIIc78i/1SZgT6ElgGuWNAx3H4EjZO4SBsYVZyhgJkpyZDwcXQltXmnnXFGM6Cli2g8IjN2k2ODjGyHDjxNyp9pDN9H1PoIqlQrCowLZ5b0LeN/MTt6nCeWGOQRzxLDUwhot7jHScl7ASBmS1trRMKed/isQ4tOGsfpYutaLhy3ygzueLSQuQ6FJ4JGRm/rxAr7IX+gN8G/PdhqFi2juxtZdzJBzvvkWmKYEMpkJjDv5Lbe4uan2RPmW5iD1AS3qzKRk/6DUK+NfYiapdIQD4D0RLEIzUf/0y+u5G/ZQ45y5MTiOJP/6oYGCaXIZNpR0+TYrqa8muI1CgeQLUN8PHhZDwZH57w9TmEz4TDkOMgRLoXEUBWOgukpRo0MSNoswvh39UTmrL1A79SvigSdMe+fSXyis9O53D+VTcBclq9rMbs8z+eEvhEhPrwiYVCjuFzZsor28/nA7MJbCqoqLqe35p0kMZMRXUu+3tG/qXBMRYSZxTHuBwYVyf/0fxVGGxv3LHxvQd7bXJdkZdAfoeMj/uSrOQBqTzuXUkA85mN/g2LqIfNS1vPjvUuB/rbNM0nvi00TdUIyJBrRwChaUasNATGA3DtMitX5jec8hP9CMQK6H4araQ1pn0MBWSwILuewbDcNNQ/nELqm+2D7DTnK/FmJqZj+FeWA9Yx54nKNfXq1ayvYrPKxqdfYP5qwLNtEtC0 Q2ThdIqM 3u1n67DfVUKtsoQIaOrdQITZbqplDevvT5tzWmsGvuPd3mw38/V4/MQNQAOsNHSj83ZS9jNMPawQTzoS0G6XGrjbER67F6SVyFwdMWMIf/U8nl8dSz7FO27IRWP2YCSR3xTy2PE+0uyUZZwXrc45h4OWXR4gK1r5HOh7OIpbGGyQhjiXm25o9BPW5Z4SK7jp22bM0tf2nOi0WIf7Qw7WXx8q1nbryE3HkiSbwmAzgEpUXj0IJnyRlUwLcdzm30W5wugAUE+JaANzc603ufSxLRYZtEKs1DCMGhJA4WZov+MYX9ufsWKx4uwgZMI3QIeQ/2bK7bTGai/EB9L4Z4UUWIQxa4A7BtMzm9zNgN9yQGz/IoJJ3DgIIXTCSeMLGTGxHPld5k+tti59Z4St9XGcDMVy8JCpZZTzPYEPEtQUER54IsZdam6EEtJ5DVoYli+iqwTbGMsVvHcklkxi9qYXLLVgDFwxEiQ3ZbrDfohYCuozefpt+S4/Jz+jbCKVUlque82cJbjzq5ymxezMCHm+9mYLr82UGsOpNJK/BTGXdve/bYCxWItKG/vnDm37akfJ0Z6uN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, On Thu, Mar 27, 2025 at 04:20:41PM +0800, kernel test robot wrote: > kernel test robot noticed a 56.4% regression of vm-scalability.throughput on: > > commit: c2f6ea38fc1b640aa7a2e155cc1c0410ff91afa2 ("mm: page_alloc: don't steal single pages from biggest buddy") > https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master > > testcase: vm-scalability > config: x86_64-rhel-9.4 > compiler: gcc-12 > test machine: 224 threads 4 sockets Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz (Cooper Lake) with 192G memory > parameters: > > runtime: 300s > test: lru-file-mmap-read > cpufreq_governor: performance Thanks for the report. Would you be able to re-test with the below patch applied? There are more details in the thread here: https://lore.kernel.org/all/20250402194425.GB198651@cmpxchg.org/ It's on top of the following upstream commit: commit acc4d5ff0b61eb1715c498b6536c38c1feb7f3c1 (origin/master, origin/HEAD) Merge: 3491aa04787f f278b6d5bb46 Author: Linus Torvalds Date: Tue Apr 1 20:00:51 2025 -0700 Merge tag 'net-6.15-rc0' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net Thanks! --- >From 13433454403e0c6f99ccc3b76c609034fe47e41c Mon Sep 17 00:00:00 2001 From: Johannes Weiner Date: Wed, 2 Apr 2025 14:23:53 -0400 Subject: [PATCH] mm: page_alloc: speed up fallbacks in rmqueue_bulk() Not-yet-signed-off-by: Johannes Weiner --- mm/page_alloc.c | 100 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 74 insertions(+), 26 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f51aa6051a99..03b0d45ed45a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2194,11 +2194,11 @@ try_to_claim_block(struct zone *zone, struct page *page, * The use of signed ints for order and current_order is a deliberate * deviation from the rest of this file, to make the for loop * condition simpler. - * - * Return the stolen page, or NULL if none can be found. */ + +/* Try to claim a whole foreign block, take a page, expand the remainder */ static __always_inline struct page * -__rmqueue_fallback(struct zone *zone, int order, int start_migratetype, +__rmqueue_claim(struct zone *zone, int order, int start_migratetype, unsigned int alloc_flags) { struct free_area *area; @@ -2236,14 +2236,26 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, page = try_to_claim_block(zone, page, current_order, order, start_migratetype, fallback_mt, alloc_flags); - if (page) - goto got_one; + if (page) { + trace_mm_page_alloc_extfrag(page, order, current_order, + start_migratetype, fallback_mt); + return page; + } } - if (alloc_flags & ALLOC_NOFRAGMENT) - return NULL; + return NULL; +} + +/* Try to steal a single page from a foreign block */ +static __always_inline struct page * +__rmqueue_steal(struct zone *zone, int order, int start_migratetype) +{ + struct free_area *area; + int current_order; + struct page *page; + int fallback_mt; + bool claim_block; - /* No luck claiming pageblock. Find the smallest fallback page */ for (current_order = order; current_order < NR_PAGE_ORDERS; current_order++) { area = &(zone->free_area[current_order]); fallback_mt = find_suitable_fallback(area, current_order, @@ -2253,25 +2265,28 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype, page = get_page_from_free_area(area, fallback_mt); page_del_and_expand(zone, page, order, current_order, fallback_mt); - goto got_one; + trace_mm_page_alloc_extfrag(page, order, current_order, + start_migratetype, fallback_mt); + return page; } return NULL; - -got_one: - trace_mm_page_alloc_extfrag(page, order, current_order, - start_migratetype, fallback_mt); - - return page; } +enum rmqueue_mode { + RMQUEUE_NORMAL, + RMQUEUE_CMA, + RMQUEUE_CLAIM, + RMQUEUE_STEAL, +}; + /* * Do the hard work of removing an element from the buddy allocator. * Call me with the zone->lock already held. */ static __always_inline struct page * __rmqueue(struct zone *zone, unsigned int order, int migratetype, - unsigned int alloc_flags) + unsigned int alloc_flags, enum rmqueue_mode *mode) { struct page *page; @@ -2290,16 +2305,47 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype, } } - page = __rmqueue_smallest(zone, order, migratetype); - if (unlikely(!page)) { - if (alloc_flags & ALLOC_CMA) + /* + * Try the different freelists, native then foreign. + * + * The fallback logic is expensive and rmqueue_bulk() calls in + * a loop with the zone->lock held, meaning the freelists are + * not subject to any outside changes. Remember in *mode where + * we found pay dirt, to save us the search on the next call. + */ + switch (*mode) { + case RMQUEUE_NORMAL: + page = __rmqueue_smallest(zone, order, migratetype); + if (page) + return page; + fallthrough; + case RMQUEUE_CMA: + if (alloc_flags & ALLOC_CMA) { page = __rmqueue_cma_fallback(zone, order); - - if (!page) - page = __rmqueue_fallback(zone, order, migratetype, - alloc_flags); + if (page) { + *mode = RMQUEUE_CMA; + return page; + } + } + fallthrough; + case RMQUEUE_CLAIM: + page = __rmqueue_claim(zone, order, migratetype, alloc_flags); + if (page) { + /* Replenished native freelist, back to normal mode */ + *mode = RMQUEUE_NORMAL; + return page; + } + fallthrough; + case RMQUEUE_STEAL: + if (!(alloc_flags & ALLOC_NOFRAGMENT)) { + page = __rmqueue_steal(zone, order, migratetype); + if (page) { + *mode = RMQUEUE_STEAL; + return page; + } + } } - return page; + return NULL; } /* @@ -2311,6 +2357,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long count, struct list_head *list, int migratetype, unsigned int alloc_flags) { + enum rmqueue_mode rmqm = RMQUEUE_NORMAL; unsigned long flags; int i; @@ -2321,7 +2368,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, } for (i = 0; i < count; ++i) { struct page *page = __rmqueue(zone, order, migratetype, - alloc_flags); + alloc_flags, &rmqm); if (unlikely(page == NULL)) break; @@ -2934,6 +2981,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, { struct page *page; unsigned long flags; + enum rmqueue_mode rmqm = RMQUEUE_NORMAL; do { page = NULL; @@ -2945,7 +2993,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { - page = __rmqueue(zone, order, migratetype, alloc_flags); + page = __rmqueue(zone, order, migratetype, alloc_flags, &rmqm); /* * If the allocation fails, allow OOM handling and -- 2.49.0