From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E620CD690C for ; Tue, 10 Oct 2023 09:23:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12D268D006D; Tue, 10 Oct 2023 05:23:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DD2D8D0002; Tue, 10 Oct 2023 05:23:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0E268D006D; Tue, 10 Oct 2023 05:23:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E19658D0002 for ; Tue, 10 Oct 2023 05:23:07 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A610640273 for ; Tue, 10 Oct 2023 09:23:07 +0000 (UTC) X-FDA: 81329012814.08.A12F988 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf20.hostedemail.com (Postfix) with ESMTP id C8F3B1C000A for ; Tue, 10 Oct 2023 09:23:05 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e1mDw0Nx; spf=pass (imf20.hostedemail.com: domain of mingo.kernel.org@gmail.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=mingo.kernel.org@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696929785; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xe8JpwsINq8culWJn2R97JOgXDbD5qqq96xuZqiOo+Y=; b=0amg6WaIpaqUCGnGHXLH83QX0puE014oT5BYNMAr4s1D7BgcLltgMWmmdXI7nakif5aNVA a7mHWFbH5xUthbWUUjAXG0XBLQSM+fUnxLJ5N+baFb55zPxikk7/t2pZoU/mN+LJQUMjqn i/8XNorof+foPQNbMeKJIzfIVbVbqqw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696929785; a=rsa-sha256; cv=none; b=5FDVMqvqyL+/ZxawenBkomKad8c3fn4OblME0web7v/GlKDy0PcDlurJ98PYoHyMARWKK8 JGAFvCutTWoisPeUgW5nUPeMjjuLiCJKdhTurVMPTHY9aE28nA75l5bWTdKwqhz/McD1Yl 2/9vqSkR1r9SstT0J7XxJd7TWSItn40= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e1mDw0Nx; spf=pass (imf20.hostedemail.com: domain of mingo.kernel.org@gmail.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=mingo.kernel.org@gmail.com; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=kernel.org (policy=none) Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4066241289bso50797145e9.0 for ; Tue, 10 Oct 2023 02:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696929784; x=1697534584; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:from:to:cc:subject:date:message-id :reply-to; bh=Xe8JpwsINq8culWJn2R97JOgXDbD5qqq96xuZqiOo+Y=; b=e1mDw0NxjxO7PwwNAgykfShfa0sAcbju8BtWVVGvRxO4HUafOFVc5bxTdewQ8tWv7q G2xY0WFrD3GwurAvNeBNTpu6+eL4qxSv3X8no58WQap7LbqLxF5zhQHiEv0xC6mIaVCu caW8c6Oi8nCy7Ckr5bE0w0BWGt/T/lHfNTEhlcyf7f47QP+nMbe+xhau8b/JhFUJs95m tCbz9y1MscA4KhDgqMEtDY0NWi8e+dofhDtZ+QsxPTFIVPigBMrGQq0Xg7P2hGGxIiz3 auz+62ly6KNs38zjZU4rk+EHjlwzAp55XtntaDrsYcngiFZESpC0ZNehUaA/sA4MVWZj AwsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696929784; x=1697534584; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:sender:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xe8JpwsINq8culWJn2R97JOgXDbD5qqq96xuZqiOo+Y=; b=NGG4bYANWQuLWrP5R6Gjmyge0UovokNoDCwqL5UFidYTHZFRvK6uqgkiIu5GB3CvNa u0B9pL1KtMejj2CR5MAVJMi3+9j6jLW0Z9Wq9NRHIvXwU3B5iPnA8qWTT03xeI3S/+9J /YnJefIpWaLRKDhMY5T62hbuUvJCG4tpn0ycLUkqZhd32hjw2Ok9/QQR/YkuEE+V1ecj OWrzgnaY+h5KmCjxv8oo8AY2OkzedDnBKvpvF8BYe10w/l2iEZmrzlIvFrWp3KIrF1Cw Uno1rrEqWVhSUrPCaAhETUogJGo+xg8VCDp3nPA/PMCTgAtNO/C2BapVlXTsS2BsZzFL s/VQ== X-Gm-Message-State: AOJu0YwZELrlBB7UfVpiZto8liud9ofKEvKHsAfyJhyBCZ8f64rVzOQ1 oYnmKFU35Da+oJM5rxSmllc= X-Google-Smtp-Source: AGHT+IGmTwB0cowYILSDxlmRY0/l5H88NPoPezYQBWK50ok4Q7Yh1g+d/KAlnBukst9DbONlji+TMA== X-Received: by 2002:adf:cf0a:0:b0:317:3f70:9dc4 with SMTP id o10-20020adfcf0a000000b003173f709dc4mr15681071wrj.31.1696929783994; Tue, 10 Oct 2023 02:23:03 -0700 (PDT) Received: from gmail.com (1F2EF237.nat.pool.telekom.hu. [31.46.242.55]) by smtp.gmail.com with ESMTPSA id n9-20020adff089000000b0031ad2f9269dsm12069341wro.40.2023.10.10.02.23.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Oct 2023 02:23:02 -0700 (PDT) Date: Tue, 10 Oct 2023 11:23:00 +0200 From: Ingo Molnar To: Mel Gorman Cc: Peter Zijlstra , Raghavendra K T , K Prateek Nayak , Bharata B Rao , Ingo Molnar , LKML , Linux-MM Subject: Re: [PATCH 6/6] sched/numa: Complete scanning of inactive VMAs when there is no alternative Message-ID: References: <20231010083143.19593-1-mgorman@techsingularity.net> <20231010083143.19593-7-mgorman@techsingularity.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231010083143.19593-7-mgorman@techsingularity.net> X-Stat-Signature: kyid1y5siegeo8zxxn7ky9aoi11i4uw3 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C8F3B1C000A X-Rspam-User: X-HE-Tag: 1696929785-460020 X-HE-Meta: U2FsdGVkX19Q/s7ze5Trr3+hWUMMnjjydLLcWbT44ZuS5Uf1h+7tmGps351ZHTyvNtPa9uNZwcD4OEX37RykHyUwJPq3ZzWOHAEcwaXYhXisY8SMTZib97uZ95olV8UCFntpNyfkrNj0nJfbNWin8pw2AxRoR/gzoacJ/ULcscMPkxpVbG+9HxdxxQ0L3TSc3Ydw/gKxtGUJcZELZPbHm3+MsFt2EX2+cIuYZT8W74nzoyBmDR4CD7ZM2OiMhPYnLr9Hg16o64cSyrmR07K5bWNcEE3q+DJ5WwK2M3c71/ivp6ySUloNi7Mvm7t86suBe0Iia/s/sVBCiGMdV1KBXN5s7Tz7uYkb+54XDD0U60R4VKKu5pcgpvKZWCp4oMwBLYjKk/5Rol7YLrEYn35c2ueLK1qBJ/HLHIdVarzHUkJOwU6b9y/U9jqdq2Twozr2vnLaZupuvK8DWqa+w7nGA32a/WRP8AqvsQI59NPMobe9mVIJ0OJcI6WN8Zb5XRyFSQtvpz/eNcirQWfChiJCZjRYcXH/Khas8bh7Mvhpwo6/uJ+UvhmiEEHnYyOFGH+1UTOY+7Yjoo2k2PyjBgi/3NTT0ris96VEUdulfvtG0KJ1zaGw5zSpkS51TEWdWPvcT2RqXnl3o9Eg683oFq54MeXOymL948IidJwnSkTC51UYTQJC7kGyhsYljs5sAcWs4+Y2qgQnKEL2jb89DYum70TcNFPhMHKXazEm7934VShPTd9fWW65AcyHB9/tP3Bl2hHDIL+Ual9gHrPyGAPswjOgky8xIjRIAei9R/+dcMYdrXBabbxKC6W5UTnAJyjZjzz7kJPqAt1F04UxED2zprmjx8OoP+TjK8Qlmrk2ds9tqbh2JDd9e9n0vJqFPRVgnb+naCOb9dkeBcL4VzXG9PTf/a1sWS3gT/ayWVzk7pTFivltTWVRA/pARD2IV02oNjtQ466zbLnkJTB/zgS Vx4/dSvt c3YQ650+IawC/CzyQtm9zMOzMHNNtefMnr6iiktiFUEoX6iZABDBd5iTybXBs64/0ahjA0ezhLgbCKdCHM/bKc1g3W8Naq/0P5G+S4R9ibVmlRUjFGemb2jxcU193dxG+VwPxuPNM44zzvLd4TqUubXroX3CPtGh3si8DZJ6/R+F9/qXtPbyiQfD07v0k+WS8SSziKifXRfyJbf3vof3ahMCEjLDw5/GX9KRkiIob00dNfFm4jaDCpIIuDMmj1YMP9UWpCBdNBhZwiKA4NO0iHBwjbDklzAwWP3xdxBcPq9GwEznP9q0kXBAk4/bx/Xd7S/fuxb/b0MDDyDwTxox3Lc03u70McWbiX77DLXr4Y6CZztIOt+4atn7f7nKw2VG++hKNpnaQupALRBv7Se2YKe4M6d9utH0YHAr3LiAXq1W3bOmdXoXEQjZJ++NwmP02bLFdYu5MvSIxH3Df0gqCI5nnQwXXW8RjiZuslgEnLnBaKtrBE8fkGHSTM9YeQPo72hj5AF0eCaeLVhKfwYgHtp8nNyzXAy4Kc0rWtFuGnwbUDf72wGMwUIplAGiMwyxx5b2X X-Bogosity: Ham, tests=bogofilter, spamicity=0.003091, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: * Mel Gorman wrote: > On a 2-socket Cascade Lake test machine, the time to complete the > workload is as follows; > > 6.6.0-rc2 6.6.0-rc2 > sched-numabtrace-v1 sched-numabselective-v1 > Min elsp-NUMA01_THREADLOCAL 174.22 ( 0.00%) 117.64 ( 32.48%) > Amean elsp-NUMA01_THREADLOCAL 175.68 ( 0.00%) 123.34 * 29.79%* > Stddev elsp-NUMA01_THREADLOCAL 1.20 ( 0.00%) 4.06 (-238.20%) > CoeffVar elsp-NUMA01_THREADLOCAL 0.68 ( 0.00%) 3.29 (-381.70%) > Max elsp-NUMA01_THREADLOCAL 177.18 ( 0.00%) 128.03 ( 27.74%) > > The time to complete the workload is reduced by almost 30% > > 6.6.0-rc2 6.6.0-rc2 > sched-numabtrace-v1 sched-numabselective-v1 / > Duration User 91201.80 63506.64 > Duration System 2015.53 1819.78 > Duration Elapsed 1234.77 868.37 > > In this specific case, system CPU time was not increased but it's not > universally true. > > From vmstat, the NUMA scanning and fault activity is as follows; > > 6.6.0-rc2 6.6.0-rc2 > sched-numabtrace-v1 sched-numabselective-v1 > Ops NUMA base-page range updates 64272.00 26374386.00 > Ops NUMA PTE updates 36624.00 55538.00 > Ops NUMA PMD updates 54.00 51404.00 > Ops NUMA hint faults 15504.00 75786.00 > Ops NUMA hint local faults % 14860.00 56763.00 > Ops NUMA hint local percent 95.85 74.90 > Ops NUMA pages migrated 1629.00 6469222.00 > > Both the number of PTE updates and hint faults is dramatically > increased. While this is superficially unfortunate, it represents > ranges that were simply skipped without the patch. As a result > of the scanning and hinting faults, many more pages were also > migrated but as the time to completion is reduced, the overhead > is offset by the gain. Nice! I've applied your series to tip:sched/core with a few non-functional edits to comment/changelog formatting/clarity. Btw., was any previous analysis done on the size of the pids_active[] hash and the hash collision rate? 64 (BITS_PER_LONG) feels a bit small, especially on larger machines running threaded workloads, and the kmalloc of numab_state likely allocates a full cacheline anyway, so we could double the hash size from 8 bytes (2x1 longs) to 32 bytes (2x2 longs) with very little real cost, and still have a long field left to spare? Thanks, Ingo