From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4EFEC64ED6 for ; Mon, 27 Feb 2023 07:55:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B3E16B0072; Mon, 27 Feb 2023 02:55:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1642C6B0073; Mon, 27 Feb 2023 02:55:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02B366B0074; Mon, 27 Feb 2023 02:55:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E4E8B6B0072 for ; Mon, 27 Feb 2023 02:55:30 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AC3CE16072B for ; Mon, 27 Feb 2023 07:55:30 +0000 (UTC) X-FDA: 80512312020.08.44BF703 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf10.hostedemail.com (Postfix) with ESMTP id B1342C0017 for ; Mon, 27 Feb 2023 07:55:27 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WmdjvY7X; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677484528; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nLylL8yzj7ZkjH+zgC0krUvvydqKQaFNrl5BuuB8x7E=; b=EiicT9daWWWGuuebOUG7h1MGn/YL1l8aGJaCNbocvOPZmjxSWK2meWF/luSU/L13o/QbdV 3AvLLN2kAPV6x4TX7nC5ZlRm4GYBD5QI+al7btN/5+6hbQPNeFAzKSDQpmg2wk74z2gwHM PEW+UJyxaGbDWMsVR7K5NJr2xjFHYQE= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WmdjvY7X; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.120 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677484528; a=rsa-sha256; cv=none; b=FwrS3n/BE7Pbv+62sJCKXEkXMC4jLekMbR6yQJ4VtZ9JgddJzYJEpZ6gG69J5RO7LOpu2i o+pZanAxKLsQC/yc9FKWNrHjlYWXrSakXrtkd7TB0J42xs90qIG3WEGITJ7XKQFqgLL7xf Fr7GONv2vlHA70jH1E+hC+qj0h5tFDc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677484527; x=1709020527; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=xQsnLwR6WvsW3HYMB5OebnDeFXQGQNEsYlVg6eCzsdo=; b=WmdjvY7X8aGd35iTesqQxKchpo2dPm4i6I1LxsjXPH/eqZDVVGi5SAh+ KBXQEfuMISm6gVYP6dMNwQ1Z64Duygl1YOpV95kdfudgVJvzumCLAfG3z 43e+nWDUbBhbbyfvq0lWqA9JcyroJIngeZNEmb7HfMWbdfsJOWWoR4Y+1 auYGU9qh3t6HCP6QFKggas9Ta49YX9ncPURMTgVo2CwK/ivm/P2P+hjSN MKKfBKfUfDABSGsc9jjrRz5JdWxjBg+dXFQuIl1BbNHCQ9qJrzqklXl2D a2Ck/2AWUh2GomTHPmZ7WXLjxuuf0OiKdQUxF307KKVDV9eY3mCdhfFm6 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="332525915" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="332525915" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2023 23:55:25 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10633"; a="623478659" X-IronPort-AV: E=Sophos;i="5.97,331,1669104000"; d="scan'208";a="623478659" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Feb 2023 23:55:20 -0800 From: "Huang, Ying" To: Bharata B Rao Cc: , , , , , , , , , , , , Subject: Re: [RFC PATCH 0/5] Memory access profiler(IBS) driven NUMA balancing References: <20230208073533.715-1-bharata@amd.com> <878rh2b5zt.fsf@yhuang6-desk2.ccr.corp.intel.com> <72b6ec8b-f141-3807-d7f2-f853b0f0b76c@amd.com> <87zg9i9iw2.fsf@yhuang6-desk2.ccr.corp.intel.com> <1547d291-1512-faae-aba5-0f84c3502be4@amd.com> <87zg9c7rrf.fsf@yhuang6-desk2.ccr.corp.intel.com> <8fea74ec-8feb-1709-14f2-cecb63fdc9ed@amd.com> Date: Mon, 27 Feb 2023 15:54:13 +0800 In-Reply-To: <8fea74ec-8feb-1709-14f2-cecb63fdc9ed@amd.com> (Bharata B. Rao's message of "Fri, 24 Feb 2023 09:06:49 +0530") Message-ID: <87v8jnbl22.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B1342C0017 X-Rspam-User: X-Stat-Signature: jb45izq1fszhomrys6zbx6bc1og69knd X-HE-Tag: 1677484527-140901 X-HE-Meta: U2FsdGVkX193eQC7jpLq0l3pc06wPLkcMo507GvMkA51mqnxIikej3jxXgGKChTmn8Bu3myasBynwzop2WxuMo5SNMZv1rm8Ul4RLmyJ4dj3Wo9/ae77oxc5g9HDVLwjduUBCZPtwggOE3gctqq0+ZpvzwXgii3b0B3G9PQbF6vma8f1Y0j7LnQlkbD0yufEsbelfzchQGHRmaICk887JB3wLJTtAsq1jrm4nFVifI1KFDLAjqOV70yHpvkYq0LGH72p+uxX1MyPpeFFnNSKzviLcgzFXTh3dzYo4zAj1+1qACuUmZsNE6MGpInsm2EzKTaXlrw54768Atn83Y445i4jqlNucg9HXcT5XW3LiCmsSdITODO/93MAzmhxiUsyKbbm+gH0Aex1yW4E0zGs3X5mkHwFnnMwsJx9cIUyBdAXG7tPG3oqz/WilY/zVYuelwYH5f++/lVzb8mOUliCiEedt5cna9lXpwum3/WN16pQra3HS38NIviUyMislsNco/aibBXykHZPty3DnFPI1fnl0xNzk/1HSNgUWLqSpVNxEDSxrJDIlfh9GOXc4J+bCRSJ1zPcQgscQbJqLe8sx+gQQdd0fA8atUqJFX9RR3PBSpxgtUGW5yE4y1q+HG1h75qjROpThujKPsFnPRJLMPzgapZImsc9z5ux+vLQY9iTc9f1uzuFyAvAut60rb2/60pDjVPkPDsvv8KeJMLiW0ZlF9H70UTid1svRF8sMENfZoEDkuK0HvwHCK2JlVsVT4/EJrYAZqywp4gGg4efgUlA4Z7BUY498T8jVpKWgMJmFL/D/8/sN8P98T1fT5J41XI/umlhWpH75mLWFFxYoHIM1zpnh9Q/e0roIkT/0PGCk+40AwSgLYown2a6OMlPa+HpMIrpptN7p9bYtKzgUQ2Sb4ZHoTpNLB6ZXnpSaUSmXv5gHe8wE7qcvE022yc0fso/GBAB5akYlj2V69q cMU5mXFE U6Qb7Z75i9QpyxdFqEne875+IHAj3Y6R97ovCLHoPhxyyg6b7pdtdzyGtSVXsxkwh74z8nVhtBwpfnlQun2gVAEJm7qW5C67JrmNsy+VzmNbwi3zmD8LFnPvwiozPoAv8KfyTo1dfwPcnCsjBRHsdWm383YPv2sLT4OgALqdoIdfWgPKxYf7LLhX6r6ibhe2vrd3R X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Bharata B Rao writes: > On 17-Feb-23 11:33 AM, Huang, Ying wrote: >> Bharata B Rao writes: >> >>> On 14-Feb-23 10:25 AM, Bharata B Rao wrote: >>>> On 13-Feb-23 12:00 PM, Huang, Ying wrote: >>>>>> I have a microbenchmark where two sets of threads bound to two >>>>>> NUMA nodes access the two different halves of memory which is >>>>>> initially allocated on the 1st node. >>>>>> >>>>>> On a two node Zen4 system, with 64 threads in each set accessing >>>>>> 8G of memory each from the initial allocation of 16G, I see that >>>>>> IBS driven NUMA balancing (i,e., this patchset) takes 50% less time >>>>>> to complete a fixed number of memory accesses. This could well >>>>>> be the best case and real workloads/benchmarks may not get this much >>>>>> uplift, but it does show the potential gain to be had. >>>>> >>>>> Can you find a way to show the overhead of the original implementation >>>>> and your method? Then we can compare between them? Because you think >>>>> the improvement comes from the reduced overhead. >>>> >>>> Sure, will measure the overhead. >>> >>> I used ftrace function_graph tracer to measure the amount of time (in us) >>> spent in fault handling and task_work handling in both the methods when >>> the above mentioned benchmark was running. >>> >>> Default IBS >>> Fault handling 29879668.71 1226770.84 >>> Task work handling 24878.894 10635593.82 >>> Sched switch handling 78159.846 >>> >>> Total 29904547.6 11940524.51 >> >> Thanks! You have shown the large overhead difference between the >> original method and your method. Can you show the number of the pages >> migrated too? I think the overhead / page can be a good overhead >> indicator too. >> >> Can it be translated to the performance improvement? Per my >> understanding, the total overhead is small compared with total run time. > > I captured some of the numbers that you wanted for two different runs. > The first case shows the data for a short run (less number of memory access > iterations) and the second one is for a long run (more number of iterations) > > Short-run > ========= > Time taken or overhead (us) for fault, task_work and sched_switch > handling > > Default IBS > Fault handling 29017953.99 1196828.67 > Task work handling 10354.40 10356778.53 > Sched switch handling 56572.21 > Total overhead 29028308.39 11610179.41 > > Benchmark score(us) 194050290 53963650 > numa_pages_migrated 2097256 662755 > Overhead / page 13.84 17.51 >From above, the overhead/page is similar. > Pages migrated per sec 72248.64 57083.95 > > Default > ------- > Total Min Max Avg > do_numa_page 29017953.99 0.1 307.63 15.97 > task_numa_work 10354.40 2.86 4573.60 175.50 > Total 29028308.39 > > IBS > --- > Total Min Max Avg > ibs_overflow_handler 1196828.67 0.15 100.28 1.26 > task_ibs_access_work 10356778.53 0.21 10504.14 28.42 > hw_access_sched_in 56572.21 0.15 16.94 1.45 > Total 11610179.41 > > > Long-run > ======== > Time taken or overhead (us) for fault, task_work and sched_switch > handling > Default IBS > Fault handling 27437756.73 901406.37 > Task work handling 1741.66 4902935.32 > Sched switch handling 100590.33 > Total overhead 27439498.38 5904932.02 > > Benchmark score(us) 306786210.0 153422489.0 > numa_pages_migrated 2097218 1746099 > Overhead / page 13.08 3.38 But from this, the overhead/page is quite different. One possibility is that there's more "local" hint page faults in the original implementation, we can check "numa_hint_faults" and "numa_hint_faults_local" in /proc/vmstat for that. If numa_hint_faults_local / numa_hint_faults is similar. For each page migrated, the number of hint page fault is similar, and the run time for each hint page fault handler is similar too. Or I made some mistake in analysis? > Pages migrated per sec 6836.08 11380.98 > > Default > ------- > Total Min Max Avg > do_numa_page 27437756.73 0.08 363.475 15.03 > task_numa_work 1741.66 3.294 1200.71 42.48 > Total 27439498.38 > > IBS > --- > Total Min Max Avg > ibs_overflow_handler 901406.37 0.15 95.51 1.06 > task_ibs_access_work 4902935.32 0.22 11013.68 9.64 > hw_access_sched_in 100590.33 0.14 91.97 1.52 > Total 5904932.02 Thank you very much for detailed data. Can you provide some analysis for your data? Best Regards, Huang, Ying