From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49A40C25B74 for ; Thu, 16 May 2024 05:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C4C46B0300; Thu, 16 May 2024 01:55:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 875136B038D; Thu, 16 May 2024 01:55:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7205C6B038C; Thu, 16 May 2024 01:55:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4D0CE6B02F3 for ; Thu, 16 May 2024 01:55:23 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9BFB58171D for ; Thu, 16 May 2024 05:55:22 +0000 (UTC) X-FDA: 82123196484.24.C8DFD88 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.14]) by imf10.hostedemail.com (Postfix) with ESMTP id 540C7C001D for ; Thu, 16 May 2024 05:55:19 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="VLdc/ARu"; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.14 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715838920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=37cpsj6SVL5XOac9iOvO2tvI9NgqrCM+fFv7hshEfHE=; b=SKGd5M9IecA2pZ6VNWk+IJYhZn8yGBzqT8aiDstGKFSWv7uVLHUUiA1D9WSdYdgPjpe+nh OuQrgVEDQM7S63VwBA6ZsNTmOdWE6c0S4bClcA2OWcTAJyuRJ7HR6G1kq/OMF1fjEsE0LG xbhscCgvrQ97Pft8u5Yz7qZFaBcAvLc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715838920; a=rsa-sha256; cv=none; b=sP4MJcmRhbDLAgno9Ue10EIV9ZCZys15ewipCuePgUxGF0QqumDjW/8nVFA3iST5tcenZo KFeaifL/ojQ4RjRZnwsontlMr15noGe8rGm95f03nvc94tnIlA/b5KmCSduOeYuEteCPBR wtmAiJXr1wxrPIEs0mdnwEfxMsRWVcQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="VLdc/ARu"; spf=pass (imf10.hostedemail.com: domain of ying.huang@intel.com designates 198.175.65.14 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715838919; x=1747374919; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=xMvyt0Z5g7e3xmtO9PR6aw0WgugOD1OHXY3O55x0INQ=; b=VLdc/ARuOq3k07p0wo+v3VL1Hph16pawnkgGiRIKBT9wVrgZFIJAGUY9 jRCBrzWOQe/PgT4DBuk85eu0OZ6G85ndBavpTYm65x+My2esZSA+jM3U1 pdQEHbdNtxiWb637BPmm4Jc3qtgIqr6++hVrRbFh1n3qzLG4aTebKhtIO u4xinNhB+4ELHAEK6XTMvKiNs+wrEGHRBm8v9UHhpOUxS+alhIVi69XB7 eNeY0Z7AStbZ8ZViWol57RZqDa+q1piR8Je+OUTSjBAXYXU7831dnnrAY 6RIDXGpxCaxew55uc5KEsB+AaPCXzXidJwxVA9o75D6OrYtoBDH7uPXUY A==; X-CSE-ConnectionGUID: d3Uy+UzvSRWXF0JMfIfKAg== X-CSE-MsgGUID: diiFF03TR8qUprwwRkBjBA== X-IronPort-AV: E=McAfee;i="6600,9927,11074"; a="15754321" X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="15754321" Received: from fmviesa008.fm.intel.com ([10.60.135.148]) by orvoesa106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 May 2024 22:55:17 -0700 X-CSE-ConnectionGUID: KTBMe29zTS+0c4w5wkmLoQ== X-CSE-MsgGUID: xc3VgJ2pQMe1HxnSw9hpOg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="31306965" Received: from unknown (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmviesa008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 May 2024 22:55:10 -0700 From: "Huang, Ying" To: Yuanchu Xie Cc: David Hildenbrand , "Aneesh Kumar K.V" , Khalid Aziz , Henry Huang , Yu Zhao , Dan Williams , Gregory Price , Kalesh Singh , Wei Xu , David Rientjes , Greg Kroah-Hartman , "Rafael J. Wysocki" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Shuah Khan , Yosry Ahmed , Matthew Wilcox , Sudarshan Rajagopalan , Kairui Song , "Michael S. Tsirkin" , Vasily Averin , Nhat Pham , Miaohe Lin , Qi Zheng , Abel Wu , "Vishal Moola (Oracle)" , Kefeng Wang , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org Subject: Re: [PATCH v1 0/7] mm: workingset reporting In-Reply-To: <20240504073011.4000534-1-yuanchu@google.com> (Yuanchu Xie's message of "Sat, 4 May 2024 00:30:04 -0700") References: <20240504073011.4000534-1-yuanchu@google.com> Date: Thu, 16 May 2024 13:53:17 +0800 Message-ID: <87v83eb8vm.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspam-User: X-Stat-Signature: dxuyyw1z3mwqguw6q58piyuhn4rrdp48 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 540C7C001D X-HE-Tag: 1715838919-107857 X-HE-Meta: U2FsdGVkX19LcHoY5pSwUExklLZ3jMYowrD1oJBgKPMO2r1BK1p22h4FiriYCzM4EhZjUXSK0vIJRYcHmq1b+wpsW+T0Pv+wZ2FBAZ/jK3NnEFLwLbkUqlcsMqRsfq/FRPNIt4k9bef/A1rZGtY7ztm2eLfP0LS8MmecnxkZ/hAblwSbF9lWMioQ+Gkmb9ylqN4UUBBWdZa0f7jD7hs6SQjXk2CNKq8exYh0+8n5JiEiI9Bvxp4156AudroUf/C6ioF196/WlYUmPAldmSsItb+7CxIliVqAjXcGymxkR8/fPrVKM8B+uybWhN99huSFytCAlgyLcR+L2W1DZghvgbzmwGMxSy9O6eGDdusxHU8upD+P506LpQhZ7UJhACQIhhhlhr7F/fvB8W+2SyAi5SjfFZzVfNp3ceDtoQ4Enj4Ro3mvsiIglvVBVyYYm7PU8maNyWXUZgeVVfvZMnwGyZ8/ZUHFiMPQYTZaR3rYIGALNZlUtmOn70ymlcb7finJIcB2Ggq1E5oI52IU9znVY11R6rLjIj5nJNWImNNbHejUTEJoMWCwE5sftnuQ5iHKNva5IRrJjgZ91Y3Oh2RKzYeA9a8PrgXt5cSwPvlDw06bB2suPORan891rrBkbM9kVIoggJNQc+/EPaU/llx4CubwYY1GycBLqhuoaz3yyP/7Z0mRqLsPh6iD5/DepXKAB6PidCM3mn0iopFiB2NH/YNEnyQjy8lBP7/R8wcY/KREcoyUZjKGrE5i76/VUL//dEceQHhmO1t6VieNAR12S3bofb1E2cpYUwMZtcFXMYY1aprAnBleg32xPzh0yKZGgVovJJvMPztgd4o1qh0+esGKCKGYtmF5/e+badA2DHtxL/GarUm1+q2TLdYhxBY4Z595JKCDP9LYQZwwBtydU5VN/oBBzk/hRzwW4+oRC95v+mYrqE1842qg4YWmk4SOVOv70sZDKL7XF6s5F4z gChHz46Z yRPFJuiJQ3M5tmja6gYegvvfRkru8oOhJ4pzMwKp7YZjo69YkGL159WQygPmFJy9twS7UnR53H7FJvdsqtngDJtFFGF7jO4KubQIelAVCSkiKKBCQBy34830jo1HqldqZo9YlXy8Gun15CyxTkBmFgQeU6sjsgG87O4+LU7W9JR/+tGNO17t4Pc0cO4MJ4E1+k1ZCYPYAbghHnm9XOhS4tjkmdqnkfIVTvMeS7wZR7UjPBckUkckLw2kbzFrPWGHVVLOOsd6sio/r24BobwHqmtg9F4Bcm75iU6BfQzGiST/W8tZAqflLcNPAV5386CGvpbVuQDWFbmJO8G1z5jHYxhPdRE1KQF/C1PB5ao4nshyx844= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, Yuanchu, Yuanchu Xie writes: > Changes from RFC v3 -> PATCH v1: > - Updated selftest to use ksft_print_msg instead of fprintf(stderr, ...) > (Muhammad Usama Anjum) > - Included more detail in patch skipping pmd_young with force_scan > (Huang, Ying) > - Deferred reaccess histogram as a followup > - Removed per-memcg page age interval configs for simplicity > > Changes from RFC v2 -> RFC v3: > - Update to v6.8 > - Added an aging kernel thread (gated behind config) > - Added basic selftests for sysfs interface files > - Track swapped out pages for reaccesses > - Refactoring and cleanup > - Dropped the virtio-balloon extension to make things manageable > > Changes from RFC v1 -> RFC v2: > - Refactored the patchs into smaller pieces > - Renamed interfaces and functions from wss to wsr (Working Set Reporting) > - Fixed build errors when CONFIG_WSR is not set > - Changed working_set_num_bins to u8 for virtio-balloon > - Added support for per-NUMA node reporting for virtio-balloon > > [rfc v1] > https://lore.kernel.org/linux-mm/20230509185419.1088297-1-yuanchu@google.com/ > [rfc v2] > https://lore.kernel.org/linux-mm/20230621180454.973862-1-yuanchu@google.com/ > [rfc v3] > https://lore.kernel.org/linux-mm/20240327213108.2384666-1-yuanchu@google.com/ > > This patch series provides workingset reporting of user pages in > lruvecs, of which coldness can be tracked by accessed bits and fd > references. However, the concept of workingset applies generically to > all types of memory, which could be kernel slab caches, discardable > userspace caches (databases), or CXL.mem. Therefore, data sources might > come from slab shrinkers, device drivers, or the userspace. IMO, the > kernel should provide a set of workingset interfaces that should be > generic enough to accommodate the various use cases, and be extensible > to potential future use cases. The current proposed interfaces are not > sufficient in that regard, but I would like to start somewhere, solicit > feedback, and iterate. > > Use cases > ========== > Job scheduling > On overcommitted hosts, workingset information allows the job scheduler > to right-size each job and land more jobs on the same host or NUMA node, > and in the case of a job with increasing workingset, policy decisions > can be made to migrate other jobs off the host/NUMA node, or oom-kill > the misbehaving job. If the job shape is very different from the machine > shape, knowing the workingset per-node can also help inform page > allocation policies. > > Proactive reclaim > Workingset information allows the a container manager to proactively > reclaim memory while not impacting a job's performance. While PSI may > provide a reactive measure of when a proactive reclaim has reclaimed too > much, workingset reporting allows the policy to be more accurate and > flexible. > > Ballooning (similar to proactive reclaim) > While this patch series does not extend the virtio-balloon device, > balloon policies benefit from workingset to more precisely determine > the size of the memory balloon. On desktops/laptops/mobile devices where > memory is scarce and overcommitted, the balloon sizing in multiple VMs > running on the same device can be orchestrated with workingset reports > from each one. > > Promotion/Demotion > Similar to proactive reclaim, a workingset report enables demotion to a > slower tier of memory. > For promotion, the workingset report interfaces need to be extended to > report hotness and gather hotness information from the devices[1]. > > [1] > https://www.opencompute.org/documents/ocp-cms-hotness-tracking-requirements-white-paper-pdf-1 This mechanism can also provide a way to control the pages ping-pong caused by promotion and demotion. If different methods are used for promotion and demotion, some pages may be migrated between the fast and slow memory back and forth. Because there's no connection between promotion side and demotion side. For example, if we use NUMA balancing based promotion and MGLRU based demotion. If we use workingset reporting on the fast memory nodes, then we can establish the connection between demotion and promotion with it. For example, the promotion hot pages threshold (determined as accessed every N seconds) should be set to make 80% of the fast memory pages pass the threshold. This can be calculated with the workingset reporting statistics. > Sysfs and Cgroup Interfaces > ========== > The interfaces are detailed in the patches that introduce them. The main > idea here is we break down the workingset per-node per-memcg into time > intervals (ms), e.g. > > 1000 anon=137368 file=24530 > 20000 anon=34342 file=0 > 30000 anon=353232 file=333608 > 40000 anon=407198 file=206052 > 9223372036854775807 anon=4925624 file=892892 > > I realize this does not generalize well to hotness information, but I > lack the intuition for an abstraction that presents hotness in a useful > way. Please advise. > > Implementation > ========== > Currently, the reporting of user pages is based off of MGLRU, and > therefore requires CONFIG_LRU_GEN=y. We would benefit from more MGLRU > generations for a more fine-grained workingset report. I will make the > generation count configurable in the next version. The workingset > reporting mechanism is gated behind CONFIG_WORKINGSET_REPORT, and the > aging thread is behind CONFIG_WORKINGSET_REPORT_AGING. > > Yuanchu Xie (7): > mm: multi-gen LRU: ignore non-leaf pmd_young for force_scan=true > mm: aggregate working set information into histograms > mm: use refresh interval to rate-limit workingset report aggregation > mm: report workingset during memory pressure driven scanning > mm: extend working set reporting to memcgs > mm: add kernel aging thread for workingset reporting > selftest: test system-wide workingset reporting > > drivers/base/node.c | 6 + > include/linux/memcontrol.h | 5 + > include/linux/mmzone.h | 9 + > include/linux/workingset_report.h | 97 ++++ > mm/Kconfig | 15 + > mm/Makefile | 2 + > mm/internal.h | 17 + > mm/memcontrol.c | 184 +++++- > mm/mm_init.c | 2 + > mm/mmzone.c | 2 + > mm/vmscan.c | 85 ++- > mm/workingset_report.c | 545 ++++++++++++++++++ > mm/workingset_report_aging.c | 127 ++++ > tools/testing/selftests/mm/.gitignore | 1 + > tools/testing/selftests/mm/Makefile | 3 + > .../testing/selftests/mm/workingset_report.c | 317 ++++++++++ > .../testing/selftests/mm/workingset_report.h | 39 ++ > .../selftests/mm/workingset_report_test.c | 332 +++++++++++ > 18 files changed, 1786 insertions(+), 2 deletions(-) > create mode 100644 include/linux/workingset_report.h > create mode 100644 mm/workingset_report.c > create mode 100644 mm/workingset_report_aging.c > create mode 100644 tools/testing/selftests/mm/workingset_report.c > create mode 100644 tools/testing/selftests/mm/workingset_report.h > create mode 100644 tools/testing/selftests/mm/workingset_report_test.c -- Best Regards, Huang, Ying