From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A6BC54EBE for ; Tue, 10 Jan 2023 06:25:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8151B8E0002; Tue, 10 Jan 2023 01:25:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C5BD8E0001; Tue, 10 Jan 2023 01:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68DA08E0002; Tue, 10 Jan 2023 01:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5A64B8E0001 for ; Tue, 10 Jan 2023 01:25:55 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1ECA3A3763 for ; Tue, 10 Jan 2023 06:25:55 +0000 (UTC) X-FDA: 80337903870.07.91E41C6 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf19.hostedemail.com (Postfix) with ESMTP id B414A1A0003 for ; Tue, 10 Jan 2023 06:25:51 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=kCcwpnH4; spf=pass (imf19.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673331952; a=rsa-sha256; cv=none; b=apRF8gmndFP/pUYyMM/Xdhw27jtus+Pgje10r4jp4v6KKV1wuzTjG/yAna5Y1YuKFjiLkB WlWrVJcFH5u4t03Q7QSY/g5MF3YTZGTErbPhA2+ZyP1kq6hoUdL5Y1E/SuDfbatNHlrLea WvKgcLPSnFSNFSROoPYCwSg80qms2NU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=kCcwpnH4; spf=pass (imf19.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673331952; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8JkCv+NV6PqjEH1P0gWr8z4uuHVpN6agMR3v9eR6cvI=; b=msRqruH2iqeFYyZsi7n87g+0uky8u1Ha/a23T2l3C4087MJKu80sn/FRsjutoAhaeruIwj zncj3nNmqkIq1LgAMNlngaPOweZifUf5DuYNBjD43QNa/iKVDbLk6LDUJ3Pep6mR92qmmh 9FHtymATcDNrFAyNCjrRawb8iL0UQWQ= Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 30A5HmNF011788; Tue, 10 Jan 2023 06:25:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : in-reply-to : references : date : message-id : mime-version : content-type; s=pp1; bh=8JkCv+NV6PqjEH1P0gWr8z4uuHVpN6agMR3v9eR6cvI=; b=kCcwpnH4+b1RbstOT3uVHLY4VspyvOm/FKnghDg7Rqzqr48xe0pGs3eMBTJtHzcdiWTR EbBQyv3BGiFOXSby2qPvL+O+hVLNeVQO5SA3j+p/0E1gteNizesk0omen8N4HKDCbleT /hHE3GiZEM+ie3k+EGqYJ+nLPDEyKnz3Hy+mZfH6zEjkMjdxzNfXJnGzzHnfnwckeXr2 4qYFZACJNxHHyWPVI+mRrVQzMOdEbtps7/i7H9k9a8qsdgjPBMazC0kePgx6jd4G/i5f BKrsvHhVxtmjTc3d3LzT7GHj2t0Bh3dDGh/jDWW7Rl6sy3ziW9lWILYL4YacZ3A1dSik 8Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3n111qt7a6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 Jan 2023 06:25:28 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 30A6LxiU005977; Tue, 10 Jan 2023 06:25:28 GMT Received: from ppma02dal.us.ibm.com (a.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.10]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3n111qt79y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 Jan 2023 06:25:28 +0000 Received: from pps.filterd (ppma02dal.us.ibm.com [127.0.0.1]) by ppma02dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 30A3U4Ev007443; Tue, 10 Jan 2023 06:25:27 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([9.208.129.120]) by ppma02dal.us.ibm.com (PPS) with ESMTPS id 3my0c7m6d7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 10 Jan 2023 06:25:27 +0000 Received: from smtpav03.wdc07v.mail.ibm.com (smtpav03.wdc07v.mail.ibm.com [10.39.53.230]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 30A6PQnL66388314 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 10 Jan 2023 06:25:26 GMT Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 06EF95805D; Tue, 10 Jan 2023 06:25:26 +0000 (GMT) Received: from smtpav03.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 083335805C; Tue, 10 Jan 2023 06:25:22 +0000 (GMT) Received: from skywalker.linux.ibm.com (unknown [9.109.205.160]) by smtpav03.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 10 Jan 2023 06:25:21 +0000 (GMT) X-Mailer: emacs 29.0.60 (via feedmail 11-beta-1 I) From: "Aneesh Kumar K.V" To: Yuanchu Xie , Johannes Weiner , Michal Hocko , Roman Gushchin , Yu Zhao Cc: Andrew Morton , Shakeel Butt , Muchun Song , linux-kernel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, Yuanchu Xie Subject: Re: [RFC PATCH 0/2] mm: multi-gen LRU: working set extensions In-Reply-To: <20221214225123.2770216-1-yuanchu@google.com> References: <20221214225123.2770216-1-yuanchu@google.com> Date: Tue, 10 Jan 2023 11:55:18 +0530 Message-ID: <87k01ulxdd.fsf@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: NbtpzACN7KtQ6ZN1evE6-1wqWOfuGVBn X-Proofpoint-GUID: 17EcHdWVxE5URnXUJJgzbWcprYSI_xok X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2023-01-10_01,2023-01-09_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 phishscore=0 impostorscore=0 mlxscore=0 mlxlogscore=506 lowpriorityscore=0 clxscore=1011 adultscore=0 bulkscore=0 spamscore=0 malwarescore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2301100038 X-Rspam-User: X-Rspamd-Queue-Id: B414A1A0003 X-Rspamd-Server: rspam01 X-Stat-Signature: ha6aiko6pzc3ucbjzpesinjzseuinjfs X-HE-Tag: 1673331951-826952 X-HE-Meta: U2FsdGVkX1/JSB3o01zB9BwdO35nfmOe8pYLFJ0ZCQmrA92SLWGnl9HB6Q6JwP35hbjeEmifqQtlogmUBcWKx5jz3X1UYNzbzaOXCCaUhJKIBIbU2DidkPPb3kUhzNSqBgf3Y/B05Vcf2ie765QRPK2//7ZlnffdtFUSneaxFSeh3vR6klrlxbg+eZsOyi/gHITeWUfzJ7vPIakgYw4huuKgooJVuMWC9FKQL5Z/11QwQZFFGHNrrT1XAADh6d8ay9/6j6QF/BDUUIvbuSGN7JWaXA1ztrjcJHnJZ3u2Xzzz3u36cXRv+JdtpzlvQ6LynyMK+qArtJ+f41r/KDWPdCuVMNnukT4fEDuaKvniyzL3J+rcsk+R5dCGGJVubYroC7Lc2UqUk8/bRAqBFxN8li4evWwQ6TrxTg88lFCf+DNRDq3l1FhpdUCb1KsjStjK9IFUGFQhWiwfwPhSnlsfobHn/oeSmtHjYixkWX3UyyekB0umwVJWzDlIvq+HTxbtVrlQ0oKK3wFcmDvZpWqjkKDzvIZirwTHec2inZgzprCa/Yl+Tz990koWoeHf5BGOSGodyyjBnrCnAvwc1JEC8KvIN0X4UPZG17CGLN1/lFLIU/fSypYVSQGV3VnkRx5kIxol1lA6eBLs3lmsROBcoGv+GVAb8mbsrEbHl8RWvqm/cUzn55Z74z9m6s9cJf02YFz5VJpHQBV+tSIlmiPKfOmWaRXmHoczBCnY6rGzoaBLIw+knA59grcb/0dt4k6LXpHTEHSd+Rh+j2Zx79R9rRloYqeC9/ckWcUyOCHk76naxkpiR+iZh/PcqlG8oV9pLrHz3OkW/OkXL991vqMp4V2NuV3zmwfq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Yuanchu Xie writes: > Introduce a way of monitoring the working set of a workload, per page > type and per NUMA node, with granularity in minutes. It has page-level > granularity and minimal memory overhead by building on the > Multi-generational LRU framework, which already has most of the > infrastructure and is just missing a useful interface. > > MGLRU organizes pages in generations, where an older generation contains > colder pages, and aging promotes the recently used pages into the young > generation and creates a new one. The working set size is how much > memory an application needs to keep working, the amount of "hot" memory > that's frequently used. The only missing pieces between MGLRU > generations and working set estimation are a consistent aging cadence > and an interface; we introduce the two additions. So with kold kthread do we need aging in reclaim ? Should we switch reciam to wakeup up kold kthread to do aging instead of doing try_to_inc_max_seq? This would also help us to try different aging mechanism which can run better in a kthread. > > Periodic aging > ====== > MGLRU Aging is currently driven by reclaim, so the amount of time > between generations is non-deterministic. With memcgs being aged > regularly, MGLRU generations become time-based working set information. > > - memory.periodic_aging: a new root-level only file in cgroupfs > Writing to memory.periodic_aging sets the aging interval and opts into > periodic aging. > - kold: a new kthread that ages memcgs based on the set aging interval. > > Page idle age stats > ====== > - memory.page_idle_age: we group pages into idle age ranges, and present > the number of pages per node per pagetype in each range. This > aggregates the time information from MGLRU generations hierarchically. > > Use case: proactive reclaimer > ====== > The proactive reclaimer sets the aging interval, and periodically reads > the page idle age stats, forming a working set estimation, which it then > calculates an amount to write to memory.reclaim. > > With the page idle age stats, a proactive reclaimer could calculate a > precise amount of memory to reclaim without continuously probing and > inducing reclaim. > > A proactive reclaimer that uses a similar interface is used in the > Google data centers. > > Use case: workload introspection > ====== > A workload may use the working set estimates to adjust application > behavior as needed, e.g. preemptively killing some of its workers to > avoid its working set thrashing, or dropping caches to fit within a > limit. > It can also be valuable to application developers, who can benefit from > an out-of-the-box overview of the application's usage behaviors. > > TODO List > ====== > - selftests > - a userspace demonstrator combining periodic aging, page idle age > stats, memory.reclaim, and/or PSI > > Open questions > ====== > - MGLRU aging mechanism has a flag called force_scan. With > force_scan=false, invoking MGLRU aging when an lruvec has a maximum > number of generations does not actually perform aging. > However, with force_scan=true, MGLRU moves the pages in the oldest > generation to the second oldest generation. The force_scan=true flag > also disables some optimizations in MGLRU's page table walks. > The current patch sets force_scan=true, so that periodic aging would > work without a proactive reclaimer evicting the oldest generation. > > - The page idle age format uses a fixed set of time ranges in seconds. > I have considered having it be based on the aging interval, or just > compiling the raw timestamps. > With the age ranges based on the aging interval, a memcg that's > undergoing memcg reclaim might have its generations in the 10 > seconds range, and a much longer aging interval would obscure this > fact. > The raw timestamps from MGLRU could lead to a very large file when > aggregated hierarchically. > > Yuanchu Xie (2): > mm: multi-gen LRU: periodic aging > mm: multi-gen LRU: cgroup working set stats > > include/linux/kold.h | 44 ++++++++++ > include/linux/mmzone.h | 4 +- > mm/Makefile | 3 + > mm/kold.c | 150 ++++++++++++++++++++++++++++++++ > mm/memcontrol.c | 188 +++++++++++++++++++++++++++++++++++++++++ > mm/vmscan.c | 35 +++++++- > 6 files changed, 422 insertions(+), 2 deletions(-) > create mode 100644 include/linux/kold.h > create mode 100644 mm/kold.c > > -- > 2.39.0.314.g84b9a713c41-goog