From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D93C07E97 for ; Mon, 27 Nov 2023 23:46:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E656E6B0257; Mon, 27 Nov 2023 18:46:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E39B96B0258; Mon, 27 Nov 2023 18:46:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3E366B025A; Mon, 27 Nov 2023 18:46:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AE02F6B0258 for ; Mon, 27 Nov 2023 18:46:06 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 8AA60C0124 for ; Mon, 27 Nov 2023 23:46:06 +0000 (UTC) X-FDA: 81505369932.22.C436134 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf30.hostedemail.com (Postfix) with ESMTP id 3539780476 for ; Mon, 27 Nov 2023 23:46:01 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GTj1mejU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701128762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=0naXIH0JGObfzirOc6Nz1+BDeP5C+xEUqfta7BLJ2ro=; b=hV0E1fBvXgdkicrhl5UrXnQyc7ji1z5Pfa+ALWTP6+mFeUpEak4VXfaXadjHo4KcI0SFux 1ngcXTjhJr0dZevy1yFCzzCEOL1Jv261RNhXzRO6941sRnf/auZcpA+puotnO6sum0Doi6 MGj+P/y3lfONT1XQNf6BNyRei7rRhhk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=GTj1mejU; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701128762; a=rsa-sha256; cv=none; b=3sjwdKoed3B7QgcihpH5sDGu+ZqGEmEYNi9FuoYKaqZJdXPphi2nf4yZUsFT1DIrQJEYlr m4EV9HLuy1Ts29ji0EwmlCAdBA66nUxKO2ZHduZ5U/Fi6iHyPNwkSqt5UNMoBMUpj4xgz3 8164BL7TZOExJywRWzSzcCly/vKuwcI= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1ce3084c2d1so40259135ad.3 for ; Mon, 27 Nov 2023 15:46:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701128761; x=1701733561; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=0naXIH0JGObfzirOc6Nz1+BDeP5C+xEUqfta7BLJ2ro=; b=GTj1mejUIJnUiepf1LpbSFtyxVu3y14BGve7EGdro9PlGmFx5xf9rP3pFzw4UQo9I9 TaX9zVUj6pjGMr8zo7sImN4rVPwwiAjVAqJmUiX3Pzu7j/lg1z7Tq60z0R8+kZTqzp0M 4MZyE+25DxjU1kUbp43LXxyrDxTovk+WMA3ERyGPTDsrZcuko+Nm+EBkwmBdEnRDFCcw bXkKOXmli9wo6+COsEzqzBpR4qSGz4nDxnPVB9upeUYGXp7+94isbTIGdjoctm4bvIUG NKMAfAKDUq9QgOFO31wYYuM+mDUipOsnHHlpVsd2+g/ayg+zO81THwfWRQAt0YkccXoR xTBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701128761; x=1701733561; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0naXIH0JGObfzirOc6Nz1+BDeP5C+xEUqfta7BLJ2ro=; b=PUsCyBLHigoSG8xck7M0lyE4bsUklga0x1BLfCd83ZOdzv7ZpEMqFRPAUWL3udPvf4 5ZlVNmSvjW/myWNgPqMS4OlWiQSpCHpF2ucUNRUfUaimZIlTwjGw68ogNlsCSw3RpamR 9wNUOZ75/BUKfFVizQz27Pi2CeN6m3Aop8BYMtUMc8bhAYWjYkoaOZoy/hQEoNZhZjp1 qpX2JLejLTpwMgz8FfEYNBqnKK7JTa5slCqJJC82RSLDcJhwKNqu7b2y+33UUkpmc0go V9iYv75001pSO7AmAWB6W+tug8R0Dj7T04tUqjGAbZrcrjFu2i06yzAAFjwoUlcMOGDG Zivw== X-Gm-Message-State: AOJu0Yz/bkPtZPM0dd40rNxHRXwr1xrMglyyEIFkzrTsQFGA7NgV6QmN lazHREDmxtp2qymbR3oyP3ZsKmmxN70= X-Google-Smtp-Source: AGHT+IG4eMKvbXes5v/gD3pRMvH+WaXrJYYV2FbfNJ0lvXkjI+dG00dYmERvEMll3xJo4Y61Ab/OvA== X-Received: by 2002:a17:902:a707:b0:1cf:f1a0:c600 with SMTP id w7-20020a170902a70700b001cff1a0c600mr1039143plq.33.1701128760949; Mon, 27 Nov 2023 15:46:00 -0800 (PST) Received: from localhost (fwdproxy-prn-020.fbsv.net. [2a03:2880:ff:14::face:b00c]) by smtp.gmail.com with ESMTPSA id a9-20020a170902ee8900b001ca86a9caccsm8857868pld.228.2023.11.27.15.46.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Nov 2023 15:46:00 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, cerasuolodomenico@gmail.com, yosryahmed@google.com, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, chrisl@kernel.org, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, shuah@kernel.org Subject: [PATCH v7 0/6] workload-specific and memory pressure-driven zswap writeback Date: Mon, 27 Nov 2023 15:45:54 -0800 Message-Id: <20231127234600.2971029-1-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 3539780476 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: d4qja6gqhkbsqadjb7czj9sedapsf5au X-HE-Tag: 1701128761-196655 X-HE-Meta: U2FsdGVkX1/0jTlgUucWkQJ6f6OJC8wyrM49eNt4PI0p8IqbcpWWoDHzEz/2/GHf0FK5tdzrkTPR9Z5lZCmVjvw5zuv4eYqW6NQZlV45PTVS+G+b5lBsGYwkICqOq2o/V/ba3FNvzYYfLS3ucTOeQoycJ0K8SI4kKPCm/q8IS6g2o2qWxd1HXJV1+0NMQjs8RV5+d19jSixwknBpZAlU27+OEeRI3wh9k3Wh8a3odHyoaXeRy9KUv8jjiEHhhg0n29Sapqk+wvUK/aXsR0pTaq/eom5K9GaXubByiZclyG7IYw/cWTWcLV6+o/agWU2Ic8mY0sX95ehhYtIZYsolUN4gTNfZ6+AZBcoxntwD8kmV9s4bG/enF1brGXZc5Leyss+lt2Gh9zUGQQWCBo2nN0P3lHBuySOX4Ex3SI9bgJvzAtJsKEKx+9LiZkzPbd70TPsC918DCPbobdi0WNO+g9NPGAF7m/nq7oE5hz/TXSDSbHKy5K8qNCG2M5KcjFIsrkn/2odQD4z8KrQOqo23oYPF0vteUU7HPuxrBIxpWGdRJc0El4n07muInfHNkH95ynLWaAX+8Gz8H0W/EpPGk9SJZSQlzVDOo0cJ6Fi5+sV3XHV45CSe8EQPRnTvqhDzLr1IbzZ9VoT6U0rD28yQW8oufTBHbJk3MvMn9yF1HQuFq3Ous17QIWvhR3OsPyrvQL8iuPM27G06VSrxbPslPf0geYySObcT/36nQoMBGUdO9/y7X4MbFAFRlDQ+4CmceP6WZk0X0n3ZidfElJPs2B4JrepkfRNshTEBvItS0YYZLo55QL7LBw0M7Q1zndosLMlh13RxQgIFELdZGXtPbeqKX8Q4fDZFPBMf1+E2amEPgkTssMDzOB9/LFYEelkY/QG7r63e0bzIvGNDDykjJzfJwQ7mrDUmWuR/AXhlGcIiMetkgrWovBe4988Bo55BouJtvPLJ5gl9eFH7Wi2 MKQ4tSYd Obsr2sd6cT2u6fp2HYzlWHJtCKwVW3nceE6DeaYBp+RXLMyBUPGI3Jnb0COCw1H0kd2Adu97mSd3xDcR53j/ZgbV74wJ3etZzFTzOaMAlbv5K8RwRqyhGuQmyuGNgz4ZjHSUwC70SZj4AtFH2bYXytOsKO0lbTNJQiyoNc8Rc26+LcCXtxMfykMGUC/exZJWfEd4g/zcAeNjbJg0+hjXpYa9u8u9/J4wgVDClt+WCQuqr83B/vayFaLWioqup59aF734/9VhNj1sRXg+qvgg1sx682F3MZNNWnoUjfU2dAPPji5iFYutmTTAK0ACMUyE8T9ZD8OeaPPlIAAWRjs2udGCiKgQSS1ylK5g/2Kaf8v2PqwJ5Ac/pbBo9ao6oVtfadu2Z9uvGGtJu9iYGfi5+Cu+pN3i2Yvnv73UVpvgP7GrKta77uu3kSBfzvAaPt3HSmpHn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changelog: v7: * Added the mem_cgroup_iter_online() function to the API for the new behavior (suggested by Andrew Morton) (patch 2) * Fixed a missing list_lru_del -> list_lru_del_obj (patch 1) v6: * Rebase on top of latest mm-unstable. * Fix/improve the in-code documentation of the new list_lru manipulation functions (patch 1) v5: * Replace reference getting with an rcu_read_lock() section for zswap lru modifications (suggested by Yosry) * Add a new prep patch that allows mem_cgroup_iter() to return online cgroup. * Add a callback that updates pool->next_shrink when the cgroup is offlined (suggested by Yosry Ahmed, Johannes Weiner) v4: * Rename list_lru_add to list_lru_add_obj and __list_lru_add to list_lru_add (patch 1) (suggested by Johannes Weiner and Yosry Ahmed) * Some cleanups on the memcg aware LRU patch (patch 2) (suggested by Yosry Ahmed) * Use event interface for the new per-cgroup writeback counters. (patch 3) (suggested by Yosry Ahmed) * Abstract zswap's lruvec states and handling into zswap_lruvec_state (patch 5) (suggested by Yosry Ahmed) v3: * Add a patch to export per-cgroup zswap writeback counters * Add a patch to update zswap's kselftest * Separate the new list_lru functions into its own prep patch * Do not start from the top of the hierarchy when encounter a memcg that is not online for the global limit zswap writeback (patch 2) (suggested by Yosry Ahmed) * Do not remove the swap entry from list_lru in __read_swapcache_async() (patch 2) (suggested by Yosry Ahmed) * Removed a redundant zswap pool getting (patch 2) (reported by Ryan Roberts) * Use atomic for the nr_zswap_protected (instead of lruvec's lock) (patch 5) (suggested by Yosry Ahmed) * Remove the per-cgroup zswap shrinker knob (patch 5) (suggested by Yosry Ahmed) v2: * Fix loongarch compiler errors * Use pool stats instead of memcg stats when !CONFIG_MEMCG_KEM There are currently several issues with zswap writeback: 1. There is only a single global LRU for zswap, making it impossible to perform worload-specific shrinking - an memcg under memory pressure cannot determine which pages in the pool it owns, and often ends up writing pages from other memcgs. This issue has been previously observed in practice and mitigated by simply disabling memcg-initiated shrinking: https://lore.kernel.org/all/20230530232435.3097106-1-nphamcs@gmail.com/T/#u But this solution leaves a lot to be desired, as we still do not have an avenue for an memcg to free up its own memory locked up in the zswap pool. 2. We only shrink the zswap pool when the user-defined limit is hit. This means that if we set the limit too high, cold data that are unlikely to be used again will reside in the pool, wasting precious memory. It is hard to predict how much zswap space will be needed ahead of time, as this depends on the workload (specifically, on factors such as memory access patterns and compressibility of the memory pages). This patch series solves these issues by separating the global zswap LRU into per-memcg and per-NUMA LRUs, and performs workload-specific (i.e memcg- and NUMA-aware) zswap writeback under memory pressure. The new shrinker does not have any parameter that must be tuned by the user, and can be opted in or out on a per-memcg basis. As a proof of concept, we ran the following synthetic benchmark: build the linux kernel in a memory-limited cgroup, and allocate some cold data in tmpfs to see if the shrinker could write them out and improved the overall performance. Depending on the amount of cold data generated, we observe from 14% to 35% reduction in kernel CPU time used in the kernel builds. Domenico Cerasuolo (3): zswap: make shrinking memcg-aware mm: memcg: add per-memcg zswap writeback stat selftests: cgroup: update per-memcg zswap writeback selftest Nhat Pham (3): list_lru: allows explicit memcg and NUMA node selection memcontrol: add a new function to traverse online-only memcg hierarchy zswap: shrinks zswap pool based on memory pressure Documentation/admin-guide/mm/zswap.rst | 7 + drivers/android/binder_alloc.c | 7 +- fs/dcache.c | 8 +- fs/gfs2/quota.c | 6 +- fs/inode.c | 4 +- fs/nfs/nfs42xattr.c | 8 +- fs/nfsd/filecache.c | 4 +- fs/xfs/xfs_buf.c | 6 +- fs/xfs/xfs_dquot.c | 2 +- fs/xfs/xfs_qm.c | 2 +- include/linux/list_lru.h | 54 ++- include/linux/memcontrol.h | 18 + include/linux/mmzone.h | 2 + include/linux/vm_event_item.h | 1 + include/linux/zswap.h | 27 +- mm/list_lru.c | 48 ++- mm/memcontrol.c | 32 +- mm/mmzone.c | 1 + mm/swap.h | 3 +- mm/swap_state.c | 26 +- mm/vmstat.c | 1 + mm/workingset.c | 4 +- mm/zswap.c | 426 +++++++++++++++++--- tools/testing/selftests/cgroup/test_zswap.c | 74 ++-- 24 files changed, 641 insertions(+), 130 deletions(-) base-commit: 5cdba94229e58a39ca389ad99763af29e6b0c5a5 -- 2.34.1