From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DA22CD1284 for ; Tue, 2 Apr 2024 07:33:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E569B6B008A; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E06B86B008C; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA73B6B0092; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A713C6B008A for ; Tue, 2 Apr 2024 03:33:06 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 78C12A07A4 for ; Tue, 2 Apr 2024 07:33:06 +0000 (UTC) X-FDA: 81963775572.03.A77B062 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf03.hostedemail.com (Postfix) with ESMTP id 9FB0A2000E for ; Tue, 2 Apr 2024 07:33:04 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ppy3xq+n; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=ZVCw2B7KQxpocjK9IGI0Ao8QivgURa3CaChKhLCc9m9EF6GPV3SPP9lgPFuxvM5UI8O0VR 9kke/X6If+kbAcovoTxsIPyoK+6JBFo+WLbGPdzStk1ypZrleTWbtpAIlbnUxkJ6Hhit75 pnwjMne1UE6A4QtXhk2ERJJixKWgjhM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043184; a=rsa-sha256; cv=none; b=jT6EUIi2IzMgLn8/A/GsEkwSNY+GaVZpqCBp2NNTxP9XpyBHF4fGmu+HdjJarQ7VllAVcj WcRuS2GF7WEVYfIa/3hkTbM2krZ/CU3RxEpiaoJ2WC9eWxSMEtZ11Xi+GiCiPj5R/rDsb4 PNx53RpR/ZdFMvzkMjw5vHdBEuSXd8E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ppy3xq+n; spf=pass (imf03.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.178 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6eae2b57ff2so3689541b3a.2 for ; Tue, 02 Apr 2024 00:33:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043183; x=1712647983; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=Ppy3xq+nySyif7WjjV1VhXIyv8GeN1b24SQfA4UlPe0U3z/HCwtrmXGxJrWEw2PAMd GjnghLc31enQ2YxeqA0bLG+OBhV8ADiLQ0caqFJ1rSDN7FyUTAB36L4P/y5oCEahnQ4C EWaUOjFBL1EjVwfz8AOqe0MAdQeE1UUSQhzCwxqCw2IJT8heAQV2jmP2gCYfGS6P9bP/ shHxgfnyfAY3a10hSexxyfrGMgvM2vhVyw+gRHCRFr73W7hIs640ZoohUUgy3lv/mDcg 5Bqb5r6vddXsf1IPhoGDkLYMGY8vaIETIGEj+r9lzpAHOzxC/bOYUTsLD1wt9Iwq8m+Q 5sIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043183; x=1712647983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HqLi1Li9mPDQzeMhmTcE3iTZttLxYb10NaxNd2/X0Ys=; b=V+RMqPsCzCAZ/zIHppJAR/6HZMyxPP0gkW9t5rTWm2TPOxvAuRQDszxsx3V9X4V2BD gdEvLlqQbdtkgbWuKMF7V5N5pU2nuBvVSxZh7CcusFsBg81SlUkZIEza9wAVxtaAQyGH 1gZM7AMoxngPrvBwiT+gxuGdy8lwwXYoGEySMfY6pXv2z5H3ZETNMBByBWfdRUinld6Q Dn+ntpEXVA4chcDNn/6zayb+JqUvBRRYXlKMqwK6+Wq4cgkCJn9539+Npul9urKWw+2m Z5orACavSFEyVAHkXSO5iF9HZZqhLMD+nlJH+BBOPeKb3saQOCsFbwpc9SnD5PhXQzJr aT1g== X-Forwarded-Encrypted: i=1; AJvYcCXe5j0bAiuOvi4byEOQRY9fG2kw0YTNlLnuXdmqxhxsRhAFm1VtgS7BgIZg8EOWgge/HlR09MDhQzn/2BKCkCANOlo= X-Gm-Message-State: AOJu0Yy231Dq2AdSdZAmjMP+8luP3yzLDyYfTa2OfA992I24n6oYAnav uEBkWAlk9xCeFUyLZOHWLxflYORDg65go1RdiStG4aREB76QyIkB X-Google-Smtp-Source: AGHT+IHQS2pMgzGS3ZZ9BOQYMyBw3FyEprxhgztZHlVlDCuWgKVjsoIbCWo8BMdbjCYd4MvwvVp6QQ== X-Received: by 2002:a05:6a20:17a7:b0:1a3:34c4:b184 with SMTP id bl39-20020a056a2017a700b001a334c4b184mr11043769pzb.19.1712043183375; Tue, 02 Apr 2024 00:33:03 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.32.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:03 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 1/4] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Tue, 2 Apr 2024 20:32:34 +1300 Message-Id: <20240402073237.240995-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 9FB0A2000E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 547nmisqf55ob3ebrx76y8tcoinq6onh X-HE-Tag: 1712043184-942191 X-HE-Meta: U2FsdGVkX1//awXXxhsC4RF91qyGYLGH3n16n87Hethj7OpiMWU2VVknvpRf1ulQ9Fhb6rcFwrzFJJn7Xj4+RIshC5YlIaZlalLrB0lqopcKl6/Fxe2z+sE1eR7zGn/aCOPURQ4902XcYFrpq/8Wfc8jOO0FwTTgsCU1j//Wbkytps6iQjXGJd0TguopIJqeiOwRtsnc05xS0BPcmnmLxBRvq+xgYHFAhXTBKjrzMrS/+hU6HL2IQ8+AEpnvYDnUHrSAX6PDGWslz0apU2F2C+B4QHXTy1VQpITYvJxW29knOM0oVtGTxUW1AMpJC4znJfS0+6q7N3Xs+x8dzW1l4Cj7lOdctaiRV1nxAhtBM+clam9zkmPPft+A1KwXS/fhFhFpU/vBgeKYKyOXJ4/OAGYbi7qHgP9dZaAnSdDFJ0L7y1Rr/4u4NdgV/TQ1PZv3vRSPMpbfVnaa97BtPoe1Cvk1cCW/amwpPnb4L1XkK8ApgnvtO+iQwo1aOAbY45a9580j8ezvax03RmEOty5K1DZ+WepCJqqFW3SHVhShurhyNiPVmufXIpVnN7BBaJoj5/1xQcaFtAq1Vj9FPmWmAVEQGqvKY+ylKKcbj57eMcJhZv8IZrBc7BGovoeUuLRC8WAIVHCObKykmxtTOyZ0CpBV7TKc6L/3HQVfMOUQ6BhpG7ABtDbIx0CcqjwcWGeGgGDzSAZUrAcko36fgO8J7tdpFxnM4FX4RSLLYunfmb+VgTCF+QKIBCr5CWC9jkkFnQ6utb+8ZWFxy0HmVpfX2WEe1esAWaLIwNmXOdZ9Kta1r/GDT2XVP/j9sHiqQcZEt+2COfC0lYEF6zZ3hosbv+eYs6zCuw1VKyqdul8WGluGSrES4+P2HMM5qyU3TPZ30+vIRscy8495d9GsO8Dy/xw+2+lSqeAvgrFXL/NZiKs1fXZ2ZUxsxnaXKoDqx4FeC8ZpOYct4rFgVT0x+4K 8aB1zPBI GlMeKkucVxR58qjJHK4m6cIfDba24hwP70Jz1SzMkOaM00a/8qJTLOZC53AQyOy4YZ3oaspCNblY0XxnGjpGOkBhgA7G4mI+ySNqZ8OSbc8hv8T/mGfjAMGixxGwR31Ki9rg+x7GZpRu2XPnpeUB6UBfJM/xgAAXICd5HZE6aDgNpQyYd5/FJwrE5KxafKHDmtapjBS4Oh87F6vsa7AGemIRL3o5cdGtbGhtcXJP/Nsp+hUhgn0JWW6BOq5n8/natlzhdDGX8PbRZL4bFd7hWK2MJuM/qOP8hyyu59UsBpC5CP115RXhlG27z9xh8cNQ9dD8I8qApJsV1YIZEE6MRe9nVVgpG6dNO7HZ7qVXjvjYp0iEChd8AXaRmC2+skRRB3i8cPizZL97yhvB1DUNssW92ezeZvxNFS+Cz5ZLaSd/ws5OULuGgOsS2pW5FyIjiZRw19Mz6iv+dDzdzMJZwWMxlsHRY8bHaRFz0Fmepu4iu7QH1u1b7CDqgf5CiSWtyRYPFgtpahz3+HH4sY+twWzAC4NkANyavktIi X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..b7a107e983b8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index d56cdc547a06..b6a63095ae67 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1357,6 +1357,57 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +/* + * Free up the maximum number of swap entries at once to limit the + * maximum kernel stack usage. + */ +#define SWAP_BATCH_NR (SWAPFILE_CLUSTER > 512 ? 512 : SWAPFILE_CLUSTER) + +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int i, j; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned int type = swp_type(entry); + unsigned long offset = swp_offset(entry); + int batch_nr, remain_nr; + DECLARE_BITMAP(usage, SWAP_BATCH_NR) = { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages == 1) { + swap_free(entry); + return; + } + + remain_nr = nr_pages; + p = _swap_info_get(entry); + if (p) { + for (i = 0; i < nr_pages; i += batch_nr) { + batch_nr = min_t(int, SWAP_BATCH_NR, remain_nr); + + ci = lock_cluster_or_swap_info(p, offset); + for (j = 0; j < batch_nr; j++) { + if (__swap_entry_free_locked(p, offset + i * SWAP_BATCH_NR + j, 1)) + __bitmap_set(usage, j, 1); + } + unlock_cluster_or_swap_info(p, ci); + + for_each_clear_bit(j, usage, batch_nr) + free_swap_slot(swp_entry(type, offset + i * SWAP_BATCH_NR + j)); + + bitmap_clear(usage, 0, SWAP_BATCH_NR); + remain_nr -= batch_nr; + } + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ -- 2.34.1