From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FDE3C47258 for ; Thu, 1 Feb 2024 01:17:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 709416B0080; Wed, 31 Jan 2024 20:17:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B93D6B0081; Wed, 31 Jan 2024 20:17:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 559DA6B0082; Wed, 31 Jan 2024 20:17:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4414A6B0080 for ; Wed, 31 Jan 2024 20:17:43 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E0F1AA018C for ; Thu, 1 Feb 2024 01:17:42 +0000 (UTC) X-FDA: 81741472764.17.CF3CBFE Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf15.hostedemail.com (Postfix) with ESMTP id 7A41BA000F for ; Thu, 1 Feb 2024 01:17:40 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vG4LSPlr; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706750261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=MhI0cjIpTwfZPdyOEuEpxlkv3Wjoztz4ObYONqohGBM=; b=m/7Kaed3pnvXHP7djJ16HAxI3JxBfXwHa+8y9CiffD4Q7nLS9h5KA9UqFpXjPG5+Qmt0zO dGrrH9eI9X/ww5R3aEx46/PAp0QlbZ5bEw4kqxZgiCPECpPNJz3vdVMZTHIr+fPvlgUHkB mTNpzGNk/bBtHsErqN+Nlg2T8X5Kk24= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=vG4LSPlr; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf15.hostedemail.com: domain of chrisl@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=chrisl@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706750261; a=rsa-sha256; cv=none; b=mScsvaJUcA7WUSBBhYgLO7vnlSaOTnMAcvuuMMaKfkkaxWzNHnzUft0Nur/pMonhSpbcdw IOJyZx8g4vg46YIIRPcaeCEXMtDZHByABOrnQpFfC3OQm3xJUdHDaRo6dxz6xwLZEiJnGI 8wz7uyqOV0WtuPCnmINgzf/1+aTzFQs= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 742BBCE2366; Thu, 1 Feb 2024 01:17:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2EA9C433F1; Thu, 1 Feb 2024 01:17:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1706750255; bh=UKxm5P0VYwSzMdnn3vI88JphP4KA0MGzRb/TUQaS9vc=; h=From:Date:Subject:To:Cc:From; b=vG4LSPlrX45rnD1bfgTx0HQ4lPQPb8wZ7wWwb307hOP36/lrjxAj72zaZUFWHIdIW b2my9Q++9VM3K3cjNKxImmgtdlsnny0K/DsqoWpfmQYDKQhHSGJvVnPmsN2kZuYuFa 29CdyRmBuBam6fJ2ZI43sSO5VKS80jxc+qD7Y5Hb98A1eiw6Rc+SneUcSBtc70hsTA qdOIiUGZXGTZjWNxO11459aGwyz5seIpa/EF/2tLbklYVe/v2k9C3PidIoXv1Wl31/ kxN8eDWsj8gXiiNMkAdqfQjKiaP4qN2M+K7RYbNou/lqKZzZNdWfUjFpvg2OrPkdum QCTvUBs079l8w== From: Chris Li Date: Wed, 31 Jan 2024 17:17:31 -0800 Subject: [PATCH v2] mm: swap: async free swap slot cache entries MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit Message-Id: <20240131-async-free-v2-1-525f03e07184@kernel.org> X-B4-Tracking: v=1; b=H4sIACrxumUC/03MQQ6CMBCF4auQWVvTmaKkrryHYQE4hUbTkqkhE tK7W4kLl/9L3rdBYvGc4FJtILz45GMoQYcKhqkLIyt/Lw2kySDhWXVpDYNywqx6dsaSxlNtCMp hFnb+vWO3tvTk0yvKutsLftcfQ/jPLKhQ2bqnprGWhl5fHyyBn8coI7Q55w+zHmippQAAAA== To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, =?utf-8?q?Wei_Xu=EF=BF=BC?= , =?utf-8?q?Yu_Zhao=EF=BF=BC?= , Greg Thelen , Chun-Tse Shao , =?utf-8?q?Suren_Baghdasaryan=EF=BF=BC?= , =?utf-8?q?Yosry_Ahmed=EF=BF=BC?= , Brain Geffon , Minchan Kim , Michal Hocko , Mel Gorman , Huang Ying , Nhat Pham , Johannes Weiner , Kairui Song , Zhongkun He , Kemeng Shi , Barry Song , Chris Li X-Mailer: b4 0.12.3 X-Rspamd-Queue-Id: 7A41BA000F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qqgdzgmfupc9tfoiukc7ouznibc5sa6w X-HE-Tag: 1706750260-955008 X-HE-Meta: U2FsdGVkX18UPr14Ozfc9PGcndSvfS8hMXyeC8255FFOHy/TJNk2bE2EUOuOmg/77ZkH8zIF6+FDc+NCzID9AasixzAOxIFPbQYZq+4OBT4H4l8wjX3epKyW53ncO38RPAFefSy0vnYdPzt0iqI3qBWDJZW5AW0luOfuB+uZokX/PstppWkYhK+Tdh4sOPH5IP0f7P2xC6tK1c9LY+BbCRjhchAgpVu6dBS9bE2zLo3TTtHLwNrdoBMB5esA4N8uA4P686wdBgozS9b01ZgbST4s1sV8O3golxzcH8NqpBiLU2kb7bHuB+COGvdkRWg4a91z5/f1f/5TWFwc6Ui2J9tUo4VeuM7WLn9widlVq4FF3DlD/4/RLdKof73Jo29onAKoQ7wVjIc2t5OnbOzF/yi7s3Rrgifl/mVD3HBA5Ha/uZI3yWCqMJC5DqBQG4SXh5JOI+gpURdNvGGQAMyExYjXLzlAbzk0e92qT4lqyzkP/NDNnMSt2m/OrafAMM/qQKqDkoMRprdK1wwUNaQZH7yAV8NtpK1b3hO8TGd+Fj4GlyQzV+ehlAO1puCm2SA8AHOJ6OW3PQJgzvO1Vq8li1H8UXmjGJ+Ap3gcgLSAkdzhuJyAEb22KFEQX4mlvzycZdomApuAT+smFCdGSzJYI82ylDS52SF6cLqwLOPlRZpJK5yR+c0nj78MLj83pbRDHUbT3ucK4J3qf/P/+z5+31UcZQvhh0oeam9HSd4u4ZJ0LSfJz4CpQy3ZmLpQPpb7ceEo8KKl0jckq7KN7Yf7idNVu/CYajEJTo3k3WVsglFHauh7ozeXxfKYwX+NjKW3kKzlClhFEqy6n6ltj0Z+8ExE7M6WZ2V4QsaPxXtuKndKluMVF/0rr/XDniqGodCWKfIFZ5/cREzMfTQEmK0S/TO4pn3CA+11D+b+QeC2mT2UsYH0AbvsLJQxUvLMR0PpMTmTb7IPSGv3s7Lb/q1 lH1aFtCl k+36HtsuAK/MngwJc2sNklIAWyDkQ8PA5FSFaauNbjFZ9Hcc3QROTIVzf5/KkePcvk3dfY881ZCjF08xcK/4Ae4xpcGlCkv/7vasvUladxFDOG/4gs4TIdbZcmgEPTJTNDuItQUu5It1UCVGiavHQxEA/EfSE1PrpVYRmqyy8fmGnaYuuKQH8JiBxdS2JXXAHaelQ9w7u9NKtMd3fK0ozD/R4UqPkV6c7oi2jwcv3yB4xQV5mwiA3W6lo8eSRrrQ+vvvj7J3ONUeZF3zQtbxtfcWqAxHJkZukof+xPlpijxhcVyLJTiQEqSf6akF7LouOEJuHpG7SElKh+a6Ogqp+RW/rJeoJFaFU2g+ekSu2sE4f1yzQK210oOULbThmMWBFu9lRv45qOwVE0E45vAbgso/JQ2XH25d06kZ7E+pxr1e5DHiDSxG1x/kG68AoYiZ1j7y3k21wc8JtR50pWidOqXfuYyrHE2zPWaUFw5jZrAOfl+H8eYVQnfBVDE7qwztcuwq8BsdBF0jZB6R6DB2gslXb+R+4AtfVy0tlVGA9RPxNsuS60GKdZc0BFpmZisuSlZ6uZ9VJaRaVxcnrWm0otknEEC2h1jtdOWFZPhu4Au9iR5E+Ti+IGOMxU6ME5KXGkACjrQENjJ2WKLCyZqLnVFbZBYSNjCCNhrK2Gy42tHXLIA921Fh1p6KZRy1Ksj2//HzgH3Tc6BbQPhj0NQ94RGpizeSMQa4BINspoXoHQP5vzU7wuJAkFx0C5PQnQ3sJ9KfrtM6wDerStB8+hJvI2mx+ApSREVEWOPis0uPQ8VelGKbVTbzqgVJln5FenuCYZGNzL0cUYW+WLqyhpyGYGmKBtVZuIRSVs6ketkT37kO5koZQfS6SV0zdAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We discovered that 1% swap page fault is 100us+ while 50% of the swap fault is under 20us. Further investigation show that a large portion of the time spent in the free_swap_slots() function for the long tail case. The percpu cache of swap slots is freed in a batch of 64 entries inside free_swap_slots(). These cache entries are accumulated from previous page faults, which may not be related to the current process. Doing the batch free in the page fault handler causes longer tail latencies and penalizes the current process. Move free_swap_slots() outside of the swapin page fault handler into an async work queue to avoid such long tail latencies. The batch free of swap slots is typically at 100us level. Such a short time will not have a significant impact on the CPU accounting. Notice that the previous swap slot batching behavior already performs a delayed batch free. It waits for the entries accumulated to 64. Adding the async scheduling time does not change the original free timing significantly. Testing: Chun-Tse did some benchmark in chromebook, showing that zram_wait_metrics improve about 15% with 80% and 95% confidence. I recently ran some experiments on about 1000 Google production machines. It shows swapin latency drops in the long tail 100us - 500us bucket dramatically. platform (100-500us) (0-100us) A 1.12% -> 0.36% 98.47% -> 99.22% B 0.65% -> 0.15% 98.96% -> 99.46% C 0.61% -> 0.23% 98.96% -> 99.38% Signed-off-by: Chris Li To: Andrew Morton Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org Cc: Wei Xu Cc: Yu Zhao Cc: Greg Thelen Cc: Chun-Tse Shao Cc: Suren Baghdasaryan Cc: Yosry Ahmed Cc: Brain Geffon Cc: Minchan Kim Cc: Michal Hocko Cc: Mel Gorman Cc: Huang Ying Cc: Nhat Pham Cc: Johannes Weiner Cc: Kairui Song Cc: Zhongkun He Cc: Kemeng Shi Cc: Barry Song remove create_work queue remove another work queue usage --- Changes in v2: - Add description of the impact of time changing suggest by Ying. - Remove create_workqueue() and use schedule_work() - Link to v1: https://lore.kernel.org/r/20231221-async-free-v1-1-94b277992cb0@kernel.org --- include/linux/swap_slots.h | 1 + mm/swap_slots.c | 29 +++++++++++++++++++++-------- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/include/linux/swap_slots.h b/include/linux/swap_slots.h index 15adfb8c813a..67bc8fa30d63 100644 --- a/include/linux/swap_slots.h +++ b/include/linux/swap_slots.h @@ -19,6 +19,7 @@ struct swap_slots_cache { spinlock_t free_lock; /* protects slots_ret, n_ret */ swp_entry_t *slots_ret; int n_ret; + struct work_struct async_free; }; void disable_swap_slots_cache_lock(void); diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 0bec1f705f8e..71d344564e55 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -44,6 +44,7 @@ static DEFINE_MUTEX(swap_slots_cache_mutex); static DEFINE_MUTEX(swap_slots_cache_enable_mutex); static void __drain_swap_slots_cache(unsigned int type); +static void swapcache_async_free_entries(struct work_struct *data); #define use_swap_slot_cache (swap_slot_cache_active && swap_slot_cache_enabled) #define SLOTS_CACHE 0x1 @@ -149,6 +150,7 @@ static int alloc_swap_slot_cache(unsigned int cpu) spin_lock_init(&cache->free_lock); cache->lock_initialized = true; } + INIT_WORK(&cache->async_free, swapcache_async_free_entries); cache->nr = 0; cache->cur = 0; cache->n_ret = 0; @@ -269,6 +271,20 @@ static int refill_swap_slots_cache(struct swap_slots_cache *cache) return cache->nr; } +static void swapcache_async_free_entries(struct work_struct *data) +{ + struct swap_slots_cache *cache; + + cache = container_of(data, struct swap_slots_cache, async_free); + spin_lock_irq(&cache->free_lock); + /* Swap slots cache may be deactivated before acquiring lock */ + if (cache->slots_ret) { + swapcache_free_entries(cache->slots_ret, cache->n_ret); + cache->n_ret = 0; + } + spin_unlock_irq(&cache->free_lock); +} + void free_swap_slot(swp_entry_t entry) { struct swap_slots_cache *cache; @@ -282,17 +298,14 @@ void free_swap_slot(swp_entry_t entry) goto direct_free; } if (cache->n_ret >= SWAP_SLOTS_CACHE_SIZE) { - /* - * Return slots to global pool. - * The current swap_map value is SWAP_HAS_CACHE. - * Set it to 0 to indicate it is available for - * allocation in global pool - */ - swapcache_free_entries(cache->slots_ret, cache->n_ret); - cache->n_ret = 0; + spin_unlock_irq(&cache->free_lock); + schedule_work(&cache->async_free); + goto direct_free; } cache->slots_ret[cache->n_ret++] = entry; spin_unlock_irq(&cache->free_lock); + if (cache->n_ret >= SWAP_SLOTS_CACHE_SIZE) + schedule_work(&cache->async_free); } else { direct_free: swapcache_free_entries(&entry, 1); --- base-commit: eacce8189e28717da6f44ee492b7404c636ae0de change-id: 20231216-async-free-bef392015432 Best regards, -- Chris Li