From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A003DC48292 for ; Mon, 5 Feb 2024 18:15:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E159D6B0072; Mon, 5 Feb 2024 13:15:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC56B6B0078; Mon, 5 Feb 2024 13:15:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB49D6B007B; Mon, 5 Feb 2024 13:15:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BD6536B0072 for ; Mon, 5 Feb 2024 13:15:09 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 6CDC9160B65 for ; Mon, 5 Feb 2024 18:15:09 +0000 (UTC) X-FDA: 81758551938.02.B754879 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.10]) by imf18.hostedemail.com (Postfix) with ESMTP id D309A1C001E for ; Mon, 5 Feb 2024 18:15:06 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SAXz77Vp; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 198.175.65.10) smtp.mailfrom=tim.c.chen@linux.intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707156907; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=buurFExRzrw7WJp0+xeMHkWXa2TAUgVl9c6QIjxHktU=; b=HfFV+vVnw2VRwnrCx+V9dUF8aV+e00sYH+S7m4qKvfMcrKHRb7trtFH1fKYkcYkoi/T5T7 fbu3WwUnHxNMgMK7rehyHMHVN/yhIBEVSbBIu9Vbk9EDzmhtJSayT6FbjyXO3Qb6y6Zolr hQ+cRunlMQp4CzHlj0q0/Vwd3KOJbgA= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SAXz77Vp; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf18.hostedemail.com: domain of tim.c.chen@linux.intel.com has no SPF policy when checking 198.175.65.10) smtp.mailfrom=tim.c.chen@linux.intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707156907; a=rsa-sha256; cv=none; b=mqj8tH5h+PabYqWzzxA2FWEz/7C1aycAbXY32ZimFg244o2U+GgXOZxNWAeBsSqCqQwkWp axjM6FYYBBwTm7kP4UOD0/KlZTdo8jYYt4Mtyzkj9iZgnMqnauyavZ/JyLEZJlrdT9lnxR qWopfAvTWu3Cxu+xthOC+nVmuq4XhYM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707156908; x=1738692908; h=message-id:subject:from:to:cc:date:in-reply-to: references:content-transfer-encoding:mime-version; bh=conKQZ6wiYv2oi5ekMEn/OIRAFGKnZQbNRkFZLGjHCQ=; b=SAXz77Vp/Ps43MeknPvIa27CXY2y5RLhXbHZvXthQpKVL2hluH3JtoVr Y7N2UkqWea23NbAEeixS9tMlOkAsAWi1EeCOKkLGz9/X6qUHLr55HPpAq yBGJO4fEt4WX5zhfwsSzN22C0HwDZU4PpPCOVTVnNJX5w8Qz1dfciqCn8 qSgdBAbIXnoqEvJcL9+6Q/vWAfubFilyTtqNZ7XMg2wZpxd/A/glvx8sq dWdXv6wvWDfl4LNwgM2N0+n8RN2ZGUnadgt1BkhulYGRfJoKdFzNVzFKY r9DyprKsO5hY01TzWG1ZjiK93nvIE8k/yJ+Fcz9VFm7epHjx4k7fp264I w==; X-IronPort-AV: E=McAfee;i="6600,9927,10975"; a="17995535" X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="17995535" Received: from orviesa004.jf.intel.com ([10.64.159.144]) by orvoesa102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 10:15:06 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,245,1701158400"; d="scan'208";a="5536044" Received: from cacunnin-mobl2.amr.corp.intel.com (HELO [10.209.92.172]) ([10.209.92.172]) by orviesa004-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2024 10:15:05 -0800 Message-ID: <1fa1da19b0b929efec46bd02a6fc358fef1b9c42.camel@linux.intel.com> Subject: Re: [PATCH v2] mm: swap: async free swap slot cache entries From: Tim Chen To: Chris Li Cc: "Huang, Ying" , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Wei =?UTF-8?Q?Xu=EF=BF=BC?= , Yu =?UTF-8?Q?Zhao=EF=BF=BC?= , Greg Thelen , Chun-Tse Shao , Suren =?UTF-8?Q?Baghdasaryan=EF=BF=BC?= , Yosry =?UTF-8?Q?Ahmed=EF=BF=BC?= , Brain Geffon , Minchan Kim , Michal Hocko , Mel Gorman , Nhat Pham , Johannes Weiner , Kairui Song , Zhongkun He , Kemeng Shi , Barry Song Date: Mon, 05 Feb 2024 10:15:04 -0800 In-Reply-To: References: <20240131-async-free-v2-1-525f03e07184@kernel.org> <87sf2ceoks.fsf@yhuang6-desk2.ccr.corp.intel.com> <7f19b4d69ff20efe8260a174c7866b4819532b1f.camel@linux.intel.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.44.4 (3.44.4-2.fc36) MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D309A1C001E X-Stat-Signature: 9o7r5kq6n8t4618dmney6hk5castyh9r X-Rspam-User: X-HE-Tag: 1707156906-210297 X-HE-Meta: U2FsdGVkX1/uZrKFIQ4DW1cuoDkIgFVy6LXGEbD/3qfZ3mRw/CkMV14pm12j3EMK+IHbR9zk1KTdbHzzh6HdXP7+aGfxt8CbvwBX+pLZIWAjbR08wB/5FRDD4GG1Zu8zHgZ/FDFW57WvR6Tze4jxt0lZvGNRPfrd8jxwhzqpW6K792mxnI0XCtQiLsDFdjIRDtIIzJWQQK54YU5wiIweL7Fbdo7rORl6kqoaa1+Tsn9xIGzyB/1IGXxv+Uxy0JyTF5gJdF/wRZ2jjTBxsDAglsOR1qJrBRIu6kNq7oD0XoCz0wULubN7Y0RQ1f1gs/M/f91N+EEtR3cbKU06TnjlnnQabIz4D6L4RSz9yVRjFBCcYmxQdfE1ur1fz/zwH8hL/awV2achpAAlJL1DNceqvfUxY4/MOq4gpstrK9ZmUSemH7PSXKk/whaN7zIWxn3oIhBscLNihaY7ELtwNhPCTu4yUngHHOMa9lnxIbW0OCOosk5DdL44oWKw+KF0YdmnfeFSz2LGnkF7ieHnetbICUAykMw/2sTMWgiVjERKzbhuor/iLb19KUZaIDwX1F9Ok3VODfGyl6GwPdI2cJwcaiJ3Xl8NXdamo7k2TUDfj2+TBGAwsX2GA61E2BdIfS6dQ7FYn9pG5vh4Ba1TTibsIFPJ4/L08Q5T3pa38HLvWt+uC8SS3bJpakm9zbhgNgyEmEsGwlgcfofXkmCheLrD9yYOau/1X7ql+iyVbBSlGGmXvKV2BPO1pMZbe86UyuZ0w8YoyTEcR3UgU7AtOLuB38akxeGz/Ft+4lvOf3Txf15J7xMV7t/Vo66p35kq+fZ9yK+L/taGUM94zfY1a54cR8bh4RYSt2ed0t6LCCUyZEjpr4OMlGXjITRmYA4CLjIGLVNnnbQ77SBUwGy2vx6dJfetBhVuqH1zdPIjM8Y+S0Pcc5+9k5Y48rnSRY+AV3jfmfAFd6vh4yXe5J6OChH A7UU5FJ1 /yxYcodfiJyYehLlwaJLk0A4nIb452RZDWGyX/odFJCwPD7RLiONeEtl5SxrBtjyAWFmuNRi42BRamdWgjuPDa4bXTXMxjoaFfn9Uwt6WIZ4vOu8s6ZOFuA1/nTYtA5z78ReS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, 2024-02-03 at 10:12 -0800, Chris Li wrote: >=20 > > > > { > > > > struct swap_slots_cache *cache; > > > > @@ -282,17 +298,14 @@ void free_swap_slot(swp_entry_t entry) > > > > goto direct_free; > > > > } > > > > if (cache->n_ret >=3D SWAP_SLOTS_CACHE_SIZE) { > > > > - /* > > > > - * Return slots to global pool. > > > > - * The current swap_map value is SWAP_HAS_CACHE= . > > > > - * Set it to 0 to indicate it is available for > > > > - * allocation in global pool > > > > - */ > > > > - swapcache_free_entries(cache->slots_ret, cache-= >n_ret); > > > > - cache->n_ret =3D 0; > > > > + spin_unlock_irq(&cache->free_lock); > > > > + schedule_work(&cache->async_free); > > > > + goto direct_free; > > > > } > > > > cache->slots_ret[cache->n_ret++] =3D entry; > > > > spin_unlock_irq(&cache->free_lock); > > > > + if (cache->n_ret >=3D SWAP_SLOTS_CACHE_SIZE) > > > > + schedule_work(&cache->async_free); > >=20 > >=20 > > I have some concerns about the current patch with the change above. > > We could hit the direct_free path very often. > >=20 > > By delaying the freeing of entries in the return > > cache, we have to do more freeing of swap entry one at a time. When > > we try to free an entry, we can find the return cache still full, waiti= ng to be freed. >=20 > You are describing the async free is not working. In that case it will al= ways > hit the direct free path one by one. >=20 > >=20 > > So we have fewer batch free of swap entries, resulting in an increase i= n > > number of sis->lock acquisitions overall. This could have the > > effect of reducing swap throughput overall when swap is under heavy > > operations and sis->lock is contended. >=20 > I can change the direct free path to free all entries. If the async > free hasn't freed up the batch by the time the next swap fault comes in. > The new swap fault will take the hit, just free the whole batch. It will = behave > closer to the original batch free behavior in this path. >=20 Will that negate the benefit you are looking for? A hack is to double the SWAP_SLOTS_CACHE_SIZE to 128, and trigger the background reclaim when entries reach 64. This will allow you to avoid the one by one relcaim direct path and hopefully the delayed job would have done its job when slots accumulate between 64 and 128. However, I am unsure how well this hack is under really heavy swap load. It means that the background reclaim will need to=C2=A0 work through a larger backlog and hold the sis->lock longer. So if you hit the direct path while the background reclaim is underway, you may have longer tail latency to acquire the sis->lock.=20 Tim