From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09018E77188 for ; Mon, 30 Dec 2024 17:47:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 811F86B0099; Mon, 30 Dec 2024 12:47:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C23F6B009A; Mon, 30 Dec 2024 12:47:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63B7D6B009B; Mon, 30 Dec 2024 12:47:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 408D16B0099 for ; Mon, 30 Dec 2024 12:47:01 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ACBBC141CA1 for ; Mon, 30 Dec 2024 17:47:00 +0000 (UTC) X-FDA: 82952355696.28.60B7BCC Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf06.hostedemail.com (Postfix) with ESMTP id CC0A0180005 for ; Mon, 30 Dec 2024 17:46:25 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QNeSMpXy; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735580797; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Doyl8vlzndLuJUSibGhSy11BH/2aXPWDa6ojLDfUXzc=; b=m1Ros6TTZfdL+8oFeDfPkgM1HxzbXgMJjtg0OFMSrpc20FLECn8KjFYnhr+e+TVIEEnKU7 hBKj2wCGW6gWHf5zWxeV6aZV1KIBC1qGsoFgthaQEy3bWyS7F+VE2FVDejlYlzWfCPasOu 2teClSwBTecIseYGs/Aq/6XMoHorzS8= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=QNeSMpXy; spf=pass (imf06.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.44 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735580797; a=rsa-sha256; cv=none; b=ZrgOKwreptgez7RReferAog214ArndX4CdA5iQnL66ywxHNymQv0PBFnao5MCE/SfUeQ19 5K3LDXODW+8SVcRJKsbHTG8Dzma/RoN8Hi3T2/apZik1kTL1aY3fADTAov3ny80V882MYG wQnTrDl1cpmElOCLX2mJd2hPGu+4Cd0= Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2ee9a780de4so9130943a91.3 for ; Mon, 30 Dec 2024 09:46:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735580817; x=1736185617; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=Doyl8vlzndLuJUSibGhSy11BH/2aXPWDa6ojLDfUXzc=; b=QNeSMpXy9WsEfX9Xt65dEHGOQhuWYMRAZgAwF0RPL9TcrkWSp/FFxUqBCs5Xu62U3s ahpE5Axt2Rk5EKJ62ug5N3lI8C2LBa6EGh3AiMhu5r8ruTwGlshw8jEAQPij2zZXk9ms O0U2qbG35Pa6Z4jpDNH6ncqpxTYIdcd540epf+HYK3O45H6s9KR/tMg2rohaptB3hLMM IQ9HEQcNTMQC517W3Gmg2yVz9B3EQnfPJDqkjAVzxNzIykSR1sgU3xLqp2wwYvip8tNT UGZcYyz7IM3ZUAoVWrntM9AiTjMmnAvxjV0ZsynNv9HclxFFFxb8qZaKPTCxjSfYStMI PL5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735580817; x=1736185617; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Doyl8vlzndLuJUSibGhSy11BH/2aXPWDa6ojLDfUXzc=; b=Iqg+u5jdIknDpJ7UOkKGad0UbRLWC5692b0Sed+eOmNE21EmTwDRFZ4oPeiQCv+/pm A3YD5m7Kby3uH7UqL692NxS2gw9UmCpDdhfyNIKPsZohIYjR5FbuR8e6aKujYfRPmA8w +KYjINUXw8Qoyr1aFpZenXoX6TE+8s5ofA+B3ASbb3LlmewRBGcFST4/YJerNdeifHI1 L5cBkTUDbSZ2C2Scn5iZTFQhjqEKX0LJYM/OBMUChk+nfJLZt9o3qQoxmiF+yJppzzYt r5K5hrm9+AV0PH3g1o6oV+wIdlcWJ9oxeVA9u2PJzz9+7Lb2sxcXvRFiFjdEaFSTFufc J3vA== X-Gm-Message-State: AOJu0Yx4HrHf2DKgkf8/phR2ITnidz8EhHqK/xVAmf6SmvvEX7E6i/oO Or6FI6LG8yiKVmWLtiXXAhDyb/zaZLFnHY97SralLfvo0ne07iTASuhjJyaSgm7QSA== X-Gm-Gg: ASbGncty0NymCuX1Tt8grb3ezyeBvgE3KPArl/1erVmXQg7ISokYt2zNDaZJrqqT75c Ztxg3D25IlFyqMZIEkphyhjeoYq67zgAt4w5UugtjhJKTcsn19GcuCtgd0Rzbz8VkRIkrDD+41d ctQeSPsGIVcyQXEXkJXk1/UjZc1Yjv3tBqPk4GeJ9UX2LYIsqnzeuxYb8OD5gsbGgZu7F6M4zlG r0VPGjdSanN6FdJwj9u7RsWS3td9rqYFHkoAcGjC2BhrYc8PHERycgtiGIz/ZcZ9RHnd6zJBjQF X-Google-Smtp-Source: AGHT+IEQCRDdPRMDf5+KaUHNnqYVgW+MsNaTPV4FGBEKTqRoTkDO1ySzMl4v5+j0v2zilPtNA/JWpA== X-Received: by 2002:a17:90b:51c4:b0:2ef:114d:7bf8 with SMTP id 98e67ed59e1d1-2f452dfcadfmr51154287a91.6.1735580816950; Mon, 30 Dec 2024 09:46:56 -0800 (PST) Received: from KASONG-MC4.tencent.com ([1.203.117.231]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc9cdf25sm180687695ad.118.2024.12.30.09.46.53 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 30 Dec 2024 09:46:56 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 06/13] mm, swap: clean up plist removal and adding Date: Tue, 31 Dec 2024 01:46:14 +0800 Message-ID: <20241230174621.61185-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241230174621.61185-1-ryncsn@gmail.com> References: <20241230174621.61185-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: CC0A0180005 X-Stat-Signature: 9n5br6hd7it5eyrr718mjgwzq5f78fmk X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1735580785-789170 X-HE-Meta: U2FsdGVkX1+qVk85/0JbJKNlAxT7+li/JCpRl1hqi7Xe0Dhp/UiXMDDq1qfIHHAVUzbcHyVYb15iNJSvN1K8/D4YQFXak4akCDUDqSfE1EsP6JqT0c0IwXQRqGnbKvbPOIjwVYainzi7/z2I8sFTiXxKnkytQzFUFSgPzorKzGhOOw4IxPjDm/J4wHcay3gUSTPvq2ef8L041kCxfg5IiEuSPpsv/OcjFYyjjDaMRrnI64+S9vgdy0fnOkTRmStH42c50c2Gji80OZ6iVOKJfXzb4Vrx5B7IZEB4WEtKjp1+3k+G0S6rmiE4ShEsbdBegbBj/S36gNGzzEggG0ZzQV9t+fRn5SBDhVccWoeujiyIelJDL0Z5WhWauXEcSmNTwhgKZvHtRD9Bh/F/OA1DfLYidjjweoYv00PYZzD7D8s6SJIA1HGwri/rzflfzSC5QY3FxC//OlYVhLn79gcdBcZxmEitU8oYhVzDAIt82//mjJzfCCiNtnMMhTaBsloDPyQwWYnEvjv08Lck/U11abZVg+SLhNv7e2YSDAkeTbDx/c/UMOZibQXjuYj244KErHOl2BUKx6baMHBPXPyJ8EeBGuPap467JKo9qsS4cLZIiGjMBEKbnmBN5FFLgcSeFX+fv4o8XKK0HGJokrLVbaN58PbPIa70gWxkpfi6WF6rhNez+XsT5RDwFCnPr3/oA/kTjlp6FXbYzKM1nqP89gAJ1n9+P35/9F3OEG3a7Ay0Rr1ozwnuTvlfWRhduY27iVd0rriKvhWuiIrcy4fE5l4jb1CD6eZDZryHj/x5hvcRc7nkMda4WoG1KbA3PALqx0kOe8mC0TMhC7aWFCwCfgz15J+3Z0DdG1Ny7bOAOWtJfByGhExHHPFtthU+2Xvs0Sj5P/R8BbKkrEEmLo2dDjPrVXinM4jap4jCvItNOjALyXzUikXtGstzrc6hJuT07La2BwWObIYmobRbaV9 YsMUebmt piNJG5xM59JuLUKxrcy+4gz2JtnoPNcFE43ZSuQeHpe1ZOliLOpRceHSJkYhnqNTVBYgK8M0knV1bgke5vHKsgf3HHqKM5a/KqmQz7de8vZ7f9IKLoiHkOQYuNLw+AukKybv5GQmXj4fOGdh6RzH4QZFFFQbQ9wY/CEVY2WJsYDyubqgV+7sGfiwriXvAqEL/6Yr48gNUvTR174xq3Jw4ILGi4XLw9Ll4c8KMbvPH2MSKvdVEjT/AQ9G+e4DUASpPoJQQdAZ659LhcnNhmTuwA/9HcpDj+Z6fGmgnvbxM8SN8Dz2NpJGxtqg/3b+fRXdxWBBLt4GnJhFwBm9wX9wMC51la6HoddkRWM1DPMPL6DznnLmP+i6T9VT7lsZqkpgumeMyxSG0q0nMVenMTLrOhWFRkCAm2eMmeQwom8Ozmtfhxk/p8wGMx+vOZiGTWGWuNBIUPe4gFMUD/GHUU3RH9fYcnBogcywV8Zl7FTS+s6GtZTogI7IGzAn49VVUZQYpP1FN4i5Ba+DCHEnE8owSpW4LXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When the swap device is full (inuse_pages == pages), it should be removed from the allocation available plist. If any slot is freed, the swap device should be added back to the plist. Additionally, during swapon or swapoff, the swap device is forcefully added or removed. Currently, the condition (inuse_pages == pages) is checked after every counter update, then remove or add the device accordingly. This is serialized by si->lock. This commit decouples it from the protection of si->lock and reworked plist removal and adding, making it possible to get rid of the hard dependency on si->lock in allocation path in later commits. To achieve this, simply using another lock is not an optimal approach, as the overhead is observable for a hot counter, and may cause complex locking issues. Thus, this commit manages to make it a lock-free atomic operation, by embedding the plist state into the second highest bit of the atomic counter. Simply making the counter an atomic will not work, if the update and plist status check are not performed atomically, we may miss an addition or removal. With the embedded info we can update the counter and check the plist status with single atomic operations, and avoid any extra overheads: If the counter is full (inuse_pages == pages) and the off-list bit is unset, we attempt to remove it from the plist. If the counter is not full (inuse_pages != pages) and the off-list bit is set, we attempt to add it to the plist. Removing, adding and bit update is serialized with a lock, which is a cold path. Ordinary counter updates will be lock-free. Signed-off-by: Kairui Song --- include/linux/swap.h | 2 +- mm/swapfile.c | 188 +++++++++++++++++++++++++++++++------------ 2 files changed, 139 insertions(+), 51 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0c222017b5c6..e1eeea6307cd 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -307,7 +307,7 @@ struct swap_info_struct { /* list of cluster that are fragmented or contented */ unsigned int frag_cluster_nr[SWAP_NR_ORDERS]; unsigned int pages; /* total of usable pages of swap */ - unsigned int inuse_pages; /* number of those currently in use */ + atomic_long_t inuse_pages; /* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev; /* swap device or bdev of swap file */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 7963a0c646a4..e6e58cfb5178 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -128,6 +128,26 @@ static inline unsigned char swap_count(unsigned char ent) return ent & ~SWAP_HAS_CACHE; /* may include COUNT_CONTINUED flag */ } +/* + * Use the second highest bit of inuse_pages counter as the indicator + * of if one swap device is on the available plist, so the atomic can + * still be updated arithmetic while having special data embedded. + * + * inuse_pages counter is the only thing indicating if a device should + * be on avail_lists or not (except swapon / swapoff). By embedding the + * on-list bit in the atomic counter, updates no longer need any lock + * to check the list status. + * + * This bit will be set if the device is not on the plist and not + * usable, will be cleared if the device is on the plist. + */ +#define SWAP_USAGE_OFFLIST_BIT (1UL << (BITS_PER_TYPE(atomic_t) - 2)) +#define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT) +static long swap_usage_in_pages(struct swap_info_struct *si) +{ + return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK; +} + /* Reclaim the swap entry anyway if possible */ #define TTRS_ANYWAY 0x1 /* @@ -717,7 +737,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) int nr_reclaim; if (force) - to_scan = si->inuse_pages / SWAPFILE_CLUSTER; + to_scan = swap_usage_in_pages(si) / SWAPFILE_CLUSTER; while (!list_empty(&si->full_clusters)) { ci = list_first_entry(&si->full_clusters, struct swap_cluster_info, list); @@ -872,42 +892,128 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o return found; } -static void __del_from_avail_list(struct swap_info_struct *si) +/* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */ +static void del_from_avail_list(struct swap_info_struct *si, bool swapoff) { int nid; + unsigned long pages; + + spin_lock(&swap_avail_lock); + + if (swapoff) { + /* + * Forcefully remove it. Clear the SWP_WRITEOK flags for + * swapoff here so it's synchronized by both si->lock and + * swap_avail_lock, to ensure the result can be seen by + * add_to_avail_list. + */ + lockdep_assert_held(&si->lock); + si->flags &= ~SWP_WRITEOK; + atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); + } else { + /* + * If not called by swapoff, take it off-list only if it's + * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly + * si->inuse_pages == pages), any concurrent slot freeing, + * or device already removed from plist by someone else + * will make this return false. + */ + pages = si->pages; + if (!atomic_long_try_cmpxchg(&si->inuse_pages, &pages, + pages | SWAP_USAGE_OFFLIST_BIT)) + goto skip; + } - assert_spin_locked(&si->lock); for_each_node(nid) plist_del(&si->avail_lists[nid], &swap_avail_heads[nid]); + +skip: + spin_unlock(&swap_avail_lock); } -static void del_from_avail_list(struct swap_info_struct *si) +/* SWAP_USAGE_OFFLIST_BIT can only be set by this helper. */ +static void add_to_avail_list(struct swap_info_struct *si, bool swapon) { + int nid; + long val; + unsigned long pages; + spin_lock(&swap_avail_lock); - __del_from_avail_list(si); + + /* Corresponding to SWP_WRITEOK clearing in del_from_avail_list */ + if (swapon) { + lockdep_assert_held(&si->lock); + si->flags |= SWP_WRITEOK; + } else { + if (!(READ_ONCE(si->flags) & SWP_WRITEOK)) + goto skip; + } + + if (!(atomic_long_read(&si->inuse_pages) & SWAP_USAGE_OFFLIST_BIT)) + goto skip; + + val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); + + /* + * When device is full and device is on the plist, only one updater will + * see (inuse_pages == si->pages) and will call del_from_avail_list. If + * that updater happen to be here, just skip adding. + */ + pages = si->pages; + if (val == pages) { + /* Just like the cmpxchg in del_from_avail_list */ + if (atomic_long_try_cmpxchg(&si->inuse_pages, &pages, + pages | SWAP_USAGE_OFFLIST_BIT)) + goto skip; + } + + for_each_node(nid) + plist_add(&si->avail_lists[nid], &swap_avail_heads[nid]); + +skip: spin_unlock(&swap_avail_lock); } -static void swap_range_alloc(struct swap_info_struct *si, - unsigned int nr_entries) +/* + * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock + * within each cluster, so the total contribution to the global counter should + * always be positive and cannot exceed the total number of usable slots. + */ +static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries) { - WRITE_ONCE(si->inuse_pages, si->inuse_pages + nr_entries); - if (si->inuse_pages == si->pages) { - del_from_avail_list(si); + long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages); - if (si->cluster_info && vm_swap_full()) - schedule_work(&si->reclaim_work); + /* + * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set, + * remove it from the plist. + */ + if (unlikely(val == si->pages)) { + del_from_avail_list(si, false); + return true; } + + return false; } -static void add_to_avail_list(struct swap_info_struct *si) +static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries) { - int nid; + long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages); - spin_lock(&swap_avail_lock); - for_each_node(nid) - plist_add(&si->avail_lists[nid], &swap_avail_heads[nid]); - spin_unlock(&swap_avail_lock); + /* + * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, + * remove it from the plist. + */ + if (unlikely(val & SWAP_USAGE_OFFLIST_BIT)) + add_to_avail_list(si, false); +} + +static void swap_range_alloc(struct swap_info_struct *si, + unsigned int nr_entries) +{ + if (swap_usage_add(si, nr_entries)) { + if (si->cluster_info && vm_swap_full()) + schedule_work(&si->reclaim_work); + } } static void swap_range_free(struct swap_info_struct *si, unsigned long offset, @@ -925,8 +1031,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, for (i = 0; i < nr_entries; i++) clear_bit(offset + i, si->zeromap); - if (si->inuse_pages == si->pages) - add_to_avail_list(si); if (si->flags & SWP_BLKDEV) swap_slot_free_notify = si->bdev->bd_disk->fops->swap_slot_free_notify; @@ -946,7 +1050,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, */ smp_wmb(); atomic_long_add(nr_entries, &nr_swap_pages); - WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries); + swap_usage_sub(si, nr_entries); } static int cluster_alloc_swap(struct swap_info_struct *si, @@ -1036,19 +1140,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order) plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]); spin_unlock(&swap_avail_lock); spin_lock(&si->lock); - if ((si->inuse_pages == si->pages) || !(si->flags & SWP_WRITEOK)) { - spin_lock(&swap_avail_lock); - if (plist_node_empty(&si->avail_lists[node])) { - spin_unlock(&si->lock); - goto nextsi; - } - WARN(!(si->flags & SWP_WRITEOK), - "swap_info %d in list but !SWP_WRITEOK\n", - si->type); - __del_from_avail_list(si); - spin_unlock(&si->lock); - goto nextsi; - } n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE, n_goal, swp_entries, order); spin_unlock(&si->lock); @@ -1057,7 +1148,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order) cond_resched(); spin_lock(&swap_avail_lock); -nextsi: /* * if we got here, it's likely that si was almost full before, * and since scan_swap_map_slots() can drop the si->lock, @@ -1789,7 +1879,7 @@ unsigned int count_swap_pages(int type, int free) if (sis->flags & SWP_WRITEOK) { n = sis->pages; if (free) - n -= sis->inuse_pages; + n -= swap_usage_in_pages(sis); } spin_unlock(&sis->lock); } @@ -2124,7 +2214,7 @@ static int try_to_unuse(unsigned int type) swp_entry_t entry; unsigned int i; - if (!READ_ONCE(si->inuse_pages)) + if (!swap_usage_in_pages(si)) goto success; retry: @@ -2137,7 +2227,7 @@ static int try_to_unuse(unsigned int type) spin_lock(&mmlist_lock); p = &init_mm.mmlist; - while (READ_ONCE(si->inuse_pages) && + while (swap_usage_in_pages(si) && !signal_pending(current) && (p = p->next) != &init_mm.mmlist) { @@ -2165,7 +2255,7 @@ static int try_to_unuse(unsigned int type) mmput(prev_mm); i = 0; - while (READ_ONCE(si->inuse_pages) && + while (swap_usage_in_pages(si) && !signal_pending(current) && (i = find_next_to_unuse(si, i)) != 0) { @@ -2200,7 +2290,7 @@ static int try_to_unuse(unsigned int type) * folio_alloc_swap(), temporarily hiding that swap. It's easy * and robust (though cpu-intensive) just to keep retrying. */ - if (READ_ONCE(si->inuse_pages)) { + if (swap_usage_in_pages(si)) { if (!signal_pending(current)) goto retry; return -EINTR; @@ -2209,7 +2299,7 @@ static int try_to_unuse(unsigned int type) success: /* * Make sure that further cleanups after try_to_unuse() returns happen - * after swap_range_free() reduces si->inuse_pages to 0. + * after swap_range_free() reduces inuse_pages to 0. */ smp_mb(); return 0; @@ -2227,7 +2317,7 @@ static void drain_mmlist(void) unsigned int type; for (type = 0; type < nr_swapfiles; type++) - if (swap_info[type]->inuse_pages) + if (swap_usage_in_pages(swap_info[type])) return; spin_lock(&mmlist_lock); list_for_each_safe(p, next, &init_mm.mmlist) @@ -2406,7 +2496,6 @@ static void setup_swap_info(struct swap_info_struct *si, int prio, static void _enable_swap_info(struct swap_info_struct *si) { - si->flags |= SWP_WRITEOK; atomic_long_add(si->pages, &nr_swap_pages); total_swap_pages += si->pages; @@ -2423,9 +2512,8 @@ static void _enable_swap_info(struct swap_info_struct *si) */ plist_add(&si->list, &swap_active_head); - /* add to available list if swap device is not full */ - if (si->inuse_pages < si->pages) - add_to_avail_list(si); + /* Add back to available list */ + add_to_avail_list(si, true); } static void enable_swap_info(struct swap_info_struct *si, int prio, @@ -2523,7 +2611,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) goto out_dput; } spin_lock(&p->lock); - del_from_avail_list(p); + del_from_avail_list(p, true); if (p->prio < 0) { struct swap_info_struct *si = p; int nid; @@ -2541,7 +2629,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) plist_del(&p->list, &swap_active_head); atomic_long_sub(p->pages, &nr_swap_pages); total_swap_pages -= p->pages; - p->flags &= ~SWP_WRITEOK; spin_unlock(&p->lock); spin_unlock(&swap_lock); @@ -2721,7 +2808,7 @@ static int swap_show(struct seq_file *swap, void *v) } bytes = K(si->pages); - inuse = K(READ_ONCE(si->inuse_pages)); + inuse = K(swap_usage_in_pages(si)); file = si->swap_file; len = seq_file_path(swap, file, " \t\n\\"); @@ -2838,6 +2925,7 @@ static struct swap_info_struct *alloc_swap_info(void) } spin_lock_init(&p->lock); spin_lock_init(&p->cont_lock); + atomic_long_set(&p->inuse_pages, SWAP_USAGE_OFFLIST_BIT); init_completion(&p->comp); return p; @@ -3335,7 +3423,7 @@ void si_swapinfo(struct sysinfo *val) struct swap_info_struct *si = swap_info[type]; if ((si->flags & SWP_USED) && !(si->flags & SWP_WRITEOK)) - nr_to_be_unused += READ_ONCE(si->inuse_pages); + nr_to_be_unused += swap_usage_in_pages(si); } val->freeswap = atomic_long_read(&nr_swap_pages) + nr_to_be_unused; val->totalswap = total_swap_pages + nr_to_be_unused; -- 2.47.1