From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBA56E95A67 for ; Sun, 8 Oct 2023 09:59:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07FBC6B028C; Sun, 8 Oct 2023 05:59:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 02F4C6B028D; Sun, 8 Oct 2023 05:59:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DECB86B028E; Sun, 8 Oct 2023 05:59:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CCF5D6B028C for ; Sun, 8 Oct 2023 05:59:51 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4DE4D1CA366 for ; Sun, 8 Oct 2023 09:59:51 +0000 (UTC) X-FDA: 81321847782.08.577FC04 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf06.hostedemail.com (Postfix) with ESMTP id 5D696180004 for ; Sun, 8 Oct 2023 09:59:49 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=transsion-corp-partner-google-com.20230601.gappssmtp.com header.s=20230601 header.b=kcankOLU; spf=pass (imf06.hostedemail.com: domain of lincheng.yang@transsion.corp-partner.google.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=lincheng.yang@transsion.corp-partner.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696759189; a=rsa-sha256; cv=none; b=WRkXoBEYRdynbKP66RoJUsA+LlJ9blFOIOJ64RKMDHnJH8+Ex2BzFU8Iw+hACMPc3z+xYl GNmKSS8DyjvM6+JZP4JNzSx1vDioDwhan79NO/QhgZIA2DAygYyW9o0x6gyrmjyWrMTHeY ea4KCgYplpdjpqLOLs75WBUbQ3L/KJk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=transsion-corp-partner-google-com.20230601.gappssmtp.com header.s=20230601 header.b=kcankOLU; spf=pass (imf06.hostedemail.com: domain of lincheng.yang@transsion.corp-partner.google.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=lincheng.yang@transsion.corp-partner.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696759189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SChUNcWFLO06mB7MjDh24sWW8FnMNVexunJGIAKn/hI=; b=KAx60+wh1mkWA3JcbOG0hCItf6b4d2WcYSBH8DJgfmPKK1X1AUYJGWmolT4fCaiTYExoxR qzMH+JQZBamO6mMkXLRS5fp4FUccTpFUp83zPvk725G4XDWPm6U4VXzx5NHXPZNKq6UAc/ ohDaSw0DY2U7Zi9n9Mf1A6pdmb67+hg= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1c724577e1fso29195035ad.0 for ; Sun, 08 Oct 2023 02:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=transsion-corp-partner-google-com.20230601.gappssmtp.com; s=20230601; t=1696759188; x=1697363988; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SChUNcWFLO06mB7MjDh24sWW8FnMNVexunJGIAKn/hI=; b=kcankOLUFwg++ZCWWvz+hcNpF8NNQIadz9nNovRIBDTmQisA0K2wQ4kpd6GdVS6wty sL4EFVoCfS2GSnbnEIPFxCGb8G9FRH1oyvz+CwyiW6/03kd/lYI53LhicudFjLB81LVI SKPTBASMdn03VOkbYyyrzk+yueSV0nHc87zssWQUhhp7I8wR5aiIiqD7qMq7WgmSa25u qqQnh1ltwEi7nqsTr8x6uHo02o6KtdsJavrcuSUVsnDQULkS4ZaU6IcQZ63MgYLr/aeu +Mrv9s81J+21ZikAU/Owt9ydwCgNb+vuef86ckSACM5B5svAxvCrEicqA7mG961a0yAO Ccog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696759188; x=1697363988; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SChUNcWFLO06mB7MjDh24sWW8FnMNVexunJGIAKn/hI=; b=RS0g1xHVweXFxGpNprySX1uimKyHLDPGKR9Xwu842b71PVZ1bmUttw1Lc+pD9PdjJe rlmtzkcFb7iIx2UGdsF9+HajvzWpfzS9UoTTQptT8mA0fXNIhGCpZc7+crwvWZVOubbR PhN5D4Mf37rvd9aWSBdZTcsXnZ7F9RZ6YZf2ZXvvuxvsnEnSg8386Qu5HuQuF+UgtooV lA9FiusL6pHZn42VANDwLoYUWVrJn4nEH5xM36eZXPj9S8bWUrb56ot27zpqaDzImtO8 FvmGOQTmL8X4WG/pgrB4jRWMwu6nMjdSSXkyMtoemiPZFjeZ2T2FB2629KJxpRGizbqv FXHg== X-Gm-Message-State: AOJu0YzCb+cADrdSaWTN/ZHAlWzkgt48G8c6984tsvnjSh6VjuSJEDuD bajG4VbBe7I8Ort3Uj5WgLqbSQ== X-Google-Smtp-Source: AGHT+IHWj5ieXF51u86WtnTGUmM97ximEOSq/JAnuFIVLNU+TG0E7YJgYgcAod7pxCcPBQbkiZ8ecw== X-Received: by 2002:a17:902:ec91:b0:1c8:9d32:3395 with SMTP id x17-20020a170902ec9100b001c89d323395mr3850640plg.41.1696759188247; Sun, 08 Oct 2023 02:59:48 -0700 (PDT) Received: from linchengyang-VB.. ([103.40.76.4]) by smtp.gmail.com with ESMTPSA id u9-20020a17090282c900b001c72c07c9d9sm7126343plz.308.2023.10.08.02.59.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 08 Oct 2023 02:59:48 -0700 (PDT) From: Lincheng Yang X-Google-Original-From: Lincheng Yang To: akpm@linux-foundation.org, rostedt@goodmis.org, mhiramat@kernel.org, willy@infradead.org, hughd@google.com, peterx@redhat.com, mike.kravetz@oracle.com, jgg@ziepe.ca, surenb@google.com, steven.price@arm.com, pasha.tatashin@soleen.com, kirill.shutemov@linux.intel.com, yuanchu@google.com, david@redhat.com, mathieu.desnoyers@efficios.com, dhowells@redhat.com, shakeelb@google.com, pcc@google.com, tytso@mit.edu, 42.hyeyoo@gmail.com, vbabka@suse.cz, catalin.marinas@arm.com, lrh2000@pku.edu.cn, ying.huang@intel.com, mhocko@suse.com, vishal.moola@gmail.com, yosryahmed@google.com, findns94@gmail.com, neilb@suse.de Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, wanbin.wang@transsion.com, chunlei.zhuang@transsion.com, jinsheng.zhao@transsion.com, jiajun.ling@transsion.com, dongyun.liu@transsion.com, Lincheng Yang Subject: [RFC PATCH 1/5] mm/swap_slots: cleanup swap slot cache Date: Sun, 8 Oct 2023 17:59:20 +0800 Message-Id: <20231008095924.1165106-2-lincheng.yang@transsion.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231008095924.1165106-1-lincheng.yang@transsion.com> References: <20231008095924.1165106-1-lincheng.yang@transsion.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5D696180004 X-Stat-Signature: w18yj9wpeza9aeo8a6nob5fhq8xea5ec X-Rspam-User: X-HE-Tag: 1696759189-742275 X-HE-Meta: U2FsdGVkX1/eypD/9YFjRUkBSQvwXphfmmJeRJCK+Y+AULI/CZHyBMJQa2UrYvvi+GPfu9rYlafMUTmsARv1Lz4UWM/qZr3mgmNHN9CGBmJyoCt2mtXX9gHrw5JbH1yWtj1GztD5Fj9bpf6Cd3ENrLTDE67+L/vv5zP4Y0N8aamF9xo3d+P3G0N3wcJ9aPS0sfWcJAvxOMsMBbRvelg1nMS+qau0YFEIvWotqnGGV7X8aAVzftFAm4zORGLgtAKXyjvvX4SB9m81lPfIKIaYKEmGsk6fSbjyXvb9m1YCQPrsD2tPlcFo2tDHqIv2K4aGRbNh7XEvgVBOCMhomULeJ8RghOh5J0IEjzXGanyUliwxvil8g99B4/xbO23yTOGNNt0bgv70vY+5cxrwbL5IBc6saLYr60pewp9dR1rh2Uc/zgh3eqvqc6lbDP9UfQwvJIZUh4XLfjzZNf8OeLuvgj9ThkoVWnbRLuOxqiniUOZ2cLxXZLpOrsMVpZAA4GNnG3AsUesSHY8ZjsK7/fdaWJaSyiT/6M/3lGQB8xW2rFaBbtoxZcyAZsCB7W+VzQujOp8EfMPaN5XRx0XFMwiqZdYfF5esHJy984VGLN3VNlx7THPM6A4j0wLQZ6hT/FcumCLPkdYFahAFD/Z1Is41irm6jt5PuMbSCzBsn9pBJhz5DKc6O3mvDpQfvgKWTY5+qUJ2wnV20U2gUowpiq6F7Mx/PT4QcPgceQNQjyBZq2yRJt1CtCOpcYsTvDyGPT9qby8fmhAN8mAqhSVFLbXofO6/xCUzjCqRBhvh2n15AJWc/6xZDdDiM2gaHP7G6aAwkWDn2P6eU2r7Z2OiZ/RmVT7tbWMe99XmYrTmUtCoMvXHUrbLRuaEUarnfyOBuRjX7UfJyQPHzxOcZrJXklph1VYcnXyx/c76i3mhMsC2F6DNuJwKV9VTAX99PK+MDZKRxm0rjrAYNWKf5kPWXVA OqtXtrSs AARfmZDtjmlrvHgs9k3kkGDh9xkd9rjeWgXEo6ExvNn9N7yUSbQ/OYWqLrkdjMjaDN+CJMfKmb9lGAlD/dvqMFc10v95S0lkm4p5m2A7CCLAY13cWKTnWRg0rUO66JHRne897MYsE4i0r2EVrr4GTCuwAs8+AfjfChlzdLL9BG/QS8GxAyA3bYaUFdbu3DE312MNJTUiscvCZoCx5P15H671EDJpYRkric+EtOEHZ1+ZqGMrWkcHP8+D2bVGhm7jJnfoK4amViz4SRfK/7LdopKxn0WgzPKpgu63JWosfDcpZCkrnWS31cJlW3YBxi6SqDIQcNa2SDzsplk3L3Qc2DNJUKXLgX4ylbk3IoXYB/LbAE2oCqQO2JNMWj5/G7dzGrui9dkjNm1EZrCzRs1JbUwa+HrWR7UZtynSTv0XmxkCNHfXW9XIA4S+Ndxp7XeVH3os0mCODkWJTx53wF3dtSTJn/GRHZF/tgZjkjYAPeyj7bzDc+5YALqMhNoH/8muT7do80XIlItsAl3HcJ4A/w5kzuwh0RRZz+6WqkVg5X98jUXPC4qJvy0P3ZIhTI/nutkY1WFVKXQdE+cTJD4fG4zT7UY6pJZlLrGom X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function of the swap slot cache will be cleaned to prepare for subsequent modifications. Signed-off-by: Lincheng Yang --- mm/swap_slots.c | 111 ++++++++++++++++++++++++++++-------------------- 1 file changed, 66 insertions(+), 45 deletions(-) diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 0bec1f705f8e..bb41c8460b62 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -110,11 +110,13 @@ static bool check_cache_active(void) return swap_slot_cache_active; } -static int alloc_swap_slot_cache(unsigned int cpu) +static int __alloc_swap_slot_cache(struct swap_slots_cache *cache) { - struct swap_slots_cache *cache; swp_entry_t *slots, *slots_ret; + if (!cache) + return 0; + /* * Do allocation outside swap_slots_cache_mutex * as kvzalloc could trigger reclaim and folio_alloc_swap, @@ -133,17 +135,6 @@ static int alloc_swap_slot_cache(unsigned int cpu) } mutex_lock(&swap_slots_cache_mutex); - cache = &per_cpu(swp_slots, cpu); - if (cache->slots || cache->slots_ret) { - /* cache already allocated */ - mutex_unlock(&swap_slots_cache_mutex); - - kvfree(slots); - kvfree(slots_ret); - - return 0; - } - if (!cache->lock_initialized) { mutex_init(&cache->alloc_lock); spin_lock_init(&cache->free_lock); @@ -165,13 +156,26 @@ static int alloc_swap_slot_cache(unsigned int cpu) return 0; } -static void drain_slots_cache_cpu(unsigned int cpu, unsigned int type, - bool free_slots) +static int alloc_swap_slot_cache(unsigned int cpu) { struct swap_slots_cache *cache; - swp_entry_t *slots = NULL; + mutex_lock(&swap_slots_cache_mutex); cache = &per_cpu(swp_slots, cpu); + if (cache->slots || cache->slots_ret) /* cache already allocated */ + cache = NULL; + mutex_unlock(&swap_slots_cache_mutex); + + __alloc_swap_slot_cache(cache); + + return 0; +} + +static void __drain_slots_cache_cpu(struct swap_slots_cache *cache, + unsigned int type, bool free_slots) +{ + swp_entry_t *slots = NULL; + if ((type & SLOTS_CACHE) && cache->slots) { mutex_lock(&cache->alloc_lock); swapcache_free_entries(cache->slots + cache->cur, cache->nr); @@ -196,6 +200,15 @@ static void drain_slots_cache_cpu(unsigned int cpu, unsigned int type, } } +static void drain_slots_cache_cpu(unsigned int cpu, unsigned int type, + bool free_slots) +{ + struct swap_slots_cache *cache; + + cache = &per_cpu(swp_slots, cpu); + __drain_slots_cache_cpu(cache, type, free_slots); +} + static void __drain_swap_slots_cache(unsigned int type) { unsigned int cpu; @@ -269,11 +282,8 @@ static int refill_swap_slots_cache(struct swap_slots_cache *cache) return cache->nr; } -void free_swap_slot(swp_entry_t entry) +static void __free_swap_slot(struct swap_slots_cache *cache, swp_entry_t entry) { - struct swap_slots_cache *cache; - - cache = raw_cpu_ptr(&swp_slots); if (likely(use_swap_slot_cache && cache->slots_ret)) { spin_lock_irq(&cache->free_lock); /* Swap slots cache may be deactivated before acquiring lock */ @@ -299,18 +309,18 @@ void free_swap_slot(swp_entry_t entry) } } -swp_entry_t folio_alloc_swap(struct folio *folio) +void free_swap_slot(swp_entry_t entry) { - swp_entry_t entry; struct swap_slots_cache *cache; - entry.val = 0; + cache = raw_cpu_ptr(&swp_slots); + __free_swap_slot(cache, entry); +} - if (folio_test_large(folio)) { - if (IS_ENABLED(CONFIG_THP_SWAP) && arch_thp_swp_supported()) - get_swap_pages(1, &entry, folio_nr_pages(folio)); - goto out; - } +static int __folio_alloc_swap(struct swap_slots_cache *cache, swp_entry_t *entry) +{ + if (unlikely(!check_cache_active() || !cache->slots)) + return -EINVAL; /* * Preemption is allowed here, because we may sleep @@ -321,26 +331,37 @@ swp_entry_t folio_alloc_swap(struct folio *folio) * The alloc path here does not touch cache->slots_ret * so cache->free_lock is not taken. */ - cache = raw_cpu_ptr(&swp_slots); - - if (likely(check_cache_active() && cache->slots)) { - mutex_lock(&cache->alloc_lock); - if (cache->slots) { + mutex_lock(&cache->alloc_lock); repeat: - if (cache->nr) { - entry = cache->slots[cache->cur]; - cache->slots[cache->cur++].val = 0; - cache->nr--; - } else if (refill_swap_slots_cache(cache)) { - goto repeat; - } - } - mutex_unlock(&cache->alloc_lock); - if (entry.val) - goto out; + if (cache->nr) { + *entry = cache->slots[cache->cur]; + cache->slots[cache->cur++].val = 0; + cache->nr--; + } else if (refill_swap_slots_cache(cache)) { + goto repeat; } + mutex_unlock(&cache->alloc_lock); + + return !!entry->val; +} + +swp_entry_t folio_alloc_swap(struct folio *folio) +{ + swp_entry_t entry; + struct swap_slots_cache *cache; + + entry.val = 0; + + if (folio_test_large(folio)) { + if (IS_ENABLED(CONFIG_THP_SWAP) && arch_thp_swp_supported()) + get_swap_pages(1, &entry, folio_nr_pages(folio)); + goto out; + } + + cache = raw_cpu_ptr(&swp_slots); + if (__folio_alloc_swap(cache, &entry)) + get_swap_pages(1, &entry, 1); - get_swap_pages(1, &entry, 1); out: if (mem_cgroup_try_charge_swap(folio, entry)) { put_swap_folio(folio, entry); -- 2.34.1