From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9AA1C3ABD8 for ; Wed, 14 May 2025 20:19:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B41F6B0098; Wed, 14 May 2025 16:19:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0379A6B00BB; Wed, 14 May 2025 16:19:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF5946B00BC; Wed, 14 May 2025 16:19:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BB7056B00BA for ; Wed, 14 May 2025 16:19:46 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C498916110C for ; Wed, 14 May 2025 20:19:46 +0000 (UTC) X-FDA: 83442629172.16.7BC7CF4 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf14.hostedemail.com (Postfix) with ESMTP id EA889100009 for ; Wed, 14 May 2025 20:19:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O6DuVofA; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747253985; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6yvNaKoiyhdGzQTNSpSTtFq3TIsydM+hZJJMZKRox3o=; b=MiMHojoI/OodrkTpKmivGiN7xiQftIeW6T4Ii+5mgTIzNMCJWiCRcObh2quBpCiALiegmO mryBBzciZX41JQEGMBaE5tCio83x2IMQC9wzQF+bggosLw380fmDe47/xYDUW42CpCs1+m PJOTcG155IFWKaP80FXnJyw9OxNotjk= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O6DuVofA; spf=pass (imf14.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747253985; a=rsa-sha256; cv=none; b=avC8Y3DnbGD6KM6Vo2VHN5HbKSLrLxYALoGmYmYAGuHGxLqrdtYz8nBtos7YYPv1oJniBy l7wz3chke/CbEyAgXaDvUx0ZbMPnqFkriQyysCyCXm70qk6KL3JDB1bn+nMgj78vNlnF/8 DSAlfP3o7s4skSywv1msvbuGMLHNMKk= Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-30a9cd61159so252896a91.1 for ; Wed, 14 May 2025 13:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747253983; x=1747858783; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=6yvNaKoiyhdGzQTNSpSTtFq3TIsydM+hZJJMZKRox3o=; b=O6DuVofALqAnCcEbvvYkp8iWORLu4TFKLMCQUDHzXdYLxVYOpsPsr+6UEAbybI3/+z pcCJhK1Jp9s77dqQHtBM0KyfAqmUfgzJm5ok8rFEqloyuybeI3e+aaTN8BffYZ797dmm vOogJpbKuNE7VikoHgD44s5A/dbVnZAsUEBrAOiWkvBS1Df7LBTKxKRJGhJBg517OJtb e697r5FvI0m2mH5c8WzAfqYMIntT218BlhTyv7lbsf3GCgWApwHOTyDWpVEuFBA37J5w jG0TwIKAFMyri9WZNiQjL/ATjiAUYMhxWCcLKHdWq0zJHcyafVRGPOICDuFzQTHDiJGP FOeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747253983; x=1747858783; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=6yvNaKoiyhdGzQTNSpSTtFq3TIsydM+hZJJMZKRox3o=; b=AJoW4bypaL97kRYoFxovREApI60ZpaOx486wT2cTXp64nZHQoE526IqO+xUSwxblsW QLastv9BjzfytM0NULZ7GVhkdRhV/ND3wTaSrhms3F/tXprwdrORZ0SUwrxVPkDfs10o kHAeXPD9917YyQBOMIIyylgPxkZ80oLzvkHXQuWghZm2fZfIM45KlF0JZMDQgafVXlHq sDVzoiBkPqB8i4rBMtKlqErARtJeJmz/jRL2RyEq2r738XHk3dC2hj2YB3gkylyigzvW GkHiqlEzVGX1q4dvuO8R/p/AGiW7DU0zjDqikpLIISFntIusP98912RVzsvNW3vWPItU b8dg== X-Gm-Message-State: AOJu0YzhPSHZzPoR3OySz6prpyC/Gp5Skecj4uRms1H+o5Pl191wng97 dVV7G5U8FfMNieAXYbqekT9nT0bWrVnRF7tImsKJhGo9j2pNkO4ZD7tA/BaLVt3/9M8= X-Gm-Gg: ASbGncs7bUsSs1R2n5uhyIueflrGPxFptCy1nZlViMTeiHNPD1pp9mjApSCNcnVNZY8 mwM5QMfYLMsN74BwleXcEgrRlIpQr5QbhFqH5Y5dpF3PiJAuF4/z5SCzBikrVik0mktwkl/QNJI Efvh0XiCf052jnpM8Z6t2sR40+vLPjhxKJRENuUoUWI8Oj1qrbt1DXDki9msMLyaPyLwkWtm4sH RzCPxw1Ww2QiM0V+bXG/EW+jQqzOSMdEFPMAk0oVlKdI2FTkze9EzjMzJ+tQ1td7PyfrCdk1zzT 1z99dn8hF1gpz81CeGvcrz9I5RvRoDJkR8tR35U79lt5uUqtG5Fff7JQI6Y7q2OYoZGFv17y X-Google-Smtp-Source: AGHT+IGzwbynZkVxEox6mUaD39pJPSF0MdkK7lxUBurI2jti1PYOMDflNME+e3xh2xNygeW7R+QKAg== X-Received: by 2002:a17:90b:3d10:b0:30a:882f:3120 with SMTP id 98e67ed59e1d1-30e4db0fc4emr1291920a91.5.1747253983159; Wed, 14 May 2025 13:19:43 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([101.32.222.185]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-30e33401934sm2003692a91.9.2025.05.14.13.19.38 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 14 May 2025 13:19:42 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Matthew Wilcox , Hugh Dickins , Chris Li , David Hildenbrand , Yosry Ahmed , "Huang, Ying" , Nhat Pham , Johannes Weiner , Baolin Wang , Baoquan He , Barry Song , Kalesh Singh , Kemeng Shi , Tim Chen , Ryan Roberts , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 24/28] mm, swap: implement helpers for reserving data in swap table Date: Thu, 15 May 2025 04:17:24 +0800 Message-ID: <20250514201729.48420-25-ryncsn@gmail.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250514201729.48420-1-ryncsn@gmail.com> References: <20250514201729.48420-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam10 X-Stat-Signature: xbsic6cgy777pp4h4e7ax6dte3u9im7s X-Rspamd-Queue-Id: EA889100009 X-Rspam-User: X-HE-Tag: 1747253984-311100 X-HE-Meta: U2FsdGVkX1/ouxsZqnioDzRHwQ4Da88VX3Qb9NRWSJjd9/Oz6wC2j3n10398auW9nNkE7AkgafkvHrS8f9m65vKxB8sv8M0J15BGiVXAE7N+CLx9T3i5NDA8dMpstq2hlpDLkMVkDQ5t8fuXKtecIXeoV/S5rRg7tK7pvAR9LSKvMrm7kywB1G5my6oFMRAoFHH4X/mBH4SVZDtMpnllfGFFlalGHU0O9zMfiBtaLxAwX25p69Q1rSZ91D0bviFXIxsLOQi1/W0UTTrMTIf9ofBYEZVvEeAP+Mg3ozTEqiQAd/kxWP+EQZAEo6lYIzxI3aorQqxpF9yczFBFdPfs02qXDwgzHNV+3R5l53MXrw1SicykZJeJQ/U2DGL/Rdam+8Us5s6ZPzXaIYQcs/TwlEOcGiv8OtUuK5WEW3NENv1px6SYSw5E70BSgCoiYr85RS5VKKbA9FzdIY75MsLBDCXfQNlKQUhoBXzQ5OOO1n55Eus3JYeLcuFg5NzXMuIwlZeivJIFyYJW9pngRekqdsndnsxN08yHJiaPJyf+Avbt1TDtEg4roaFKfXVuUR/Nzf5NS1pDhRLPluoyz9Oy60neoEpaohXt3e8Ry9ZvdAUqQhD6Y6q7vh+EMspSbYAVm+Pa4cCmKIFS3M+638QdDp+Mn1Q0gSiO0rObbQcs9qSYqie6LbkrO/6RkV7bTk82qON0SHaL6rbPVRAJby/uj+1Ir7cFYSt4f1IZwOT0U/tL6/Ixva6e2dtOWNuslmPsC7LGjQf3kjP+QZNMp2YIFjNaclBfz5F+HPY4mfK3q5BBBfJW1pjseX2CwAct4eDFKyffgb+qDzEMWJVKK/W4xiocLrSToJ1Ztv4w2zRGyOSew9MYGB4ohUQQaiTu288gwyCwzwgRdq3VEKXWqu/bOf0EgbXVn1WAFHjAAZ5T/4uGqZ1Tnwc+o73LW27n+yBkFBjXW7C8PtsiMNoDqF4 XyzdSrQN YQbYgRvkxl6SnaYchMfn2Fe/oYOmZxlvnHIz4UFAAFb3XTpp28UnDnRVLdDtGlUSN97okdmVtOZAXA2Zrn1CwdLA9Y/LlNIY8eD0DrjicnHueajxm0WKMvkr84Rza4BJieCEtOVNytAugwp1x5nhg2TbIGpB0jjpNpGRNwpoxcnPCFC+78RGHHO8RRHc86wlJb5Sd5XC5eQ77Px/cuGq1HBOhUfIeVJYVk4G37xa9tbsP+RClb/fifXHwFhDIBlU1gTg2d5K4P0qCVWk0ibUGAUtqsiOuPSJpJsySYcKzGUla0Tz+UvszBR+vWEpQ38tsYLG+8Ky9Jl69TBWXoH2g0apk4QBzDXGicSb83AgYyhyXrsY115YZXvxr+ldzuxrw7Gf2WAFSBRZlSNIFfq378hFjteboFanZ+l4qTDXh0eKbRYOVuhSObGvEasYXodopp3+5QOWxAu/7+KiLZQXWp+ij8qKG2mJgRAZShWv2Ko1PEh7V/HXivS1n5skFPo4BUvhV23qIMO+VpqCreEDQlHIfgY8IeuWhzvgZCYj6fYT19e58LVwMp3YUJQQ5nJjb7sWK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song To prepare for using swap table as the unified swap layer, introduce macro and helpers for storing multiple kind of data in an swap table entry. Signed-off-by: Kairui Song --- mm/swap_table.h | 130 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 119 insertions(+), 11 deletions(-) diff --git a/mm/swap_table.h b/mm/swap_table.h index 69a074339444..9356004d211a 100644 --- a/mm/swap_table.h +++ b/mm/swap_table.h @@ -5,9 +5,41 @@ #include "swap.h" /* - * Swap table entry could be a pointer (folio), a XA_VALUE (shadow), or NULL. + * Swap table entry type and bit layouts: + * + * NULL: | ------------ 0 -------------| + * Shadow: | SWAP_COUNT |---- SHADOW_VAL ---|1| + * Folio: | SWAP_COUNT |------ PFN -------|10| + * Pointer: |----------- Pointer ----------|100| + * + * Usage: + * - NULL: Swap Entry is unused. + * + * - Shadow: Swap Entry is used and not cached (swapped out). + * It's reusing XA_VALUE format to be compatible with workingset + * shadows. SHADOW_VAL part could be all 0. + * + * - Folio: Swap Entry is in cache. + * + * - Pointer: Unused yet. Because only the last three bit of a pointer + * is usable so now `100` is reserved for potential pointer use. */ +#define ENTRY_COUNT_BITS BITS_PER_BYTE +#define ENTRY_SHADOW_MARK 0b1UL +#define ENTRY_PFN_MARK 0b10UL +#define ENTRY_PFN_LOW_MASK 0b11UL +#define ENTRY_PFN_SHIFT 2 +#define ENTRY_PFN_MASK ((~0UL) >> ENTRY_COUNT_BITS) +#define ENTRY_COUNT_MASK (~((~0UL) >> ENTRY_COUNT_BITS)) +#define ENTRY_COUNT_SHIFT (BITS_PER_LONG - BITS_PER_BYTE) +#define ENTRY_COUNT_MAX ((1 << ENTRY_COUNT_BITS) - 2) +#define ENTRY_COUNT_BAD ((1 << ENTRY_COUNT_BITS) - 1) /* ENTRY_BAD */ +#define ENTRY_BAD (~0UL) + +/* For shadow offset calculation */ +#define SWAP_COUNT_SHIFT ENTRY_COUNT_BITS + /* * Helpers for casting one type of info into a swap table entry. */ @@ -19,17 +51,27 @@ static inline swp_te_t null_swp_te(void) static inline swp_te_t folio_swp_te(struct folio *folio) { - BUILD_BUG_ON(sizeof(swp_te_t) != sizeof(void *)); - swp_te_t swp_te = { .counter = (unsigned long)folio }; + BUILD_BUG_ON((MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT) > + (BITS_PER_LONG - ENTRY_PFN_SHIFT - ENTRY_COUNT_BITS)); + swp_te_t swp_te = { + .counter = (folio_pfn(folio) << ENTRY_PFN_SHIFT) | ENTRY_PFN_MARK + }; return swp_te; } static inline swp_te_t shadow_swp_te(void *shadow) { - BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) != - BITS_PER_BYTE * sizeof(swp_te_t)); - VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow)); swp_te_t swp_te = { .counter = ((unsigned long)shadow) }; + BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) != BITS_PER_BYTE * sizeof(swp_te_t)); + BUILD_BUG_ON((unsigned long)xa_mk_value(0) != ENTRY_SHADOW_MARK); + VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow)); + swp_te.counter |= ENTRY_SHADOW_MARK; + return swp_te; +} + +static inline swp_te_t bad_swp_te(void) +{ + swp_te_t swp_te = { .counter = ENTRY_BAD }; return swp_te; } @@ -43,7 +85,7 @@ static inline bool swp_te_is_null(swp_te_t swp_te) static inline bool swp_te_is_folio(swp_te_t swp_te) { - return !xa_is_value((void *)swp_te.counter) && !swp_te_is_null(swp_te); + return ((swp_te.counter & ENTRY_PFN_LOW_MASK) == ENTRY_PFN_MARK); } static inline bool swp_te_is_shadow(swp_te_t swp_te) @@ -51,19 +93,63 @@ static inline bool swp_te_is_shadow(swp_te_t swp_te) return xa_is_value((void *)swp_te.counter); } +static inline bool swp_te_is_valid_shadow(swp_te_t swp_te) +{ + /* The shadow could be empty, just for holding the swap count */ + return xa_is_value((void *)swp_te.counter) && + xa_to_value((void *)swp_te.counter); +} + +static inline bool swp_te_is_bad(swp_te_t swp_te) +{ + return swp_te.counter == ENTRY_BAD; +} + +static inline bool __swp_te_is_countable(swp_te_t ent) +{ + return (swp_te_is_shadow(ent) || swp_te_is_folio(ent) || + swp_te_is_null(ent)); +} + /* * Helpers for retrieving info from swap table. */ static inline struct folio *swp_te_folio(swp_te_t swp_te) { VM_WARN_ON(!swp_te_is_folio(swp_te)); - return (void *)swp_te.counter; + return pfn_folio((swp_te.counter & ENTRY_PFN_MASK) >> ENTRY_PFN_SHIFT); } static inline void *swp_te_shadow(swp_te_t swp_te) { VM_WARN_ON(!swp_te_is_shadow(swp_te)); - return (void *)swp_te.counter; + return (void *)(swp_te.counter & ~ENTRY_COUNT_MASK); +} + +static inline unsigned char swp_te_get_count(swp_te_t swp_te) +{ + VM_WARN_ON(!__swp_te_is_countable(swp_te)); + return ((swp_te.counter & ENTRY_COUNT_MASK) >> ENTRY_COUNT_SHIFT); +} + +static inline unsigned char swp_te_try_get_count(swp_te_t swp_te) +{ + if (__swp_te_is_countable(swp_te)) + return swp_te_get_count(swp_te); + return 0; +} + +static inline swp_te_t swp_te_set_count(swp_te_t swp_te, + unsigned char count) +{ + VM_BUG_ON(!__swp_te_is_countable(swp_te)); + VM_BUG_ON(count > ENTRY_COUNT_MAX); + + swp_te.counter &= ~ENTRY_COUNT_MASK; + swp_te.counter |= ((unsigned long)count) << ENTRY_COUNT_SHIFT; + VM_BUG_ON(swp_te_get_count(swp_te) != count); + + return swp_te; } /* @@ -87,17 +173,39 @@ static inline swp_te_t __swap_table_get(struct swap_cluster_info *ci, pgoff_t of static inline void __swap_table_set_folio(struct swap_cluster_info *ci, pgoff_t off, struct folio *folio) { - __swap_table_set(ci, off, folio_swp_te(folio)); + swp_te_t swp_te; + unsigned char count; + + swp_te = __swap_table_get(ci, off); + count = swp_te_get_count(swp_te); + swp_te = swp_te_set_count(folio_swp_te(folio), count); + + __swap_table_set(ci, off, swp_te); } static inline void __swap_table_set_shadow(struct swap_cluster_info *ci, pgoff_t off, void *shadow) { - __swap_table_set(ci, off, shadow_swp_te(shadow)); + swp_te_t swp_te; + unsigned char count; + + swp_te = __swap_table_get(ci, off); + count = swp_te_get_count(swp_te); + swp_te = swp_te_set_count(shadow_swp_te(shadow), count); + + __swap_table_set(ci, off, swp_te); } static inline void __swap_table_set_null(struct swap_cluster_info *ci, pgoff_t off) { __swap_table_set(ci, off, null_swp_te()); } + +static inline void __swap_table_set_count(struct swap_cluster_info *ci, pgoff_t off, + unsigned char count) +{ + swp_te_t swp_te; + swp_te = swp_te_set_count(__swap_table_get(ci, off), count); + __swap_table_set(ci, off, swp_te); +} #endif -- 2.49.0