[PATCH v9 mm-unstable 00/18] Add zpdesc memory descriptor for zswap.zpool

[PATCH v9 mm-unstable 00/18] Add zpdesc memory descriptor for zswap.zpool

[Hyeonggon Yoo]

This patch series introduces a new memory descriptor for zswap.zpool that
currently overlaps with struct page for now. This is part of the effort
to reduce the size of struct page and to enable dynamic allocation of
memory descriptors [1].

This series does not bloat anything for zsmalloc and no functional
change is intended (except for using zpdesc and folios).

In the near future, the removal of page->index from struct page [2]
will be addressed and the project also depends on this patch series.

I think this series is now ready to be included in mm-unstable if there's
no objection. Sergey thankfully added Reviewed-by and Tested-by tags on v8.
But as I updated the patchset, could you please explicitly add them for v9
as well? A range-diff output is included at the end of this cover letter
to help review.

Thanks to everyone got involved in this series, especially, Alex who's been
pushing it forward this year.

v8: https://lore.kernel.org/linux-mm/20241205175000.3187069-1-willy@infradead.org
[1] https://lore.kernel.org/linux-mm/ZvRKzKizOfEWBtJp@casper.infradead.org
[2] https://lore.kernel.org/linux-mm/Z09hOy-UY9KC8WMb@casper.infradead.org

v8 -> v9:
Functionally very little change and most of them are comment/changelog
updates.

  - (patch 1) Added comments for basic zpdesc helper functions,
    some bits copied from struct slab.

  - (patch 4) Changed 'f_zpdesc->next = NULL' to 'f_zpdesc->handle = 0'
    as f_zpdesc here is for a huge zspage.

  - (patch 17) Fixed a mistake in a previous rebase from v6 to v7.

  - (page 11, 19) Changed reset_zpdesc() to use struct page for robustness
    against re-organizing zpdesc fields.

  - Dropped patch 20 in v8 as it does not make re-implementing
    zsdesc management easier in the glorious future. we can just
    re-implement the whole reset_zpdesc().

  - Dropped patch 21 in v8 and folded some comments of the patch into
    patch 2 that introduces zpdesc_{un,}lock().

The rest of the changes are changelog/comment cleanups.

Cheers,
Hyeonggon

Alex Shi (7):
  mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool
  mm/zsmalloc: use zpdesc in trylock_zspage()/lock_zspage()
  mm/zsmalloc: convert create_page_chain() and its users to use zpdesc
  mm/zsmalloc: convert reset_page to reset_zpdesc
  mm/zsmalloc: convert SetZsPageMovable and remove unused funcs
  mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc
  mm/zsmalloc: introduce __zpdesc_clear/set_zsmalloc()

Hyeonggon Yoo (11):
  mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc
  mm/zsmalloc: add and use pfn/zpdesc seeking funcs
  mm/zsmalloc: convert obj_malloc() to use zpdesc
  mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc
  mm/zsmalloc: convert init_zspage() to use zpdesc
  mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc
  mm/zsmalloc: add two helpers for zs_page_migrate() and make it use
    zpdesc
  mm/zsmalloc: convert __free_zspage() to use zpdesc
  mm/zsmalloc: convert location_to_obj() to take zpdesc
  mm/zsmalloc: convert migrate_zspage() to use zpdesc
  mm/zsmalloc: convert get_zspage() to take zpdesc

 mm/zpdesc.h   | 182 +++++++++++++++++++++
 mm/zsmalloc.c | 436 ++++++++++++++++++++++++++------------------------
 2 files changed, 408 insertions(+), 210 deletions(-)
 create mode 100644 mm/zpdesc.h

-- 
2.43.5

For ease of review, here I add range-diff output showing differences between
v8 and v9:

$ git range-diff zpdesc-v8...zpdesc-v9
 1:  3d74794250ab !  1:  9809a405a425 mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool
    @@ Metadata
      ## Commit message ##
         mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool
     
    -    The 1st patch introduces new memory descriptor zpdesc and rename
    -    zspage.first_page to zspage.first_zpdesc, no functional change.
    +    The 1st patch introduces new memory descriptor zpdesc and renames
    +    zspage.first_page to zspage.first_zpdesc, with no functional change.
     
    -    We removed PG_owner_priv_1 since it was moved to zspage after
    -    commit a41ec880aa7b ("zsmalloc: move huge compressed obj from
    -    page to zspage").
    +    We removed the comment about PG_owner_priv_1 since it is no longer used
    +    after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page
    +    to zspage").
     
    +    [42.hyeyoo: rework comments a little bit]
         Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zpdesc.h (new) ##
     @@
    @@ mm/zpdesc.h (new)
     +#define __MM_ZPDESC_H__
     +
     +/*
    -+ * struct zpdesc -	Memory descriptor for zpool memory
    -+ * @flags:		Page flags, mostly unused.
    -+ * @lru:		Indirectly used by page migration
    -+ * @movable_ops:	Used by page migration
    -+ * @next:		Next zpdesc in a zspage in zsmalloc zpool
    -+ * @handle:		For huge zspage in zsmalloc zpool
    -+ * @zspage:		Points to the zspage this zpdesc is a part of
    -+ * @first_obj_offset:	First object offset in zsmalloc zpool
    -+ * @_refcount:		Indirectly used by page migration
    -+ * @memcg_data:		Memory Control Group data.
    ++ * struct zpdesc -	Memory descriptor for zpool memory.
    ++ * @flags:		Page flags, mostly unused by zsmalloc.
    ++ * @lru:		Indirectly used by page migration.
    ++ * @movable_ops:	Used by page migration.
    ++ * @next:		Next zpdesc in a zspage in zsmalloc zpool.
    ++ * @handle:		For huge zspage in zsmalloc zpool.
    ++ * @zspage:		Points to the zspage this zpdesc is a part of.
    ++ * @first_obj_offset:	First object offset in zsmalloc zpool.
    ++ * @_refcount:		The number of references to this zpdesc.
     + *
     + * This struct overlays struct page for now. Do not modify without a good
    -+ * understanding of the issues.  In particular, do not expand into
    -+ * the overlap with memcg_data.
    ++ * understanding of the issues. In particular, do not expand into the overlap
    ++ * with memcg_data.
     + *
     + * Page flags used:
    -+ * * PG_private identifies the first component page
    -+ * * PG_locked is used by page migration code
    ++ * * PG_private identifies the first component page.
    ++ * * PG_locked is used by page migration code.
     + */
     +struct zpdesc {
     +	unsigned long flags;
    @@ mm/zpdesc.h (new)
     +		unsigned long handle;
     +	};
     +	struct zspage *zspage;
    ++	/*
    ++	 * Only the lower 24 bits are available for offset, limiting a page
    ++	 * to 16 MiB. The upper 8 bits are reserved for PGTY_zsmalloc.
    ++	 *
    ++	 * Do not access this field directly.
    ++	 * Instead, use {get,set}_first_obj_offset() helpers.
    ++	 */
     +	unsigned int first_obj_offset;
     +	atomic_t _refcount;
     +};
    @@ mm/zpdesc.h (new)
     +#undef ZPDESC_MATCH
     +static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
     +
    ++/*
    ++ * zpdesc_page - The first struct page allocated for a zpdesc
    ++ * @zp: The zpdesc.
    ++ *
    ++ * A convenience wrapper for converting zpdesc to the first struct page of the
    ++ * underlying folio, to communicate with code not yet converted to folio or
    ++ * struct zpdesc.
    ++ *
    ++ */
     +#define zpdesc_page(zp)			(_Generic((zp),			\
     +	const struct zpdesc *:		(const struct page *)(zp),	\
     +	struct zpdesc *:		(struct page *)(zp)))
     +
    -+/* Using folio conversion to skip compound_head checking */
    ++/**
    ++ * zpdesc_folio - The folio allocated for a zpdesc
    ++ * @zpdesc: The zpdesc.
    ++ *
    ++ * Zpdescs are descriptors for zpool memory. The zpool memory itself is
    ++ * allocated as folios that contain the zpool objects, and zpdesc uses specific
    ++ * fields in the first struct page of the folio - those fields are now accessed
    ++ * by struct zpdesc.
    ++ *
    ++ * It is occasionally necessary convert to back to a folio in order to
    ++ * communicate with the rest of the mm. Please use this helper function
    ++ * instead of casting yourself, as the implementation may change in the future.
    ++ */
     +#define zpdesc_folio(zp)		(_Generic((zp),			\
     +	const struct zpdesc *:		(const struct folio *)(zp),	\
     +	struct zpdesc *:		(struct folio *)(zp)))
    -+
    ++/**
    ++ * page_zpdesc - Converts from first struct page to zpdesc.
    ++ * @p: The first (either head of compound or single) page of zpdesc.
    ++ *
    ++ * A temporary wrapper to convert struct page to struct zpdesc in situations
    ++ * where we know the page is the compound head, or single order-0 page.
    ++ *
    ++ * Long-term ideally everything would work with struct zpdesc directly or go
    ++ * through folio to struct zpdesc.
    ++ *
    ++ * Return: The zpdesc which contains this page
    ++ */
     +#define page_zpdesc(p)			(_Generic((p),			\
     +	const struct page *:		(const struct zpdesc *)(p),	\
     +	struct page *:			(struct zpdesc *)(p)))
    @@ mm/zpdesc.h (new)
     +#endif
     
      ## mm/zsmalloc.c ##
    +@@
    +  * Released under the terms of GNU General Public License Version 2.0
    +  */
    + 
    +-/*
    +- * Following is how we use various fields and flags of underlying
    +- * struct page(s) to form a zspage.
    +- *
    +- * Usage of struct page fields:
    +- *	page->private: points to zspage
    +- *	page->index: links together all component pages of a zspage
    +- *		For the huge page, this is always 0, so we use this field
    +- *		to store handle.
    +- *	page->page_type: PGTY_zsmalloc, lower 24 bits locate the first object
    +- *		offset in a subpage of a zspage
    +- *
    +- * Usage of struct page flags:
    +- *	PG_private: identifies the first component page
    +- *	PG_owner_priv_1: identifies the huge component page
    +- *
    +- */
    +-
    + #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
    + 
    + /*
     @@
      #include <linux/pagemap.h>
      #include <linux/fs.h>
    @@ mm/zsmalloc.c: static void create_page_chain(struct size_class *class, struct zs
      			if (unlikely(class->objs_per_zspage == 1 &&
      					class->pages_per_zspage == 1))
     @@ mm/zsmalloc.c: static unsigned long obj_malloc(struct zs_pool *pool,
    + 		/* record handle in the header of allocated chunk */
      		link->handle = handle | OBJ_ALLOCATED_TAG;
      	else
    - 		/* record handle to page->index */
    +-		/* record handle to page->index */
     -		zspage->first_page->index = handle | OBJ_ALLOCATED_TAG;
     +		zspage->first_zpdesc->handle = handle | OBJ_ALLOCATED_TAG;
      
 2:  d39d4fb6ce47 !  2:  213aacce3c28 mm/zsmalloc: use zpdesc in trylock_zspage()/lock_zspage()
    @@ Metadata
      ## Commit message ##
         mm/zsmalloc: use zpdesc in trylock_zspage()/lock_zspage()
     
    -    To use zpdesc in trylock_zspage()/lock_zspage() funcs, we add couple of helpers:
    -    zpdesc_lock()/zpdesc_unlock()/zpdesc_trylock()/zpdesc_wait_locked() and
    -    zpdesc_get()/zpdesc_put() for this purpose.
    +    Convert trylock_zspage() and lock_zspage() to use zpdesc. To achieve
    +    that, introduce a couple of helper functions:
    +      - zpdesc_lock()
    +      - zpdesc_unlock()
    +      - zpdesc_trylock()
    +      - zpdesc_wait_locked()
    +      - zpdesc_get()
    +      - zpdesc_put()
     
    -    Here we use the folio series func in guts for 2 reasons, one zswap.zpool
    -    only get single page, and use folio could save some compound_head checking;
    -    two, folio_put could bypass devmap checking that we don't need.
    +    Here we use the folio version of functions for 2 reasons.
    +    First, zswap.zpool currently only uses order-0 pages and using folio
    +    could save some compound_head checks. Second, folio_put could bypass
    +    devmap checking that we don't need.
     
         BTW, thanks Intel LKP found a build warning on the patch.
     
         Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zpdesc.h ##
     @@ mm/zpdesc.h: static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
 3:  eac31b98eccb =  3:  bcc63dd123f4 mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc
 4:  88ff6685943a !  4:  44faeff2ab83 mm/zsmalloc: add and use pfn/zpdesc seeking funcs
    @@ mm/zsmalloc.c: static void obj_free(int class_size, unsigned long obj)
      		link->next = get_freeobj(zspage) << OBJ_TAG_BITS;
      	else
     -		f_page->index = 0;
    -+		f_zpdesc->next = NULL;
    ++		f_zpdesc->handle = 0;
      	set_freeobj(zspage, f_objidx);
      
      	kunmap_local(vaddr);
 5:  9c9e34caf1dc =  5:  c907377a9f73 mm/zsmalloc: convert obj_malloc() to use zpdesc
 6:  74470439e747 !  6:  561c74077136 mm/zsmalloc: convert create_page_chain() and its users to use zpdesc
    @@ Metadata
      ## Commit message ##
         mm/zsmalloc: convert create_page_chain() and its users to use zpdesc
     
    -    Introduce a few helper functions for conversion to convert create_page_chain()
    -    to use zpdesc, then use zpdesc in replace_sub_page() too.
    +    Introduce a few helper functions for conversion to convert
    +    create_page_chain() to use zpdesc, then use zpdesc in replace_sub_page().
     
         Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zpdesc.h ##
     @@ mm/zpdesc.h: static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)
 7:  999ce777e513 =  7:  5b6bb539edac mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc
 8:  42cc3fe825bc =  8:  219b638a95ac mm/zsmalloc: convert init_zspage() to use zpdesc
 9:  c2fea65391d9 =  9:  ad4cd88fb89c mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc
10:  c23d24a549dc ! 10:  931f0c1fdff8 mm/zsmalloc: add zpdesc_is_isolated()/zpdesc_zone() helper for zs_page_migrate()
    @@ Metadata
     Author: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## Commit message ##
    -    mm/zsmalloc: add zpdesc_is_isolated()/zpdesc_zone() helper for zs_page_migrate()
    +    mm/zsmalloc: add two helpers for zs_page_migrate() and make it use zpdesc
     
         To convert page to zpdesc in zs_page_migrate(), we added
         zpdesc_is_isolated()/zpdesc_zone() helpers. No functional change.
11:  f0fdf4f127a4 ! 11:  3a37599de2a1 mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it
    @@ Metadata
     Author: Alex Shi <alexs@kernel.org>
     
      ## Commit message ##
    -    mm/zsmalloc: rename reset_page to reset_zpdesc and use zpdesc in it
    +    mm/zsmalloc: convert reset_page to reset_zpdesc
     
         zpdesc.zspage matches with page.private, zpdesc.next matches with
         page.index. They will be reset in reset_page() which is called prior to
         free base pages of a zspage.
    -    Use zpdesc to replace page struct and rename it to reset_zpdesc(), few
    -    page helper still left since they are used too widely.
     
    +    Since the fields that need to be initialized are independent of the
    +    order in struct zpdesc, Keep it to use struct page to ensure robustness
    +    against potential rearrangements of struct zpdesc fields in the future.
    +
    +    [42.hyeyoo: keep reset_zpdesc() to use struct page fields]
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zsmalloc.c ##
     @@ mm/zsmalloc.c: static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
    @@ mm/zsmalloc.c: static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj
     +
      	__ClearPageMovable(page);
      	ClearPagePrivate(page);
    --	set_page_private(page, 0);
    --	page->index = 0;
    -+	zpdesc->zspage = NULL;
    -+	zpdesc->next = NULL;
    - 	__ClearPageZsmalloc(page);
    - }
    - 
    + 	set_page_private(page, 0);
     @@ mm/zsmalloc.c: static void __free_zspage(struct zs_pool *pool, struct size_class *class,
      	do {
      		VM_BUG_ON_PAGE(!PageLocked(page), page);
12:  889db23882fb = 12:  26c059dc7680 mm/zsmalloc: convert __free_zspage() to use zpdesc
13:  21398ee44728 = 13:  75da524b90b6 mm/zsmalloc: convert location_to_obj() to take zpdesc
14:  fcbdb848eafe = 14:  ffbf4cdbde74 mm/zsmalloc: convert migrate_zspage() to use zpdesc
15:  abc171445571 = 15:  c78de6e45dd4 mm/zsmalloc: convert get_zspage() to take zpdesc
16:  fc6e6df18de6 ! 16:  a409db41562c mm/zsmalloc: convert SetZsPageMovable and remove unused funcs
    @@ Commit message
     
         Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zsmalloc.c ##
     @@ mm/zsmalloc.c: static DEFINE_PER_CPU(struct mapping_area, zs_map_area) = {
17:  e3e1b9ba2739 ! 17:  a0046eec7921 mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc
    @@ Commit message
         Now that all users of get/set_first_obj_offset() are converted
         to use zpdesc, convert them to take zpdesc.
     
    -    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zsmalloc.c ##
     @@ mm/zsmalloc.c: static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
    @@ mm/zsmalloc.c: static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
     -static inline void set_first_obj_offset(struct page *page, unsigned int offset)
     +static inline void set_first_obj_offset(struct zpdesc *zpdesc, unsigned int offset)
      {
    --	/* With 24 bits available, we can support offsets into 16 MiB pages. */
    --	BUILD_BUG_ON(PAGE_SIZE > SZ_16M);
    + 	/* With 24 bits available, we can support offsets into 16 MiB pages. */
    + 	BUILD_BUG_ON(PAGE_SIZE > SZ_16M);
     -	VM_WARN_ON_ONCE(!PageZsmalloc(page));
    -+	/* With 16 bit available, we can support offsets into 64 KiB pages. */
    -+	BUILD_BUG_ON(PAGE_SIZE > SZ_64K);
     +	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
      	VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK);
     -	page->page_type &= ~FIRST_OBJ_PAGE_TYPE_MASK;
18:  ff7376e59bfd <  -:  ------------ mm/zsmalloc: introduce __zpdesc_clear_movable
19:  3d74d287da4e ! 18:  0ac98437b837 mm/zsmalloc: introduce __zpdesc_clear/set_zsmalloc()
    @@ Commit message
         __zpdesc_set_zsmalloc() for __SetPageZsmalloc(), and use them in
         callers.
     
    +    [42.hyeyoo: keep reset_zpdesc() to use struct page]
         Signed-off-by: Alex Shi <alexs@kernel.org>
    +    Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
     
      ## mm/zpdesc.h ##
    -@@ mm/zpdesc.h: static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc)
    - 	__ClearPageMovable(zpdesc_page(zpdesc));
    +@@ mm/zpdesc.h: static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
    + 	__SetPageMovable(zpdesc_page(zpdesc), mops);
      }
      
     +static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc)
    @@ mm/zpdesc.h: static inline void __zpdesc_clear_movable(struct zpdesc *zpdesc)
      	return PageIsolated(zpdesc_page(zpdesc));
     
      ## mm/zsmalloc.c ##
    -@@ mm/zsmalloc.c: static void reset_zpdesc(struct zpdesc *zpdesc)
    - 	ClearPagePrivate(page);
    - 	zpdesc->zspage = NULL;
    - 	zpdesc->next = NULL;
    --	__ClearPageZsmalloc(page);
    -+	__zpdesc_clear_zsmalloc(zpdesc);
    - }
    - 
    - static int trylock_zspage(struct zspage *zspage)
     @@ mm/zsmalloc.c: static struct zspage *alloc_zspage(struct zs_pool *pool,
      		if (!zpdesc) {
      			while (--i >= 0) {
20:  6e5528eb9957 <  -:  ------------ mm/zsmalloc: introduce zpdesc_clear_first() helper
21:  d9346ccf3749 <  -:  ------------ mm/zsmalloc: update comments for page->zpdesc changes

[PATCH v9 mm-unstable 01/18] mm/zsmalloc: add zpdesc memory descriptor for zswap.zpool

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

The 1st patch introduces new memory descriptor zpdesc and renames
zspage.first_page to zspage.first_zpdesc, with no functional change.

We removed the comment about PG_owner_priv_1 since it is no longer used
after commit a41ec880aa7b ("zsmalloc: move huge compressed obj from page
to zspage").

[42.hyeyoo: rework comments a little bit]
Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zpdesc.h   | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++
 mm/zsmalloc.c |  28 +++----------
 2 files changed, 112 insertions(+), 23 deletions(-)
 create mode 100644 mm/zpdesc.h

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
new file mode 100644
index 000000000000..e0852498aecf
--- /dev/null
+++ b/mm/zpdesc.h
@@ -0,0 +1,107 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* zpdesc.h: zswap.zpool memory descriptor
+ *
+ * Written by Alex Shi <alexs@kernel.org>
+ *	      Hyeonggon Yoo <42.hyeyoo@gmail.com>
+ */
+#ifndef __MM_ZPDESC_H__
+#define __MM_ZPDESC_H__
+
+/*
+ * struct zpdesc -	Memory descriptor for zpool memory.
+ * @flags:		Page flags, mostly unused by zsmalloc.
+ * @lru:		Indirectly used by page migration.
+ * @movable_ops:	Used by page migration.
+ * @next:		Next zpdesc in a zspage in zsmalloc zpool.
+ * @handle:		For huge zspage in zsmalloc zpool.
+ * @zspage:		Points to the zspage this zpdesc is a part of.
+ * @first_obj_offset:	First object offset in zsmalloc zpool.
+ * @_refcount:		The number of references to this zpdesc.
+ *
+ * This struct overlays struct page for now. Do not modify without a good
+ * understanding of the issues. In particular, do not expand into the overlap
+ * with memcg_data.
+ *
+ * Page flags used:
+ * * PG_private identifies the first component page.
+ * * PG_locked is used by page migration code.
+ */
+struct zpdesc {
+	unsigned long flags;
+	struct list_head lru;
+	unsigned long movable_ops;
+	union {
+		struct zpdesc *next;
+		unsigned long handle;
+	};
+	struct zspage *zspage;
+	/*
+	 * Only the lower 24 bits are available for offset, limiting a page
+	 * to 16 MiB. The upper 8 bits are reserved for PGTY_zsmalloc.
+	 *
+	 * Do not access this field directly.
+	 * Instead, use {get,set}_first_obj_offset() helpers.
+	 */
+	unsigned int first_obj_offset;
+	atomic_t _refcount;
+};
+#define ZPDESC_MATCH(pg, zp) \
+	static_assert(offsetof(struct page, pg) == offsetof(struct zpdesc, zp))
+
+ZPDESC_MATCH(flags, flags);
+ZPDESC_MATCH(lru, lru);
+ZPDESC_MATCH(mapping, movable_ops);
+ZPDESC_MATCH(index, next);
+ZPDESC_MATCH(index, handle);
+ZPDESC_MATCH(private, zspage);
+ZPDESC_MATCH(page_type, first_obj_offset);
+ZPDESC_MATCH(_refcount, _refcount);
+#undef ZPDESC_MATCH
+static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
+
+/*
+ * zpdesc_page - The first struct page allocated for a zpdesc
+ * @zp: The zpdesc.
+ *
+ * A convenience wrapper for converting zpdesc to the first struct page of the
+ * underlying folio, to communicate with code not yet converted to folio or
+ * struct zpdesc.
+ *
+ */
+#define zpdesc_page(zp)			(_Generic((zp),			\
+	const struct zpdesc *:		(const struct page *)(zp),	\
+	struct zpdesc *:		(struct page *)(zp)))
+
+/**
+ * zpdesc_folio - The folio allocated for a zpdesc
+ * @zpdesc: The zpdesc.
+ *
+ * Zpdescs are descriptors for zpool memory. The zpool memory itself is
+ * allocated as folios that contain the zpool objects, and zpdesc uses specific
+ * fields in the first struct page of the folio - those fields are now accessed
+ * by struct zpdesc.
+ *
+ * It is occasionally necessary convert to back to a folio in order to
+ * communicate with the rest of the mm. Please use this helper function
+ * instead of casting yourself, as the implementation may change in the future.
+ */
+#define zpdesc_folio(zp)		(_Generic((zp),			\
+	const struct zpdesc *:		(const struct folio *)(zp),	\
+	struct zpdesc *:		(struct folio *)(zp)))
+/**
+ * page_zpdesc - Converts from first struct page to zpdesc.
+ * @p: The first (either head of compound or single) page of zpdesc.
+ *
+ * A temporary wrapper to convert struct page to struct zpdesc in situations
+ * where we know the page is the compound head, or single order-0 page.
+ *
+ * Long-term ideally everything would work with struct zpdesc directly or go
+ * through folio to struct zpdesc.
+ *
+ * Return: The zpdesc which contains this page
+ */
+#define page_zpdesc(p)			(_Generic((p),			\
+	const struct page *:		(const struct zpdesc *)(p),	\
+	struct page *:			(struct zpdesc *)(p)))
+
+#endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 64b66a4d3e6e..00d111f011be 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -13,24 +13,6 @@
  * Released under the terms of GNU General Public License Version 2.0
  */
 
-/*
- * Following is how we use various fields and flags of underlying
- * struct page(s) to form a zspage.
- *
- * Usage of struct page fields:
- *	page->private: points to zspage
- *	page->index: links together all component pages of a zspage
- *		For the huge page, this is always 0, so we use this field
- *		to store handle.
- *	page->page_type: PGTY_zsmalloc, lower 24 bits locate the first object
- *		offset in a subpage of a zspage
- *
- * Usage of struct page flags:
- *	PG_private: identifies the first component page
- *	PG_owner_priv_1: identifies the huge component page
- *
- */
-
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
 /*
@@ -67,6 +49,7 @@
 #include <linux/pagemap.h>
 #include <linux/fs.h>
 #include <linux/local_lock.h>
+#include "zpdesc.h"
 
 #define ZSPAGE_MAGIC	0x58
 
@@ -254,7 +237,7 @@ struct zspage {
 	};
 	unsigned int inuse;
 	unsigned int freeobj;
-	struct page *first_page;
+	struct zpdesc *first_zpdesc;
 	struct list_head list; /* fullness list */
 	struct zs_pool *pool;
 	rwlock_t lock;
@@ -459,7 +442,7 @@ static inline void mod_zspage_inuse(struct zspage *zspage, int val)
 
 static inline struct page *get_first_page(struct zspage *zspage)
 {
-	struct page *first_page = zspage->first_page;
+	struct page *first_page = zpdesc_page(zspage->first_zpdesc);
 
 	VM_BUG_ON_PAGE(!is_first_page(first_page), first_page);
 	return first_page;
@@ -952,7 +935,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
 		set_page_private(page, (unsigned long)zspage);
 		page->index = 0;
 		if (i == 0) {
-			zspage->first_page = page;
+			zspage->first_zpdesc = page_zpdesc(page);
 			SetPagePrivate(page);
 			if (unlikely(class->objs_per_zspage == 1 &&
 					class->pages_per_zspage == 1))
@@ -1317,8 +1300,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 		/* record handle in the header of allocated chunk */
 		link->handle = handle | OBJ_ALLOCATED_TAG;
 	else
-		/* record handle to page->index */
-		zspage->first_page->index = handle | OBJ_ALLOCATED_TAG;
+		zspage->first_zpdesc->handle = handle | OBJ_ALLOCATED_TAG;
 
 	kunmap_local(vaddr);
 	mod_zspage_inuse(zspage, 1);
-- 
2.43.5

[PATCH v9 mm-unstable 02/18] mm/zsmalloc: use zpdesc in trylock_zspage()/lock_zspage()

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

Convert trylock_zspage() and lock_zspage() to use zpdesc. To achieve
that, introduce a couple of helper functions:
  - zpdesc_lock()
  - zpdesc_unlock()
  - zpdesc_trylock()
  - zpdesc_wait_locked()
  - zpdesc_get()
  - zpdesc_put()

Here we use the folio version of functions for 2 reasons.
First, zswap.zpool currently only uses order-0 pages and using folio
could save some compound_head checks. Second, folio_put could bypass
devmap checking that we don't need.

BTW, thanks Intel LKP found a build warning on the patch.

Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zpdesc.h   | 30 ++++++++++++++++++++++++
 mm/zsmalloc.c | 64 ++++++++++++++++++++++++++++++++++-----------------
 2 files changed, 73 insertions(+), 21 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index e0852498aecf..c866758feec3 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -104,4 +104,34 @@ static_assert(sizeof(struct zpdesc) <= sizeof(struct page));
 	const struct page *:		(const struct zpdesc *)(p),	\
 	struct page *:			(struct zpdesc *)(p)))
 
+static inline void zpdesc_lock(struct zpdesc *zpdesc)
+{
+	folio_lock(zpdesc_folio(zpdesc));
+}
+
+static inline bool zpdesc_trylock(struct zpdesc *zpdesc)
+{
+	return folio_trylock(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_unlock(struct zpdesc *zpdesc)
+{
+	folio_unlock(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_wait_locked(struct zpdesc *zpdesc)
+{
+	folio_wait_locked(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_get(struct zpdesc *zpdesc)
+{
+	folio_get(zpdesc_folio(zpdesc));
+}
+
+static inline void zpdesc_put(struct zpdesc *zpdesc)
+{
+	folio_put(zpdesc_folio(zpdesc));
+}
+
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 00d111f011be..51f4a9b78023 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -428,13 +428,17 @@ static __maybe_unused int is_first_page(struct page *page)
 	return PagePrivate(page);
 }
 
+static inline bool is_first_zpdesc(struct zpdesc *zpdesc)
+{
+	return PagePrivate(zpdesc_page(zpdesc));
+}
+
 /* Protected by class->lock */
 static inline int get_zspage_inuse(struct zspage *zspage)
 {
 	return zspage->inuse;
 }
 
-
 static inline void mod_zspage_inuse(struct zspage *zspage, int val)
 {
 	zspage->inuse += val;
@@ -448,6 +452,14 @@ static inline struct page *get_first_page(struct zspage *zspage)
 	return first_page;
 }
 
+static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
+{
+	struct zpdesc *first_zpdesc = zspage->first_zpdesc;
+
+	VM_BUG_ON_PAGE(!is_first_zpdesc(first_zpdesc), zpdesc_page(first_zpdesc));
+	return first_zpdesc;
+}
+
 #define FIRST_OBJ_PAGE_TYPE_MASK	0xffffff
 
 static inline unsigned int get_first_obj_offset(struct page *page)
@@ -734,6 +746,16 @@ static struct page *get_next_page(struct page *page)
 	return (struct page *)page->index;
 }
 
+static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
+{
+	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
+
+	if (unlikely(ZsHugePage(zspage)))
+		return NULL;
+
+	return zpdesc->next;
+}
+
 /**
  * obj_to_location - get (<page>, <obj_idx>) from encoded object value
  * @obj: the encoded object value
@@ -803,11 +825,11 @@ static void reset_page(struct page *page)
 
 static int trylock_zspage(struct zspage *zspage)
 {
-	struct page *cursor, *fail;
+	struct zpdesc *cursor, *fail;
 
-	for (cursor = get_first_page(zspage); cursor != NULL; cursor =
-					get_next_page(cursor)) {
-		if (!trylock_page(cursor)) {
+	for (cursor = get_first_zpdesc(zspage); cursor != NULL; cursor =
+					get_next_zpdesc(cursor)) {
+		if (!zpdesc_trylock(cursor)) {
 			fail = cursor;
 			goto unlock;
 		}
@@ -815,9 +837,9 @@ static int trylock_zspage(struct zspage *zspage)
 
 	return 1;
 unlock:
-	for (cursor = get_first_page(zspage); cursor != fail; cursor =
-					get_next_page(cursor))
-		unlock_page(cursor);
+	for (cursor = get_first_zpdesc(zspage); cursor != fail; cursor =
+					get_next_zpdesc(cursor))
+		zpdesc_unlock(cursor);
 
 	return 0;
 }
@@ -1635,7 +1657,7 @@ static int putback_zspage(struct size_class *class, struct zspage *zspage)
  */
 static void lock_zspage(struct zspage *zspage)
 {
-	struct page *curr_page, *page;
+	struct zpdesc *curr_zpdesc, *zpdesc;
 
 	/*
 	 * Pages we haven't locked yet can be migrated off the list while we're
@@ -1647,24 +1669,24 @@ static void lock_zspage(struct zspage *zspage)
 	 */
 	while (1) {
 		migrate_read_lock(zspage);
-		page = get_first_page(zspage);
-		if (trylock_page(page))
+		zpdesc = get_first_zpdesc(zspage);
+		if (zpdesc_trylock(zpdesc))
 			break;
-		get_page(page);
+		zpdesc_get(zpdesc);
 		migrate_read_unlock(zspage);
-		wait_on_page_locked(page);
-		put_page(page);
+		zpdesc_wait_locked(zpdesc);
+		zpdesc_put(zpdesc);
 	}
 
-	curr_page = page;
-	while ((page = get_next_page(curr_page))) {
-		if (trylock_page(page)) {
-			curr_page = page;
+	curr_zpdesc = zpdesc;
+	while ((zpdesc = get_next_zpdesc(curr_zpdesc))) {
+		if (zpdesc_trylock(zpdesc)) {
+			curr_zpdesc = zpdesc;
 		} else {
-			get_page(page);
+			zpdesc_get(zpdesc);
 			migrate_read_unlock(zspage);
-			wait_on_page_locked(page);
-			put_page(page);
+			zpdesc_wait_locked(zpdesc);
+			zpdesc_put(zpdesc);
 			migrate_read_lock(zspage);
 		}
 	}
-- 
2.43.5

[PATCH v9 mm-unstable 03/18] mm/zsmalloc: convert __zs_map_object/__zs_unmap_object to use zpdesc

[Hyeonggon Yoo]

These two functions take a pointer to an array of struct page.
Make __zs_{map,unmap}_object() take pointer to an array of zpdesc
instead of page.

Add silly type casting when calling them. Casting will be removed later.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 51f4a9b78023..c038caaef3a8 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1049,7 +1049,7 @@ static inline void __zs_cpu_down(struct mapping_area *area)
 }
 
 static void *__zs_map_object(struct mapping_area *area,
-			struct page *pages[2], int off, int size)
+			struct zpdesc *zpdescs[2], int off, int size)
 {
 	size_t sizes[2];
 	char *buf = area->vm_buf;
@@ -1065,14 +1065,14 @@ static void *__zs_map_object(struct mapping_area *area,
 	sizes[1] = size - sizes[0];
 
 	/* copy object to per-cpu buffer */
-	memcpy_from_page(buf, pages[0], off, sizes[0]);
-	memcpy_from_page(buf + sizes[0], pages[1], 0, sizes[1]);
+	memcpy_from_page(buf, zpdesc_page(zpdescs[0]), off, sizes[0]);
+	memcpy_from_page(buf + sizes[0], zpdesc_page(zpdescs[1]), 0, sizes[1]);
 out:
 	return area->vm_buf;
 }
 
 static void __zs_unmap_object(struct mapping_area *area,
-			struct page *pages[2], int off, int size)
+			struct zpdesc *zpdescs[2], int off, int size)
 {
 	size_t sizes[2];
 	char *buf;
@@ -1090,8 +1090,8 @@ static void __zs_unmap_object(struct mapping_area *area,
 	sizes[1] = size - sizes[0];
 
 	/* copy per-cpu buffer to object */
-	memcpy_to_page(pages[0], off, buf, sizes[0]);
-	memcpy_to_page(pages[1], 0, buf + sizes[0], sizes[1]);
+	memcpy_to_page(zpdesc_page(zpdescs[0]), off, buf, sizes[0]);
+	memcpy_to_page(zpdesc_page(zpdescs[1]), 0, buf + sizes[0], sizes[1]);
 
 out:
 	/* enable page faults to match kunmap_local() return conditions */
@@ -1230,7 +1230,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	pages[1] = get_next_page(page);
 	BUG_ON(!pages[1]);
 
-	ret = __zs_map_object(area, pages, off, class->size);
+	ret = __zs_map_object(area, (struct zpdesc **)pages, off, class->size);
 out:
 	if (likely(!ZsHugePage(zspage)))
 		ret += ZS_HANDLE_SIZE;
@@ -1265,7 +1265,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 		pages[1] = get_next_page(page);
 		BUG_ON(!pages[1]);
 
-		__zs_unmap_object(area, pages, off, class->size);
+		__zs_unmap_object(area, (struct zpdesc **)pages, off, class->size);
 	}
 	local_unlock(&zs_map_area.lock);
 
-- 
2.43.5

[PATCH v9 mm-unstable 04/18] mm/zsmalloc: add and use pfn/zpdesc seeking funcs

[Hyeonggon Yoo]

Add pfn_zpdesc(), pfn_zpdesc() and kmap_local_zpdesc().  Convert
obj_to_location() to take zpdesc and also convert its users to use zpdesc.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zpdesc.h   | 14 ++++++++++
 mm/zsmalloc.c | 75 ++++++++++++++++++++++++++-------------------------
 2 files changed, 52 insertions(+), 37 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index c866758feec3..223d0381a444 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -134,4 +134,18 @@ static inline void zpdesc_put(struct zpdesc *zpdesc)
 	folio_put(zpdesc_folio(zpdesc));
 }
 
+static inline void *kmap_local_zpdesc(struct zpdesc *zpdesc)
+{
+	return kmap_local_page(zpdesc_page(zpdesc));
+}
+
+static inline unsigned long zpdesc_pfn(struct zpdesc *zpdesc)
+{
+	return page_to_pfn(zpdesc_page(zpdesc));
+}
+
+static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)
+{
+	return page_zpdesc(pfn_to_page(pfn));
+}
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index c038caaef3a8..e71da84ad73a 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -757,15 +757,15 @@ static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 }
 
 /**
- * obj_to_location - get (<page>, <obj_idx>) from encoded object value
+ * obj_to_location - get (<zpdesc>, <obj_idx>) from encoded object value
  * @obj: the encoded object value
- * @page: page object resides in zspage
+ * @zpdesc: zpdesc object resides in zspage
  * @obj_idx: object index
  */
-static void obj_to_location(unsigned long obj, struct page **page,
+static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
 				unsigned int *obj_idx)
 {
-	*page = pfn_to_page(obj >> OBJ_INDEX_BITS);
+	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
 	*obj_idx = (obj & OBJ_INDEX_MASK);
 }
 
@@ -1181,13 +1181,13 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 			enum zs_mapmode mm)
 {
 	struct zspage *zspage;
-	struct page *page;
+	struct zpdesc *zpdesc;
 	unsigned long obj, off;
 	unsigned int obj_idx;
 
 	struct size_class *class;
 	struct mapping_area *area;
-	struct page *pages[2];
+	struct zpdesc *zpdescs[2];
 	void *ret;
 
 	/*
@@ -1200,8 +1200,8 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	/* It guarantees it can get zspage from handle safely */
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
-	obj_to_location(obj, &page, &obj_idx);
-	zspage = get_zspage(page);
+	obj_to_location(obj, &zpdesc, &obj_idx);
+	zspage = get_zspage(zpdesc_page(zpdesc));
 
 	/*
 	 * migration cannot move any zpages in this zspage. Here, class->lock
@@ -1220,17 +1220,17 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	area->vm_mm = mm;
 	if (off + class->size <= PAGE_SIZE) {
 		/* this object is contained entirely within a page */
-		area->vm_addr = kmap_local_page(page);
+		area->vm_addr = kmap_local_zpdesc(zpdesc);
 		ret = area->vm_addr + off;
 		goto out;
 	}
 
 	/* this object spans two pages */
-	pages[0] = page;
-	pages[1] = get_next_page(page);
-	BUG_ON(!pages[1]);
+	zpdescs[0] = zpdesc;
+	zpdescs[1] = get_next_zpdesc(zpdesc);
+	BUG_ON(!zpdescs[1]);
 
-	ret = __zs_map_object(area, (struct zpdesc **)pages, off, class->size);
+	ret = __zs_map_object(area, zpdescs, off, class->size);
 out:
 	if (likely(!ZsHugePage(zspage)))
 		ret += ZS_HANDLE_SIZE;
@@ -1242,7 +1242,7 @@ EXPORT_SYMBOL_GPL(zs_map_object);
 void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
-	struct page *page;
+	struct zpdesc *zpdesc;
 	unsigned long obj, off;
 	unsigned int obj_idx;
 
@@ -1250,8 +1250,8 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 	struct mapping_area *area;
 
 	obj = handle_to_obj(handle);
-	obj_to_location(obj, &page, &obj_idx);
-	zspage = get_zspage(page);
+	obj_to_location(obj, &zpdesc, &obj_idx);
+	zspage = get_zspage(zpdesc_page(zpdesc));
 	class = zspage_class(pool, zspage);
 	off = offset_in_page(class->size * obj_idx);
 
@@ -1259,13 +1259,13 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 	if (off + class->size <= PAGE_SIZE)
 		kunmap_local(area->vm_addr);
 	else {
-		struct page *pages[2];
+		struct zpdesc *zpdescs[2];
 
-		pages[0] = page;
-		pages[1] = get_next_page(page);
-		BUG_ON(!pages[1]);
+		zpdescs[0] = zpdesc;
+		zpdescs[1] = get_next_zpdesc(zpdesc);
+		BUG_ON(!zpdescs[1]);
 
-		__zs_unmap_object(area, (struct zpdesc **)pages, off, class->size);
+		__zs_unmap_object(area, zpdescs, off, class->size);
 	}
 	local_unlock(&zs_map_area.lock);
 
@@ -1406,23 +1406,24 @@ static void obj_free(int class_size, unsigned long obj)
 {
 	struct link_free *link;
 	struct zspage *zspage;
-	struct page *f_page;
+	struct zpdesc *f_zpdesc;
 	unsigned long f_offset;
 	unsigned int f_objidx;
 	void *vaddr;
 
-	obj_to_location(obj, &f_page, &f_objidx);
+
+	obj_to_location(obj, &f_zpdesc, &f_objidx);
 	f_offset = offset_in_page(class_size * f_objidx);
-	zspage = get_zspage(f_page);
+	zspage = get_zspage(zpdesc_page(f_zpdesc));
 
-	vaddr = kmap_local_page(f_page);
+	vaddr = kmap_local_zpdesc(f_zpdesc);
 	link = (struct link_free *)(vaddr + f_offset);
 
 	/* Insert this object in containing zspage's freelist */
 	if (likely(!ZsHugePage(zspage)))
 		link->next = get_freeobj(zspage) << OBJ_TAG_BITS;
 	else
-		f_page->index = 0;
+		f_zpdesc->handle = 0;
 	set_freeobj(zspage, f_objidx);
 
 	kunmap_local(vaddr);
@@ -1467,7 +1468,7 @@ EXPORT_SYMBOL_GPL(zs_free);
 static void zs_object_copy(struct size_class *class, unsigned long dst,
 				unsigned long src)
 {
-	struct page *s_page, *d_page;
+	struct zpdesc *s_zpdesc, *d_zpdesc;
 	unsigned int s_objidx, d_objidx;
 	unsigned long s_off, d_off;
 	void *s_addr, *d_addr;
@@ -1476,8 +1477,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 
 	s_size = d_size = class->size;
 
-	obj_to_location(src, &s_page, &s_objidx);
-	obj_to_location(dst, &d_page, &d_objidx);
+	obj_to_location(src, &s_zpdesc, &s_objidx);
+	obj_to_location(dst, &d_zpdesc, &d_objidx);
 
 	s_off = offset_in_page(class->size * s_objidx);
 	d_off = offset_in_page(class->size * d_objidx);
@@ -1488,8 +1489,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 	if (d_off + class->size > PAGE_SIZE)
 		d_size = PAGE_SIZE - d_off;
 
-	s_addr = kmap_local_page(s_page);
-	d_addr = kmap_local_page(d_page);
+	s_addr = kmap_local_zpdesc(s_zpdesc);
+	d_addr = kmap_local_zpdesc(d_zpdesc);
 
 	while (1) {
 		size = min(s_size, d_size);
@@ -1514,17 +1515,17 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 		if (s_off >= PAGE_SIZE) {
 			kunmap_local(d_addr);
 			kunmap_local(s_addr);
-			s_page = get_next_page(s_page);
-			s_addr = kmap_local_page(s_page);
-			d_addr = kmap_local_page(d_page);
+			s_zpdesc = get_next_zpdesc(s_zpdesc);
+			s_addr = kmap_local_zpdesc(s_zpdesc);
+			d_addr = kmap_local_zpdesc(d_zpdesc);
 			s_size = class->size - written;
 			s_off = 0;
 		}
 
 		if (d_off >= PAGE_SIZE) {
 			kunmap_local(d_addr);
-			d_page = get_next_page(d_page);
-			d_addr = kmap_local_page(d_page);
+			d_zpdesc = get_next_zpdesc(d_zpdesc);
+			d_addr = kmap_local_zpdesc(d_zpdesc);
 			d_size = class->size - written;
 			d_off = 0;
 		}
@@ -1763,7 +1764,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	struct zs_pool *pool;
 	struct size_class *class;
 	struct zspage *zspage;
-	struct page *dummy;
+	struct zpdesc *dummy;
 	void *s_addr, *d_addr, *addr;
 	unsigned int offset;
 	unsigned long handle;
-- 
2.43.5

[PATCH v9 mm-unstable 05/18] mm/zsmalloc: convert obj_malloc() to use zpdesc

[Hyeonggon Yoo]

Use get_first_zpdesc/get_next_zpdesc to replace
get_first_page/get_next_page. no functional change.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index e71da84ad73a..b7fab2e28d87 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1295,12 +1295,12 @@ EXPORT_SYMBOL_GPL(zs_huge_class_size);
 static unsigned long obj_malloc(struct zs_pool *pool,
 				struct zspage *zspage, unsigned long handle)
 {
-	int i, nr_page, offset;
+	int i, nr_zpdesc, offset;
 	unsigned long obj;
 	struct link_free *link;
 	struct size_class *class;
 
-	struct page *m_page;
+	struct zpdesc *m_zpdesc;
 	unsigned long m_offset;
 	void *vaddr;
 
@@ -1308,14 +1308,14 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	obj = get_freeobj(zspage);
 
 	offset = obj * class->size;
-	nr_page = offset >> PAGE_SHIFT;
+	nr_zpdesc = offset >> PAGE_SHIFT;
 	m_offset = offset_in_page(offset);
-	m_page = get_first_page(zspage);
+	m_zpdesc = get_first_zpdesc(zspage);
 
-	for (i = 0; i < nr_page; i++)
-		m_page = get_next_page(m_page);
+	for (i = 0; i < nr_zpdesc; i++)
+		m_zpdesc = get_next_zpdesc(m_zpdesc);
 
-	vaddr = kmap_local_page(m_page);
+	vaddr = kmap_local_zpdesc(m_zpdesc);
 	link = (struct link_free *)vaddr + m_offset / sizeof(*link);
 	set_freeobj(zspage, link->next >> OBJ_TAG_BITS);
 	if (likely(!ZsHugePage(zspage)))
@@ -1327,7 +1327,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_local(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(m_page, obj);
+	obj = location_to_obj(zpdesc_page(m_zpdesc), obj);
 	record_obj(handle, obj);
 
 	return obj;
-- 
2.43.5

[PATCH v9 mm-unstable 06/18] mm/zsmalloc: convert create_page_chain() and its users to use zpdesc

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

Introduce a few helper functions for conversion to convert
create_page_chain() to use zpdesc, then use zpdesc in replace_sub_page().

Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zpdesc.h   |   6 +++
 mm/zsmalloc.c | 109 ++++++++++++++++++++++++++++++++------------------
 2 files changed, 76 insertions(+), 39 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index 223d0381a444..9aca8d307796 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -148,4 +148,10 @@ static inline struct zpdesc *pfn_zpdesc(unsigned long pfn)
 {
 	return page_zpdesc(pfn_to_page(pfn));
 }
+
+static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
+					const struct movable_operations *mops)
+{
+	__SetPageMovable(zpdesc_page(zpdesc), mops);
+}
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index b7fab2e28d87..59a30c61160f 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -228,6 +228,35 @@ struct zs_pool {
 	atomic_t compaction_in_progress;
 };
 
+static inline void zpdesc_set_first(struct zpdesc *zpdesc)
+{
+	SetPagePrivate(zpdesc_page(zpdesc));
+}
+
+static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc)
+{
+	inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
+}
+
+static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc)
+{
+	dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
+}
+
+static inline struct zpdesc *alloc_zpdesc(gfp_t gfp)
+{
+	struct page *page = alloc_page(gfp);
+
+	return page_zpdesc(page);
+}
+
+static inline void free_zpdesc(struct zpdesc *zpdesc)
+{
+	struct page *page = zpdesc_page(zpdesc);
+
+	__free_page(page);
+}
+
 struct zspage {
 	struct {
 		unsigned int huge:HUGE_BITS;
@@ -937,35 +966,35 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 }
 
 static void create_page_chain(struct size_class *class, struct zspage *zspage,
-				struct page *pages[])
+				struct zpdesc *zpdescs[])
 {
 	int i;
-	struct page *page;
-	struct page *prev_page = NULL;
-	int nr_pages = class->pages_per_zspage;
+	struct zpdesc *zpdesc;
+	struct zpdesc *prev_zpdesc = NULL;
+	int nr_zpdescs = class->pages_per_zspage;
 
 	/*
 	 * Allocate individual pages and link them together as:
-	 * 1. all pages are linked together using page->index
-	 * 2. each sub-page point to zspage using page->private
+	 * 1. all pages are linked together using zpdesc->next
+	 * 2. each sub-page point to zspage using zpdesc->zspage
 	 *
-	 * we set PG_private to identify the first page (i.e. no other sub-page
+	 * we set PG_private to identify the first zpdesc (i.e. no other zpdesc
 	 * has this flag set).
 	 */
-	for (i = 0; i < nr_pages; i++) {
-		page = pages[i];
-		set_page_private(page, (unsigned long)zspage);
-		page->index = 0;
+	for (i = 0; i < nr_zpdescs; i++) {
+		zpdesc = zpdescs[i];
+		zpdesc->zspage = zspage;
+		zpdesc->next = NULL;
 		if (i == 0) {
-			zspage->first_zpdesc = page_zpdesc(page);
-			SetPagePrivate(page);
+			zspage->first_zpdesc = zpdesc;
+			zpdesc_set_first(zpdesc);
 			if (unlikely(class->objs_per_zspage == 1 &&
 					class->pages_per_zspage == 1))
 				SetZsHugePage(zspage);
 		} else {
-			prev_page->index = (unsigned long)page;
+			prev_zpdesc->next = zpdesc;
 		}
-		prev_page = page;
+		prev_zpdesc = zpdesc;
 	}
 }
 
@@ -977,7 +1006,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 					gfp_t gfp)
 {
 	int i;
-	struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE];
+	struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE];
 	struct zspage *zspage = cache_alloc_zspage(pool, gfp);
 
 	if (!zspage)
@@ -987,25 +1016,25 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 	migrate_lock_init(zspage);
 
 	for (i = 0; i < class->pages_per_zspage; i++) {
-		struct page *page;
+		struct zpdesc *zpdesc;
 
-		page = alloc_page(gfp);
-		if (!page) {
+		zpdesc = alloc_zpdesc(gfp);
+		if (!zpdesc) {
 			while (--i >= 0) {
-				dec_zone_page_state(pages[i], NR_ZSPAGES);
-				__ClearPageZsmalloc(pages[i]);
-				__free_page(pages[i]);
+				zpdesc_dec_zone_page_state(zpdescs[i]);
+				__ClearPageZsmalloc(zpdesc_page(zpdescs[i]));
+				free_zpdesc(zpdescs[i]);
 			}
 			cache_free_zspage(pool, zspage);
 			return NULL;
 		}
-		__SetPageZsmalloc(page);
+		__SetPageZsmalloc(zpdesc_page(zpdesc));
 
-		inc_zone_page_state(page, NR_ZSPAGES);
-		pages[i] = page;
+		zpdesc_inc_zone_page_state(zpdesc);
+		zpdescs[i] = zpdesc;
 	}
 
-	create_page_chain(class, zspage, pages);
+	create_page_chain(class, zspage, zpdescs);
 	init_zspage(class, zspage);
 	zspage->pool = pool;
 	zspage->class = class->index;
@@ -1725,26 +1754,28 @@ static void migrate_write_unlock(struct zspage *zspage)
 static const struct movable_operations zsmalloc_mops;
 
 static void replace_sub_page(struct size_class *class, struct zspage *zspage,
-				struct page *newpage, struct page *oldpage)
+				struct zpdesc *newzpdesc, struct zpdesc *oldzpdesc)
 {
-	struct page *page;
-	struct page *pages[ZS_MAX_PAGES_PER_ZSPAGE] = {NULL, };
+	struct zpdesc *zpdesc;
+	struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE] = {NULL, };
+	unsigned int first_obj_offset;
 	int idx = 0;
 
-	page = get_first_page(zspage);
+	zpdesc = get_first_zpdesc(zspage);
 	do {
-		if (page == oldpage)
-			pages[idx] = newpage;
+		if (zpdesc == oldzpdesc)
+			zpdescs[idx] = newzpdesc;
 		else
-			pages[idx] = page;
+			zpdescs[idx] = zpdesc;
 		idx++;
-	} while ((page = get_next_page(page)) != NULL);
+	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
 
-	create_page_chain(class, zspage, pages);
-	set_first_obj_offset(newpage, get_first_obj_offset(oldpage));
+	create_page_chain(class, zspage, zpdescs);
+	first_obj_offset = get_first_obj_offset(zpdesc_page(oldzpdesc));
+	set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset);
 	if (unlikely(ZsHugePage(zspage)))
-		newpage->index = oldpage->index;
-	__SetPageMovable(newpage, &zsmalloc_mops);
+		newzpdesc->handle = oldzpdesc->handle;
+	__zpdesc_set_movable(newzpdesc, &zsmalloc_mops);
 }
 
 static bool zs_page_isolate(struct page *page, isolate_mode_t mode)
@@ -1817,7 +1848,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	}
 	kunmap_local(s_addr);
 
-	replace_sub_page(class, zspage, newpage, page);
+	replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page));
 	/*
 	 * Since we complete the data copy and set up new zspage structure,
 	 * it's okay to release migration_lock.
-- 
2.43.5

[PATCH v9 mm-unstable 07/18] mm/zsmalloc: convert obj_allocated() and related helpers to use zpdesc

[Hyeonggon Yoo]

Convert obj_allocated(), and related helpers to take zpdesc. Also make
its callers to cast (struct page *) to (struct zpdesc *) when calling them.
The users will be converted gradually as there are many.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 59a30c61160f..a3e2e596b4f3 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -823,15 +823,15 @@ static unsigned long handle_to_obj(unsigned long handle)
 	return *(unsigned long *)handle;
 }
 
-static inline bool obj_allocated(struct page *page, void *obj,
+static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 				 unsigned long *phandle)
 {
 	unsigned long handle;
-	struct zspage *zspage = get_zspage(page);
+	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
 
 	if (unlikely(ZsHugePage(zspage))) {
-		VM_BUG_ON_PAGE(!is_first_page(page), page);
-		handle = page->index;
+		VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc));
+		handle = zpdesc->handle;
 	} else
 		handle = *(unsigned long *)obj;
 
@@ -1569,18 +1569,18 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
  * return handle.
  */
 static unsigned long find_alloced_obj(struct size_class *class,
-				      struct page *page, int *obj_idx)
+				      struct zpdesc *zpdesc, int *obj_idx)
 {
 	unsigned int offset;
 	int index = *obj_idx;
 	unsigned long handle = 0;
-	void *addr = kmap_local_page(page);
+	void *addr = kmap_local_zpdesc(zpdesc);
 
-	offset = get_first_obj_offset(page);
+	offset = get_first_obj_offset(zpdesc_page(zpdesc));
 	offset += class->size * index;
 
 	while (offset < PAGE_SIZE) {
-		if (obj_allocated(page, addr + offset, &handle))
+		if (obj_allocated(zpdesc, addr + offset, &handle))
 			break;
 
 		offset += class->size;
@@ -1604,7 +1604,7 @@ static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage,
 	struct size_class *class = pool->size_class[src_zspage->class];
 
 	while (1) {
-		handle = find_alloced_obj(class, s_page, &obj_idx);
+		handle = find_alloced_obj(class, page_zpdesc(s_page), &obj_idx);
 		if (!handle) {
 			s_page = get_next_page(s_page);
 			if (!s_page)
@@ -1837,7 +1837,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 	for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE;
 					addr += class->size) {
-		if (obj_allocated(page, addr, &handle)) {
+		if (obj_allocated(page_zpdesc(page), addr, &handle)) {
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-- 
2.43.5

[PATCH v9 mm-unstable 08/18] mm/zsmalloc: convert init_zspage() to use zpdesc

[Hyeonggon Yoo]

Replace get_first/next_page func series and kmap_atomic to new helper,
no functional change.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index a3e2e596b4f3..83d48cffe96f 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -925,16 +925,16 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 {
 	unsigned int freeobj = 1;
 	unsigned long off = 0;
-	struct page *page = get_first_page(zspage);
+	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
 
-	while (page) {
-		struct page *next_page;
+	while (zpdesc) {
+		struct zpdesc *next_zpdesc;
 		struct link_free *link;
 		void *vaddr;
 
-		set_first_obj_offset(page, off);
+		set_first_obj_offset(zpdesc_page(zpdesc), off);
 
-		vaddr = kmap_local_page(page);
+		vaddr = kmap_local_zpdesc(zpdesc);
 		link = (struct link_free *)vaddr + off / sizeof(*link);
 
 		while ((off += class->size) < PAGE_SIZE) {
@@ -947,8 +947,8 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		 * page, which must point to the first object on the next
 		 * page (if present)
 		 */
-		next_page = get_next_page(page);
-		if (next_page) {
+		next_zpdesc = get_next_zpdesc(zpdesc);
+		if (next_zpdesc) {
 			link->next = freeobj++ << OBJ_TAG_BITS;
 		} else {
 			/*
@@ -958,7 +958,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 			link->next = -1UL << OBJ_TAG_BITS;
 		}
 		kunmap_local(vaddr);
-		page = next_page;
+		zpdesc = next_zpdesc;
 		off %= PAGE_SIZE;
 	}
 
-- 
2.43.5

[PATCH v9 mm-unstable 09/18] mm/zsmalloc: convert obj_to_page() and zs_free() to use zpdesc

[Hyeonggon Yoo]

Rename obj_to_page() to obj_to_zpdesc() and also convert it and
its user zs_free() to use zpdesc.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 83d48cffe96f..112603f9449f 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -798,9 +798,9 @@ static void obj_to_location(unsigned long obj, struct zpdesc **zpdesc,
 	*obj_idx = (obj & OBJ_INDEX_MASK);
 }
 
-static void obj_to_page(unsigned long obj, struct page **page)
+static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 {
-	*page = pfn_to_page(obj >> OBJ_INDEX_BITS);
+	*zpdesc = pfn_zpdesc(obj >> OBJ_INDEX_BITS);
 }
 
 /**
@@ -1462,7 +1462,7 @@ static void obj_free(int class_size, unsigned long obj)
 void zs_free(struct zs_pool *pool, unsigned long handle)
 {
 	struct zspage *zspage;
-	struct page *f_page;
+	struct zpdesc *f_zpdesc;
 	unsigned long obj;
 	struct size_class *class;
 	int fullness;
@@ -1476,8 +1476,8 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	 */
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
-	obj_to_page(obj, &f_page);
-	zspage = get_zspage(f_page);
+	obj_to_zpdesc(obj, &f_zpdesc);
+	zspage = get_zspage(zpdesc_page(f_zpdesc));
 	class = zspage_class(pool, zspage);
 	spin_lock(&class->lock);
 	read_unlock(&pool->migrate_lock);
-- 
2.43.5

[PATCH v9 mm-unstable 10/18] mm/zsmalloc: add two helpers for zs_page_migrate() and make it use zpdesc

[Hyeonggon Yoo]

To convert page to zpdesc in zs_page_migrate(), we added
zpdesc_is_isolated()/zpdesc_zone() helpers. No functional change.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zpdesc.h   | 11 +++++++++++
 mm/zsmalloc.c | 30 ++++++++++++++++--------------
 2 files changed, 27 insertions(+), 14 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index 9aca8d307796..c7c52e05e737 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -154,4 +154,15 @@ static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
 {
 	__SetPageMovable(zpdesc_page(zpdesc), mops);
 }
+
+static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc)
+{
+	return PageIsolated(zpdesc_page(zpdesc));
+}
+
+static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc)
+{
+	return page_zone(zpdesc_page(zpdesc));
+}
+
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 112603f9449f..432e78e61d2e 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1796,19 +1796,21 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	struct size_class *class;
 	struct zspage *zspage;
 	struct zpdesc *dummy;
+	struct zpdesc *newzpdesc = page_zpdesc(newpage);
+	struct zpdesc *zpdesc = page_zpdesc(page);
 	void *s_addr, *d_addr, *addr;
 	unsigned int offset;
 	unsigned long handle;
 	unsigned long old_obj, new_obj;
 	unsigned int obj_idx;
 
-	VM_BUG_ON_PAGE(!PageIsolated(page), page);
+	VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc));
 
 	/* We're committed, tell the world that this is a Zsmalloc page. */
-	__SetPageZsmalloc(newpage);
+	__SetPageZsmalloc(zpdesc_page(newzpdesc));
 
 	/* The page is locked, so this pointer must remain valid */
-	zspage = get_zspage(page);
+	zspage = get_zspage(zpdesc_page(zpdesc));
 	pool = zspage->pool;
 
 	/*
@@ -1825,30 +1827,30 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	/* the migrate_write_lock protects zpage access via zs_map_object */
 	migrate_write_lock(zspage);
 
-	offset = get_first_obj_offset(page);
-	s_addr = kmap_local_page(page);
+	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	s_addr = kmap_local_zpdesc(zpdesc);
 
 	/*
 	 * Here, any user cannot access all objects in the zspage so let's move.
 	 */
-	d_addr = kmap_local_page(newpage);
+	d_addr = kmap_local_zpdesc(newzpdesc);
 	copy_page(d_addr, s_addr);
 	kunmap_local(d_addr);
 
 	for (addr = s_addr + offset; addr < s_addr + PAGE_SIZE;
 					addr += class->size) {
-		if (obj_allocated(page_zpdesc(page), addr, &handle)) {
+		if (obj_allocated(zpdesc, addr, &handle)) {
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(newpage,
+			new_obj = (unsigned long)location_to_obj(zpdesc_page(newzpdesc),
 								obj_idx);
 			record_obj(handle, new_obj);
 		}
 	}
 	kunmap_local(s_addr);
 
-	replace_sub_page(class, zspage, page_zpdesc(newpage), page_zpdesc(page));
+	replace_sub_page(class, zspage, newzpdesc, zpdesc);
 	/*
 	 * Since we complete the data copy and set up new zspage structure,
 	 * it's okay to release migration_lock.
@@ -1857,14 +1859,14 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	spin_unlock(&class->lock);
 	migrate_write_unlock(zspage);
 
-	get_page(newpage);
-	if (page_zone(newpage) != page_zone(page)) {
-		dec_zone_page_state(page, NR_ZSPAGES);
-		inc_zone_page_state(newpage, NR_ZSPAGES);
+	zpdesc_get(newzpdesc);
+	if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) {
+		zpdesc_dec_zone_page_state(zpdesc);
+		zpdesc_inc_zone_page_state(newzpdesc);
 	}
 
 	reset_page(page);
-	put_page(page);
+	zpdesc_put(zpdesc);
 
 	return MIGRATEPAGE_SUCCESS;
 }
-- 
2.43.5

[PATCH v9 mm-unstable 11/18] mm/zsmalloc: convert reset_page to reset_zpdesc

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

zpdesc.zspage matches with page.private, zpdesc.next matches with
page.index. They will be reset in reset_page() which is called prior to
free base pages of a zspage.

Since the fields that need to be initialized are independent of the
order in struct zpdesc, Keep it to use struct page to ensure robustness
against potential rearrangements of struct zpdesc fields in the future.

[42.hyeyoo: keep reset_zpdesc() to use struct page fields]
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zsmalloc.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 432e78e61d2e..dded6d1f3b7a 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -843,8 +843,10 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 	return true;
 }
 
-static void reset_page(struct page *page)
+static void reset_zpdesc(struct zpdesc *zpdesc)
 {
+	struct page *page = zpdesc_page(zpdesc);
+
 	__ClearPageMovable(page);
 	ClearPagePrivate(page);
 	set_page_private(page, 0);
@@ -887,7 +889,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 	do {
 		VM_BUG_ON_PAGE(!PageLocked(page), page);
 		next = get_next_page(page);
-		reset_page(page);
+		reset_zpdesc(page_zpdesc(page));
 		unlock_page(page);
 		dec_zone_page_state(page, NR_ZSPAGES);
 		put_page(page);
@@ -1865,7 +1867,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 		zpdesc_inc_zone_page_state(newzpdesc);
 	}
 
-	reset_page(page);
+	reset_zpdesc(zpdesc);
 	zpdesc_put(zpdesc);
 
 	return MIGRATEPAGE_SUCCESS;
-- 
2.43.5

[PATCH v9 mm-unstable 12/18] mm/zsmalloc: convert __free_zspage() to use zpdesc

[Hyeonggon Yoo]

Introduce zpdesc_is_locked() and convert __free_zspage() to use zpdesc.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zpdesc.h   |  4 ++++
 mm/zsmalloc.c | 20 ++++++++++----------
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index c7c52e05e737..fa80c50993c6 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -165,4 +165,8 @@ static inline struct zone *zpdesc_zone(struct zpdesc *zpdesc)
 	return page_zone(zpdesc_page(zpdesc));
 }
 
+static inline bool zpdesc_is_locked(struct zpdesc *zpdesc)
+{
+	return folio_test_locked(zpdesc_folio(zpdesc));
+}
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index dded6d1f3b7a..e1f501d51226 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -878,23 +878,23 @@ static int trylock_zspage(struct zspage *zspage)
 static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 				struct zspage *zspage)
 {
-	struct page *page, *next;
+	struct zpdesc *zpdesc, *next;
 
 	assert_spin_locked(&class->lock);
 
 	VM_BUG_ON(get_zspage_inuse(zspage));
 	VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0);
 
-	next = page = get_first_page(zspage);
+	next = zpdesc = get_first_zpdesc(zspage);
 	do {
-		VM_BUG_ON_PAGE(!PageLocked(page), page);
-		next = get_next_page(page);
-		reset_zpdesc(page_zpdesc(page));
-		unlock_page(page);
-		dec_zone_page_state(page, NR_ZSPAGES);
-		put_page(page);
-		page = next;
-	} while (page != NULL);
+		VM_BUG_ON_PAGE(!zpdesc_is_locked(zpdesc), zpdesc_page(zpdesc));
+		next = get_next_zpdesc(zpdesc);
+		reset_zpdesc(zpdesc);
+		zpdesc_unlock(zpdesc);
+		zpdesc_dec_zone_page_state(zpdesc);
+		zpdesc_put(zpdesc);
+		zpdesc = next;
+	} while (zpdesc != NULL);
 
 	cache_free_zspage(pool, zspage);
 
-- 
2.43.5

[PATCH v9 mm-unstable 13/18] mm/zsmalloc: convert location_to_obj() to take zpdesc

[Hyeonggon Yoo]

As all users of location_to_obj() now use zpdesc, convert
location_to_obj() to take zpdesc.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index e1f501d51226..37212964a365 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -804,15 +804,15 @@ static void obj_to_zpdesc(unsigned long obj, struct zpdesc **zpdesc)
 }
 
 /**
- * location_to_obj - get obj value encoded from (<page>, <obj_idx>)
- * @page: page object resides in zspage
+ * location_to_obj - get obj value encoded from (<zpdesc>, <obj_idx>)
+ * @zpdesc: zpdesc object resides in zspage
  * @obj_idx: object index
  */
-static unsigned long location_to_obj(struct page *page, unsigned int obj_idx)
+static unsigned long location_to_obj(struct zpdesc *zpdesc, unsigned int obj_idx)
 {
 	unsigned long obj;
 
-	obj = page_to_pfn(page) << OBJ_INDEX_BITS;
+	obj = zpdesc_pfn(zpdesc) << OBJ_INDEX_BITS;
 	obj |= obj_idx & OBJ_INDEX_MASK;
 
 	return obj;
@@ -1358,7 +1358,7 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	kunmap_local(vaddr);
 	mod_zspage_inuse(zspage, 1);
 
-	obj = location_to_obj(zpdesc_page(m_zpdesc), obj);
+	obj = location_to_obj(m_zpdesc, obj);
 	record_obj(handle, obj);
 
 	return obj;
@@ -1845,8 +1845,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 
 			old_obj = handle_to_obj(handle);
 			obj_to_location(old_obj, &dummy, &obj_idx);
-			new_obj = (unsigned long)location_to_obj(zpdesc_page(newzpdesc),
-								obj_idx);
+			new_obj = (unsigned long)location_to_obj(newzpdesc, obj_idx);
 			record_obj(handle, new_obj);
 		}
 	}
-- 
2.43.5

[PATCH v9 mm-unstable 14/18] mm/zsmalloc: convert migrate_zspage() to use zpdesc

[Hyeonggon Yoo]

Use get_first_zpdesc/get_next_zpdesc to replace get_first/next_page. No
functional change.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 37212964a365..19c1ca3957f2 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1602,14 +1602,14 @@ static void migrate_zspage(struct zs_pool *pool, struct zspage *src_zspage,
 	unsigned long used_obj, free_obj;
 	unsigned long handle;
 	int obj_idx = 0;
-	struct page *s_page = get_first_page(src_zspage);
+	struct zpdesc *s_zpdesc = get_first_zpdesc(src_zspage);
 	struct size_class *class = pool->size_class[src_zspage->class];
 
 	while (1) {
-		handle = find_alloced_obj(class, page_zpdesc(s_page), &obj_idx);
+		handle = find_alloced_obj(class, s_zpdesc, &obj_idx);
 		if (!handle) {
-			s_page = get_next_page(s_page);
-			if (!s_page)
+			s_zpdesc = get_next_zpdesc(s_zpdesc);
+			if (!s_zpdesc)
 				break;
 			obj_idx = 0;
 			continue;
-- 
2.43.5

[PATCH v9 mm-unstable 15/18] mm/zsmalloc: convert get_zspage() to take zpdesc

[Hyeonggon Yoo]

Now that all users except get_next_page() (which will be removed in
later patch) use zpdesc, convert get_zspage() to take zpdesc instead
of page.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
---
 mm/zsmalloc.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 19c1ca3957f2..a1a620192596 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -757,9 +757,9 @@ static int fix_fullness_group(struct size_class *class, struct zspage *zspage)
 	return newfg;
 }
 
-static struct zspage *get_zspage(struct page *page)
+static struct zspage *get_zspage(struct zpdesc *zpdesc)
 {
-	struct zspage *zspage = (struct zspage *)page_private(page);
+	struct zspage *zspage = zpdesc->zspage;
 
 	BUG_ON(zspage->magic != ZSPAGE_MAGIC);
 	return zspage;
@@ -767,7 +767,7 @@ static struct zspage *get_zspage(struct page *page)
 
 static struct page *get_next_page(struct page *page)
 {
-	struct zspage *zspage = get_zspage(page);
+	struct zspage *zspage = get_zspage(page_zpdesc(page));
 
 	if (unlikely(ZsHugePage(zspage)))
 		return NULL;
@@ -777,7 +777,7 @@ static struct page *get_next_page(struct page *page)
 
 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 {
-	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
+	struct zspage *zspage = get_zspage(zpdesc);
 
 	if (unlikely(ZsHugePage(zspage)))
 		return NULL;
@@ -827,7 +827,7 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
 				 unsigned long *phandle)
 {
 	unsigned long handle;
-	struct zspage *zspage = get_zspage(zpdesc_page(zpdesc));
+	struct zspage *zspage = get_zspage(zpdesc);
 
 	if (unlikely(ZsHugePage(zspage))) {
 		VM_BUG_ON_PAGE(!is_first_zpdesc(zpdesc), zpdesc_page(zpdesc));
@@ -1232,7 +1232,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle,
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
 	obj_to_location(obj, &zpdesc, &obj_idx);
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 
 	/*
 	 * migration cannot move any zpages in this zspage. Here, class->lock
@@ -1282,7 +1282,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle)
 
 	obj = handle_to_obj(handle);
 	obj_to_location(obj, &zpdesc, &obj_idx);
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 	class = zspage_class(pool, zspage);
 	off = offset_in_page(class->size * obj_idx);
 
@@ -1445,7 +1445,7 @@ static void obj_free(int class_size, unsigned long obj)
 
 	obj_to_location(obj, &f_zpdesc, &f_objidx);
 	f_offset = offset_in_page(class_size * f_objidx);
-	zspage = get_zspage(zpdesc_page(f_zpdesc));
+	zspage = get_zspage(f_zpdesc);
 
 	vaddr = kmap_local_zpdesc(f_zpdesc);
 	link = (struct link_free *)(vaddr + f_offset);
@@ -1479,7 +1479,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle)
 	read_lock(&pool->migrate_lock);
 	obj = handle_to_obj(handle);
 	obj_to_zpdesc(obj, &f_zpdesc);
-	zspage = get_zspage(zpdesc_page(f_zpdesc));
+	zspage = get_zspage(f_zpdesc);
 	class = zspage_class(pool, zspage);
 	spin_lock(&class->lock);
 	read_unlock(&pool->migrate_lock);
@@ -1812,7 +1812,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	__SetPageZsmalloc(zpdesc_page(newzpdesc));
 
 	/* The page is locked, so this pointer must remain valid */
-	zspage = get_zspage(zpdesc_page(zpdesc));
+	zspage = get_zspage(zpdesc);
 	pool = zspage->pool;
 
 	/*
-- 
2.43.5

[PATCH v9 mm-unstable 16/18] mm/zsmalloc: convert SetZsPageMovable and remove unused funcs

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

Convert SetZsPageMovable() to use zpdesc, and then remove unused
funcs: get_next_page()/get_first_page()/is_first_page().

Originally-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zsmalloc.c | 33 +++++----------------------------
 1 file changed, 5 insertions(+), 28 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index a1a620192596..1801dce2f7ca 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -452,11 +452,6 @@ static DEFINE_PER_CPU(struct mapping_area, zs_map_area) = {
 	.lock	= INIT_LOCAL_LOCK(lock),
 };
 
-static __maybe_unused int is_first_page(struct page *page)
-{
-	return PagePrivate(page);
-}
-
 static inline bool is_first_zpdesc(struct zpdesc *zpdesc)
 {
 	return PagePrivate(zpdesc_page(zpdesc));
@@ -473,14 +468,6 @@ static inline void mod_zspage_inuse(struct zspage *zspage, int val)
 	zspage->inuse += val;
 }
 
-static inline struct page *get_first_page(struct zspage *zspage)
-{
-	struct page *first_page = zpdesc_page(zspage->first_zpdesc);
-
-	VM_BUG_ON_PAGE(!is_first_page(first_page), first_page);
-	return first_page;
-}
-
 static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
 {
 	struct zpdesc *first_zpdesc = zspage->first_zpdesc;
@@ -765,16 +752,6 @@ static struct zspage *get_zspage(struct zpdesc *zpdesc)
 	return zspage;
 }
 
-static struct page *get_next_page(struct page *page)
-{
-	struct zspage *zspage = get_zspage(page_zpdesc(page));
-
-	if (unlikely(ZsHugePage(zspage)))
-		return NULL;
-
-	return (struct page *)page->index;
-}
-
 static struct zpdesc *get_next_zpdesc(struct zpdesc *zpdesc)
 {
 	struct zspage *zspage = get_zspage(zpdesc);
@@ -1936,13 +1913,13 @@ static void init_deferred_free(struct zs_pool *pool)
 
 static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage)
 {
-	struct page *page = get_first_page(zspage);
+	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
 
 	do {
-		WARN_ON(!trylock_page(page));
-		__SetPageMovable(page, &zsmalloc_mops);
-		unlock_page(page);
-	} while ((page = get_next_page(page)) != NULL);
+		WARN_ON(!zpdesc_trylock(zpdesc));
+		__zpdesc_set_movable(zpdesc, &zsmalloc_mops);
+		zpdesc_unlock(zpdesc);
+	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
 }
 #else
 static inline void zs_flush_migration(struct zs_pool *pool) { }
-- 
2.43.5

[PATCH v9 mm-unstable 17/18] mm/zsmalloc: convert get/set_first_obj_offset() to take zpdesc

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

Now that all users of get/set_first_obj_offset() are converted
to use zpdesc, convert them to take zpdesc.

Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zsmalloc.c | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 1801dce2f7ca..3a841e16746e 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -478,20 +478,20 @@ static struct zpdesc *get_first_zpdesc(struct zspage *zspage)
 
 #define FIRST_OBJ_PAGE_TYPE_MASK	0xffffff
 
-static inline unsigned int get_first_obj_offset(struct page *page)
+static inline unsigned int get_first_obj_offset(struct zpdesc *zpdesc)
 {
-	VM_WARN_ON_ONCE(!PageZsmalloc(page));
-	return page->page_type & FIRST_OBJ_PAGE_TYPE_MASK;
+	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
+	return zpdesc->first_obj_offset & FIRST_OBJ_PAGE_TYPE_MASK;
 }
 
-static inline void set_first_obj_offset(struct page *page, unsigned int offset)
+static inline void set_first_obj_offset(struct zpdesc *zpdesc, unsigned int offset)
 {
 	/* With 24 bits available, we can support offsets into 16 MiB pages. */
 	BUILD_BUG_ON(PAGE_SIZE > SZ_16M);
-	VM_WARN_ON_ONCE(!PageZsmalloc(page));
+	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
 	VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK);
-	page->page_type &= ~FIRST_OBJ_PAGE_TYPE_MASK;
-	page->page_type |= offset & FIRST_OBJ_PAGE_TYPE_MASK;
+	zpdesc->first_obj_offset &= ~FIRST_OBJ_PAGE_TYPE_MASK;
+	zpdesc->first_obj_offset |= offset & FIRST_OBJ_PAGE_TYPE_MASK;
 }
 
 static inline unsigned int get_freeobj(struct zspage *zspage)
@@ -911,7 +911,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		struct link_free *link;
 		void *vaddr;
 
-		set_first_obj_offset(zpdesc_page(zpdesc), off);
+		set_first_obj_offset(zpdesc, off);
 
 		vaddr = kmap_local_zpdesc(zpdesc);
 		link = (struct link_free *)vaddr + off / sizeof(*link);
@@ -1555,7 +1555,7 @@ static unsigned long find_alloced_obj(struct size_class *class,
 	unsigned long handle = 0;
 	void *addr = kmap_local_zpdesc(zpdesc);
 
-	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	offset = get_first_obj_offset(zpdesc);
 	offset += class->size * index;
 
 	while (offset < PAGE_SIZE) {
@@ -1750,8 +1750,8 @@ static void replace_sub_page(struct size_class *class, struct zspage *zspage,
 	} while ((zpdesc = get_next_zpdesc(zpdesc)) != NULL);
 
 	create_page_chain(class, zspage, zpdescs);
-	first_obj_offset = get_first_obj_offset(zpdesc_page(oldzpdesc));
-	set_first_obj_offset(zpdesc_page(newzpdesc), first_obj_offset);
+	first_obj_offset = get_first_obj_offset(oldzpdesc);
+	set_first_obj_offset(newzpdesc, first_obj_offset);
 	if (unlikely(ZsHugePage(zspage)))
 		newzpdesc->handle = oldzpdesc->handle;
 	__zpdesc_set_movable(newzpdesc, &zsmalloc_mops);
@@ -1806,7 +1806,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	/* the migrate_write_lock protects zpage access via zs_map_object */
 	migrate_write_lock(zspage);
 
-	offset = get_first_obj_offset(zpdesc_page(zpdesc));
+	offset = get_first_obj_offset(zpdesc);
 	s_addr = kmap_local_zpdesc(zpdesc);
 
 	/*
-- 
2.43.5

[PATCH v9 mm-unstable 18/18] mm/zsmalloc: introduce __zpdesc_clear/set_zsmalloc()

[Hyeonggon Yoo]

From: Alex Shi <alexs@kernel.org>

Add helper __zpdesc_clear_zsmalloc() for __ClearPageZsmalloc(),
__zpdesc_set_zsmalloc() for __SetPageZsmalloc(), and use them in
callers.

[42.hyeyoo: keep reset_zpdesc() to use struct page]
Signed-off-by: Alex Shi <alexs@kernel.org>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zpdesc.h   | 10 ++++++++++
 mm/zsmalloc.c |  6 +++---
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/mm/zpdesc.h b/mm/zpdesc.h
index fa80c50993c6..2da58339ac5b 100644
--- a/mm/zpdesc.h
+++ b/mm/zpdesc.h
@@ -155,6 +155,16 @@ static inline void __zpdesc_set_movable(struct zpdesc *zpdesc,
 	__SetPageMovable(zpdesc_page(zpdesc), mops);
 }
 
+static inline void __zpdesc_set_zsmalloc(struct zpdesc *zpdesc)
+{
+	__SetPageZsmalloc(zpdesc_page(zpdesc));
+}
+
+static inline void __zpdesc_clear_zsmalloc(struct zpdesc *zpdesc)
+{
+	__ClearPageZsmalloc(zpdesc_page(zpdesc));
+}
+
 static inline bool zpdesc_is_isolated(struct zpdesc *zpdesc)
 {
 	return PageIsolated(zpdesc_page(zpdesc));
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 3a841e16746e..dae32e051779 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1001,13 +1001,13 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 		if (!zpdesc) {
 			while (--i >= 0) {
 				zpdesc_dec_zone_page_state(zpdescs[i]);
-				__ClearPageZsmalloc(zpdesc_page(zpdescs[i]));
+				__zpdesc_clear_zsmalloc(zpdescs[i]);
 				free_zpdesc(zpdescs[i]);
 			}
 			cache_free_zspage(pool, zspage);
 			return NULL;
 		}
-		__SetPageZsmalloc(zpdesc_page(zpdesc));
+		__zpdesc_set_zsmalloc(zpdesc);
 
 		zpdesc_inc_zone_page_state(zpdesc);
 		zpdescs[i] = zpdesc;
@@ -1786,7 +1786,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
 	VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc));
 
 	/* We're committed, tell the world that this is a Zsmalloc page. */
-	__SetPageZsmalloc(zpdesc_page(newzpdesc));
+	__zpdesc_set_zsmalloc(newzpdesc);
 
 	/* The page is locked, so this pointer must remain valid */
 	zspage = get_zspage(zpdesc);
-- 
2.43.5

Re: [PATCH v9 mm-unstable 00/18] Add zpdesc memory descriptor for zswap.zpool

[Sergey Senozhatsky]

On (24/12/17 00:04), Hyeonggon Yoo wrote:
> This patch series introduces a new memory descriptor for zswap.zpool that
> currently overlaps with struct page for now. This is part of the effort
> to reduce the size of struct page and to enable dynamic allocation of
> memory descriptors [1].
> 
> This series does not bloat anything for zsmalloc and no functional
> change is intended (except for using zpdesc and folios).
> 
> In the near future, the removal of page->index from struct page [2]
> will be addressed and the project also depends on this patch series.
> 
> I think this series is now ready to be included in mm-unstable if there's
> no objection. Sergey thankfully added Reviewed-by and Tested-by tags on v8.
> But as I updated the patchset, could you please explicitly add them for v9
> as well? A range-diff output is included at the end of this cover letter
> to help review.
> 
> Thanks to everyone got involved in this series, especially, Alex who's been
> pushing it forward this year.
> 
> v8: https://lore.kernel.org/linux-mm/20241205175000.3187069-1-willy@infradead.org
> [1] https://lore.kernel.org/linux-mm/ZvRKzKizOfEWBtJp@casper.infradead.org
> [2] https://lore.kernel.org/linux-mm/Z09hOy-UY9KC8WMb@casper.infradead.org
> 
> v8 -> v9:
> Functionally very little change and most of them are comment/changelog
> updates.

Huge thanks to everyone.

FWIW
Acked-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Tested-by: Sergey Senozhatsky <senozhatsky@chromium.org>

Re: [PATCH v9 mm-unstable 11/18] mm/zsmalloc: convert reset_page to reset_zpdesc

[Matthew Wilcox]

On Tue, Dec 17, 2024 at 12:04:42AM +0900, Hyeonggon Yoo wrote:
> From: Alex Shi <alexs@kernel.org>
> 
> zpdesc.zspage matches with page.private, zpdesc.next matches with
> page.index. They will be reset in reset_page() which is called prior to
> free base pages of a zspage.
> 
> Since the fields that need to be initialized are independent of the
> order in struct zpdesc, Keep it to use struct page to ensure robustness
> against potential rearrangements of struct zpdesc fields in the future.
> 
> [42.hyeyoo: keep reset_zpdesc() to use struct page fields]

Ummm ... why did you make this change?  Now page->index still has a user
in zsmalloc ;-(

> Signed-off-by: Alex Shi <alexs@kernel.org>
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> ---
>  mm/zsmalloc.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
> index 432e78e61d2e..dded6d1f3b7a 100644
> --- a/mm/zsmalloc.c
> +++ b/mm/zsmalloc.c
> @@ -843,8 +843,10 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
>  	return true;
>  }
>  
> -static void reset_page(struct page *page)
> +static void reset_zpdesc(struct zpdesc *zpdesc)
>  {
> +	struct page *page = zpdesc_page(zpdesc);
> +
>  	__ClearPageMovable(page);
>  	ClearPagePrivate(page);
>  	set_page_private(page, 0);
> @@ -887,7 +889,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
>  	do {
>  		VM_BUG_ON_PAGE(!PageLocked(page), page);
>  		next = get_next_page(page);
> -		reset_page(page);
> +		reset_zpdesc(page_zpdesc(page));
>  		unlock_page(page);
>  		dec_zone_page_state(page, NR_ZSPAGES);
>  		put_page(page);
> @@ -1865,7 +1867,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
>  		zpdesc_inc_zone_page_state(newzpdesc);
>  	}
>  
> -	reset_page(page);
> +	reset_zpdesc(zpdesc);
>  	zpdesc_put(zpdesc);
>  
>  	return MIGRATEPAGE_SUCCESS;
> -- 
> 2.43.5
> 

Re: [PATCH v9 mm-unstable 11/18] mm/zsmalloc: convert reset_page to reset_zpdesc

[Hyeonggon Yoo]

On 1/10/2025 1:43 PM, Matthew Wilcox wrote:
> On Tue, Dec 17, 2024 at 12:04:42AM +0900, Hyeonggon Yoo wrote:
>> From: Alex Shi <alexs@kernel.org>
>>
>> zpdesc.zspage matches with page.private, zpdesc.next matches with
>> page.index. They will be reset in reset_page() which is called prior to
>> free base pages of a zspage.
>>
>> Since the fields that need to be initialized are independent of the
>> order in struct zpdesc, Keep it to use struct page to ensure robustness
>> against potential rearrangements of struct zpdesc fields in the future.
>>
>> [42.hyeyoo: keep reset_zpdesc() to use struct page fields]
> 
> Ummm ... why did you make this change?  Now page->index still has a user
> in zsmalloc ;-(

Ohhh.., my bad :/

I was thinking of a situation when someone re-arranges zpdesc fields
and forgets to reset the field that overlaps page->index now.
And when writing it did not consider that we're going to remove usages
of page->index anyway...

Let me fix this - will send an updated version of this patch or series
(whatever way that is convenient for You and Andrew)

---
Hyeonggon

>> Signed-off-by: Alex Shi <alexs@kernel.org>
>> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
>> ---
>>   mm/zsmalloc.c | 8 +++++---
>>   1 file changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
>> index 432e78e61d2e..dded6d1f3b7a 100644
>> --- a/mm/zsmalloc.c
>> +++ b/mm/zsmalloc.c
>> @@ -843,8 +843,10 @@ static inline bool obj_allocated(struct zpdesc *zpdesc, void *obj,
>>   	return true;
>>   }
>>   
>> -static void reset_page(struct page *page)
>> +static void reset_zpdesc(struct zpdesc *zpdesc)
>>   {
>> +	struct page *page = zpdesc_page(zpdesc);
>> +
>>   	__ClearPageMovable(page);
>>   	ClearPagePrivate(page);
>>   	set_page_private(page, 0);
>> @@ -887,7 +889,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
>>   	do {
>>   		VM_BUG_ON_PAGE(!PageLocked(page), page);
>>   		next = get_next_page(page);
>> -		reset_page(page);
>> +		reset_zpdesc(page_zpdesc(page));
>>   		unlock_page(page);
>>   		dec_zone_page_state(page, NR_ZSPAGES);
>>   		put_page(page);
>> @@ -1865,7 +1867,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page,
>>   		zpdesc_inc_zone_page_state(newzpdesc);
>>   	}
>>   
>> -	reset_page(page);
>> +	reset_zpdesc(zpdesc);
>>   	zpdesc_put(zpdesc);
>>   
>>   	return MIGRATEPAGE_SUCCESS;
>> -- 
>> 2.43.5
>>
> 

Re: [PATCH v9 mm-unstable 11/18] mm/zsmalloc: convert reset_page to reset_zpdesc

[Andrew Morton]

On Fri, 10 Jan 2025 15:08:44 +0900 Hyeonggon Yoo <hyeonggon.yoo@sk.com> wrote:

> >> [42.hyeyoo: keep reset_zpdesc() to use struct page fields]
> > 
> > Ummm ... why did you make this change?  Now page->index still has a user
> > in zsmalloc ;-(
> 
> Ohhh.., my bad :/
> 
> I was thinking of a situation when someone re-arranges zpdesc fields
> and forgets to reset the field that overlaps page->index now.
> And when writing it did not consider that we're going to remove usages
> of page->index anyway...
> 
> Let me fix this - will send an updated version of this patch or series
> (whatever way that is convenient for You and Andrew)

A little fixup patch would be preferred at this stage, please.  Or a
new version of this patch, from which I'll create a little fixup patch.

[PATCH v9 mm-unstable 19/19] mm/zsmalloc: reset zpdesc fields in reset_zpdesc()

[Hyeonggon Yoo]

To prepare for the future removal of struct page fields (e.g. the removal
of page->index [1]), update reset_zpdesc() to reset the descriptor
via struct zpdesc instead of struct page.

As struct zpdesc overlays struct page for now,
no functional change is intended.

[1] https://lore.kernel.org/linux-mm/Z09hOy-UY9KC8WMb@casper.infradead.org

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/zsmalloc.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index dae32e051779..817626a351f8 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -826,8 +826,8 @@ static void reset_zpdesc(struct zpdesc *zpdesc)
 
 	__ClearPageMovable(page);
 	ClearPagePrivate(page);
-	set_page_private(page, 0);
-	page->index = 0;
+	zpdesc->zspage = NULL;
+	zpdesc->next = NULL;
 	__ClearPageZsmalloc(page);
 }
 
-- 
2.43.5