From: "Huang, Ying" <ying.huang@intel.com>
To: Chris Li <chrisl@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Kairui Song <kasong@tencent.com>,
Hugh Dickins <hughd@google.com>,
Ryan Roberts <ryan.roberts@arm.com>,
Kalesh Singh <kaleshsingh@google.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
Barry Song <baohua@kernel.org>
Subject: Re: [PATCH v4 0/3] mm: swap: mTHP swap allocator base on swap cluster order
Date: Thu, 18 Jul 2024 13:50:25 +0800 [thread overview]
Message-ID: <87cynbxn8e.fsf@yhuang6-desk2.ccr.corp.intel.com> (raw)
In-Reply-To: <20240711-swap-allocator-v4-0-0295a4d4c7aa@kernel.org> (Chris Li's message of "Thu, 11 Jul 2024 00:29:04 -0700")
Chris Li <chrisl@kernel.org> writes:
> This is the short term solutions "swap cluster order" listed
> in my "Swap Abstraction" discussion slice 8 in the recent
> LSF/MM conference.
>
> When commit 845982eb264bc "mm: swap: allow storage of all mTHP
> orders" is introduced, it only allocates the mTHP swap entries
> from the new empty cluster list. It has a fragmentation issue
> reported by Barry.
>
> https://lore.kernel.org/all/CAGsJ_4zAcJkuW016Cfi6wicRr8N9X+GJJhgMQdSMp+Ah+NSgNQ@mail.gmail.com/
>
> The reason is that all the empty clusters have been exhausted while
> there are plenty of free swap entries in the cluster that are
> not 100% free.
>
> Remember the swap allocation order in the cluster.
> Keep track of the per order non full cluster list for later allocation.
>
> The patch 3 of this series gives the swap SSD allocation
> a new separate code path from the HDD allocation. The new allocator
> use cluster list only and do not global scan swap_map[] without lock
> any more.
>
> This streamline the swap allocation for SSD. The code matches the execution
> flow much better.
>
> User impact: For users that allocate and free mix order mTHP swapping,
> It greatly improves the success rate of the mTHP swap allocation after the
> initial phase.
>
> It also performs faster when the swapfile is close to full, because the
> allocator can get the non full cluster from a list rather than scanning
> a lot of swap_map entries.
>
> This series still lacks the swap cache reclaim feature. The reclaim series
> of patches are under development and testing right now. Will post the
> mail list soon. For this reason, the patch 3 is consider RFC and not
> ready to merge.
>
> With Barry's mthp test program V2:
>
> Without:
> $ ./thp_swap_allocator_test -a
> Iteration 1: swpout inc: 32, swpout fallback inc: 192, Fallback percentage: 85.71%
> Iteration 2: swpout inc: 0, swpout fallback inc: 231, Fallback percentage: 100.00%
> Iteration 3: swpout inc: 0, swpout fallback inc: 227, Fallback percentage: 100.00%
> ...
> Iteration 98: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00%
> Iteration 99: swpout inc: 0, swpout fallback inc: 215, Fallback percentage: 100.00%
> Iteration 100: swpout inc: 0, swpout fallback inc: 222, Fallback percentage: 100.00%
>
> $ ./thp_swap_allocator_test -a -s
> Iteration 1: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00%
> Iteration 2: swpout inc: 0, swpout fallback inc: 218, Fallback percentage: 100.00%
> Iteration 3: swpout inc: 0, swpout fallback inc: 222, Fallback percentage: 100.00%
> ..
> Iteration 98: swpout inc: 0, swpout fallback inc: 228, Fallback percentage: 100.00%
> Iteration 99: swpout inc: 0, swpout fallback inc: 230, Fallback percentage: 100.00%
> Iteration 100: swpout inc: 0, swpout fallback inc: 229, Fallback percentage: 100.00%
>
> $ ./thp_swap_allocator_test -s
> Iteration 1: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00%
> Iteration 2: swpout inc: 0, swpout fallback inc: 218, Fallback percentage: 100.00%
> Iteration 3: swpout inc: 0, swpout fallback inc: 222, Fallback percentage: 100.00%
> ..
> Iteration 98: swpout inc: 0, swpout fallback inc: 228, Fallback percentage: 100.00%
> Iteration 99: swpout inc: 0, swpout fallback inc: 230, Fallback percentage: 100.00%
> Iteration 100: swpout inc: 0, swpout fallback inc: 229, Fallback percentage: 100.00%
>
> $ ./thp_swap_allocator_test
> Iteration 1: swpout inc: 0, swpout fallback inc: 224, Fallback percentage: 100.00%
> Iteration 2: swpout inc: 0, swpout fallback inc: 218, Fallback percentage: 100.00%
> Iteration 3: swpout inc: 0, swpout fallback inc: 222, Fallback percentage: 100.00%
> ..
> Iteration 98: swpout inc: 0, swpout fallback inc: 228, Fallback percentage: 100.00%
> Iteration 99: swpout inc: 0, swpout fallback inc: 230, Fallback percentage: 100.00%
> Iteration 100: swpout inc: 0, swpout fallback inc: 229, Fallback percentage: 100.00%
>
> With:
> $ ./thp_swap_allocator_test -a
> Iteration 1: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 2: swpout inc: 231, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 3: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00%
> ...
> Iteration 98: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 99: swpout inc: 215, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 100: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
>
> $ ./thp_swap_allocator_test -a -s
> Iteration 1: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 2: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 3: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 4: swpout inc: 226, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 5: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 6: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 7: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 8: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 9: swpout inc: 217, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 10: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 11: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 12: swpout inc: 232, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 13: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 14: swpout inc: 223, swpout fallback inc: 3, Fallback percentage: 1.33%
> Iteration 15: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 16: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 17: swpout inc: 212, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 18: swpout inc: 234, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 19: swpout inc: 220, swpout fallback inc: 6, Fallback percentage: 2.65%
> Iteration 20: swpout inc: 231, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 21: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 22: swpout inc: 226, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 23: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 24: swpout inc: 232, swpout fallback inc: 1, Fallback percentage: 0.43%
> Iteration 25: swpout inc: 215, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 26: swpout inc: 230, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 27: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 28: swpout inc: 225, swpout fallback inc: 1, Fallback percentage: 0.44%
> Iteration 29: swpout inc: 226, swpout fallback inc: 2, Fallback percentage: 0.88%
> Iteration 30: swpout inc: 224, swpout fallback inc: 1, Fallback percentage: 0.44%
> Iteration 31: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 32: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 33: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 34: swpout inc: 226, swpout fallback inc: 2, Fallback percentage: 0.88%
> Iteration 35: swpout inc: 230, swpout fallback inc: 3, Fallback percentage: 1.29%
> Iteration 36: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 37: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 38: swpout inc: 221, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 39: swpout inc: 229, swpout fallback inc: 1, Fallback percentage: 0.43%
> Iteration 40: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 41: swpout inc: 231, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 42: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 43: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 44: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 45: swpout inc: 221, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 46: swpout inc: 221, swpout fallback inc: 2, Fallback percentage: 0.90%
> Iteration 47: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 48: swpout inc: 220, swpout fallback inc: 1, Fallback percentage: 0.45%
> Iteration 49: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 50: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 51: swpout inc: 224, swpout fallback inc: 2, Fallback percentage: 0.88%
> Iteration 52: swpout inc: 229, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 53: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 54: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 55: swpout inc: 226, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 56: swpout inc: 226, swpout fallback inc: 2, Fallback percentage: 0.88%
> Iteration 57: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 58: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 59: swpout inc: 224, swpout fallback inc: 1, Fallback percentage: 0.44%
> Iteration 60: swpout inc: 229, swpout fallback inc: 1, Fallback percentage: 0.43%
> Iteration 61: swpout inc: 217, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 62: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 63: swpout inc: 223, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 64: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 65: swpout inc: 226, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 66: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 67: swpout inc: 220, swpout fallback inc: 2, Fallback percentage: 0.90%
> Iteration 68: swpout inc: 224, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 69: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 70: swpout inc: 219, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 71: swpout inc: 225, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 72: swpout inc: 231, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 73: swpout inc: 218, swpout fallback inc: 5, Fallback percentage: 2.24%
> Iteration 74: swpout inc: 223, swpout fallback inc: 5, Fallback percentage: 2.19%
> Iteration 75: swpout inc: 222, swpout fallback inc: 7, Fallback percentage: 3.06%
> Iteration 76: swpout inc: 226, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 77: swpout inc: 229, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 78: swpout inc: 215, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 79: swpout inc: 223, swpout fallback inc: 2, Fallback percentage: 0.89%
> Iteration 80: swpout inc: 222, swpout fallback inc: 1, Fallback percentage: 0.45%
> Iteration 81: swpout inc: 218, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 82: swpout inc: 228, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 83: swpout inc: 229, swpout fallback inc: 1, Fallback percentage: 0.43%
> Iteration 84: swpout inc: 222, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 85: swpout inc: 213, swpout fallback inc: 1, Fallback percentage: 0.47%
> Iteration 86: swpout inc: 215, swpout fallback inc: 8, Fallback percentage: 3.59%
> Iteration 87: swpout inc: 222, swpout fallback inc: 1, Fallback percentage: 0.45%
> Iteration 88: swpout inc: 227, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 89: swpout inc: 222, swpout fallback inc: 6, Fallback percentage: 2.63%
> Iteration 90: swpout inc: 224, swpout fallback inc: 1, Fallback percentage: 0.44%
> Iteration 91: swpout inc: 214, swpout fallback inc: 1, Fallback percentage: 0.47%
> Iteration 92: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 93: swpout inc: 221, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 94: swpout inc: 223, swpout fallback inc: 2, Fallback percentage: 0.89%
> Iteration 95: swpout inc: 222, swpout fallback inc: 1, Fallback percentage: 0.45%
> Iteration 96: swpout inc: 223, swpout fallback inc: 4, Fallback percentage: 1.76%
> Iteration 97: swpout inc: 223, swpout fallback inc: 7, Fallback percentage: 3.04%
> Iteration 98: swpout inc: 227, swpout fallback inc: 1, Fallback percentage: 0.44%
> Iteration 99: swpout inc: 229, swpout fallback inc: 1, Fallback percentage: 0.43%
> Iteration 100: swpout inc: 229, swpout fallback inc: 0, Fallback percentage: 0.00%
>
> $ ./thp_swap_allocator_test
> Iteration 1: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 2: swpout inc: 134, swpout fallback inc: 98, Fallback percentage: 42.24%
> Iteration 3: swpout inc: 72, swpout fallback inc: 154, Fallback percentage: 68.14%
> Iteration 4: swpout inc: 40, swpout fallback inc: 183, Fallback percentage: 82.06%
> Iteration 5: swpout inc: 27, swpout fallback inc: 199, Fallback percentage: 88.05%
> Iteration 6: swpout inc: 22, swpout fallback inc: 202, Fallback percentage: 90.18%
> Iteration 7: swpout inc: 12, swpout fallback inc: 216, Fallback percentage: 94.74%
> Iteration 8: swpout inc: 14, swpout fallback inc: 214, Fallback percentage: 93.86%
> Iteration 9: swpout inc: 5, swpout fallback inc: 221, Fallback percentage: 97.79%
> Iteration 10: swpout inc: 10, swpout fallback inc: 218, Fallback percentage: 95.61%
> ...
> Iteration 97: swpout inc: 12, swpout fallback inc: 207, Fallback percentage: 94.52%
> Iteration 98: swpout inc: 8, swpout fallback inc: 219, Fallback percentage: 96.48%
> Iteration 99: swpout inc: 16, swpout fallback inc: 218, Fallback percentage: 93.16%
> Iteration 100: swpout inc: 10, swpout fallback inc: 218, Fallback percentage: 95.61%
>
> $ ./thp_swap_allocator_test -s
> Iteration 1: swpout inc: 233, swpout fallback inc: 0, Fallback percentage: 0.00%
> Iteration 2: swpout inc: 84, swpout fallback inc: 148, Fallback percentage: 63.79%
> Iteration 3: swpout inc: 39, swpout fallback inc: 195, Fallback percentage: 83.33%
> Iteration 4: swpout inc: 16, swpout fallback inc: 217, Fallback percentage: 93.13%
> Iteration 5: swpout inc: 11, swpout fallback inc: 214, Fallback percentage: 95.11%
> Iteration 6: swpout inc: 10, swpout fallback inc: 218, Fallback percentage: 95.61%
> ...
> Iteration 96: swpout inc: 5, swpout fallback inc: 225, Fallback percentage: 97.83%
> Iteration 97: swpout inc: 2, swpout fallback inc: 215, Fallback percentage: 99.08%
> Iteration 98: swpout inc: 2, swpout fallback inc: 220, Fallback percentage: 99.10%
> Iteration 99: swpout inc: 4, swpout fallback inc: 222, Fallback percentage: 98.23%
> Iteration 100: swpout inc: 3, swpout fallback inc: 221, Fallback percentage: 98.66%
>
> Kernel compile under tmpfs with cgroup memory.max = 2G.
> 12 core 24 hyperthreading, 32 jobs.
>
> HDD swap 3 runs average, 20G swap file:
>
> Without:
> user 4186.290
> system 421.743
> real 597.317
>
> With:
> user 4113.897
> system 413.123
> real 659.543
>
> SSD swap 10 runs average, 20G swap partition:
>
> Without:
> user 4736.810
> system 500.921
> real 250.243
>
> With:
> user 4729.478
> system 500.265
> real 249.633
>
> Two zram swap:
> zram0 1.4G zram1 20G.
> The idea is forcing the zram0 almost
> full then overflow to zram1:
>
> Two zram 10 runs average:
>
> Without:
> user 4600.693
> system 384.105
> real 238.735
>
> With:
> user 4604.502
> system 382.087
> real 239.063
>
> Reported-by: Barry Song <21cnbao@gmail.com>
> Signed-off-by: Chris Li <chrisl@kernel.org>
> ---
> Changes in v4:
> - Remove a warning in patch 2.
> - Allocating from the free cluster list before the nonfull list. Revert the v3 behavior.
> - Add cluster_index and cluster_offset function.
> - Patch 3 has a new allocating path for SSD.
> - HDD swap allocation does not need to consider clusters any more.
It appears that my comments in the following emails are ignored?
https://lore.kernel.org/linux-mm/87bk3pzr5p.fsf@yhuang6-desk2.ccr.corp.intel.com/
https://lore.kernel.org/linux-mm/874j9hzqr3.fsf@yhuang6-desk2.ccr.corp.intel.com/
> changes in v3:
> - Using V1 as base.
> - Rename "next" to "list" for the list field, suggested by Ying.
> - Update comment for the locking rules for cluster fields and list,
> suggested by Ying.
> - Allocate from the nonfull list before attempting free list, suggested
> by Kairui.
> - Link to v2: https://lore.kernel.org/r/20240614-swap-allocator-v2-0-2a513b4a7f2f@kernel.org
>
> Changes in v2:
> - Abandoned.
> - Link to v1: https://lore.kernel.org/r/20240524-swap-allocator-v1-0-47861b423b26@kernel.org
>
> ---
> Chris Li (3):
> mm: swap: swap cluster switch to double link list
> mm: swap: mTHP allocate swap entries from nonfull list
> RFC: mm: swap: seperate SSD allocation from scan_swap_map_slots()
>
> include/linux/swap.h | 30 ++--
> mm/swapfile.c | 490 +++++++++++++++++++++++----------------------------
> 2 files changed, 238 insertions(+), 282 deletions(-)
> ---
> base-commit: ff3a648ecb9409aff1448cf4f6aa41d78c69a3bc
> change-id: 20240523-swap-allocator-1534c480ece4
>
--
Best Regards,
Huang, Ying
next prev parent reply other threads:[~2024-07-18 5:54 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-07-11 7:29 Chris Li
2024-07-11 7:29 ` [PATCH v4 1/3] mm: swap: swap cluster switch to double link list Chris Li
2024-07-15 14:57 ` Ryan Roberts
2024-07-16 22:11 ` Chris Li
2024-07-18 6:26 ` Huang, Ying
2024-07-26 5:46 ` Chris Li
2024-07-11 7:29 ` [PATCH v4 2/3] mm: swap: mTHP allocate swap entries from nonfull list Chris Li
2024-07-15 15:40 ` Ryan Roberts
2024-07-16 22:46 ` Chris Li
2024-07-17 10:14 ` Ryan Roberts
2024-07-17 15:41 ` Chris Li
2024-07-18 7:53 ` Huang, Ying
2024-07-19 10:30 ` Ryan Roberts
2024-07-22 2:14 ` Huang, Ying
2024-07-22 7:51 ` Ryan Roberts
2024-07-22 8:49 ` Huang, Ying
2024-07-22 9:54 ` Ryan Roberts
2024-07-23 6:27 ` Huang, Ying
2024-07-24 8:33 ` Ryan Roberts
2024-07-24 22:41 ` Chris Li
2024-07-25 6:43 ` Huang, Ying
2024-07-25 8:09 ` Chris Li
2024-07-26 2:09 ` Huang, Ying
2024-07-26 5:09 ` Chris Li
2024-07-26 6:02 ` Huang, Ying
2024-07-26 7:15 ` Chris Li
2024-07-26 7:42 ` Huang, Ying
2024-07-25 6:53 ` Huang, Ying
2024-07-25 8:26 ` Chris Li
2024-07-26 2:04 ` Huang, Ying
2024-07-26 4:50 ` Chris Li
2024-07-26 5:52 ` Huang, Ying
2024-07-26 7:10 ` Chris Li
2024-07-26 7:18 ` Huang, Ying
2024-07-26 7:26 ` Chris Li
2024-07-26 7:37 ` Huang, Ying
2024-07-11 7:29 ` [PATCH v4 3/3] RFC: mm: swap: seperate SSD allocation from scan_swap_map_slots() Chris Li
2024-07-11 10:02 ` [PATCH v4 0/3] mm: swap: mTHP swap allocator base on swap cluster order Ryan Roberts
2024-07-11 14:08 ` Chris Li
2024-07-15 14:10 ` Ryan Roberts
2024-07-15 18:14 ` Chris Li
2024-07-18 5:50 ` Huang, Ying [this message]
2024-07-26 5:51 ` Chris Li
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87cynbxn8e.fsf@yhuang6-desk2.ccr.corp.intel.com \
--to=ying.huang@intel.com \
--cc=akpm@linux-foundation.org \
--cc=baohua@kernel.org \
--cc=chrisl@kernel.org \
--cc=hughd@google.com \
--cc=kaleshsingh@google.com \
--cc=kasong@tencent.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=ryan.roberts@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox