linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Zhongkun He <hezhongkun.hzk@bytedance.com>
To: Johannes Weiner <hannes@cmpxchg.org>,
	Yosry Ahmed <yosryahmed@google.com>,
	 Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm <linux-mm@kvack.org>,
	wuyun.abel@bytedance.com,  zhouchengming@bytedance.com,
	Nhat Pham <nphamcs@gmail.com>
Subject: [bug report] mm/zswap :memory corruption after zswap_load().
Date: Thu, 21 Mar 2024 12:34:41 +0800	[thread overview]
Message-ID: <CACSyD1N+dUvsu8=zV9P691B9bVq33erwOXNTmEaUbi9DrDeJzw@mail.gmail.com> (raw)

Hey folks,

Recently, I tested the zswap with memory reclaiming in the mainline
(6.8) and found a memory corruption issue related to exclusive loads.


root@**:/sys/fs/cgroup/zz# stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep
stress: info: [31753] dispatching hogs: 0 cpu, 0 io, 5 vm, 0 hdd
stress: FAIL: [31758] (522) memory corruption at: 0x7f347ed1a010
stress: FAIL: [31753] (394) <-- worker 31758 returned error 1
stress: WARN: [31753] (396) now reaping child worker processes
stress: FAIL: [31753] (451) failed run completed in 14s


1. Test step(the frequency of memory reclaiming has been accelerated):
-------------------------
a. set up the zswap, zram and cgroup V2
b. echo 0 > /sys/kernel/mm/lru_gen/enabled
      (Increase the probability of problems occurring)
c.  mkdir /sys/fs/cgroup/zz
     echo $$  > /sys/fs/cgroup/zz/cgroup.procs
     cd  /sys/fs/cgroup/zz/
     stress --vm 5 --vm-bytes 1g --vm-hang 3 --vm-keep

e. in other shell:
   while :;do for i in {1..5};do echo 20g >
/sys/fs/cgroup/zz/memory.reclaim & done;sleep 1;done

2. Root cause:
--------------------------
With a small probability, the page fault will occur twice with the
original pte, even if a new pte has been successfully set.
Unfortunately, zswap_entry has been released during the first page fault
with exclusive loads, so zswap_load will fail, and there is no corresponding
data in swap space, memory corruption occurs.

bpftrace -e'k:zswap_load {printf("%lld, %lld\n", ((struct page
*)arg0)->private,nsecs)}'
--include linux/mm_types.h  > a.txt

look up the same index:

index            nsecs
1318876, 8976040736819
1318876, 8976040746078

4123110, 8976234682970
4123110, 8976234689736

2268896, 8976660124792
2268896, 8976660130607

4634105, 8976662117938
4634105, 8976662127596

3. Solution

Should we free zswap_entry in batches so that zswap_entry will be
valid when the next page fault occurs with the
original pte? It would be great if there are other better solutions.


             reply	other threads:[~2024-03-21  4:34 UTC|newest]

Thread overview: 30+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-03-21  4:34 Zhongkun He [this message]
2024-03-21  4:42 ` Chengming Zhou
2024-03-21  5:09   ` [External] " Zhongkun He
2024-03-21  5:24     ` Chengming Zhou
2024-03-21  6:36       ` Zhongkun He
2024-03-21  9:28         ` Chengming Zhou
2024-03-21 15:25           ` Nhat Pham
2024-03-21 18:32             ` Yosry Ahmed
2024-03-22  3:27               ` Chengming Zhou
2024-03-22  3:16             ` Zhongkun He
2024-03-22  3:04           ` Zhongkun He
2024-03-22 19:34             ` Yosry Ahmed
2024-03-22 23:04               ` Barry Song
2024-03-22 23:08                 ` Yosry Ahmed
2024-03-22 23:18                   ` Barry Song
2024-03-22 23:22                     ` Yosry Ahmed
2024-03-22 23:32                       ` Barry Song
2024-03-22 23:34                         ` Yosry Ahmed
2024-03-22 23:38                           ` Barry Song
2024-03-22 23:41                             ` Yosry Ahmed
2024-03-23  0:34                               ` Barry Song
2024-03-23  0:42                                 ` Yosry Ahmed
2024-03-23 10:48                                 ` Chris Li
2024-03-23 11:27                                   ` Chris Li
2024-03-23 12:41                                   ` Zhongkun He
2024-03-23  1:34               ` Zhongkun He
2024-03-23  1:36                 ` Yosry Ahmed
2024-03-23 10:52                 ` Chris Li
2024-03-23 10:55                   ` Barry Song
2024-03-28 15:45 Trisha Busch

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CACSyD1N+dUvsu8=zV9P691B9bVq33erwOXNTmEaUbi9DrDeJzw@mail.gmail.com' \
    --to=hezhongkun.hzk@bytedance.com \
    --cc=akpm@linux-foundation.org \
    --cc=hannes@cmpxchg.org \
    --cc=linux-mm@kvack.org \
    --cc=nphamcs@gmail.com \
    --cc=wuyun.abel@bytedance.com \
    --cc=yosryahmed@google.com \
    --cc=zhouchengming@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox