From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13811C52D7C for ; Thu, 15 Aug 2024 18:29:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B4256B015B; Thu, 15 Aug 2024 14:29:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 863796B015E; Thu, 15 Aug 2024 14:29:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72B948D0005; Thu, 15 Aug 2024 14:29:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4F8866B015B for ; Thu, 15 Aug 2024 14:29:19 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 14C711C53AB for ; Thu, 15 Aug 2024 18:29:19 +0000 (UTC) X-FDA: 82455317238.28.3A06EAF Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf21.hostedemail.com (Postfix) with ESMTP id 0BF461C000B for ; Thu, 15 Aug 2024 18:29:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ALq1RqNM; spf=pass (imf21.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723746483; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4HAC4rsTwT91LtGfC3aWTKc7iRuE79+JItbuf7AJ9AY=; b=ktq9PnQ/qsF8B2cn9hyxG2e6EEJoaqYWMQdJlZoDZoq08ezwtZvizRu3Mwi/srf/d0u+93 0afWbmilNdFgA31AeWOZdulW4iPuuncUTXmCfmm+LqSYxw1n93OMH5QoEr/qKFsNrNcM5B deg0UjMCSeFcStiBtrQbwfMqJyVG8YU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723746483; a=rsa-sha256; cv=none; b=Y9U6ye5QLqqOU+xIgUiBT2k/ehy0+swuxrYWViaLwfMfnNOQWjbjJdoKZDTGMebbhWHFeL U7RYwjwjX0TmzfQmf7UJ1spxTKkzSfOPiJlf+QaQfvPdZFrWrxYtgCZVhCqaVmtUNkZPt0 Ah4fDOA+gwbZ9N8gJ0nT5b5vTKlZbNQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=ALq1RqNM; spf=pass (imf21.hostedemail.com: domain of chrisl@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=chrisl@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id E778C61F8F for ; Thu, 15 Aug 2024 18:29:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EB15C4AF12 for ; Thu, 15 Aug 2024 18:29:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723746554; bh=h31zZOjFDlK55Fqr0SJYQuM0oWmJH9bcjbwnWnmM13w=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=ALq1RqNMn/XMhCL4dce8UuioBNT1CeNsbgNCGPKQqBu+YoTf+kiQ1+9P01R/YMsLU 0gOVhz7/39zk1Kj40IXb6AITrARQ7fSUqgki1erJKUZ1kxHmUmGQxvLL/Ls+PBGHqP piZniysUdBQYpYoztCtM8YJb/H+oEwyq/DgshUnePLKi1n5q7kT6qxNl5PrNVppOja s8C0C5uyDE0NSGEE5v6DZvAP9v6ebwzjdbvbBwAQtN/zuWY5zK7tofY/M/FGgBMC1V u3j1w3GZImszHEhHbRgKdsWaAVyk/cw2ku8EUrxD8+W4BdTv4VA584Hm1wXxm0TEtF RDu+I5SXBz0vg== Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-651da7c1531so11881027b3.0 for ; Thu, 15 Aug 2024 11:29:14 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCURJ3SEWfJH/zKagX/GGJqoHm9wL4xIwQl3a8y/tveKTHLy0tBXzqzuSp8kiFaPC+B0jTVD1L5nmo7g7OnWKIB3KkI= X-Gm-Message-State: AOJu0Yy2HTea8Om71ZJX8OGzGufsv/5MyB9fHMYMecnZR3OTMyz07Jg5 GqY7LDO+QQ0hBVScalV0VuDGOBk5DLF0+1EhMNdqa/ub4wd/qS65uOGLZ06tFmVY/TQ5xOXgfal dc6hnCst+mARS8Idf5fWzVmIk6EUcmJzkrfQheA== X-Google-Smtp-Source: AGHT+IE9yHMXcrs2p2OFSk4mZ0uKUmCDuQP0kBih3PHrirKmSu4uSNJw5528bUEA6aRcIKVPHAoBbvJnEYRuifUsgSY= X-Received: by 2002:a05:690c:2c8d:b0:6b0:8f62:bc77 with SMTP id 00721157ae682-6b1bc5e461bmr4933687b3.41.1723746553605; Thu, 15 Aug 2024 11:29:13 -0700 (PDT) MIME-Version: 1.0 References: <20240807215859.57491-1-21cnbao@gmail.com> <20240807215859.57491-3-21cnbao@gmail.com> In-Reply-To: <20240807215859.57491-3-21cnbao@gmail.com> From: Chris Li Date: Thu, 15 Aug 2024 11:29:02 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v3 2/2] mm: attempt to batch free swap entries for zap_pte_range() To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, linux-mm@kvack.org, david@redhat.com, hughd@google.com, justinjiang@vivo.com, kaleshsingh@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, v-songbaohua@oppo.com, ying.huang@intel.com, Yosry Ahmed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0BF461C000B X-Stat-Signature: 7xwsphh1tjudix69wb8gsjhc5krqjo6p X-HE-Tag: 1723746555-500554 X-HE-Meta: U2FsdGVkX1/UAEgtON5Xmd7wXl9pNJg6uTWIKTm75GWkzjAPrb9GEuuK6f31CtSNKSVACzv5d/Nj+aE3VkxHY+1GNxy5A2jflmqImlcrwGYJtX1qG1Eh0YoCEIPRBCJFlV+FyZ0oAeaccPSvF65dfTgJzQ1vPnWycQyBBdrSDYNJTg3ELUCG/SX9MVGcDhk4B//p/1FYHMWPBfCM2cBSGt4feh+0JNGJd/cYrHyC5W9yRcSX/cuFNEcREujJNCCqgf0++oSSiWD+DMbABH96b2ocdc6T2kMyZ/gbYl5ArnIu/sJ1v4A+gqdtwcYzPjmUc0XHWEbt+MsF6BWaSWP48ZFEtJ5yj8+cBLkbmACM4jy9Vx45q+Qv7UCNEgPQbwBFpOui94LlGqplwjwWmKhg/twRbxiVWWWdX2cH5ZkmFYMN1e3RKAvHWlNIOOQrDbFA6s3g90c4AOhTvYprblmALCSLMzEo9bCiPwFtOz7DnAMT+FUZDi/CFPDkNM91WK/FMrZOmk+P6M6dXcLTdLyO8vy8CGPV3rugIUEHhMQ9e/9supN0VofV1/1JyVpEBHvhz0cWs+9+e5SC92VCeGssC91Bde05hJWpHlXhOSi2aO4/OCFE2Ibm8oG82NYxC80GuXpTuqgoPYL36hZFjfVQRVe6opvUv1IWisduAHunhgYOw2WMPX69QfIrVvQUcTC9e9JgbZ3R5EejwiwCQeofkfeg/Dt1K2HksrQ5E7UNsuqUnClqrtF2u7uXpjZXbPmjUzUTvtt4/hzy3C4R4JDxWAhP16Gi9FWoVhgRrvRjPRFj15wFSL/Bsl4gsvaEAP4DP6ToCVGJPYXXNw66HYulFZija425pKnHMfpeqKikjFawhyYqtek9J5Gr6ht//G50yiP7mPc2o/lth2l5OV6Fp5Qk0xKGG2TBv5St4ScRsOMI8i5MIaeoqfaM1gty9QM0wS8IGbGGYBq6fPxThuc qCHK3Azz K79ew5eREim+g/bijEJBKkI/ZGY5gIKp2t7JU0JfYC5Ygsm6sZqC9f2/yu9Z5G4G42k40RCdHB7JLVHKs8rF5OdOn86SPDLJ52RV7P4Q2FiDrHGGxgQ3wGrVWjOyqFP9ZmGH1LmIjh22607Lo43N7UqtJMZGPB7UUx3tZj+tlOdNtepQr5YbR7niSAZQu4Rrh0Fh7dKNckuiLnVmdLqzMDFOXEHqIuId1SBje7OXnPz2cKZ/SR9zZbH9XIsQ3ViSw/l59/8AhfbyMBU9mJhpugxC44Xx8+UXQTKRRwfR3IPLMqCjA4INsv+fMg8t43YNp+g+hIrjK/MUaWmPfZ5eoUrw8OLyc6zTVI+fHy8EZZSXMv730N4iniAj/4UWmjcxqvF6ZTB+l5jpM85I/QHhlJ0wJzdHgieF6sDHDWSp28uCbDPsoVI0mef8KllpHZnpD6gg3xdiewh2umhh2gR77rsyWmpLlS5JL5daZ/TDuR/bjz4xeFXIFPyD9uf6lGk5tkUg9bmSZcnnyu8tQMRajLmsWCz4WqembLMA2fBnCBbmxDxK9gpYVBM9ug8/xHasiQGr3xvL3/GAI5Tmof7gNftf7p5p0bxrAST/33vvJ7uacPlCel1ExqRn5BTmBtwklpJYNTOvuPhxjEUZxHY8ravT+Hyn3H7g/YPLRBkhlMBWDbvcOt5JKV3aafw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Barry, We got a crash report from syzbot that has been bisect into this change. Please see the comment below. ------------[ cut here ]------------ kernel BUG at mm/swap_cgroup.c:141 ! Oops: invalid opcode: 0000 [#1] PREEMPT SMP KASAN PTI CPU: 1 UID: 0 PID: 5371 Comm: syz.0.15 Not tainted 6.11.0-rc3-next-20240812-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024 RIP: 0010:swap_cgroup_record+0x2cd/0x2d0 mm/swap_cgroup.c:141 Code: e7 e8 a7 c9 f6 ff e9 64 fe ff ff e8 cd 41 8e ff 48 c7 c7 c0 db a5 8e 48 89 de e8 2e 8c e8 02 e9 7a fd ff ff e8 b4 41 8e ff 90 <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f RSP: 0018:ffffc90003e172f8 EFLAGS: 00010093 RAX: ffffffff82054c9c RBX: 000000000000000b RCX: ffff88802298bc00 RDX: 0000000000000000 RSI: 000000000000000a RDI: 0000000000000000 RBP: 0000000000000001 R08: ffffffff82054b43 R09: fffff520007c2e3c R10: dffffc0000000000 R11: fffff520007c2e3c R12: ffff88801cf0f014 R13: 0000000000000000 R14: 000000000000000a R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff8880b9100000(0000) knlGS:000000000000000= 0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f9332107a8c CR3: 000000000e734000 CR4: 00000000003506f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __mem_cgroup_uncharge_swap+0x84/0x2e0 mm/memcontrol.c:5118 mem_cgroup_uncharge_swap include/linux/swap.h:668 [inline] swap_entry_range_free+0x45f/0x1120 mm/swapfile.c:1556 __swap_entries_free mm/swapfile.c:1518 [inline] free_swap_and_cache_nr+0xa65/0xae0 mm/swapfile.c:1876 zap_pte_range mm/memory.c:1653 [inline] zap_pmd_range mm/memory.c:1736 [inline] zap_pud_range mm/memory.c:1765 [inline] zap_p4d_range mm/memory.c:1786 [inline] unmap_page_range+0x1924/0x42c0 mm/memory.c:1807 unmap_vmas+0x3cc/0x5f0 mm/memory.c:1897 exit_mmap+0x267/0xc20 mm/mmap.c:1923 __mmput+0x115/0x390 kernel/fork.c:1347 exit_mm+0x220/0x310 kernel/exit.c:571 do_exit+0x9b2/0x28e0 kernel/exit.c:926 do_group_exit+0x207/0x2c0 kernel/exit.c:1088 __do_sys_exit_group kernel/exit.c:1099 [inline] __se_sys_exit_group kernel/exit.c:1097 [inline] __x64_sys_exit_group+0x3f/0x40 kernel/exit.c:1097 x64_sys_call+0x2634/0x2640 arch/x86/include/generated/asm/syscalls_64.h:2= 32 do_syscall_x64 arch/x86/entry/common.c:52 [inline] do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83 entry_SYSCALL_64_after_hwframe+0x77/0x7f On Wed, Aug 7, 2024 at 2:59=E2=80=AFPM Barry Song <21cnbao@gmail.com> wrote= : > > From: Barry Song > > Zhiguo reported that swap release could be a serious bottleneck > during process exits[1]. With mTHP, we have the opportunity to > batch free swaps. > Thanks to the work of Chris and Kairui[2], I was able to achieve > this optimization with minimal code changes by building on their > efforts. > If swap_count is 1, which is likely true as most anon memory are > private, we can free all contiguous swap slots all together. > > Ran the below test program for measuring the bandwidth of munmap > using zRAM and 64KiB mTHP: > > #include > #include > #include > > unsigned long long tv_to_ms(struct timeval tv) > { > return tv.tv_sec * 1000 + tv.tv_usec / 1000; > } > > main() > { > struct timeval tv_b, tv_e; > int i; > #define SIZE 1024*1024*1024 > void *p =3D mmap(NULL, SIZE, PROT_READ | PROT_WRITE, > MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); > if (!p) { > perror("fail to get memory"); > exit(-1); > } > > madvise(p, SIZE, MADV_HUGEPAGE); > memset(p, 0x11, SIZE); /* write to get mem */ > > madvise(p, SIZE, MADV_PAGEOUT); > > gettimeofday(&tv_b, NULL); > munmap(p, SIZE); > gettimeofday(&tv_e, NULL); > > printf("munmap in bandwidth: %ld bytes/ms\n", > SIZE/(tv_to_ms(tv_e) - tv_to_ms(tv_b))); > } > > The result is as below (munmap bandwidth): > mm-unstable mm-unstable-with-patch > round1 21053761 63161283 > round2 21053761 63161283 > round3 21053761 63161283 > round4 20648881 67108864 > round5 20648881 67108864 > > munmap bandwidth becomes 3X faster. > > [1] https://lore.kernel.org/linux-mm/20240731133318.527-1-justinjiang@viv= o.com/ > [2] https://lore.kernel.org/linux-mm/20240730-swap-allocator-v5-0-cb9c148= b9297@kernel.org/ > > Cc: Kairui Song > Cc: Chris Li > Cc: "Huang, Ying" > Cc: Hugh Dickins > Cc: Kalesh Singh > Cc: Ryan Roberts > Cc: David Hildenbrand > Signed-off-by: Barry Song > --- > mm/swapfile.c | 76 +++++++++++++++++++++++++++++++++++++++++++-------- > 1 file changed, 65 insertions(+), 11 deletions(-) > > diff --git a/mm/swapfile.c b/mm/swapfile.c > index 35cb58373493..52e941b6d626 100644 > --- a/mm/swapfile.c > +++ b/mm/swapfile.c > @@ -156,6 +156,25 @@ static bool swap_is_has_cache(struct swap_info_struc= t *si, > return true; > } > > +static bool swap_is_last_map(struct swap_info_struct *si, > + unsigned long offset, int nr_pages, bool *has_cache) > +{ > + unsigned char *map =3D si->swap_map + offset; > + unsigned char *map_end =3D map + nr_pages; > + unsigned char count =3D *map; > + > + if (swap_count(count) !=3D 1) > + return false; > + > + while (++map < map_end) { > + if (*map !=3D count) > + return false; > + } > + > + *has_cache =3D !!(count & SWAP_HAS_CACHE); > + return true; > +} > + > /* > * returns number of pages in the folio that backs the swap entry. If po= sitive, > * the folio was reclaimed. If negative, the folio was not reclaimed. If= 0, no > @@ -1469,6 +1488,51 @@ static unsigned char __swap_entry_free(struct swap= _info_struct *si, > return usage; > } > > +static bool __swap_entries_free(struct swap_info_struct *si, > + swp_entry_t entry, int nr) > +{ > + unsigned long offset =3D swp_offset(entry); > + unsigned int type =3D swp_type(entry); > + struct swap_cluster_info *ci; > + bool has_cache =3D false; > + unsigned char count; > + int i; > + > + if (nr <=3D 1 || swap_count(data_race(si->swap_map[offset])) !=3D= 1) > + goto fallback; > + /* cross into another cluster */ > + if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER) > + goto fallback; > + > + ci =3D lock_cluster_or_swap_info(si, offset); > + if (!swap_is_last_map(si, offset, nr, &has_cache)) { > + unlock_cluster_or_swap_info(si, ci); > + goto fallback; > + } > + for (i =3D 0; i < nr; i++) > + WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); > + unlock_cluster_or_swap_info(si, ci); > + > + if (!has_cache) { > + spin_lock(&si->lock); > + swap_entry_range_free(si, entry, nr); Here it calls swap_entry_range_free() to free a range of the swap entry. However the swap_entry_range_free() has the assumption that all entries belong to the same folio and charge to the same memcg. It eventually pass down to swap_cgroup_record(), which BUG on this line: VM_BUG_ON(sc->id !=3D old); The root cause is that the swap entries are not from the same memcg. Thankos Yosry for finding the root cause. > + spin_unlock(&si->lock); > + } > + return has_cache; > + > +fallback: > + for (i =3D 0; i < nr; i++) { > + if (data_race(si->swap_map[offset + i])) { > + count =3D __swap_entry_free(si, swp_entry(type, o= ffset + i)); > + if (count =3D=3D SWAP_HAS_CACHE) > + has_cache =3D true; > + } else { > + WARN_ON_ONCE(1); > + } > + } > + return has_cache; > +} > + > /* > * Drop the last HAS_CACHE flag of swap entries, caller have to > * ensure all entries belong to the same cgroup. > @@ -1792,11 +1856,9 @@ void free_swap_and_cache_nr(swp_entry_t entry, int= nr) > { > const unsigned long start_offset =3D swp_offset(entry); > const unsigned long end_offset =3D start_offset + nr; > - unsigned int type =3D swp_type(entry); > struct swap_info_struct *si; > bool any_only_cache =3D false; > unsigned long offset; > - unsigned char count; > > if (non_swap_entry(entry)) > return; > @@ -1811,15 +1873,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int= nr) > /* > * First free all entries in the range. > */ > - for (offset =3D start_offset; offset < end_offset; offset++) { > - if (data_race(si->swap_map[offset])) { > - count =3D __swap_entry_free(si, swp_entry(type, o= ffset)); > - if (count =3D=3D SWAP_HAS_CACHE) > - any_only_cache =3D true; > - } else { > - WARN_ON_ONCE(1); > - } > - } > + any_only_cache =3D __swap_entries_free(si, entry, nr); Here we are just doing a page table walk, there is no guarantee the 'nr' number of swap entries came from the same folio and previously charged to the same memcg. The swap_pte_batch() only checks they are the same swap type, does not check they charge to the same memcg. Chris > > /* > * Short-circuit the below loop if none of the entries had their > -- > 2.34.1 >