From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEEF1C27C5E for ; Mon, 10 Jun 2024 18:36:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 663816B0089; Mon, 10 Jun 2024 14:36:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EC456B008A; Mon, 10 Jun 2024 14:36:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 465BD6B0092; Mon, 10 Jun 2024 14:36:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1FFDB6B0089 for ; Mon, 10 Jun 2024 14:36:31 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D8C7BA1778 for ; Mon, 10 Jun 2024 18:36:30 +0000 (UTC) X-FDA: 82215834540.02.770D716 Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by imf15.hostedemail.com (Postfix) with ESMTP id CE7EFA0008 for ; Mon, 10 Jun 2024 18:36:28 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DyrvQUbz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718044588; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1IvVlgOmmYIMKKD/vlpvl6SApIUUrJQmJ1vq7fEcBk=; b=VSbPl8n5mkv/TIeTFQ2l7bTxK6Bu73Rl2w5iojTB6SvOLMXxdRj8E7OQq3E5MXHIG7FFcy DNzS2MbCDrSzBYg0qw9nFNfjNCGyfjyX6ycytcRlRX3KdKF3/Vc1gOEeaaDKLjjFF+RPBJ wizUYZNKSepwDs4Dk/4jDASwGZ6Gtz0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=DyrvQUbz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf15.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718044588; a=rsa-sha256; cv=none; b=mzeLVl33Fthsts3mgWfm43C0hrUE2vDC2SXJIqCrNwGRKPda7OjTs1waFIFKbgmWTTOVuN mKJqRkRAeGTmwvDhHEBteXGDoVWrSrg5I+zfG9wprh/Ep0j03L2VMBmgdfVr0+YGm7rlQu qY1ZYkTXJXg2jTXVLFUTNruN8GcggyA= Received: by mail-ej1-f41.google.com with SMTP id a640c23a62f3a-a6f11a2d18aso208534266b.2 for ; Mon, 10 Jun 2024 11:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718044587; x=1718649387; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=t1IvVlgOmmYIMKKD/vlpvl6SApIUUrJQmJ1vq7fEcBk=; b=DyrvQUbzkbvXYA3frRUJlIDRbSaEcwLa9Nc1TXVl4uAYPg7+FtgThIXOpK2gvNSnup pNlUsmP7kqwcP38BcvJs+vRvlbtc0vqxNeME47SW4W3xh8L1j9gU3ITzyJ175vjZaO9p 9xrZEqGn3vxwW9PKQKkulE0GWMYm4nau9L+w40N9ItZ/DxZTBzfezARj3dffeZyDWc6u Hky8lxiTdEO21EX3WRg3AcxfQhsg9T9OFt8bef18RMncwCGE/zRmx2JHjM4b10awydvI WHuGIhqkVztYA0e3j49WV89P60ZnpoHacPJEd20U0cGSp9QeVIziBo7U6u+hzwtqvkoJ zuVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718044587; x=1718649387; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t1IvVlgOmmYIMKKD/vlpvl6SApIUUrJQmJ1vq7fEcBk=; b=KnGpsKeg3jh45r7lISnnaepopV7nNEop6Z+RUzEhSKCb9jE7FQsKq1Jtk+u3jMm2XW aQsI3nNGdVZvuK3xZRQjG98Xon9RAAPiO/YDZnStij93E+uGc/POxij+EKniurZMjEGk HZOLQmnqcH39AWSmtpiO3ru2xJh4oq0HQZT5Vqr3fv79FFjQjlUahyhMgIdqSOb4LhqY QsBZ5kaZWAMSYUEHyX0UUi5nGgpXv2M6a4M3KBwP5iYpSFNHLKW2SElTG83AedJMC9Uo Z94so3uf9WRxH4N4pEXvi2Ld33vmpF909DByJFW2Gi8Vgg6ry/HETku9gFZhDfiFi5im WU6A== X-Forwarded-Encrypted: i=1; AJvYcCW7R3F67nZpt3i2y9GrtFSuXnA4hnfrJCZopPyLCBHDZ8UrZoV5QxjSXo2+2plzkk7C4sq/Tn1dzMk5ozI11RTSqWM= X-Gm-Message-State: AOJu0YyTQAkr5dpjpufJMdWSyqI4HT9sejJOpNZJM2MCjyhyKV0xstXv eLdZ+UZsT9uV8B245So8mfOWWNpL6r2Hbkonn5jwRISbnokikJ+B X-Google-Smtp-Source: AGHT+IGcadJSdVKqCwum8WSNxSBkYV9jXj9BaZDZiJ4pZh7Y8BFEMwxsQwH9Ex+5Q591ypVYYwRc0A== X-Received: by 2002:a17:906:4690:b0:a6e:fb27:8cde with SMTP id a640c23a62f3a-a6efb27aaa0mr417202366b.19.1718044586690; Mon, 10 Jun 2024 11:36:26 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:eb:d0d0:c7fd:c82c? ([2620:10d:c092:500::7:493]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a6f2195a2a3sm132335066b.99.2024.06.10.11.36.26 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 10 Jun 2024 11:36:26 -0700 (PDT) Message-ID: Date: Mon, 10 Jun 2024 19:36:25 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 1/2] mm: store zero pages to be swapped out in a bitmap To: Yosry Ahmed Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, david@redhat.com, ying.huang@intel.com, hughd@google.com, willy@infradead.org, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com References: <20240610121820.328876-1-usamaarif642@gmail.com> <20240610121820.328876-2-usamaarif642@gmail.com> Content-Language: en-US From: Usama Arif In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: CE7EFA0008 X-Stat-Signature: tjzynwdn7d1nehgg8c7nrmoqqzmn777y X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1718044588-869169 X-HE-Meta: U2FsdGVkX185rU2udAMLU84vKWpVpNQYxoaIfPF8KOrTtLdMbJtgp9U3FJBjqfBpDOsiuVsHyPfJPCGjlVareIkyZ+SggtQeL/kp1NB8PTkmP/L+GqK8k5mt11lLn3P1BTXJqnPqIR1yNxvBz5l2wKp/cGH5hYMghMY96ujFzxfxbWZOU0RyZ2ImkFGNsZZhdAkr+UpMu8gkdweSK0Aa/aCebOGdgOKs2dSsFPJ5uwV4GqWaNdHp+bNtxhICzvPX07lGxHNlz6jIznJs5NXUcf8bf6TGxD3YwGfJMtg8oa/sLC7pNEyZWuCVTVePSs/M4e2W8zLUx5BSscTabogvNfWeGTxCznjMrZjbyG1b1ge4WAR3XsoLlzM6YFYaXcoDfr070Q6UtlbX0HBlG+k7JC9ajUZbdkxtdb+xBWmL7bI3jBqyLNBMu+PyNWfXvt5TFt+618Nj01Cymm+L6s0w0DhyNpL6BMVXkykAbpW6AzXYkc5ATB8eMZJ3NqeOfWd7axxQtMopcqA3eBrjXsQLQm3gLysr9MMFavNuX0RDEmzwDufVYn2FVMGp73Nek2+BiTUFUE80pUPyoe18HqpNApgJiAuxDzUXmRuO6vQjtlg+9QMziEsEDrqm962sgnNay09vpIr/lJEfc3ofhUU1BnatO0v1JeKbhuyRg0sgXnsAuRP391wgvjSyWGw7nEgbIpvp4RrtXef32H71oTsp6ZPofFTTCJ8Nz8rNYUO5Vp6Y1WteEKLRAz6lmVlkR6meikh70QYfGEzFfXW7so0K6qgtrDjBb0AJMHlGdtpV+Wbb3ND5akefS3wvpwCvKuc31M9/375feVKIunwbDBWFSxkcaKaZ6JV+SkgXe+NHZJvbVHbO545DDO5fF7ruX2q1JhXea4YOvX36VOxJ7t3x1v8BtsIXATMQj1sQ42fp6oWnMDiz4emSGUGeTBIAWcSZyh2YLXy91fx3G7gq8gf P48r42ZA +XVBoM9k2hmVfp91eqy9mga0Or/np3VQcz2CGbqcVIbKH2Ez3wDHwaOS0O+SSPPTd88Alb1t3Wx++F2Wxx+ue7yf8ZMpDfwBEkplgqb94yQ5fYbHrG6b6j5XhVvpcxhw/qsvfWJWT9BrwL7J495MpJf/b0rcR19BOfNQvDRmCb6Rq4qrrSTIWKcKtwyOh9+m0VOvgmmdb3JZt+NQF2NBodk/IQcaeqvBvzratBEG+VALKmY9aV8r66oAPPUXK9JHbx2dsXAP2Fv7EHwVDD4tSaIW7vCgtzkdkf6XqwtlTAyF65TpbnD+nLCmDJW2ma75QyNYUgTH0f2ra80rcOOlK25PERHgySof4vZGowuTpYoqNde0SSxaJKWkXblowVgFAvOy0pDETYFkkaB0x2E8sWjxSHrd6E3sHB94WoUc/4HOvJRrljlGZm8paCNVir5RMybypRaFmDc2Na5Iai4XZT8/aTibfqO4UtGBRZM+kDKxfwyAEJvPgCsty/fi0WChR89JCkO2AZSaj2clP2cfqcGvmEUvL7WL5wc6GWDqoysrdOH7p6EJkTeclTkBUd7slJ8eTfJFZXctDGfhsMFmyNsCJaNkM57bMJhh5hP1iakTW67HNWRUukV8PxYiwM2EI9lFhZdS5nKbcuYDCWPG4Z3l6DHvLdA1H/29yuqxmmTaVrdE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 10/06/2024 18:57, Yosry Ahmed wrote: > On Mon, Jun 10, 2024 at 5:18 AM Usama Arif wrote: >> Approximately 10-20% of pages to be swapped out are zero pages [1]. >> Rather than reading/writing these pages to flash resulting >> in increased I/O and flash wear, a bitmap can be used to mark these >> pages as zero at write time, and the pages can be filled at >> read time if the bit corresponding to the page is set. >> With this patch, NVMe writes in Meta server fleet decreased >> by almost 10% with conventional swap setup (zswap disabled). >> >> [1]https://lore.kernel.org/all/20171018104832epcms5p1b2232e2236258de3d03d1344dde9fce0@epcms5p1/ >> >> Signed-off-by: Usama Arif >> --- >> include/linux/swap.h | 1 + >> mm/page_io.c | 92 +++++++++++++++++++++++++++++++++++++++++++- >> mm/swapfile.c | 21 +++++++++- >> 3 files changed, 111 insertions(+), 3 deletions(-) >> >> diff --git a/include/linux/swap.h b/include/linux/swap.h >> index a11c75e897ec..e88563978441 100644 >> --- a/include/linux/swap.h >> +++ b/include/linux/swap.h >> @@ -299,6 +299,7 @@ struct swap_info_struct { >> signed char type; /* strange name for an index */ >> unsigned int max; /* extent of the swap_map */ >> unsigned char *swap_map; /* vmalloc'ed array of usage counts */ >> + unsigned long *zeromap; /* vmalloc'ed bitmap to track zero pages */ >> struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ >> struct swap_cluster_list free_clusters; /* free clusters list */ >> unsigned int lowest_bit; /* index of first free in swap_map */ >> diff --git a/mm/page_io.c b/mm/page_io.c >> index a360857cf75d..2cac1e11fb85 100644 >> --- a/mm/page_io.c >> +++ b/mm/page_io.c >> @@ -172,6 +172,82 @@ int generic_swapfile_activate(struct swap_info_struct *sis, >> goto out; >> } >> >> +static bool is_folio_page_zero_filled(struct folio *folio, int i) >> +{ >> + unsigned long *data; >> + unsigned int pos, last_pos = PAGE_SIZE / sizeof(*data) - 1; >> + bool ret = false; >> + >> + data = kmap_local_folio(folio, i * PAGE_SIZE); >> + if (data[last_pos]) >> + goto out; >> + for (pos = 0; pos < PAGE_SIZE / sizeof(*data); pos++) { >> + if (data[pos]) >> + goto out; >> + } >> + ret = true; >> +out: >> + kunmap_local(data); >> + return ret; >> +} >> + >> +static bool is_folio_zero_filled(struct folio *folio) >> +{ >> + unsigned int i; >> + >> + for (i = 0; i < folio_nr_pages(folio); i++) { >> + if (!is_folio_page_zero_filled(folio, i)) >> + return false; >> + } >> + return true; >> +} > Is there any benefit to iterating on the folio in pages (i.e. have > both is_folio_zero_filled() and is_folio_page_zero_filled())? Why > don't we just kmap the entire folio and check it all at once? Is there an API to kmap an entire folio? I could just do data = page_address(&folio->page) in above and iterate through folio_nr_pages(folio) * PAGE_SIZE, and do it all in one function, but this just looks much nicer and much more readable? In other places in the kernel, the folio iteration is through pages as well: https://elixir.bootlin.com/linux/latest/source/arch/csky/abiv2/cacheflush.c#L29 https://elixir.bootlin.com/linux/latest/source/arch/mips/mm/cache.c#L162 and others as well. >> + >> +static void folio_zero_fill(struct folio *folio) >> +{ >> + unsigned int i; >> + >> + for (i = 0; i < folio_nr_pages(folio); i++) >> + clear_highpage(folio_page(folio, i)); >> +} >> + >> +static void swap_zeromap_folio_set(struct folio *folio) >> +{ >> + struct swap_info_struct *sis = swp_swap_info(folio->swap); >> + swp_entry_t entry; >> + unsigned int i; >> + >> + for (i = 0; i < folio_nr_pages(folio); i++) { >> + entry = page_swap_entry(folio_page(folio, i)); >> + set_bit(swp_offset(entry), sis->zeromap); >> + } >> +} >> + >> +static void swap_zeromap_folio_clear(struct folio *folio) >> +{ >> + struct swap_info_struct *sis = swp_swap_info(folio->swap); >> + swp_entry_t entry; >> + unsigned int i; >> + >> + for (i = 0; i < folio_nr_pages(folio); i++) { >> + entry = page_swap_entry(folio_page(folio, i)); >> + clear_bit(swp_offset(entry), sis->zeromap); >> + } >> +} >> + >> +static bool swap_zeromap_folio_test(struct folio *folio) >> +{ >> + struct swap_info_struct *sis = swp_swap_info(folio->swap); >> + swp_entry_t entry; >> + unsigned int i; >> + >> + for (i = 0; i < folio_nr_pages(folio); i++) { >> + entry = page_swap_entry(folio_page(folio, i)); >> + if (!test_bit(swp_offset(entry), sis->zeromap)) >> + return false; >> + } >> + return true; >> +} >> + >> /* >> * We may have stale swap cache pages in memory: notice >> * them here and get rid of the unnecessary final write. >> @@ -195,6 +271,15 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) >> folio_unlock(folio); >> return ret; >> } >> + >> + if (is_folio_zero_filled(folio)) { >> + swap_zeromap_folio_set(folio); >> + folio_start_writeback(folio); >> + folio_unlock(folio); >> + folio_end_writeback(folio); >> + return 0; >> + } >> + swap_zeromap_folio_clear(folio); >> if (zswap_store(folio)) { >> folio_start_writeback(folio); >> folio_unlock(folio); >> @@ -515,8 +600,11 @@ void swap_read_folio(struct folio *folio, bool synchronous, >> psi_memstall_enter(&pflags); >> } >> delayacct_swapin_start(); >> - >> - if (zswap_load(folio)) { >> + if (swap_zeromap_folio_test(folio)) { >> + folio_zero_fill(folio); >> + folio_mark_uptodate(folio); >> + folio_unlock(folio); > We don't currently support swapping in large folios, but it is a work > in progress, and this will break once we have it. > swap_zeromap_folio_test() will return false even if parts of the folio > are in fact zero-filled. Then, we will go read those from disk swap, > essentially corrupting data. So yes, with this patch I tested swap out of large zero folio, but when swapping in it was page by page. My assumption was that swap_read_folio (when support is added) would only pass a large folio that was earlier swapped out as a large folio. So if a zero filled large folio was swapped out, the zeromap will be set for all the pages in that folio, and at large folio swap in (when it is supported), it will see that all the bits in the zeromap for that folio are set,  and will just folio_zero_fill. If even a single page in large folio has non-zero data then zeromap will not store it and it will go to either zswap or disk, and at read time as all the bits in zeromap are not set, it will still goto either zswap or disk, so I think this works? Is my assumption wrong that only large folios can be swapped in only if they were swapped out as large? I think this code works in that case. > > The same problem can happen for zswap, if a large folio being swapped > is only partially in zswap. In both cases, it's really easy to miss > the problem if we're testing with zswap disabled, with incompressible > data, or with non-zero data. Silent data corruption is not very > debuggable. > > I proposed adding handling for this case in zswap here: > https://lore.kernel.org/lkml/20240608023654.3513385-1-yosryahmed@google.com/ > > The discussions there hadn't settled yet, but depending on how it pans > out I suspect we will want something similar for the zeromap case as > well. > >> + } else if (zswap_load(folio)) { >> folio_mark_uptodate(folio); >> folio_unlock(folio); >> } else if (data_race(sis->flags & SWP_FS_OPS)) { >> diff --git a/mm/swapfile.c b/mm/swapfile.c >> index f1e559e216bd..90451174fe34 100644 >> --- a/mm/swapfile.c >> +++ b/mm/swapfile.c >> @@ -453,6 +453,8 @@ static unsigned int cluster_list_del_first(struct swap_cluster_list *list, >> static void swap_cluster_schedule_discard(struct swap_info_struct *si, >> unsigned int idx) >> { >> + unsigned int i; >> + >> /* >> * If scan_swap_map_slots() can't find a free cluster, it will check >> * si->swap_map directly. To make sure the discarding cluster isn't >> @@ -461,6 +463,13 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, >> */ >> memset(si->swap_map + idx * SWAPFILE_CLUSTER, >> SWAP_MAP_BAD, SWAPFILE_CLUSTER); >> + /* >> + * zeromap can see updates from concurrent swap_writepage() and swap_read_folio() >> + * call on other slots, hence use atomic clear_bit for zeromap instead of the >> + * non-atomic bitmap_clear. >> + */ >> + for (i = 0; i < SWAPFILE_CLUSTER; i++) >> + clear_bit(idx * SWAPFILE_CLUSTER + i, si->zeromap); >> >> cluster_list_add_tail(&si->discard_clusters, si->cluster_info, idx); >> >> @@ -482,7 +491,7 @@ static void __free_cluster(struct swap_info_struct *si, unsigned long idx) >> static void swap_do_scheduled_discard(struct swap_info_struct *si) >> { >> struct swap_cluster_info *info, *ci; >> - unsigned int idx; >> + unsigned int idx, i; >> >> info = si->cluster_info; >> >> @@ -498,6 +507,8 @@ static void swap_do_scheduled_discard(struct swap_info_struct *si) >> __free_cluster(si, idx); >> memset(si->swap_map + idx * SWAPFILE_CLUSTER, >> 0, SWAPFILE_CLUSTER); >> + for (i = 0; i < SWAPFILE_CLUSTER; i++) >> + clear_bit(idx * SWAPFILE_CLUSTER + i, si->zeromap); >> unlock_cluster(ci); >> } >> } >> @@ -1336,6 +1347,7 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) >> count = p->swap_map[offset]; >> VM_BUG_ON(count != SWAP_HAS_CACHE); >> p->swap_map[offset] = 0; >> + clear_bit(offset, p->zeromap); >> dec_cluster_info_page(p, p->cluster_info, offset); >> unlock_cluster(ci); >> >> @@ -2597,6 +2609,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) >> free_percpu(p->cluster_next_cpu); >> p->cluster_next_cpu = NULL; >> vfree(swap_map); >> + bitmap_free(p->zeromap); >> kvfree(cluster_info); >> /* Destroy swap account information */ >> swap_cgroup_swapoff(p->type); >> @@ -3123,6 +3136,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) >> goto bad_swap_unlock_inode; >> } >> >> + p->zeromap = bitmap_zalloc(maxpages, GFP_KERNEL); >> + if (!p->zeromap) { >> + error = -ENOMEM; >> + goto bad_swap_unlock_inode; >> + } >> + >> if (p->bdev && bdev_stable_writes(p->bdev)) >> p->flags |= SWP_STABLE_WRITES; >> >> -- >> 2.43.0 >>