From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F78FCA0ED3 for ; Wed, 4 Sep 2024 11:14:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3C298D0251; Wed, 4 Sep 2024 07:14:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEE358D0003; Wed, 4 Sep 2024 07:14:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 965298D0251; Wed, 4 Sep 2024 07:14:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 76CAA8D0003 for ; Wed, 4 Sep 2024 07:14:47 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2D029A1166 for ; Wed, 4 Sep 2024 11:14:47 +0000 (UTC) X-FDA: 82526798214.08.B1E6E2F Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by imf11.hostedemail.com (Postfix) with ESMTP id 3406340014 for ; Wed, 4 Sep 2024 11:14:44 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HUbFlwbl; spf=pass (imf11.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725448461; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Me3O/kZ+odhMmOaQ3WjDbn57wTW+RVJQ6ZiktI6MK90=; b=RGhfHqEvGdgSJo90rVjzH9A68Qn61aeDfhtehh4UwNiD9Eae8nO29bPsag69y1OdSSrKv3 70Igphl4jdUomGJt+MObw0/tbUpQUKZekUhJRB5j2puM0kNMrw69HMLyInqI2xES38+9nt MZ4XJAlorrtw4Z+ciHUHEE4qHnV/NhI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HUbFlwbl; spf=pass (imf11.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725448461; a=rsa-sha256; cv=none; b=lXYSJxPf3A30dkkLGvSasm73oVknWywzpFWipenSz07GLPQnGGKJsoB0TKMGKfjvIEkUOF u3fmoOesQ2Uohjq8AJemhUkpGaDFfcWlSo0xotmqa6JgWawpxTfkpIwmApY7IAOTm9cvOM v5bi3bEjJoxrSga/BT9zrXMW2hsLzsw= Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-a8682bb5e79so88556666b.2 for ; Wed, 04 Sep 2024 04:14:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1725448483; x=1726053283; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=Me3O/kZ+odhMmOaQ3WjDbn57wTW+RVJQ6ZiktI6MK90=; b=HUbFlwbl4sCDB/vfvSsl1n9z1mX/Swoa0GLQTFWryphW/e68Whkws9R8HACGmI3rLe QuVHHXRgPpqpv+ItxilNHQcCJkaGykSlqrUWAJ1b0MFDtJKAIeqzrFXgcN89HFPgwMeB M8VGjDjP+6yxCO3Eo5FBeuZAQM0QqjqmzpRJeKTDUpi2sLS/35//NRucUCWVT9UZUWlP EiRjZzu5Xb6cCPFIOfswUGd4GIZEekEYmOI/AqJEoL/k52ETu4vU0jjc2sgEXR/wmK+X tPVMFdzMMIaWgKMwX6HJQfhvyL4UqLRU8ttTCVqCOr2vsyYW5s0n13LbceNM4/hsXxmA X/Hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725448483; x=1726053283; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Me3O/kZ+odhMmOaQ3WjDbn57wTW+RVJQ6ZiktI6MK90=; b=EiaAovAnZKsqq9z5EOBItvFa29Ioecr4/YisLg/CH20mbJoZeNvq0qsNlj2E0euxQD TCG6wP/KW/2HAWgygmuVaX5oEVvFtKvytwbmNNEbNLHYnxTQ+5SDkyASIzUgKu9MsD8D DCcpLzmkwDjXZYuOYH7Mq30Qm7SRP1CiKo3OYfXyuD4UAUpiN2CcFyfn3CZyJ8sfmpLu RJ9fkSP79P3FWV6YVdgjfrMeKBjFu2MlW147WkE/zPvILSF6lMQgiSDlPiAEX91PFY/2 hPodpesDBXh8vZXJPd3WudacCo8XU4WOdpwQFhSB7DnbaYAAvA6FOZzGmax20XS4dVQ+ yqkA== X-Forwarded-Encrypted: i=1; AJvYcCUDu2Oopuh0qSlmd4KuUKnyc+drVuJWlEVL3zpSOtwyNo87uesJmLNyEC5n0HLwZPy4EqfUrD9vzg==@kvack.org X-Gm-Message-State: AOJu0YyUAq/he0Q3tsQSpT2PHvYwlk0ELzfrNjrvQcxKVbHma304gFgw c2UHD+h3R45aKamysOUntFRP6j2xjnh9rwyrK1+5b9uZXlHdRoHf X-Google-Smtp-Source: AGHT+IH+yCLKOd1Vd8GUN8WoEMx3Yl/2nFAfnlE3qR4hFA+lOVGZsr8xBSsCs7s9IdWnooVlk6MyHw== X-Received: by 2002:a17:907:7d93:b0:a77:b3c4:cd28 with SMTP id a640c23a62f3a-a8a3f0ef33amr167341866b.9.1725448482461; Wed, 04 Sep 2024 04:14:42 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:eb:d0d0:c7fd:c82c? ([2620:10d:c092:500::6:4d49]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a8988feacd7sm797121266b.34.2024.09.04.04.14.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 04 Sep 2024 04:14:42 -0700 (PDT) Message-ID: <7a91ff31-1f56-4d0c-a4a7-a305331ba97a@gmail.com> Date: Wed, 4 Sep 2024 12:14:41 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 1/2] mm: store zero pages to be swapped out in a bitmap To: Barry Song <21cnbao@gmail.com> Cc: akpm@linux-foundation.org, chengming.zhou@linux.dev, david@redhat.com, hannes@cmpxchg.org, hughd@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, shakeel.butt@linux.dev, willy@infradead.org, ying.huang@intel.com, yosryahmed@google.com, hanchuanhua@oppo.com References: <20240612124750.2220726-2-usamaarif642@gmail.com> <20240904055522.2376-1-21cnbao@gmail.com> Content-Language: en-US From: Usama Arif In-Reply-To: <20240904055522.2376-1-21cnbao@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: mspocykdznedeao61u854cnsbud97aot X-Rspamd-Queue-Id: 3406340014 X-Rspamd-Server: rspam11 X-HE-Tag: 1725448484-65590 X-HE-Meta: U2FsdGVkX1+HsBG7gRXmye0pYy27H+DP/rLcER1HowdM76GaGMiYFVn30xC18NGzYdfzzEI81nIrhPN+8Wz42/3yiwJvkiBDOieW+IL8NzuEwNkI78tZYElw63jPKa2hwzwUTMhb8pfSbbZdwKVv7SpAtbNBilljxf0DfrV8lJZ4C6xYlquk7ANClYVn7974e/p3jEN8+BLjA292yfH1muXEYJG9NGZlgUrkEJLTyxLKKU+GqBc7vkcZcNfD2yz4G+XApRaNQ050sAKjAdSfwxwJ2wVTlFihtOLeABxm751Eclecd6G54aZDr3kR5Q0eHKajN2Qhh32JxJsGpY1Eo2cnV2lfg3HDGMloyqWy/RHyfzWHhcstnHsHI+hQCjZTNHkxyUWiyQiufeePPK5Kkw8JaFjWbU58HvQ0cW3I6CqgSXfYCXqBEs4lDrn00tV7g5M+OayxRDmeEgx6B4rXqAf7DcdpuelYWfVX9vBMVY+lDxU/po2ioto5Y6KrKEDqqxJVqrEly3I9KWiFQATMUq8TfDDnfu0SWxdYkrI8rr6GNayxb0J7LoYoI87m5iIkYeWatWXN+uoThcPpEDH8FH9kTxfkHQy89gO53js0el5CmOLZGL14HbrZ50RD/BHg0w+v+3HPtaKwyRt4FCQ0PI5R955ay1Qf96PfJaLajnhBOn8BSBnxUwBA8akUkTB87zNjTkH/lK4aPtWFuEq7I/LNkGc/ZXmnwbfhMAkQFdspU+pZ9pfZA+6NMHnsatz4nSvjPaO9Fn698HHpwzEZTgTs/KslvdovagCrY/9s+OV+EA757HCbuYOhrUpJF7+sz4MB3k1a9XEyZPpdpFl1bGzCq2i4MWSj3eMjP0OaTI/gNt4qNAv/1b0zy3OGSmZDLPUmjIQXr8EPwwT25g3ttJxYAExdQ4BiAsc34w2r5c7FVmfis189fOldhvtynDYd73Rs7M6+7vAGMg7Qx5q hO14a28W YyKSsJSLU0I+dsvV6RQMDRHhB0sLr38uC27L4T7MfLrWoA0XuSI1dyuO9aOdqFddDRKofD4u1Ozjum1LLJFfrVkBObVXI9L52I5Mmq4iV1uWBbAsk18AMSG8Oqdwy+zlHpM8cvcwu/niCvoYrFdYwRvdKTmtzkxmH/dPCR4qgvKppxlNYyik/ha3tRN46lYwhkFjpsIFlJKMrAbPQm10R3ocnVEUk4Mj73znm4r9PX7ig9WCklMwRiUpo7SxyVthHJ/Iv1mJv1QVVF7l3ab07z5tHssea5MTEVZBt1IhsehflI4zZXJ+JzcNbT6AIIjixZFnG41QMw1fDDH4C19WQ7yVs77RmzOKvd7b8UhpRFwVIdo2XjcZBJud79yN2vz3YhL3JTjtR1Z/4rbCZ3A55NUSCtCtkj10/kzYNdoINNn8D4VofMAhrUyOc5diB817NvhG/PVW4GR3S25ms6MRjJnV91znSD8xQdHFwrL5SewBp/5v4viZoHPEcKzcxczqtrQx1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 04/09/2024 06:55, Barry Song wrote: > On Thu, Jun 13, 2024 at 12:48 AM Usama Arif wrote: >> >> Approximately 10-20% of pages to be swapped out are zero pages [1]. >> Rather than reading/writing these pages to flash resulting >> in increased I/O and flash wear, a bitmap can be used to mark these >> pages as zero at write time, and the pages can be filled at >> read time if the bit corresponding to the page is set. >> With this patch, NVMe writes in Meta server fleet decreased >> by almost 10% with conventional swap setup (zswap disabled). >> >> [1] https://lore.kernel.org/all/20171018104832epcms5p1b2232e2236258de3d03d1344dde9fce0@epcms5p1/ >> >> Signed-off-by: Usama Arif >> --- >>  include/linux/swap.h |   1 + >>  mm/page_io.c         | 114 ++++++++++++++++++++++++++++++++++++++++++- >>  mm/swapfile.c        |  24 ++++++++- >>  3 files changed, 136 insertions(+), 3 deletions(-) >> >> diff --git a/include/linux/swap.h b/include/linux/swap.h >> index a11c75e897ec..e88563978441 100644 >> --- a/include/linux/swap.h >> +++ b/include/linux/swap.h >> @@ -299,6 +299,7 @@ struct swap_info_struct { >>         signed char     type;           /* strange name for an index */ >>         unsigned int    max;            /* extent of the swap_map */ >>         unsigned char *swap_map;        /* vmalloc'ed array of usage counts */ >> +       unsigned long *zeromap;         /* vmalloc'ed bitmap to track zero pages */ >>         struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */ >>         struct swap_cluster_list free_clusters; /* free clusters list */ >>         unsigned int lowest_bit;        /* index of first free in swap_map */ >> diff --git a/mm/page_io.c b/mm/page_io.c >> index a360857cf75d..39fc3919ce15 100644 >> --- a/mm/page_io.c >> +++ b/mm/page_io.c >> @@ -172,6 +172,88 @@ int generic_swapfile_activate(struct swap_info_struct *sis, >>         goto out; >>  } >> >> +static bool is_folio_page_zero_filled(struct folio *folio, int i) >> +{ >> +       unsigned long *data; >> +       unsigned int pos, last_pos = PAGE_SIZE / sizeof(*data) - 1; >> +       bool ret = false; >> + >> +       data = kmap_local_folio(folio, i * PAGE_SIZE); >> +       if (data[last_pos]) >> +               goto out; >> +       for (pos = 0; pos < PAGE_SIZE / sizeof(*data); pos++) { >> +               if (data[pos]) >> +                       goto out; >> +       } >> +       ret = true; >> +out: >> +       kunmap_local(data); >> +       return ret; >> +} >> + >> +static bool is_folio_zero_filled(struct folio *folio) >> +{ >> +       unsigned int i; >> + >> +       for (i = 0; i < folio_nr_pages(folio); i++) { >> +               if (!is_folio_page_zero_filled(folio, i)) >> +                       return false; >> +       } >> +       return true; >> +} >> + >> +static void folio_zero_fill(struct folio *folio) >> +{ >> +       unsigned int i; >> + >> +       for (i = 0; i < folio_nr_pages(folio); i++) >> +               clear_highpage(folio_page(folio, i)); >> +} >> + >> +static void swap_zeromap_folio_set(struct folio *folio) >> +{ >> +       struct swap_info_struct *sis = swp_swap_info(folio->swap); >> +       swp_entry_t entry; >> +       unsigned int i; >> + >> +       for (i = 0; i < folio_nr_pages(folio); i++) { >> +               entry = page_swap_entry(folio_page(folio, i)); >> +               set_bit(swp_offset(entry), sis->zeromap); >> +       } >> +} >> + >> +static void swap_zeromap_folio_clear(struct folio *folio) >> +{ >> +       struct swap_info_struct *sis = swp_swap_info(folio->swap); >> +       swp_entry_t entry; >> +       unsigned int i; >> + >> +       for (i = 0; i < folio_nr_pages(folio); i++) { >> +               entry = page_swap_entry(folio_page(folio, i)); >> +               clear_bit(swp_offset(entry), sis->zeromap); >> +       } >> +} >> + >> +/* >> + * Return the index of the first subpage which is not zero-filled >> + * according to swap_info_struct->zeromap. >> + * If all pages are zero-filled according to zeromap, it will return >> + * folio_nr_pages(folio). >> + */ >> +static unsigned int swap_zeromap_folio_test(struct folio *folio) >> +{ >> +       struct swap_info_struct *sis = swp_swap_info(folio->swap); >> +       swp_entry_t entry; >> +       unsigned int i; >> + >> +       for (i = 0; i < folio_nr_pages(folio); i++) { >> +               entry = page_swap_entry(folio_page(folio, i)); >> +               if (!test_bit(swp_offset(entry), sis->zeromap)) >> +                       return i; >> +       } >> +       return i; >> +} >> + >>  /* >>   * We may have stale swap cache pages in memory: notice >>   * them here and get rid of the unnecessary final write. >> @@ -195,6 +277,13 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) >>                 folio_unlock(folio); >>                 return ret; >>         } >> + >> +       if (is_folio_zero_filled(folio)) { >> +               swap_zeromap_folio_set(folio); >> +               folio_unlock(folio); >> +               return 0; >> +       } >> +       swap_zeromap_folio_clear(folio); >>         if (zswap_store(folio)) { >>                 folio_start_writeback(folio); >>                 folio_unlock(folio); >> @@ -426,6 +515,26 @@ static void sio_read_complete(struct kiocb *iocb, long ret) >>         mempool_free(sio, sio_pool); >>  } >> >> +static bool swap_read_folio_zeromap(struct folio *folio) >> +{ >> +       unsigned int idx = swap_zeromap_folio_test(folio); >> + >> +       if (idx == 0) >> +               return false; >> + >> +       /* >> +        * Swapping in a large folio that is partially in the zeromap is not >> +        * currently handled. Return true without marking the folio uptodate so >> +        * that an IO error is emitted (e.g. do_swap_page() will sigbus). >> +        */ >> +       if (WARN_ON_ONCE(idx < folio_nr_pages(folio))) >> +               return true; > > Hi Usama, Yosry, > > I feel the warning is wrong as we could have the case where idx==0 > is not zeromap but idx=1 is zeromap. idx == 0 doesn't necessarily > mean we should return false. > > What about the below change which both fixes the warning and unblocks > large folios swap-in? > Hi Barry, I remembered when resending the zeromap series about the comment Yosry had made earlier, but checked that the mTHP swap-in was not in mm-unstable. I should have checked the mailing list and commented! I have not tested the below diff yet (will do in a few hours). But there might be a small issue with it. Have commented inline. > diff --git a/mm/page_io.c b/mm/page_io.c > index 4bc77d1c6bfa..7d7ff7064e2b 100644 > --- a/mm/page_io.c > +++ b/mm/page_io.c > @@ -226,26 +226,6 @@ static void swap_zeromap_folio_clear(struct folio *folio) > } > } > > -/* > - * Return the index of the first subpage which is not zero-filled > - * according to swap_info_struct->zeromap. > - * If all pages are zero-filled according to zeromap, it will return > - * folio_nr_pages(folio). > - */ > -static unsigned int swap_zeromap_folio_test(struct folio *folio) > -{ > - struct swap_info_struct *sis = swp_swap_info(folio->swap); > - swp_entry_t entry; > - unsigned int i; > - > - for (i = 0; i < folio_nr_pages(folio); i++) { > - entry = page_swap_entry(folio_page(folio, i)); > - if (!test_bit(swp_offset(entry), sis->zeromap)) > - return i; > - } > - return i; > -} > - > /* > * We may have stale swap cache pages in memory: notice > * them here and get rid of the unnecessary final write. > @@ -524,9 +504,10 @@ static void sio_read_complete(struct kiocb *iocb, long ret) > > static bool swap_read_folio_zeromap(struct folio *folio) > { > - unsigned int idx = swap_zeromap_folio_test(folio); > + unsigned int nr_pages = folio_nr_pages(folio); > + unsigned int nr = swap_zeromap_entries_count(folio->swap, nr_pages); > > - if (idx == 0) > + if (nr == 0) > return false; > > /* > @@ -534,7 +515,7 @@ static bool swap_read_folio_zeromap(struct folio *folio) > * currently handled. Return true without marking the folio uptodate so > * that an IO error is emitted (e.g. do_swap_page() will sigbus). > */ > - if (WARN_ON_ONCE(idx < folio_nr_pages(folio))) > + if (WARN_ON_ONCE(nr < nr_pages)) > return true; > > folio_zero_range(folio, 0, folio_size(folio)); > diff --git a/mm/swap.h b/mm/swap.h > index f8711ff82f84..2d59e9d89e95 100644 > --- a/mm/swap.h > +++ b/mm/swap.h > @@ -80,6 +80,32 @@ static inline unsigned int folio_swap_flags(struct folio *folio) > { > return swp_swap_info(folio->swap)->flags; > } > + > +/* > + * Return the number of entries which are zero-filled according to > + * swap_info_struct->zeromap. It isn't precise if the return value > + * is larger than 0 and smaller than nr to avoid extra iterations, > + * In this case, it means entries haven't consistent zeromap. > + */ > +static inline unsigned int swap_zeromap_entries_count(swp_entry_t entry, int nr) > +{ > + struct swap_info_struct *sis = swp_swap_info(entry); > + unsigned long offset = swp_offset(entry); > + unsigned int type = swp_type(entry); > + unsigned int n = 0; > + > + for (int i = 0; i < nr; i++) { > + entry = swp_entry(type, offset + i); > + if (test_bit(offset + i, sis->zeromap)) { Should this be if (test_bit(swp_offset(entry), sis->zeromap)) Also, are you going to use this in alloc_swap_folio? You mentioned above that this unblocks large folios swap-in, but I don't see it in the diff here. I am guessing there is some change in alloc_swap_info that uses swap_zeromap_entries_count? Thanks Usama > + if (i != n) > + return i; > + n++; > + } > + } > + > + return n; > +} > + > #else /* CONFIG_SWAP */ > struct swap_iocb; > static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) > @@ -171,6 +197,11 @@ static inline unsigned int folio_swap_flags(struct folio *folio) > { > return 0; > } > + > +static inline unsigned int swap_zeromap_entries_count(swp_entry_t entry, int nr) > +{ > + return 0; > +} > #endif /* CONFIG_SWAP */ > > #endif /* _MM_SWAP_H */ >