From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 729CACD3431 for ; Wed, 4 Sep 2024 07:12:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3FB38D022F; Wed, 4 Sep 2024 03:12:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCFA68D0228; Wed, 4 Sep 2024 03:12:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7F618D022F; Wed, 4 Sep 2024 03:12:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 867B88D0228 for ; Wed, 4 Sep 2024 03:12:54 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 0538A80E05 for ; Wed, 4 Sep 2024 07:12:54 +0000 (UTC) X-FDA: 82526188668.08.C44A071 Received: from mail-ej1-f51.google.com (mail-ej1-f51.google.com [209.85.218.51]) by imf29.hostedemail.com (Postfix) with ESMTP id 1CC3C120015 for ; Wed, 4 Sep 2024 07:12:51 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Il6tbxsx; spf=pass (imf29.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1725433923; a=rsa-sha256; cv=none; b=kLTdfiYY94UEQeN2Emq0dFVJHv4p0VCK1TSAWDmCmZspaKXxPbD4LBUHyGmilWeelWX28A ZYvTYfjX8IjaCM/42KUJprIo2H680RE0pIVyyEu4yNbCj3WItXyTqqntMxxNPBoRWRvrRB CfVGSW9yVvru0IUYH7HaiRfmECh98Ng= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Il6tbxsx; spf=pass (imf29.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.51 as permitted sender) smtp.mailfrom=yosryahmed@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1725433923; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sF9gTMDRoHUxjCGKUN8c1P0f0J3cAsJjaAYSD6V8Zuc=; b=T2K3LiF6F9bZF+XG4h07MVN9V5Ht2qf2NrhOCcu1LVM1OONJZD1GlB3Wl60++t/rDI/en8 llBsu1ZQPXznvJzSxIu9O2galEvSeQtP6tSs0/gPP6BdkzDpoBNKAUgKTiqXzkMhVt0Mfl Rn8cotfeIyKnJ0jh1SQXvmBRb29CUYU= Received: by mail-ej1-f51.google.com with SMTP id a640c23a62f3a-a866cea40c4so719457866b.0 for ; Wed, 04 Sep 2024 00:12:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1725433970; x=1726038770; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=sF9gTMDRoHUxjCGKUN8c1P0f0J3cAsJjaAYSD6V8Zuc=; b=Il6tbxsxCaNCnxAKRa9sPBKd9RLVcQRoNhDRPZDwGwPa0DEJhwQNjSnPqvVySVW6JC HNPFHkYM0EnEip+aE4DerkhKau/4xu4tNjjplHQH99Gx+aO2xAwhfebfaqpjlokfZztL k+qScwmAq37D5nGsXgYu/Bgo01QXXZkT5tWhP9EoKKZUpi6fDQWC2iFk3ucqx/Fb6oG3 PolNsC90dh0gCicJ648ges6Hf4dqP7oUerNUr2KwzVqSzpWJ2VQTc1IwTSKAdZC3nt/S ta7cYaD4k8jjCFXMQXSco5EawlE3B2Vne3uPobAOF9gOaK+dFmWTpzi3c52gypVIdlEF 6TLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725433970; x=1726038770; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sF9gTMDRoHUxjCGKUN8c1P0f0J3cAsJjaAYSD6V8Zuc=; b=BjhPgaw2USpSLNaNSF7xbMx7Sm3egi7gxZZ/hYSoon5oTFpVV9s5M7S3eMnb2UMRyR nrgNgW+iDXr+KcJLb56E0cX8tL/2tHg6rJNK5wgGqPmzxPwLvbm/uIBTQasnVgJRJ5us p3eHw2ly4/83vMBm8v0+X7W8yBGz1Q0qAAYTA4fL6QwoVnaFwU4qh28AjXcYl5X1zLBv 6yYyDZuessOIqr2bitmncfyLIyqCQPDSN0/Msj8fewdv/2tLPQPWBGFGTx9mW1d/LPdZ YkuGeiotZ+Y/OSyMhF7QotFWI9q+H/LVBednBcVFSbq4PGqMUb0sJOKDRSuW92gbsdEO //vg== X-Forwarded-Encrypted: i=1; AJvYcCWDzqr1GTyU8QnunhiGbq+1JBtyR+XH5ufpSjWmdq51u80+/QLiF/uiErzT3vysyu0fo2jRX+O/gQ==@kvack.org X-Gm-Message-State: AOJu0Yz6qtkk5g0Bem1Uqlrk8Y4tsRFcrTtt3z7I/D6wnKYzh5LRe5lf gtliwBSnRmpuKVYcDNH3WqimqJXwKII0ILtaMLFS9D234ABC4jtLaY5BsM9VxNbMmfCPyIJsSIS CT1AnWa+B8s+wKYJC2TfK/ZgvMu91dk5re8KT X-Google-Smtp-Source: AGHT+IGiKM8mRZKEWT/jJk3tEjBXHT0LpeR7ppkJxsh5gN6YpzFOyN1Wbo9+rxShvu6DKlg4OS7p30BqqlN/dPBXoOk= X-Received: by 2002:a17:907:7290:b0:a86:ac86:1e17 with SMTP id a640c23a62f3a-a8a1d4c3496mr419921866b.54.1725433969516; Wed, 04 Sep 2024 00:12:49 -0700 (PDT) MIME-Version: 1.0 References: <20240612124750.2220726-2-usamaarif642@gmail.com> <20240904055522.2376-1-21cnbao@gmail.com> In-Reply-To: <20240904055522.2376-1-21cnbao@gmail.com> From: Yosry Ahmed Date: Wed, 4 Sep 2024 00:12:13 -0700 Message-ID: Subject: Re: [PATCH v4 1/2] mm: store zero pages to be swapped out in a bitmap To: Barry Song <21cnbao@gmail.com> Cc: usamaarif642@gmail.com, akpm@linux-foundation.org, chengming.zhou@linux.dev, david@redhat.com, hannes@cmpxchg.org, hughd@google.com, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nphamcs@gmail.com, shakeel.butt@linux.dev, willy@infradead.org, ying.huang@intel.com, hanchuanhua@oppo.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: n1qtghdr67t9tjtkib81kn4rrfzkn4eg X-Rspamd-Queue-Id: 1CC3C120015 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1725433971-490259 X-HE-Meta: U2FsdGVkX18eTm7SOa1xrjSbqjCoowFFeFQAzcWTVtnY4n1oMNSvaO1AA/7PcjU/GrpRLOVKSQTr+1qhosb0kCOonHKL8JJPiFAF/hsKI07YV8O+MRsgYmX64hpYoFt2zegvnjlpSNpGOFwFj4vN4pNfz3C0ZCZE63GP54pnrmeg1qzTZgXrmdP0DeCY8qiYtZtNn7niUTwzrwUJAiCJs1/zNbxYIvA+1rczL9YKEm/Ct48YAsaLsh0Jq0bumkBAJ86CtnJ4Kwkodb31ezHidEmS81lW0xDt24x4riZGh6XZqUnyn3LSr977nebJloYOHIaVKTXSqPcfmcWJDgvhCVreecBNzrzNmiP4tPFumyPZULPcc4bDzlC8SXRpjAGiwwRi9QBI81Ys/dOaZmMG3THx37/cNeVgyJR/G0kpi2ErkOyiHLXtGlZAjJksP0bKa1nTUSFyWYuxLeIoG7hCFJV8ldsAlXOQ9v/nwyoQo6pPhbtl7qI7QCwGdHe2M0a5o2+byaIZAmmv1aJgDXHi58fS6cPkqG8N+L5YeCKCB4pkc9ahdq33o4p4porOLWHMoI2pLpgmdr78BFB79PuYQusg8MRQLc8mTPnE08Gn7GnlLWEEE//sEGNZwd2lFGlBtVrZjciYNsq35rvMnjudzM3jMCF9y9r+jF3OE6XRO55eDwnZvBDwrDHv6N3EPP6+fXHFqqta4fcoL8rnTuBtSRUtdngrHZWFxtoPfjo8/o6w3btXT/XKK/XxtsP6AbQmhuX77bOI8A0vO1VSPy4WTpvlaTkK0bESAGPEuDWYcowQ6sbTssdWrBD3jsQCs2mTtoKrRpqMGu76hYhzkmvR8kLauQuYALrk4qGmk8dJWuAzfAP2kli4Nwtv1zjtJw4KZ2gH0Wg8DG7kR7wo70lDsp/zK3+RZ/pWLEEn01WLJau9YkdUPLIS4nrdWTz2bH6j2A57wW8aTuQZ63nVoGo oVaSlsKF DSpjW/LdClT3LyCWI0vwBtEc3qwF+aC+XEaG1J3sbME2Fkrm+zxfh259V2P15qZNmR9gesrf64CqT8YcBcdpm/WjkOmuDJrxLa+3PcHQ+sEy1N3/ZFFVb9cGmnGKbpZJKtKYbU1cXi6G6F9e2JZG0VL0u+VIEK1oSZYlSqD03Y//5QSht/Q7lWs9RdVHypoS/qNTK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: [..] > > @@ -426,6 +515,26 @@ static void sio_read_complete(struct kiocb *iocb, long ret) > > mempool_free(sio, sio_pool); > > } > > > > +static bool swap_read_folio_zeromap(struct folio *folio) > > +{ > > + unsigned int idx = swap_zeromap_folio_test(folio); > > + > > + if (idx == 0) > > + return false; > > + > > + /* > > + * Swapping in a large folio that is partially in the zeromap is not > > + * currently handled. Return true without marking the folio uptodate so > > + * that an IO error is emitted (e.g. do_swap_page() will sigbus). > > + */ > > + if (WARN_ON_ONCE(idx < folio_nr_pages(folio))) > > + return true; > > Hi Usama, Yosry, > > I feel the warning is wrong as we could have the case where idx==0 > is not zeromap but idx=1 is zeromap. idx == 0 doesn't necessarily > mean we should return false. Good catch. Yeah if idx == 0 is not in the zeromap but other indices are we will mistakenly read the entire folio from swap. > > What about the below change which both fixes the warning and unblocks > large folios swap-in? But I don't see how that unblocks the large folios swap-in work? We still need to actually handle the case where a large folio being swapped in is partially in the zeromap. Right now we warn and unlock the folio without calling folio_mark_uptodate(), which emits an IO error. > > diff --git a/mm/page_io.c b/mm/page_io.c > index 4bc77d1c6bfa..7d7ff7064e2b 100644 > --- a/mm/page_io.c > +++ b/mm/page_io.c > @@ -226,26 +226,6 @@ static void swap_zeromap_folio_clear(struct folio *folio) > } > } > > -/* > - * Return the index of the first subpage which is not zero-filled > - * according to swap_info_struct->zeromap. > - * If all pages are zero-filled according to zeromap, it will return > - * folio_nr_pages(folio). > - */ > -static unsigned int swap_zeromap_folio_test(struct folio *folio) > -{ > - struct swap_info_struct *sis = swp_swap_info(folio->swap); > - swp_entry_t entry; > - unsigned int i; > - > - for (i = 0; i < folio_nr_pages(folio); i++) { > - entry = page_swap_entry(folio_page(folio, i)); > - if (!test_bit(swp_offset(entry), sis->zeromap)) > - return i; > - } > - return i; > -} > - > /* > * We may have stale swap cache pages in memory: notice > * them here and get rid of the unnecessary final write. > @@ -524,9 +504,10 @@ static void sio_read_complete(struct kiocb *iocb, long ret) > > static bool swap_read_folio_zeromap(struct folio *folio) > { > - unsigned int idx = swap_zeromap_folio_test(folio); > + unsigned int nr_pages = folio_nr_pages(folio); > + unsigned int nr = swap_zeromap_entries_count(folio->swap, nr_pages); > > - if (idx == 0) > + if (nr == 0) > return false; > > /* > @@ -534,7 +515,7 @@ static bool swap_read_folio_zeromap(struct folio *folio) > * currently handled. Return true without marking the folio uptodate so > * that an IO error is emitted (e.g. do_swap_page() will sigbus). > */ > - if (WARN_ON_ONCE(idx < folio_nr_pages(folio))) > + if (WARN_ON_ONCE(nr < nr_pages)) > return true; > > folio_zero_range(folio, 0, folio_size(folio)); > diff --git a/mm/swap.h b/mm/swap.h > index f8711ff82f84..2d59e9d89e95 100644 > --- a/mm/swap.h > +++ b/mm/swap.h > @@ -80,6 +80,32 @@ static inline unsigned int folio_swap_flags(struct folio *folio) > { > return swp_swap_info(folio->swap)->flags; > } > + > +/* > + * Return the number of entries which are zero-filled according to > + * swap_info_struct->zeromap. It isn't precise if the return value > + * is larger than 0 and smaller than nr to avoid extra iterations, > + * In this case, it means entries haven't consistent zeromap. > + */ > +static inline unsigned int swap_zeromap_entries_count(swp_entry_t entry, int nr) > +{ > + struct swap_info_struct *sis = swp_swap_info(entry); > + unsigned long offset = swp_offset(entry); > + unsigned int type = swp_type(entry); > + unsigned int n = 0; > + > + for (int i = 0; i < nr; i++) { > + entry = swp_entry(type, offset + i); > + if (test_bit(offset + i, sis->zeromap)) { > + if (i != n) > + return i; > + n++; > + } > + } > + > + return n; > +} > + > #else /* CONFIG_SWAP */ > struct swap_iocb; > static inline void swap_read_folio(struct folio *folio, struct swap_iocb **plug) > @@ -171,6 +197,11 @@ static inline unsigned int folio_swap_flags(struct folio *folio) > { > return 0; > } > + > +static inline unsigned int swap_zeromap_entries_count(swp_entry_t entry, int nr) > +{ > + return 0; > +} > #endif /* CONFIG_SWAP */ > > #endif /* _MM_SWAP_H */ > > > + > > + folio_zero_fill(folio); > > + folio_mark_uptodate(folio); > > + return true; > > +} > > + > > static void swap_read_folio_fs(struct folio *folio, struct swap_iocb **plug) > > { > > struct swap_info_struct *sis = swp_swap_info(folio->swap); > > @@ -515,8 +624,9 @@ void swap_read_folio(struct folio *folio, bool synchronous, > > psi_memstall_enter(&pflags); > > } > > delayacct_swapin_start(); > > - > > - if (zswap_load(folio)) { > > + if (swap_read_folio_zeromap(folio)) { > > + folio_unlock(folio); > > + } else if (zswap_load(folio)) { > > folio_mark_uptodate(folio); > > folio_unlock(folio); > > } else if (data_race(sis->flags & SWP_FS_OPS)) { > > diff --git a/mm/swapfile.c b/mm/swapfile.c > > index f1e559e216bd..48d8dca0b94b 100644 > > --- a/mm/swapfile.c > > +++ b/mm/swapfile.c > > @@ -453,6 +453,8 @@ static unsigned int cluster_list_del_first(struct swap_cluster_list *list, > > static void swap_cluster_schedule_discard(struct swap_info_struct *si, > > unsigned int idx) > > { > > + unsigned int i; > > + > > /* > > * If scan_swap_map_slots() can't find a free cluster, it will check > > * si->swap_map directly. To make sure the discarding cluster isn't > > @@ -461,6 +463,13 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si, > > */ > > memset(si->swap_map + idx * SWAPFILE_CLUSTER, > > SWAP_MAP_BAD, SWAPFILE_CLUSTER); > > + /* > > + * zeromap can see updates from concurrent swap_writepage() and swap_read_folio() > > + * call on other slots, hence use atomic clear_bit for zeromap instead of the > > + * non-atomic bitmap_clear. > > + */ > > + for (i = 0; i < SWAPFILE_CLUSTER; i++) > > + clear_bit(idx * SWAPFILE_CLUSTER + i, si->zeromap); > > > > cluster_list_add_tail(&si->discard_clusters, si->cluster_info, idx); > > > > @@ -482,7 +491,7 @@ static void __free_cluster(struct swap_info_struct *si, unsigned long idx) > > static void swap_do_scheduled_discard(struct swap_info_struct *si) > > { > > struct swap_cluster_info *info, *ci; > > - unsigned int idx; > > + unsigned int idx, i; > > > > info = si->cluster_info; > > > > @@ -498,6 +507,8 @@ static void swap_do_scheduled_discard(struct swap_info_struct *si) > > __free_cluster(si, idx); > > memset(si->swap_map + idx * SWAPFILE_CLUSTER, > > 0, SWAPFILE_CLUSTER); > > + for (i = 0; i < SWAPFILE_CLUSTER; i++) > > + clear_bit(idx * SWAPFILE_CLUSTER + i, si->zeromap); > > unlock_cluster(ci); > > } > > } > > @@ -1059,9 +1070,12 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) > > { > > unsigned long offset = idx * SWAPFILE_CLUSTER; > > struct swap_cluster_info *ci; > > + unsigned int i; > > > > ci = lock_cluster(si, offset); > > memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); > > + for (i = 0; i < SWAPFILE_CLUSTER; i++) > > + clear_bit(offset + i, si->zeromap); > > cluster_set_count_flag(ci, 0, 0); > > free_cluster(si, idx); > > unlock_cluster(ci); > > @@ -1336,6 +1350,7 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) > > count = p->swap_map[offset]; > > VM_BUG_ON(count != SWAP_HAS_CACHE); > > p->swap_map[offset] = 0; > > + clear_bit(offset, p->zeromap); > > dec_cluster_info_page(p, p->cluster_info, offset); > > unlock_cluster(ci); > > > > @@ -2597,6 +2612,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) > > free_percpu(p->cluster_next_cpu); > > p->cluster_next_cpu = NULL; > > vfree(swap_map); > > + bitmap_free(p->zeromap); > > kvfree(cluster_info); > > /* Destroy swap account information */ > > swap_cgroup_swapoff(p->type); > > @@ -3123,6 +3139,12 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) > > goto bad_swap_unlock_inode; > > } > > > > + p->zeromap = bitmap_zalloc(maxpages, GFP_KERNEL); > > + if (!p->zeromap) { > > + error = -ENOMEM; > > + goto bad_swap_unlock_inode; > > + } > > + > > if (p->bdev && bdev_stable_writes(p->bdev)) > > p->flags |= SWP_STABLE_WRITES; > > > > -- > > 2.43.0 > > > >