From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CD85C25B74 for ; Thu, 30 May 2024 19:50:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D40EC6B0085; Thu, 30 May 2024 15:50:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF0E96B009A; Thu, 30 May 2024 15:50:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB8666B009B; Thu, 30 May 2024 15:50:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9DD536B0085 for ; Thu, 30 May 2024 15:50:18 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1130FC0C5C for ; Thu, 30 May 2024 19:50:18 +0000 (UTC) X-FDA: 82176103716.14.FAD302E Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) by imf16.hostedemail.com (Postfix) with ESMTP id 2CD67180002 for ; Thu, 30 May 2024 19:50:15 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DK18J5hQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717098616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c6wRNe3Cnz+DyCo7wP/3ZRDce4Mvllz18ZChzx9cU4c=; b=ALiMGogcjaxBlQebFlwpKKcuRKIpkSw21x+X1lNmHWi6Kuiry08Oogi56FA/47VnpkFYqe kk5pfQpuHiKRov7lBfwMa6Q8ifluetpw64ceKxhWBIdjSv9t6KOubgyt8W4lWOJ+WP6jQv iRwZnm0VYtjK53i8OQDI6+gEJM40OBk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717098616; a=rsa-sha256; cv=none; b=xuPn3O03PhoeKkqPLP5JnVefCqUCL2pSGJuGEyYl2pnhIQ9NN7Q0BfX1TCOWl6jFhOt1X9 UlygXjcsXZTar54CeqFe2ThyT71OaA+dA+Kiy1hXwMcDaYHDWHVpN5dEz5MnQ2ZCD5qMQY aCqzNp8wvN9sdw5p1I1Z7cgqDncZF+A= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=DK18J5hQ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.44 as permitted sender) smtp.mailfrom=yosryahmed@google.com Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-a59cc765c29so122578166b.3 for ; Thu, 30 May 2024 12:50:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717098614; x=1717703414; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=c6wRNe3Cnz+DyCo7wP/3ZRDce4Mvllz18ZChzx9cU4c=; b=DK18J5hQ/7omIT1AE5qEVd7VN6GqcOKt9UfQEeJ+haPtZmWhNC3Y0hOai0v5T4qdct 0faq+7awtw+LvHpBz4P4IglOLxpkevfGYFVeOrznvIC9lXt5wk5cuyJtG+GXj33PuKKP AdHei6wsrZaEBe5yO4AYBHee7PJrpfRQD19GUjXhyoVaYZTNDIn7Dahz9zDZDtIbrw6c rD8BtaHb8J6+l8Swt3GNUKzXzUK3eGkUtIM4V4U5qDEpwki6Wi9jzuQS5fX2jYFCp4OR UNZ2xR/Xy3rbj1f4uQPblcCq/9hs0sOCdJLWr5jvF+mX/eJYjbboHFnz0vnfreFePwBY KH7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717098614; x=1717703414; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c6wRNe3Cnz+DyCo7wP/3ZRDce4Mvllz18ZChzx9cU4c=; b=WP7vY532HO1s+Knho8LxJlS9YyHFJccLv2iWWwjcSy10AEi3WhO99kxdZQOwuh+axu /g9XQNfqoDgwB7BZNl00I8sqG6zDNtFHYf+TqXfPUVRSA6fXSkeg73rc0q7GVma4FoPs npZPow4mrj0p4Wl1ouVyYoUlGOhYyv3BCx9fP+oK9ZiXBK2Z4alhdZuBo7B8Y4lyyfOo n5+8QfU1hM7BIFtvrjQgxEd2t84CqwAFJ+sxXtvAdBgg1NFkleXZ3y6hk8r2KoXbWPam +9ivWqVbCgfrrWOd9AeqqShc5PKxnyZUyDeUo+pTL4OnB5MSzEeJ6K+/+Qt5BUjkKtph mXlA== X-Forwarded-Encrypted: i=1; AJvYcCXCcnKdHBMFFWu/zEmfXWekiM/EsRZAoXifN6CkIZWKyOz8WGHLroTSAV8Z0XRlvtZy0J2vFR5r5xtHCRC8Kr0FW1w= X-Gm-Message-State: AOJu0YwnpI0rJE1Lye7tUXdCHniSMpLT6aac+co4rHYphZqjAKNXqoFE 7mF2bclOLeBcggHtx5P6HfV1QM9hAMASSRMJH9q+Q9HhFtdleR0++jf/6R+L/z8xH3xYbzzrUns uNar0GRkoXawXEesJUgroBB9P/23gOTQRXsng X-Google-Smtp-Source: AGHT+IF1c1eWKrOm8daSYlxTJweKPmwyeDE632pzkKk1rxsIUu5gUd8ChEDGZNeFvJNrqUFCwlz77fCk9SnWFf3XsQs= X-Received: by 2002:a17:906:3b98:b0:a67:6bdb:10bf with SMTP id a640c23a62f3a-a676bdb1519mr60256266b.16.1717098613679; Thu, 30 May 2024 12:50:13 -0700 (PDT) MIME-Version: 1.0 References: <20240530102126.357438-1-usamaarif642@gmail.com> <20240530102126.357438-2-usamaarif642@gmail.com> <20240530122715.GB1222079@cmpxchg.org> In-Reply-To: From: Yosry Ahmed Date: Thu, 30 May 2024 12:49:35 -0700 Message-ID: Subject: Re: [PATCH 1/2] mm: store zero pages to be swapped out in a bitmap To: Nhat Pham Cc: Johannes Weiner , Usama Arif , akpm@linux-foundation.org, chengming.zhou@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Hugh Dickins , Huang Ying Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: grbb9za5a9aynikpas5atst5zbs1j3tw X-Rspamd-Queue-Id: 2CD67180002 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1717098615-933829 X-HE-Meta: U2FsdGVkX1/IdpIJQUyHnhlfp+pwwB8hUnI1mn0gqzTckNj+fUrgGV0BmiPQiSFqfIjCTDQnHtp0fn2M6jCLy6XWwZhlp1VHwBSa990E3rR7xgrlfQnmc2qyQqLgvkIR45D2V5IztN3NC90V61pmzoJB2kfPuJUdhl4gabrvX5BrY+DvCWbGVmLyR1xqLk65Lq5vj8PcEOEAhrvfUYPrzIPm7Qn+oPAHG5WNYFUFiB82BIP4gVVZyKxowR7vL2UhnVkNAMx7VuKTz0QHVE0FxgKpZrmb8oYOIS8IKGjy6fDrLV2i1BAIr8MpzvE0lxfYAdaFrF22EJRLhxObghTeUnTxCFIv1q2tCPzYgW8tz61f5LmG/vx64vYf44H9wnYOYwCtGQh+IC2MSPPA1KwGZT3NUzstjUK6lXpHa6CwY57b4WeXF/LiuAsWLWHZQxyOSBsrjG0gnd3iYXhyI14Ip2H0n7ymooMhYtinfuUgL6DbNDxXX8XAM5i95X2FWlJnsL5pBTrDCEg/rgUrF/VJmyzg+isx5eKbP8lH7DC/82ZrlbRc1fN8zHSlewOXoM186enIIMjzd7qsjl7DAYuBOYhcHbI5fCfCBV/g/jhunF6yeeIQObxfvKsNJb5alGyUoShnpXOOykC2V/e5UM8HaEQBhBRftrPMol7+DdtWFRNuxO7Bxhebtrh1+RL+8qFjHw744QGo4AW24tWcasXhga9rsCZ8hVrwl4I7LRoyB1/uab4ScHpjcnwy9BjV1+BUkrTVeTdJsXlje+CtrAkHjZRtZIEYENiuS6qrdn3tY0qGFTUiEF85XH4w2xCrJZPltMwDcVLiRIgw/ujz10EgtemcYqVaL/CNNaZ+p8lZYG+kW32ve3ylJbpoH9T+mR0AtZXbmGHumWf5ztMV5l4eEZpSzVtJETdn48RXfly6GTBbyLA5o0yschmRd/Q2VgPfa9PgrmD/LHDF+pR17o7 lXSdq+Ky bmwT/bQZzy+6bv8C4CJarbusLNnFmfOeTmZOCeV+HueTgc0oxvWtnSiaTW+bW/OEyz9sBX6ehiXps9nQCDK8wddKayUUeZ5uIcPM4rH1/KMz5ow3qYCqWhBJhJZyNTuwGfng/v1nwx8XW78mpf5qYbyVcfEbYDvRqYHExPcCbTUD1XVGf2qYFztAKa4wgfCU2aeEfImkWHyRuYNerga4zr+mMiOc3l5p17NY7gMJcIBn8ktil08sSxkP8MHtwXZ1r6doGmkVrkQ9eWQ88rDG6DC8JkwNS4LCcx/R5aPqrI4WvuZmjqyJn2u5bYRWcQx6Dfqo5LpHadhBKdAEuJbbCoBJTpC4a4MTPN8Z8WoVvscrb9a5dJQGm1VeDU7C0ClIOPiP+uOBh5Br7iII= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, May 30, 2024 at 12:18=E2=80=AFPM Nhat Pham wrot= e: > > On Thu, May 30, 2024 at 9:24=E2=80=AFAM Yosry Ahmed wrote: > > > > On Thu, May 30, 2024 at 5:27=E2=80=AFAM Johannes Weiner wrote: > > > > > > On Thu, May 30, 2024 at 11:19:07AM +0100, Usama Arif wrote: > > > > Approximately 10-20% of pages to be swapped out are zero pages [1]. > > > > Rather than reading/writing these pages to flash resulting > > > > in increased I/O and flash wear, a bitmap can be used to mark these > > > > pages as zero at write time, and the pages can be filled at > > > > read time if the bit corresponding to the page is set. > > > > With this patch, NVMe writes in Meta server fleet decreased > > > > by almost 10% with conventional swap setup (zswap disabled). > > > > > > > > [1]https://lore.kernel.org/all/20171018104832epcms5p1b2232e2236258d= e3d03d1344dde9fce0@epcms5p1/ > > > > > > > > Signed-off-by: Usama Arif > > > > > > This is awesome. > > > > > > > --- > > > > include/linux/swap.h | 1 + > > > > mm/page_io.c | 86 ++++++++++++++++++++++++++++++++++++++++= ++-- > > > > mm/swapfile.c | 10 ++++++ > > > > 3 files changed, 95 insertions(+), 2 deletions(-) > > > > > > > > diff --git a/include/linux/swap.h b/include/linux/swap.h > > > > index a11c75e897ec..e88563978441 100644 > > > > --- a/include/linux/swap.h > > > > +++ b/include/linux/swap.h > > > > @@ -299,6 +299,7 @@ struct swap_info_struct { > > > > signed char type; /* strange name for an index = */ > > > > unsigned int max; /* extent of the swap_map */ > > > > unsigned char *swap_map; /* vmalloc'ed array of usage = counts */ > > > > + unsigned long *zeromap; /* vmalloc'ed bitmap to track= zero pages */ > > > > > > One bit per swap slot, so 1 / (4096 * 8) =3D 0.003% static memory > > > overhead for configured swap space. That seems reasonable for what > > > appears to be a fairly universal 10% reduction in swap IO. > > > > > > An alternative implementation would be to reserve a bit in > > > swap_map. This would be no overhead at idle, but would force > > > continuation counts earlier on heavily shared page tables, and AFAICS > > > would get complicated in terms of locking, whereas this one is pretty > > > simple (atomic ops protect the map, swapcache lock protects the bit). > > > > > > So I prefer this version. But a few comments below: > > > > I am wondering if it's even possible to take this one step further and > > avoid reclaiming zero-filled pages in the first place. Can we just > > unmap them and let the first read fault allocate a zero'd page like > > uninitialized memory, or point them at the zero page and make them > > read-only, or something? Then we could free them directly without > > going into the swap code to begin with. > > > > That's how I thought about it initially when I attempted to support > > only zero-filled pages in zswap. It could be a more complex > > implementation though. > > We can aim for this eventually, but yeah the implementation will be > more complex. We'll need to be careful in handling shared zero pages, > synchronizing accesses and maintaining reference counts. I think we > will need to special-case swap cache and swap map for these zero pages > (a ghost zero swap device perhaps), or reinvent the wheel to manage > these pieces of information. Isn't there an existing mechanism to have read-only mappings pointing at the shared zero page, and do COW? Can't we just use that? I think this is already what we do for mapped areas that were never written in some cases (see do_anonymous_page()), so it would be just like that (i.e. as if the mappings were never written). Someone with more familiarity with this would know better though. > > Not impossible, but annoying :) For now, I think Usama's approach is > clean enough and does the job. Yeah, I am not against Usama's approach at all. I just want us to consider both options before we commit to one. If they are close enough in complexity, it may be worth avoiding swap completely.