From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F96CC27C6E for ; Fri, 14 Jun 2024 09:28:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 744008D0003; Fri, 14 Jun 2024 05:22:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C9E56B011B; Fri, 14 Jun 2024 05:22:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0DCB68D0005; Fri, 14 Jun 2024 05:22:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 24D0C8D0003 for ; Fri, 14 Jun 2024 05:22:08 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BA6FF1205B3 for ; Fri, 14 Jun 2024 09:22:07 +0000 (UTC) X-FDA: 82228952694.26.BE05E06 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by imf23.hostedemail.com (Postfix) with ESMTP id B90C9140004 for ; Fri, 14 Jun 2024 09:22:05 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=moVFhCUS; spf=pass (imf23.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718356923; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b5zMuaq26uVXSI7TixMTY+2oASK+Pw9TXRfbwHPPXcw=; b=QjKsRstgpdMdwqXW5YFDZqm8b9egC6+4JYn4YiE9Dley7IEd4iZy7AOb4v47GIQvXHjUSs c2QyDM6YLA4uIf/HbvSC9QnurYF2hDctgn7XR8+6fiJCuJdB19Kj/iB/HgcGMZstyVrxQn NS6bDuJNPhuz1na6H5KHBKFuNz5DeAY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=moVFhCUS; spf=pass (imf23.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.221.48 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718356923; a=rsa-sha256; cv=none; b=Gld/H48PLURR0yqpwdbG5mP9a3y3gy+tsSb5eEcVVPu3LUCn2Qwth+ow3WzEBv97fWTSCa NTG+lb0WfI2pZJSM/Q4iHuPBhxiX39yHsx4BETWIBjYpkuRZrabLTzHnUa1UjlmlqBmgco EPY2H0fJHx+iKWcVlt0z05ZFUe8HTv0= Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-36084187525so269912f8f.3 for ; Fri, 14 Jun 2024 02:22:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718356924; x=1718961724; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=b5zMuaq26uVXSI7TixMTY+2oASK+Pw9TXRfbwHPPXcw=; b=moVFhCUS2N4avjhy8T8v/916nUFXSbGf3z/NmzsVnots1PkvD2Prpiq1g24HDjMlS+ /JQwNrcZPgGCOxauvVO0vIt0V9q9srnorg44AfuMwokCSHwX66mu9CdX+2GJip3uG3uI iY5IbX1QcthKR5i7X4wtsSnukxKMwQg8GM8judB19NUZ+Z3gip/Ejkep8WEj1ni5agCp BXOL6wVP9+Og7VpUbaMGVmx4tHuZW+ACHOW7snzVQQ5pCPKdr24TpVG7kOk07SQsq8Lj gRIMb7b59m1dxJxYTOpBbUub8ttIYyxdiuMmbgrV1G5X7Z5PJgXukFl53VbNc+WI4EHY McAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718356924; x=1718961724; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b5zMuaq26uVXSI7TixMTY+2oASK+Pw9TXRfbwHPPXcw=; b=nMTaXKnZmsUDr/STf4KrllLJdbbjNmNmV4vkI1jLQ/NBFAFqn/TNUzEzQNNJ2aslGo mlK6mRkKlPpPkipYRsgkbL07Pvl/fdNXf34ETQVbG+xFxTe+Uc8jwK4X/MAPqpr48pVc gloAT0AefXz7X+osDpye/Y79ePZf0nSrJ97E2imYIeBkMLoEO/jebd4cgsD8r2QdZouk cuOyx+YLnUYHtBZlItM5XUOJD55dNdH8aenVcHZ5nQBdHTWn6WdrRoUkAV+oAHKDLjvJ QnZitD2t9z8auQyrBD+noC5qh8/HI09rlaH2Xpwwq1f8oCYQwLlA5KwgHKfulC77f1hw vj8g== X-Forwarded-Encrypted: i=1; AJvYcCXmXTxCUp4eBiGGmCf3FQlhnEqP0XKunwZCUpulUG6MayXxSksx3gfq2JEGyAMeC9QPMSt3wD0BGwvCtUCVFk8O2ec= X-Gm-Message-State: AOJu0Yz/dzjadh0TXcpTuA4oVWXY1i7S1wg4RLPcj9+Wk/4/Hw7B5x2a AtnuNyhCPJKQgrhlDdlex2JzA1QpCeR7t+UPHoV8I66v72tSMNUT X-Google-Smtp-Source: AGHT+IG6rBveLT/eg3VVBllWdrtJWTm9L9sPZCw0njzRuK0u1dysIeWh0f/MOLwOzE0dlUNEff1wTw== X-Received: by 2002:a5d:59a3:0:b0:360:827f:1c3e with SMTP id ffacd0b85a97d-360827f1f6dmr1027155f8f.3.1718356924012; Fri, 14 Jun 2024 02:22:04 -0700 (PDT) Received: from ?IPV6:2a02:6b6a:b75d:0:64:3301:4710:ec21? ([2a02:6b6a:b75d:0:64:3301:4710:ec21]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-36075104b45sm3770586f8f.109.2024.06.14.02.22.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 14 Jun 2024 02:22:03 -0700 (PDT) Message-ID: <52df55b7-2e1d-4a95-85bf-19f6680e3fec@gmail.com> Date: Fri, 14 Jun 2024 10:22:02 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 0/2] mm: store zero pages to be swapped out in a bitmap To: Yosry Ahmed Cc: akpm@linux-foundation.org, hannes@cmpxchg.org, david@redhat.com, ying.huang@intel.com, hughd@google.com, willy@infradead.org, nphamcs@gmail.com, chengming.zhou@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com References: <20240610121820.328876-1-usamaarif642@gmail.com> Content-Language: en-US From: Usama Arif In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: B90C9140004 X-Stat-Signature: 9cm87t86yb9onz5m7nd5ojhdcyh9qrk5 X-Rspam-User: X-HE-Tag: 1718356925-778521 X-HE-Meta: U2FsdGVkX199XUnAL6RCqireEA93SuTF+HZxPWsL3YMkNtKb4y5eYc370lf+kYse6apWq3TTfYqqZ1KJTEj5v9tnN747OoLz0EexVPcuUwJaVEF5g9YUKBcFl82c3xFabREzgahupT/3yqrOOk9CkUqnbJZJN024pF8rDJLtlObXlOeZgRH3tYIIMmldGn5uIENNb9RiRxqR3Ecay6Hmw9laJWaYQlx4/vLd4Pw6u5LIFLzVyDxFwSvdJOEPE1K/htKesR25GnTl96jIEzdV+DZYii5GSF4fXoBw3PAu1Lnru2QgW8XIfkoMjZGqUrhFqZ6VF/jbczCQclEGWp6lzm2f94vhQSrq33fRXXpATEV8z1Durr/+gamCGIjkwYEt6DJEv3d2XdaajiKaG956+DYZ1ZhFUV5wB83wsuu7gKzht2nWMHmwNKCNxeUhCHfI+cb+FTxvkD/VzE47XdLArjueYgYEQPiP1DRz1izRhKjWVundTM50Vu6D6OD+fNCznK+6DLqrutZtMr3mFJsbYQ7dQzVG658i91hNh8xTxBMlCr9bEZZAVzgO5RkqInBGr63LPjj3ls0exnL0XxiL+5kq8BKdW1GgtsARVzkIrDcXmsVz0oWKjeGQX5672J821z+YZhHHc4UwI/pDvO5D4HWEOjELrACKlZqFbQab1Xx+UJtNkDxvhJOwWoDDpqMBPcV2qjmdrUlvEIT+8I/XTMLG/hqm8grs4Fvc+GqMeIT/aVP9xDxhdzVxY513qwWZMp0ChIU2qZKPEjI8mzaNgwmn8WQ+Adis6p7iKL+b1VbT2r2C5oq1pMuE4dNZJAIapHG9FiUT6/5LpBXT4w18nX1OQt/IGrVp+6wDJOZ+oDlXwxBD8/PsbsBypT902immIuqPB9QVM/BWLswavOAcMhZjKao7EZNjG7uusOHSQ9Rnk8xGOM3SthGyEGQFCvtamSB9RMSlSO9W1LsmkPG BeRFeqm1 yhd5LDCieq5AFQm9Bw9E7xelikkwKRO5LMXyTxAJVtAC2aqxy7o0VGNSGTXbDugnyXnL9gIAnYhqTV/QSNFDwTBo1XID3Zi+7IndouUx5RURbL6MnIHjm2/o5Fpn5qjehL6jzfiNZ/VVkFAIpFgAScoUr5Om4tKYLC+mzRy8flSa5M18XrXymXDwCllXO0Zlkycrr5Ye+ZNRKt6zX0w4fs4jEUBy3Spmj7LOPkT9a1Rlel++D7UsU1+9bl5Um9QqXNMEQAqbezIR1t5NUOqbPbXMn+ILLFzJSxhJrA4keZMHom9tzVsl9mPJiSjBowgFd64FFMZmX1rlsxdhsKopckuqhThMJr7H0kt0wmrFs0wYC2rGrQ0K02ZVC02PbzOO7Jt+F+WPxipBUobrX32OLchDIxOHClZDlkDMeNaCiKrQqwPVwf1AMBeQO3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000010, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 13/06/2024 22:21, Yosry Ahmed wrote: > On Mon, Jun 10, 2024 at 5:18 AM Usama Arif wrote: >> Going back to the v1 implementation of the patchseries. The main reason >> is that a correct version of v2 implementation requires another rmap >> walk in shrink_folio_list to change the ptes from swap entry to zero pages to >> work (i.e. more CPU used) [1], is more complex to implement compared to v1 >> and is harder to verify correctness compared to v1, where everything is >> handled by swap. >> >> --- >> As shown in the patchseries that introduced the zswap same-filled >> optimization [2], 10-20% of the pages stored in zswap are same-filled. >> This is also observed across Meta's server fleet. >> By using VM counters in swap_writepage (not included in this >> patchseries) it was found that less than 1% of the same-filled >> pages to be swapped out are non-zero pages. >> >> For conventional swap setup (without zswap), rather than reading/writing >> these pages to flash resulting in increased I/O and flash wear, a bitmap >> can be used to mark these pages as zero at write time, and the pages can >> be filled at read time if the bit corresponding to the page is set. >> >> When using zswap with swap, this also means that a zswap_entry does not >> need to be allocated for zero filled pages resulting in memory savings >> which would offset the memory used for the bitmap. >> >> A similar attempt was made earlier in [3] where zswap would only track >> zero-filled pages instead of same-filled. >> This patchseries adds zero-filled pages optimization to swap >> (hence it can be used even if zswap is disabled) and removes the >> same-filled code from zswap (as only 1% of the same-filled pages are >> non-zero), simplifying code. >> >> This patchseries is based on mm-unstable. > Aside from saving swap/zswap space and simplifying the zswap code > (thanks for that!), did you observe any performance benefits from not > having to go into zswap code for zero-filled pages? > > In [3], I observed ~1.5% improvement in kernbench just by optimizing > zswap's handling of zero-filled pages, and that benchmark only > produced around 1.5% zero-filled pages. I imagine avoiding the zswap > code entirely, and for workloads that have 10-20% zero-filled pages, > the performance improvement should be more pronounced. > > When zswap is not being used and all swap activity translates to IO, I > imagine the benefits will be much more significant. > > I am curious if you have any numbers with or without zswap :) Apart from tracking zero-filled pages (using inaccurate counters not in this series) which had the same pattern to zswap_same_filled_pages, the nvme writes went down around 5-10% during stable points in the production experiment. The performance improved by 2-3% at some points, but this is comparing 2 sets of machines running production workloads (which can vary between machine sets), so I would take those numbers cautiously and which is why I didnt include them in the cover letter.