From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A757AD70DEB for ; Thu, 28 Nov 2024 20:57:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1DCC6B0083; Thu, 28 Nov 2024 15:57:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECD6A6B0085; Thu, 28 Nov 2024 15:57:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D95036B0088; Thu, 28 Nov 2024 15:57:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BB5C66B0083 for ; Thu, 28 Nov 2024 15:57:13 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 380C91A08C3 for ; Thu, 28 Nov 2024 20:57:13 +0000 (UTC) X-FDA: 82836713778.16.C4C08C4 Received: from mail-vs1-f44.google.com (mail-vs1-f44.google.com [209.85.217.44]) by imf10.hostedemail.com (Postfix) with ESMTP id 33C87C0003 for ; Thu, 28 Nov 2024 20:57:08 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CEfhNLtP; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.44 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1732827428; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xt25J9rLexkRSwtqiTK7ze8wNZwwl3FLbzeYYa27FBc=; b=UYPB1SVUbwhGwPFilIx7hxOFz1WMy3wz8Las9TQSh4iVr/NGBom3kud/ZDW3AM+ZDjFfC2 o+ESxBl8+lpa50jzDOqXmieR3qOQGa5DvuUwwraAvcJrniwRnkZN8LRcwLMYSFeAtoqGnK I0u+GKsyDGgQW7B3l+lHu5BixdBkDIk= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=CEfhNLtP; spf=pass (imf10.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.217.44 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1732827428; a=rsa-sha256; cv=none; b=51ruu7b7AwRJ8glqZ1gmwUQyWAiqp8f9Skwyn3rylv1DgO55Nhg/10HJ13r5WXNGXkDhx3 Ksq17d1UGGHqb8t1WX5HFcWBm1OyBVh+YBAOgBatW44kOpPNFJWgZAWAoHj+8LFjDjfIOY BRFo6jaWanU1pb1tum99wBIp/pj3QkY= Received: by mail-vs1-f44.google.com with SMTP id ada2fe7eead31-4af173cd0e5so389483137.1 for ; Thu, 28 Nov 2024 12:57:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1732827430; x=1733432230; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Xt25J9rLexkRSwtqiTK7ze8wNZwwl3FLbzeYYa27FBc=; b=CEfhNLtP3OBQg3Mv/ejlfGZrY7QX5BpgM7tRZ5loix3S+TJr6sAWW3MEF0tYX3Ns3B Hum6I32T2g2mpypzip0i5AFOdByfCv9T+F+g8iIGxAlSm09kqKppXC9bujFP05W1JmKC Q1QBTEoAbyqZIc42w86G7Xf6LnxXaB4IHx1jRHzn7X4+57J07PVWiv9E4s4tGQN29FQo Wi64R5QoAnn3wSVMvlhVNl5jB/ukVcZe9A07emPULlY6tYeH3tzquiX0QXhMduLGkK8M LK0BEfPQfkkc4rO8yKeicIOOiAC2OLJSp/sduZghs6EeKfakrM/y4WtKS8b3grcFY2+F SUcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732827430; x=1733432230; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xt25J9rLexkRSwtqiTK7ze8wNZwwl3FLbzeYYa27FBc=; b=mqnVNqM/nuEqNgpuV0CZQU06YjdABddQxGjw17o0Ubt8jAeeVgCKTuqiwFKgBbsExx iVGYR+8X8KsoWrTJTL8A6ZcXxACh18roUIhFH3HCvAZ7Qk7RaeO5HBfjOsNe+y33TVZc hwpXQ11Nh3hJRgMJlubJF7tZqJa99tqxdBfk/A/Qt4MvRaqzxR/sNWMMVU9PuF1mAib8 sgJnCudJSgx9NTPdlJ1+87UHRw/OZDZQcAaTeUurIcUQSNiUGL+6KTmm28BYgBEdqqfA +tdv9ouKahZAT1K+Hs2CciicTHywhtT//7UvtLP85vSTduBcXcsqjaWkyQGlnHejHbao h8Pg== X-Forwarded-Encrypted: i=1; AJvYcCUH8eKsqdhilzTXUgDLmyJtGER3EXGPwtK6+gMOmn+zFTNO/psm+BuQIyqGPmeiJpuqkmyf8/wZxg==@kvack.org X-Gm-Message-State: AOJu0Yz07Nh3e0CgRPq0C/e4E4ufIgrsfcieuayRyROJSar4AWrLWBLQ kp8O84FFBXoxh1j4zDAqFhNsKfCQ/W6Suf0HvCX5fMDcfKz8K3wJ73w0bv7PfoP6/ffvfK92tGo Pfpbhhz+3okpgwQAZrLoF7/5xbgVrr5Py X-Gm-Gg: ASbGncvdP9usSqyar4Q1yDK2974/My5N7epYiJRpyDFoY6ihyxJ02PjRM9TWhUc5IQy Ft1+miOD5myxgKJb+i6M6pA3hjHkKpORTWa7nUOgt6mza6HpmGakK+J+WxiGf/+XGRQ== X-Google-Smtp-Source: AGHT+IEZxsMN4gvwxqUgXwCg2TXiu2HdtGvItisIinKFnidEaJWWOzGqFHgLSy3X0awccxS7eVXYxryL8u7PEmFrQoA= X-Received: by 2002:a05:6102:6ce:b0:4af:3f27:5732 with SMTP id ada2fe7eead31-4af4498cb08mr11500273137.16.1732827430464; Thu, 28 Nov 2024 12:57:10 -0800 (PST) MIME-Version: 1.0 References: <20241121222521.83458-1-21cnbao@gmail.com> <20241126050917.GC440697@google.com> <20241126105258.GE440697@google.com> <20241127050445.GG440697@google.com> In-Reply-To: <20241127050445.GG440697@google.com> From: Barry Song <21cnbao@gmail.com> Date: Fri, 29 Nov 2024 09:56:59 +1300 Message-ID: Subject: Re: [PATCH RFC v3 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages To: Sergey Senozhatsky Cc: akpm@linux-foundation.org, linux-mm@kvack.org, axboe@kernel.dk, bala.seshasayee@linux.intel.com, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kanchana.p.sridhar@intel.com, kasong@tencent.com, linux-block@vger.kernel.org, minchan@kernel.org, nphamcs@gmail.com, ryan.roberts@arm.com, surenb@google.com, terrelln@fb.com, usamaarif642@gmail.com, v-songbaohua@oppo.com, wajdi.k.feghali@intel.com, willy@infradead.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, zhengtangquan@oppo.com, zhouchengming@bytedance.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 33C87C0003 X-Stat-Signature: wor5j7sjzp6pbxkhioec8deno8q5h4mu X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1732827427-264819 X-HE-Meta: U2FsdGVkX1/uRy0cuTKP2f+hDxULfc1j/uJ8TU/JVfEkcVlW4LX5BuSZHr8UMZN0HEnaKlaELnHH/F2e83UPGzL1gZP1ks6du8ohLj9igh1HXrjGjuX3ssZ3AFlcI9CiJkcQcj+UwGltW+fzegL+Mp+lP50YsVaw29pDbFhNrovTmLJvumXjJtsPpTAacQRhm3FULRrzC4IHvMNk10TbuLVr4npBNkAzeJriUNtFwEhlIsLVNWmMlGfOWQuwNlNCdGJ+hK8YAw12XkSUi9O0TJBYSX+Wc3spimUyOMeOz4z7pflOc11h/y2UeOaH3nKtfLd0D2PSqbtSZqb4MRGNsdq+8GMBgV+isebpo3wikF3I2QeA/mIzNpMsg3ZN9E3NRRJTcZaHG6CLJrl+AcShFIbAEG8AypM/5n6u0W7cH7KpIRcOGV/XOzOgZ900iFibxc/M8u7k6ovEU2HBt3B1l1rpEOKDqIWgtQ5JTEI29ZMFmdZa1ZCCqrxHmF99UIyvCnssC4di5WZ7w1FMqGOYu15PL+FKmH5R6H5BcKu/3Tn0xFfmEvHR247FEK4wuloC7y2WhrrqoSqoXs33OGU5lCHo9sIfuatpqwB8AMc9yjbwoWSpOFqQ4X28znwNnpXmMUusag3bCa0PUVIEsx+CUeEjd6jUbge7qG7URl+/NirtIB0JQ/6XPDup3MmJGHhyxoYRmSipO68GpQq0xdR091PdFa+i70IJUyJYqFomrSx8786W8eAGHXAO/qDlcC8gtwjGMUDhx+VRxhEMx3/LGK3bOLGIMzzQMf93YdZdtF8M/9WbMJsQeVJAobajeU5i13vvRezJL0Inn7NS6vffLw/jQT1bx64UnhzHG9qpR+iLTtcx68jkWpfSKanvtfuwFPRLm5WPoaHgwT4eYBFFrCDCo0WOK8i/O2cDR0lJ+CuaYFre8whFo2e6yrQG9GFW0BMxfiur3ltj8OSjoXN 7g7dNRj0 VUsuaKu+LZotHhUsmAbb3s7CpvoAehBiomivT9PoT4fH1zgWZF4jV9A3UnlNq01KgWeUS5c5hrgi7uI+BFkMxPBn50uPkffioJwkH44Ay2yY9koXYROVE731OQmEkcAvPnLxAOjc0vgQx6YbrU3jT+bDOT0DksRCdByC772CLjKQXlcwcrJsVDxgVKRNpMZosEGDYyhgOFTTOMnsOtdifPL2psOCI3jB5UpDYAGKjEBQxtJoCdcDh8i1APSTXP/MIo1ChEubArpyF01WjwSFODLArKF70rAK9UzbE1Pb5DlXAX0SjcKB2XSAxwUaxLfRgGLcHgiXZ3kjV+4tNk/zI39xxTNbyI/qMKiBjoX/3qNq+Ptcf/AHPPYHpJw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000148, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Nov 27, 2024 at 6:04=E2=80=AFPM Sergey Senozhatsky wrote: > > On (24/11/27 09:31), Barry Song wrote: > > On Tue, Nov 26, 2024 at 11:53=E2=80=AFPM Sergey Senozhatsky > > wrote: > > > > > > On (24/11/26 14:09), Sergey Senozhatsky wrote: > > > > > swap-out time(ms) 68711 49908 > > > > > swap-in time(ms) 30687 20685 > > > > > compression ratio 20.49% 16.9% > > > > > > I'm also sort of curious if you'd use zstd with pre-trained user > > > dictionary [1] (e.g. based on a dump of your swap-file under most > > > common workloads) would it give you desired compression ratio > > > improvements (on current zram, that does single page compression). > > > > > > [1] https://github.com/facebook/zstd?tab=3Dreadme-ov-file#the-case-fo= r-small-data-compression > > > > Not yet, but it might be worth trying. A key difference between servers= and > > Android phones is that phones have millions of different applications > > downloaded from the Google Play Store or other sources. > > Maybe yes maybe not, I don't know. It could be that that 99% of users > use the same 1% apps out of those millions. > > > In this case, would using a dictionary be a feasible approach? Apologie= s > > if my question seems too naive. > > It's a good question, and there is probably only one way to answer > it - through experiments, it's data dependent, so it's case-by-case. Sure, we may collect data on the most popular apps (e.g., the top 100) and train zstd using their anonymous data to identify patterns. We=E2=80=99ll f= ollow up with you afterward. > > > On the other hand, the advantage of a pre-trained user dictionary > > doesn't outweigh the benefits of large block compression? Can=E2=80=99t= both > > be used together? > > Well, so far the approach has many unmeasured unknowns and corner > cases, I don't think I personally even understand all of them to begin I agree we can make an effort to dig deeper and collect more data, analyzin= g as many corner cases as possible but many unknowns are a common characteristic of new things :-) > with. Not sure if I have a way to measure and analyze, that mTHP > swapout seems like a relatively new thing and it also seems that you > are still fixing some of its issues/shortcomings. A challenge is determining how to make mTHP fully transparent (e.g., not dependent on sysfs controls for enabling/disabling) across various workloads. The default policy may not always be optimal for all workloads. Despite that, there are certainly benefits we can gain from mTHP within zsmalloc/zram. Thanks Barry