From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ADD0C27C6E for ; Fri, 14 Jun 2024 23:04:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 942066B00EB; Fri, 14 Jun 2024 18:35:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CAA26B010C; Fri, 14 Jun 2024 18:35:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 745CF6B011E; Fri, 14 Jun 2024 18:35:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 52F576B00EB for ; Fri, 14 Jun 2024 18:35:08 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 00F27160620 for ; Fri, 14 Jun 2024 22:35:07 +0000 (UTC) X-FDA: 82230951096.28.08E2BF1 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) by imf10.hostedemail.com (Postfix) with ESMTP id 2BD43C000C for ; Fri, 14 Jun 2024 22:35:06 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=muI4Wz5j; spf=pass (imf10.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.52 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718404503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=E0I7vnsNiEgJ3h6oEt+1hD/h/hGcMM12yiRg06OyFvY=; b=NzYkTqvHbAnwoRHemde2sbSnBcXG+QUMc0vHHGy58OoqNswMfa5D1ENE9ZYOzlbUVXwrfF 2qDtLGRnmMqRxxdXDYfxgZc8G4dmQfIEnbuSkemGOMP+BRFYZIc55p8rw0fD8+CurTdsxm g2h3egSHcjUHanMa+4nAePBvLP1M2mo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718404503; a=rsa-sha256; cv=none; b=WgmExBZSY4Bq5zh7hPDBoj2OGqAWPvc6KQZGvSMyyqoA+2NL7RIwrY6SM2dJXdsvKCZkqV aT5hU/LU25v1CPVqROSkQeQy9t7yWHjK8QKvPtRPJN+esiWbMPrKTKkKVSdX/HBt3ou4f+ Dnyy97Yt9vLFt0moDj/kkW7rzCEd4Vo= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=muI4Wz5j; spf=pass (imf10.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.219.52 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6b09072c9d9so21280816d6.1 for ; Fri, 14 Jun 2024 15:35:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1718404505; x=1719009305; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=E0I7vnsNiEgJ3h6oEt+1hD/h/hGcMM12yiRg06OyFvY=; b=muI4Wz5j3dGO/Fv3a1T3Chw6K6b1JE6Qzy0Pzc/mAMJHBld9kN5TictacL3bcXOLYh mo88dZT/CvkeWsFPCI5uSX0h1YDjgSesMbd7fiLSimNvj/00iZUFpD1f5hBQnODURnUc kTFaqwPn9+hOA6Xg6L0Rx6riNhBKErt72Pu2Z6Pxhfugw25ybn/yHC3K4332bYoFCci0 C7nFUYZswnM1GxgBAh1hJgK6DqwjDt2wG5WUpV+9oKEJEtSJrtyYJavmmbPItCxH/Qw0 BOUnaSpsmUSP17TQ7TVEZ+bMpRfGlIa+ctiCESW0EsymPg4VznS1lqMJY9hbUE2KPSBb 6wIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718404505; x=1719009305; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=E0I7vnsNiEgJ3h6oEt+1hD/h/hGcMM12yiRg06OyFvY=; b=kvCnLk2Z3rcO0FX92gIs/q5csc9anwOlDNvY+h62DTo8/RH5vkxy9KuCeLz0bV9/1f zDdGinDbDHC+J5dNRpVFBtGWflA1js3rDPL7jxDIy4cJfEVoept8oTIeb96uCKndRSmo ClqbAO2oO9RckebwLWvRSYDTEcjBcMYvl6xOAGR8N1oLeUxEYs/gU/h0haQCGLy8inr6 oekpKpdIuLL7q026yMnIp74MiIkn6HYOBjbKWxJnXu/qEALgiF1lAyjnOydU8+nU8ryW pZtmZ6MaNjlBQlQ57rQKlamRsy8rgd31dBZfhnBEslEEfPEK/4irV0/qyC76u86TmDSX BHUw== X-Forwarded-Encrypted: i=1; AJvYcCX79pElEEF5aYnK1GIpibR2KlxNRVVkwUgqOFrU7DEuCYK0OWpbwkT/j7L4+7JteJJWa3+ZiV+W59woCweB2oejSrA= X-Gm-Message-State: AOJu0Yw7lJY0WHCcJH9K6BpM5sCf4ptN+5dfra+4fuOQA48HSivDhouE nG2HSGz+TiHpR2UWkRSKi/0/qZ2m5hp867FG1QwKHcwLaWb11+paMWaesREY6PLfOyIHQ4Jbms9 qRuc+AbyY7o8AjACcH27Nk9oa1QQ= X-Google-Smtp-Source: AGHT+IFHhWMz4bQc3O+IUtEZZ51wsvrPFXXryj9Yt8+5ij9PiDJPELvtZ5+wvOScKiEgqSJEtU7THcVmVIv2Tc6V2s0= X-Received: by 2002:a0c:b711:0:b0:6af:3d00:bbfd with SMTP id 6a1803df08f44-6b2af2fe509mr75117616d6.27.1718404505192; Fri, 14 Jun 2024 15:35:05 -0700 (PDT) MIME-Version: 1.0 References: <20240608155316.451600-1-flintglass@gmail.com> In-Reply-To: From: Nhat Pham Date: Fri, 14 Jun 2024 15:34:54 -0700 Message-ID: Subject: Re: [PATCH v1 0/3] mm: zswap: global shrinker fix and proactive shrink To: Takero Funaki Cc: Johannes Weiner , Yosry Ahmed , Chengming Zhou , Jonathan Corbet , Andrew Morton , Domenico Cerasuolo , linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 3ixfxpyoqajnt4g149bbnx4areppksoa X-Rspamd-Queue-Id: 2BD43C000C X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1718404506-898327 X-HE-Meta: U2FsdGVkX1+cD11V5IBK0Rvp7Nssqh/hjSWCJaxv1olHK3NCgAl//SYrTWTct3yhgmBaWNjgm7ya0hD+bjd9Ryw7MHFzkGjHE2XJdNrzz8Gi7S/4eGBYAZXfWH8DgQnrcT0aoh1nNcpBnsS7jpgBp6N3UKa6llHBGIsBFMMGKbzDc0ZRGrIlNYXNEcotbJEsL8fyrNIaqleol7WClLLiaPST+MWc1hnlqgqoiryp3BgIexfzmHWNmLUBmZpJXFrVc4VJQFvIRvQMezJinbAp3rtUx5Zp+VT0W3oJSiHpO+NKaB/HtNHidIva9kg7HSRlvgfoq6cjXDOyJkN3irOri9Nf1XrVm39nm4udK4suRZ1/nGxnmI53mgDqBNGZfnF2zUnrkynStfeLlHhmMQ3ULnkfuxyUzDxBcLzC4GpAlyKWPc2goQX4BK6O+1rd/k7jp58BkaBMbBV+4czK9eCAqAjosmQ7layX6mbuGLfWrwskGrK629YItRWkOLHJ+YlkJ8dwDxXUIufT/6mRv8geEDS3oy0kWCPn/yboGSPqHTARwXdaVFIObqvHm+CVDDdGcDN5rWPrlv/vY7qxAxoJMpWTTtHI/NCvMmbvdFvJoAAE+QPUw1tQA0vWXtJnnlynl/Jlljx0viAEw2iS+eNa+3qz11G8rE3zZPYePoWrrRl/mYIbvr0CTkeln0C1aN6aJ8WlpthyUIIuWgQO60e/+mjPbHbw5Ko8PZDg2L3v7QgBPxX19cePPgC15xVEBUgGRW+r42S2es+Pmbc5WbM3GDrSue7vxdlOXfTGSR5mWU/5TYEwUaQR+3sULdNos9Biolr+ip4jabwzCwu3URS6atV+vfJTI4cU8EcO9LEfD1mzpkgu2qUbC5+XHQsn7ak3DSyQXZcR8n3ODzhNCH/8xS3p9WCZLx6N0bSbo6sv5xfjVHAIYwYJ4H4+/INkrcgaYAR8e5zUE7+uKZGBeIX +G3pU1Ml +NvFTEXRcmn4mAx0JiSm8cRP6u5MZPAbO+YHjXlHdZkCMuX/P+qZ6TDYKg5wQOV0qbvYtcTvwYozTIH+Y7DzllAj/5nES2ag7cB3Ke/04AXno7bh9SSYXQU+sWU4D6Vua8fEeHPV745Ix2gyKvjsUMOICkvjPPJGOx87qYwy9eiE5SahJZ42iDbHDvCTfnmbdbCankGHXib7VFe0kIcxyjbb5HDfxWaZzxA3EZMcwBNT3QxJ8qkWwVBOWNE9F7iXq+Z9I2R4gpTcq7BQPBA/rhN58YZEX7yueTxfK8PWkG38pMtCmRjMlVohRnCGpeJhexH/n X-Bogosity: Ham, tests=bogofilter, spamicity=0.001208, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jun 13, 2024 at 9:09=E2=80=AFPM Takero Funaki wrote: > > 2024=E5=B9=B46=E6=9C=8814=E6=97=A5(=E9=87=91) 0:22 Nhat Pham : > > > > Taking a step back from the correctness conversation, could you > > include in the changelog of the patches and cover letter a realistic > > scenario, along with user space-visible metrics that show (ideally all > > 4, but at least some of the following): > > > > 1. A user problem (that affects performance, or usability, etc.) is hap= pening. > > > > 2. The root cause is what we are trying to fix (for e.g in patch 1, we > > are skipping over memcgs unnecessarily in the global shrinker loop). > > > > 3. The fix alleviates the root cause in b) > > > > 4. The userspace-visible problem goes away or is less serious. > > > > Thank you for your suggestions. > For quick response before submitting v2, > > 1. > The visible issue is that pageout/in operations from active processes > are slow when zswap is near its max pool size. This is particularly > significant on small memory systems, where total swap usage exceeds > what zswap can store. This means that old pages occupy most of the > zswap pool space, and recent pages use swap disk directly. Makes sense. You could probably check pswpin stats etc. to verify this, or maybe some sort of tracing to measure performance impact? > > 2. > This issue is caused by zswap keeping the pool size near 100%. Since > the shrinker fails to shrink the pool to accept_thr_percent and zswap > rejects incoming pages, rejection occurs more frequently than it > should. The rejected pages are directly written to disk while zswap > protects old pages from eviction, leading to slow pageout/in > performance for recent pages on the swap disk. Yeah this makes sense. The hysteresis heuristic is broken. > > 3. > If the pool size were shrunk proactively, rejection by pool limit hits > would be less likely. New incoming pages could be accepted as the pool > gains some space in advance, while older pages are written back in the > background. zswap would then be filled with recent pages, as expected > in the LRU logic. > > Patch 1 and 2 make the shrinker reduce the pool to accept_thr_percent. > Patch 3 makes zswap_store trigger the shrinker before reaching the max > pool size. With this series, zswap will prepare some space to reduce > the probability of problematic pool_limit_hit situation, thus reducing > slow reclaim and the page priority inversion against LRU. Makes sense, but do include numbers to back up your claims if you have them= ! > > 4. > Once proactive shrinking reduces the pool size, pageouts complete > instantly as long as the space prepared by shrinking can store the > direct reclaim. If an admin sees a large pool_limit_hit, lowering > accept_threshold_percent will improve active process performance. > This makes sense from a glance. Thank you for writing all of this, and please include actual numbers from a benchmark to show these in your v2, if you have them :) > Regarding the production workload, I believe we are facing different situ= ations. > > My intended workload is low-activity services distributed on small > system like t2.nano, with 0.5G to 1G of RAM. There are a significant > number of pool_limit_hits and the zswap pool usage sometimes stays > near 100% filled by background service processes. > > When I evaluated zswap and zramswap, zswap performed well. I suppose > this was because of its LRU. Once old pages occupied zramswap, there > was no way to move pages from zramswap to the swap disk. zswap could > adapt to this situation by writing back old pages in LRU order. We > could benefit from compressed swap storing recent pages and also > utilize overcommit backed by a large swap on disk. > > I think if a system has never seen a pool limit hit, there is no need > to manage the costly LRU. In such cases, I will try zramswap. > > I am developing these patches by testing with artificially allocating > large memory to fill the zswap pool (emulating background services), > and then invoking another allocation to trigger direct reclaim > (emulating short-live active process). > On real devices, seeing much less pool_limit_hit by lower > accept_thr_percent=3D50 and higher max_pool_percent=3D25. > Note that before patch 3, zswap_store() did not count some of the > rejection as pool_limit_hit properly (underestimated). we cannot > compare the counter directly. Sounds good! Did you see performance improvement from lowering pool_limit_hit etc.? > > I will try to create a more reproducible realistic benchmark, but it > might not be realistic for your workload. Your approach is not interfering with our workload - we don't encounter the global shrinker in production ever. That does not mean I will not consider your code though - this is everyone's kernel, not just Meta's kernel :) What I was concerned with was if there were any real workloads at all that benefit from it - as long as there are (which there seem to be based on your description), and the approach does not hurt other cases, I'm happy to review it for merging :) And as I have stated before, I personally believe the global shrinker needs to change. I just do not have the means to verify that any ideas/improvement actually work in a realistic scenario (since our production system does not encounter global shrinker). Thank you for picking up the work!