From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D91BC27C52 for ; Thu, 6 Jun 2024 23:04:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C5CE36B008A; Thu, 6 Jun 2024 19:04:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C0DC46B0092; Thu, 6 Jun 2024 19:04:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD6B86B009C; Thu, 6 Jun 2024 19:04:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8E2766B008A for ; Thu, 6 Jun 2024 19:04:38 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 33CE8160DA6 for ; Thu, 6 Jun 2024 23:04:38 +0000 (UTC) X-FDA: 82201995036.19.E468FA0 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by imf10.hostedemail.com (Postfix) with ESMTP id 5A927C0022 for ; Thu, 6 Jun 2024 23:04:36 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VoK9xSof; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.51 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717715076; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q8bxSKsew92PdeFJ/YvuL9+HqLlwXrZwlBtMxipH6zQ=; b=Q4zRfN3RG6v+YXLp7uB5luCM6iXM6ezGGmyTxLR22qIu4TFYv6+O9xNdWro28p58JWMEUH O+Xsn4Qc8dT1WawneA2gNAxWro/H6lhOOgY/wLrytuDHoM+tLVWf/4ilVi959OnVxWmRDL 6jBkixBBpVUB9QbSxuxBoXclVKr4oOg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VoK9xSof; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf10.hostedemail.com: domain of yosryahmed@google.com designates 209.85.221.51 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717715076; a=rsa-sha256; cv=none; b=yC2Equbx7qykoe2k+cSBmpTw8iSFT7jtKDawnGEZgDCeHGvG3U68XsrnUqEho92sf++qk+ WCiXLlKzRlhIF2eIJFGc/0WLXxPO0AeIs4tGkZEIQd9VV6oGg3fqFTckg7z6UtucxSo5Cf TmMEoFyplZPJbwRD/tIgEePJ2kFtAFc= Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-35dce610207so1724047f8f.2 for ; Thu, 06 Jun 2024 16:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717715075; x=1718319875; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Q8bxSKsew92PdeFJ/YvuL9+HqLlwXrZwlBtMxipH6zQ=; b=VoK9xSof7gFpjB5oB4j0Yg7rzGZNjyizI1Ztwp7wSfsLZQDn8EU+owYMnKqfDizvkA ui87UQKOwW8fhkdp14Xmwi/nPR9GuqCYTAkCtXIuyiLCxc5pCouQoY1BGWLq77SCJSrn NeC3YNUuM4fMtpou2F8U6x8EiQt59GsVPtDKxjF0oQb64AxCzGISFGpiC1ChX2vbfKmr vjMjG++WyPue9ybC00B/jL9JS0SPGGLc+6QWaOnrzzEP0Ja6/VMN991nFRJDWI2ZifXW eBxepwsgDtpbmkvGdavitdj9jb5KsybVE13qRMFEnQRu6liDc7iw56FmjIAer0wJpsvd wevA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717715075; x=1718319875; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q8bxSKsew92PdeFJ/YvuL9+HqLlwXrZwlBtMxipH6zQ=; b=j9+V4RSJHMof66Nip5L8mmx5xmi4w1TwFM9ICoGA2ucPWauqidv/7aLN4w9a1AS2oD 5QNkVD5xXigEbP1l0OFjaFh4ZS08TQEqh/9v0dqikiSGjv3qOojwQoQvH2pxJoWKGz/a O+dzrFY1f2GAGUMxC+dDk2a7C6MqoOgU6mraEMKX1vF7YFqjgB2+eOVTPqnT0Q1795ct Zg0eRcHS+uxUtRWIAjff/QHwPJwk1zIerWywG01wkpSQSfktkkvB3At84VxHT8IyQ6Jy BZsIR7sUX3pka2f9iSMBs15PcZraJaRnEz7YDc4UpZUf548VsVmprdfJ5eYcVpzNgFtS NgUg== X-Forwarded-Encrypted: i=1; AJvYcCVxubHL/SZMCEcxthJW01mQ/hVqYp0WUB8Ft6q2Mn1GkbGeIGgzYtVyPJ+hjHHPGHiDp52FNvqf9ndHx+SFbtH7PHM= X-Gm-Message-State: AOJu0Yzcy9yFKXswnUCOPOsW/rrEkCAdB+jQ41/bkZt64VCpUEFnHXMF zxkuX+0nsgUmOghnqKUMlERF/TfrG6WzOO55Fq47UM2oJc7INHHLWrm2sdnYBHVEyEbOjprvDBI XN5gitbekTCrVQYDBsZpibpzvn+ogNz3T1yB0 X-Google-Smtp-Source: AGHT+IGfKu8GCyHss7A0fzkmGAXuAZw8ZVlbxzPX7Md011kCn7oVjvjpflSy8Gs4VzJFgsXWjLX4rVqw9mOPvuUX7+E= X-Received: by 2002:a5d:6da2:0:b0:35d:a567:40b8 with SMTP id ffacd0b85a97d-35efee11d3fmr1050487f8f.59.1717715074495; Thu, 06 Jun 2024 16:04:34 -0700 (PDT) MIME-Version: 1.0 References: <20240604175340.218175-1-yosryahmed@google.com> In-Reply-To: From: Yosry Ahmed Date: Thu, 6 Jun 2024 16:03:55 -0700 Message-ID: Subject: Re: [PATCH] mm: zsmalloc: share slab caches for all zsmalloc zpools To: Minchan Kim Cc: Andrew Morton , Sergey Senozhatsky , Vlastimil Babka , David Rientjes , Christoph Lameter , Erhard Furtner , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 5A927C0022 X-Stat-Signature: m3j6jksw4k6p4k3mz36bsuw6jpb638p9 X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717715076-883055 X-HE-Meta: U2FsdGVkX1+GZRR3o3FyNTWlONjF1qE+reVKJ6NvvH8lHorDyMNzAA5fb73Imqw6h/4Y10dbqMo/znDqpKQA6gt5QmXesApHdDwCODda2WDBFS+XW4RZ0ddRKbQNOORG2gyhjc6cdje0c+j7GkSW4oMeXsvQmwwJrLVm/DulHQMeW5pUcZjOQ9Iq5Kkzd5vrs5P5Lj4rJvwRETL77jdQQmsWzqF7jDP6BzjS7u62LlQ6zVin+6SON6HOvWYkpVRdXlC08AsyRycY3IqkYxnSrgrHF05K2GKAEV+TIQ3dPZyzrftt/DhQOl9AIpEB3oDzv36uzZL8p1KuLCWcXZ4UpHCTuzeFQ65E82WJVQMACMgvbRMU0x/dtiEWgzJ5Xy4f6Y36lErinicZ3va4Na6VCwWsgWwbKt6KRPW9a5RVzt3KQTqB9ofRaODuEnXBCbefZu5vqX4Meay3kWLhk3gpaESdiu3fvV9B5KkQzV7wn2oqOfTJJW9cMk1BKVjQedXGPgzPgaF729p/m4qv3u3dAWo03f1BvdjFfX/0tfCEfJ58ZGZ+v9I4vZ3uKd0iS2jL62ABj2XCotKfbJ+55Fu66UiuWvU+XP2sb6RJVK6bU8f+hnv8XKzRFeIPxcDVyAKlhBZrBRFKKF7Ky9PfGPHvgVY3L/gLwhoIt767d1uEbXM/RYEam6HVCTIiZ+isuRa9TATZGPL1UpPYfKvG24W58pS//jWCAj03nk3WRO5ltCOQOckjQ9UEnQTvE10ZgW5eZQKh769+3zBp3y005dyt5wsdybbBcQvsprrNEX0Q1peQqPms70oMOrGRqlZZUDdIYDSXEvC/PB3fH+P3i4gBKIFaoOXw5tpDUvZeBacrkI1BkUmyKduZyHDJ01gAJMMkgbPbtlTK4ffaxVAFsqBIStqOLFBhtRIgGlgjTF9dGzKNHDgJlSo4qPjZ6iH2dUhcAYOZCFOX8ruFL4gzhfd k8/zYtux u7RqJv1fAutj1XbgqF4C5y8oOYIMzclh8oyeCMja2syJ4XKnazdlsodRpAZ6WFR005zGq7dawfX+bUF6+wn+Igkm0pf/2Sqlr1E3mKBHGY9B5bsDMCRXw6ilMV48DxejTX97v20oxDSQ5KDb5exREmgj/Os82iUWOx8ryezkXi95f/btK6KYC2n8lBjG/IZ9jplMLUbZvXB1adfQsnOr8LfvgDCovqU6Kz2HqJn3Tzw610++47BgAM/nDV5ZdGVO8mrcYRuOPG0fJLwedPmZzLbQsV5LPg1941ovH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jun 6, 2024 at 3:36=E2=80=AFPM Minchan Kim wro= te: > > On Tue, Jun 04, 2024 at 05:53:40PM +0000, Yosry Ahmed wrote: > > Zswap creates multiple zpools to improve concurrency. Each zsmalloc > > zpool creates its own 'zs_handle' and 'zspage' slab caches. Currently w= e > > end up with 32 slab caches of each type. > > > > Since each slab cache holds some free objects, we end up with a lot of > > free objects distributed among the separate zpool caches. Slab caches > > are designed to handle concurrent allocations by using percpu > > structures, so having a single instance of each cache should be enough, > > and avoids wasting more memory than needed due to fragmentation. > > > > Additionally, having more slab caches than needed unnecessarily slows > > down code paths that iterate slab_caches. > > > > In the results reported by Eric in [1], the amount of unused slab memor= y > > in these caches goes down from 242808 bytes to 29216 bytes (-88%). This > > is calculated by (num_objs - active_objs) * objsize for each 'zs_handle= ' > > and 'zspage' cache. Although this patch did not help with the allocatio= n > > failure reported by Eric with zswap + zsmalloc, I think it is still > > worth merging on its own. > > > > [1]https://lore.kernel.org/lkml/20240604134458.3ae4396a@yea/ > > I doubt this is the right direction. > > Zsmalloc is used for various purposes, each with different object > lifecycles. For example, swap operations relatively involve short-lived > objects, while filesystem use cases might have longer-lived objects. > This mix of lifecycles could lead to fragmentation with this approach. Even in a swapfile, some objects can be short-lived and some objects can be long-lived, and the line between swap and file systems both becomes blurry with shmem/tmpfs. I don't think having separate caches here is vital, but I am not generally familiar with the file system use cases and I don't have data to prove/disprove it. > > I believe the original problem arose when zsmalloc reduced its lock > granularity from the class level to a global level. And then, Zswap went > to mitigate the issue with multiple zpools, but it's essentially another > bandaid on top of the existing problem, IMO. IIRC we reduced the granularity when we added writeback support to zsmalloc, which was relatively recent. I think we have seen lock contention with zsmalloc long before that. We have had a similar patch internally to use multiple zpools in zswap for many years now. +Yu Zhao Yu has more historical context about this, I am hoping he will shed more light about this. > > The correct approach would be to further reduce the zsmalloc lock > granularity. I definitely agree that the correct approach should be to fix the lock contention at the source and drop zswap's usage of multiple zpools. Nonetheless, I think this patch provides value in the meantime. The fragmentation within the slab caches is real with zswap's use case. OTOH, sharing a cache between swap and file system use cases leading to fragmentation within the same slab cache is a less severe problem in my opinion. That being said, I don't feel strongly. If you really don't like this patch I am fine with dropping it.