From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 612D3C02198 for ; Thu, 6 Feb 2025 06:55:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D77EF6B0082; Thu, 6 Feb 2025 01:55:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D27CB6B0083; Thu, 6 Feb 2025 01:55:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BEF556B0085; Thu, 6 Feb 2025 01:55:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A0D496B0082 for ; Thu, 6 Feb 2025 01:55:42 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5583CB1CA1 for ; Thu, 6 Feb 2025 06:55:42 +0000 (UTC) X-FDA: 83088609324.19.8788969 Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com [209.85.208.180]) by imf29.hostedemail.com (Postfix) with ESMTP id 619ED120007 for ; Thu, 6 Feb 2025 06:55:40 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WcEy4aDm; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738824940; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EYxML1IvkX/tlwSsdd/qDyAsTAIzYWO3WovQD8yPps8=; b=t2aCTc2gX8B32/loPa1cYUnr73R7BpvuVdvhAqy7LZ2ZIPMzgPXTGiFyY8cEhArhY65dRR /NbiSFQJFU2ElZ4UuVzgNyjT18phPZtOEWSDvsawK/A2u97mYkPdIZG/z0hFbUwnQhmiJ4 YdJAs0RFotVvmlGv9jxpu3H9E8gAlk8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738824940; a=rsa-sha256; cv=none; b=ajs8DBS61ZVOwPR0Dk2FOv5QsjzkgzXzhQ6AbQ0z3RusYKNy9dXzxVUislynnzqmwLVp5S zB+yExtO1w1nSVeJOjJujJSITPIMFcEyjWvCzLPEGxqyNLEb45D3TZe3V6azZuyu6+jP+N LMAArTV+PZ+qLO2/ihFyqyc0FnofMHk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WcEy4aDm; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.208.180 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-f180.google.com with SMTP id 38308e7fff4ca-30797730cbdso5421911fa.3 for ; Wed, 05 Feb 2025 22:55:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738824938; x=1739429738; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=EYxML1IvkX/tlwSsdd/qDyAsTAIzYWO3WovQD8yPps8=; b=WcEy4aDmTspQ9PCHocwtuxawDYZnWW1YwTfuModGzOlFeZhrRA4Yl+hsF6U7pRJilL mbCM9AP0Onxvi4ZuKuMgQvyT0Ag04qQjPUYQPJL+bnj6W0AwU8QHZP+IfaiECZFLIdqs DLPC9KvVWs8LNVDdMerKkfAAxH/UTT43JqiFSyY64uIq7MFMu+4Izx/F81WHzdOgWndL ONjLRwSxN4ktrGVMpOD5D/0tkP1DDh/4iodh9u8AyNW2AVeyDbPD71GAIr2Ki0dLjFKn cnDl4Eiu7FURtdleN+xWRL7dKF5Ocz3j60BG83De8McuxYlc8w/DAUCbCPrbICu7qg8X WqIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738824938; x=1739429738; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EYxML1IvkX/tlwSsdd/qDyAsTAIzYWO3WovQD8yPps8=; b=em+YN1CNhzoCmpZokWUDORdyai6jQxGKXBd+wujyjyttob/J58XFpFQjiiwwUxGDYw b/ywHiBZwD1vR/lx4H7XQNcFMVTcwlq4t2ya6AYqE+vPjSIjbjivCrKNyyDJIVDtFXIa Lq9iXgMcP4MVW7fmCwIqZ37vhBLuiiT98vCHudt83xjIOafY4tgVuoFe17odFMHOrpKP U8Hj2r0LwhAq5PeiJAn71b/ujC7jUbv9mrHvY5y6ZyB6EB5fA5cK2CPX0oeBrkreUMtP rueC576807A5zoM0nXNO/draCuhNcrbm8E0u6Z/A+IcSwT2R0JLNYSZLbfqTZDd8S+nR J8+A== X-Forwarded-Encrypted: i=1; AJvYcCW4F1C4H7U4kbDm6CQBeRFpEAIvO7hVmudpMLNznXe2V3rFZccruhNZcMcSv/ZCGRLsakVPvGA67Q==@kvack.org X-Gm-Message-State: AOJu0YyLpv99CA6Y/8gjb/2z6ZGRc4SktyZBeuPnpcAoRHGe/gp5Z+h9 if85Js8ak6zvNxNPbR/Ql6DRYoOtkGh0PTivCuvxGzv+nYv0wYrkEEj9kXl8ndgtAH4ljjiH71Q 4CNvIW1NV0u3pIYO+ztPXNgj4cEw= X-Gm-Gg: ASbGnct8IXPSQmfvuqHv1p8ISPcjMBsVgxgOpj8rCeiQrowLK25TGePiDHP4eKRi2Pm QVPWvByQAr9cc/XQzIWd3/kJtr1Rv17Cn9wxnOJv28xwo16/Dp98ur4sILGC7LHXRPGMzFxy0 X-Google-Smtp-Source: AGHT+IEoRUgWvsN+VPa1Lnr21xtLoHTS1yZYhwFh18Q05mc2Z5sFhWyJRAYjCrkEH1KxoE2+BMFU2O8Zs1330IgxfMU= X-Received: by 2002:a05:651c:2225:b0:302:3c78:4ea4 with SMTP id 38308e7fff4ca-307cf370fefmr19656601fa.30.1738824938259; Wed, 05 Feb 2025 22:55:38 -0800 (PST) MIME-Version: 1.0 References: <20250131090658.3386285-1-senozhatsky@chromium.org> <20250131090658.3386285-3-senozhatsky@chromium.org> In-Reply-To: From: Kairui Song Date: Thu, 6 Feb 2025 14:55:21 +0800 X-Gm-Features: AWEUYZmxJd7uZ-US7tSScgg6iDTwpXgeuX_q0pRG49_fxDXdBe6tPR1AAidDZnY Message-ID: Subject: Re: [PATCHv4 02/17] zram: do not use per-CPU compression streams To: Sergey Senozhatsky Cc: Andrew Morton , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 75ejh66zmqde43qpy14ifh9o5kth75ym X-Rspam-User: X-Rspamd-Queue-Id: 619ED120007 X-Rspamd-Server: rspam03 X-HE-Tag: 1738824940-650474 X-HE-Meta: U2FsdGVkX1/pbbh3i7xB993aBHM1+ILVIIczVjAeSL4hOlRZ6OqJ4uCy5NzMTaOyMpXmVQVslX432K4/jifzXhRzLzqcAeSgZSLobbBh1xZ4oVJx+zdQfJcAgFNcsSYph5x5yoy2+0dNJkyxxxC/XFCROs28GYAWNfh18jNCU+9eBqavV86vzAH+GnBmSdTv4vBeSFmF8I2vG6BSuH6N9UpHKylbnJ4T9ac9P2Me8mXVDw7Z15kf48+vaojvEluEpqsFZhSe0sp/SyxFV2wXWKM8XSH4FPsfowy47nTfaAycsZbRcvvPgxw4FHAy4YCuuybwbIELxnTKTD+toKbNnxW3IH+q5P0P0mt4YmiI+PqJOokA9dBATao8aUuQJWYkLY7FIUrh/iziqpuAMpUsY5v/S0tGzGrsW1fWwoL+i8RhXqcMgA1Gr+MsoDF47arPbNKvx1mt1WXkHQ/E7rqvCH9FODAqszS8OFm3xKf8DYXmW57z+lkK0gfddNDWXEsfP9Vi/OVU9gNVJFW3S51KN9TFjTD/sfsLz46fhufxEvEeGGvl9xBDcqmYzYVYIbo46wB9ZCNzY+x5OjrUPJfCIh/Nucq3cyGIh8AIW1ogLtjwXy0w4FUiaKscj27zwD3lL6L0uA1KN78Kcsvtz6gnwnxoYjBlKPWUF4Vjvx70e+UYqBOXBA744EY8xG5kVSCaO28JCKqY8sYtEGjcRouWqyeDM/tv5k0TWS4omD6VsyBi01U4JqWjNtLRLK0VOwrF6HQa09rbBaMXekPi6zzaLA6A3zsP9IPgQkflGO6e4mZy/LiU2H4FKR/ZY9d8K/4I8Aiq9UbqhmN8q/11ammUQP9v1rSEUcJ8QHY9n5dPzFpHMAtBVYjDW5vN/6Vpo5+0uSk6gYhpGHzQx+iq4FXRHc2eEo2hORM+YZDecM/0vZkGJwLuU3aaSUg9ZxxDgbuWsfkRV0gNsjAy0gNQwxn myhuvmkM GtofUlKw6J+sZHhsfQnW5j00eR+kYRDiJsfNq78KGJcN6rf15G7+wcwQfFoW6Q4jv4D6wM9cR7Ji0XVNxXLhg8XhW9dmaYsSf0CAWcqtv0AkxAJ+9hHmxUOV9PZJXf8Ec6L1YJEOvx5XKtc4wL8JR5fhrdCTvvTdo1HFXi4qL/KZU0BQi+I89I4O9/Mhc40/fIbc/TGGqi+kurkiA45Hb5tr9neO708m5p/JiAjurqyPBsH1/zmeb09eHyAVNsWk/lxerz+k9/KsScIbDUWD56x86v4nH6Pu0BImZ8OvvvCGlL0u3b1Q4vay+Q0l0IbMccTPykag2scqDB/+YXTFBgPz69w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000495, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Feb 3, 2025 at 11:49=E2=80=AFAM Sergey Senozhatsky wrote: > > On (25/02/01 17:21), Kairui Song wrote: > > This seems will cause a huge regression of performance on multi core > > systems, this is especially significant as the number of concurrent > > tasks increases: > > > > Test build linux kernel using ZRAM as SWAP (1G memcg): > > > > Before: > > + /usr/bin/time make -s -j48 > > 2495.77user 2604.77system 2:12.95elapsed 3836%CPU (0avgtext+0avgdata > > 863304maxresident)k > > > > After: > > + /usr/bin/time make -s -j48 > > 2403.60user 6676.09system 3:38.22elapsed 4160%CPU (0avgtext+0avgdata > > 863276maxresident)k > > How many CPUs do you have? I assume, preemption gets into way which is > sort of expected, to be honest... Using per-CPU compression streams > disables preemption and uses CPU exclusively at a price of other tasks > not being able to run. I do tend to think that I made a mistake by > switching zram to per-CPU compression streams. > > What preemption model do you use and to what extent do you overload > your system? > > My tests don't show anything unusual (but I don't overload the system) > > CONFIG_PREEMPT I'm using CONFIG_PREEMPT_VOLUNTARY=3Dy, and there are 96 logical CPUs (48c96t), make -j48 shouldn't be considered overload I think. make -j32 also showed an obvious slow down. > > before > 1371.96user 156.21system 1:30.91elapsed 1680%CPU (0avgtext+0avgdata 82563= 6maxresident)k > 32688inputs+1768416outputs (259major+51539861minor)pagefaults 0swaps > > after > 1372.05user 155.79system 1:30.82elapsed 1682%CPU (0avgtext+0avgdata 82568= 4maxresident)k > 32680inputs+1768416outputs (273major+51541815minor)pagefaults 0swaps > > (I use zram as a block device with ext4 on it.) I'm testing with ZRAM as SWAP, and tmpfs as storage for the kernel source code, with memory pressure inside a 2G or smaller mem cgroup (depend on make -j48 or -j32). > > > `perf lock contention -ab sleep 3` also indicates the big spin lock in > > zcomp_stream_get/put is having significant contention: > > Hmm it's just > > spin_lock() > list first entry > spin_unlock() > > Shouldn't be "a big spin lock", that's very odd. I'm not familiar with > perf lock contention, let me take a look. I can debug this a bit more to figure out why the contention is huge later, but my first thought is that, as Yosry also mentioned in another reply, making it preemptable doesn't necessarily mean the per CPU stream has to be gone.