From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AEECC0218A for ; Mon, 3 Feb 2025 03:49:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F26076B007B; Sun, 2 Feb 2025 22:49:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ED65D6B0083; Sun, 2 Feb 2025 22:49:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D9E916B0085; Sun, 2 Feb 2025 22:49:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BEFF16B007B for ; Sun, 2 Feb 2025 22:49:52 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 46059B4B80 for ; Mon, 3 Feb 2025 03:49:52 +0000 (UTC) X-FDA: 83077254624.16.498A0AA Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf04.hostedemail.com (Postfix) with ESMTP id 5AFA940006 for ; Mon, 3 Feb 2025 03:49:50 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GHuzvudq; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738554590; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/lQ2Z8Dpj7uKLQom+qHTSsO8xHrv+IjBy1dtKyJZ5zY=; b=JpJU+tg9Rf+ixf+AG1ygyOI1RFhsZ4uI36P+UzJTrfEzaaK+9BgwQgB84FPR2kDjsCtuxR gyzbCN+VNEe9laLNOs+1EPqKqbWV8pbgNBcaJXmJqi42I/tWECLzO9L5EDKWWz+i0otswj mZpF+fOYLG9y4hpiUXBSLeY/mrHEadM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GHuzvudq; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738554590; a=rsa-sha256; cv=none; b=R178XAKgAh9hiCuiC1ep3e2MGPVVpYFdrWDJmRqjOXen2G5nesmzDgqaaKRd4NUlp4c84O f1vOjty33Hup1WcdSYBjdz/TwdAh1msoCCXHJnjr2a3twQaFxYn5dal4wfzNPLTtgTT8rn YtUUT80iIP79p5j/PHdyKius5BfjpGs= Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2ee709715d9so5127897a91.3 for ; Sun, 02 Feb 2025 19:49:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738554589; x=1739159389; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=/lQ2Z8Dpj7uKLQom+qHTSsO8xHrv+IjBy1dtKyJZ5zY=; b=GHuzvudqgxN58W0wTdq21kyYqk91E5o7xL1JoFPg0MB5ft3eo3UHZZhc81McfMOgq/ ZmXi59VlfK3UbTS7kbzYIlgLDyvQSkxaSvthdxXwYSinMu4cVbSMTVftdeC0p5yMGQZK VpnsmWv6J3hvXLyDdnYMYXmE48reCsxuYtViE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738554589; x=1739159389; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=/lQ2Z8Dpj7uKLQom+qHTSsO8xHrv+IjBy1dtKyJZ5zY=; b=MjXz8tXfENQCAk4qJTvE65gh1ZIV7Vdn38sJkLUygHXlQpX5cHB1nc64MrlT4WpU3R yepiGFOGv5SytsVlttAjBAzTGzMsYXLTLM9qDuGsf0mS98LDTbuktlhyJK6ZSlMT1c+J hHag11mJdm1Y/ZmcmXYzmpai2DJNjQ1cqU/76zyYYtP/KqewAgVhDanrvRsHaDhbZOqO v1QofvDyTCcnGMBtjOCuK871EBQ6Bu/Jq3zXv7NH6NzPd9xs2Jk7nK4MZpVcj7anEldA 03VdSABZKqT8wbJowkaNgj01AhOhW/8V6USVuid7GbYIDCFwaynKbA9udgM7MB344sCD /fTQ== X-Forwarded-Encrypted: i=1; AJvYcCUsjtTkbP6EJmEQftsHuEsbeQMs1Fsh06Wi7q/cncY60ORYcB069o8yMAMYw6RKpBHiQfQPQiy2JA==@kvack.org X-Gm-Message-State: AOJu0Ywui8NrQx0RA5ura8YV9l3YA3+MxLxxxI5rOoOLESSypcKfCDP0 fHnsifmaCS7qIGpK6SoHQKfUnHryrIb65sYMoxrVHGt0v466zKSI9dGpX9UQoJK2VJXkkXDNspo = X-Gm-Gg: ASbGnctbx8IJhUJUdgfBH8e+iUBHHX+uY0hyQb4+KcIPgRvokn7FiDTBE/nuTGcy3V4 ApVr7NlVxSUBEEZsuYVe9M3cwgmOOkGs7HKXTuaLEbrJvPvTru4w6aDSz4kw7BqfEThXqC6prts FR5smzqpWAZjaT0aJ4j1VikWWB2euKvBicEZJMZxdSuwQz+rRE0OGBG5bl+Zc7vIwxaFyoMkq48 fQHzK2AuAEbWxWSQpRGmsRejzOoBNKLjeIRH3GTdIhfo6d1pnnfbWrzwulCj8F9qBJ3fteXCd1r 2jVtxpfNOsSWKb1M X-Google-Smtp-Source: AGHT+IH9R5w/mBIKAQhPXyg/WVREUR/j8hfEG9f/yQjcZapINnYhqZ9Qu7LmxQntVPblYbIfaD374w== X-Received: by 2002:a17:90b:5387:b0:2ee:5111:a54b with SMTP id 98e67ed59e1d1-2f83ac8ad6emr28029765a91.31.1738554589043; Sun, 02 Feb 2025 19:49:49 -0800 (PST) Received: from google.com ([2401:fa00:8f:203:f3c0:b0:250f:9437]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f83bf95b44sm10210749a91.37.2025.02.02.19.49.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Feb 2025 19:49:48 -0800 (PST) Date: Mon, 3 Feb 2025 12:49:42 +0900 From: Sergey Senozhatsky To: Kairui Song Cc: Sergey Senozhatsky , Andrew Morton , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCHv4 02/17] zram: do not use per-CPU compression streams Message-ID: References: <20250131090658.3386285-1-senozhatsky@chromium.org> <20250131090658.3386285-3-senozhatsky@chromium.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 5AFA940006 X-Stat-Signature: fqd5fg84em88a6n6hwqsk3yxg1imp3bg X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738554590-682913 X-HE-Meta: U2FsdGVkX19SbInMeq7p/awBLHfaGN6uy3Vt4ojIRMDnD5l6T12eg7XT3OHOzwJ6R3T73yf7O3TkONgaO2Nj1iwjD81XRnxGIdJ+sKdKQxk6jRFRKEG9qU8Gct5d/seIHXFK+FPpr8suhbsrLDrCwa9Tl+bfu0a8QOmnJwjxy1TmJb5FGS6IxSB4n+BnPcvXefBUO/Jft/JTvv5iPz23u/UlwaFR5Mbd1RpzJ9i3ZsDNMJRORFmLus/AM/O0xL6Cejy7S3y5ynhJCxbGH17ZBIVCHwQnU9xNeU9leZdgYHyGC5yYs/CCbjzDGFinkDILuLNsF+Xc5edZT0JRHXa/zam9258b6oSqH14+wXwjgKElzWpcaevV0+sbNRp8ua15HQJwSYmtWo5SL3Hwrj3xAVnU3svSGW+IpNF8OM/gG42GfCF/1qGhBijL5h+vgz+XibUnoMu9PYKIibI1iRa16pNcZypxKaMVhdp4LssGuys/bn787201h92CObyAKIGE5G4EA4LLNndLfi8cFNT2IDoKuKTKB7PiZRclMyuO8nlhz5lTaQV/7xcEkeNVhEkuYM18DnX2AAub7EnfpuiwYYtj+M91K1TwyUJ6rB7mH2tWti3gQ9amnT6KY9jl+MIyB2xkVGbnvnS+I2V3P4RdMNa7DFgHrzfuyvZgCT3W0rAbKq/05G0TMQrq61AZh7kkyS4IHWJmZT2rpBwtxLvUSBGkL0hfy3XXxjwGbOEJi+mb2dULNYI02X/TkdSMckZcaaXeDYUSZ3Ya4Udr9dfDOb/0fnO5zf0YWUYZmn3rX87Me9u712LQHPIrIlT+VUqc8kd3EGB6xzxv+NgTAfu9UOeUYitn0WXUDnnA+4RJXP1+y8ViBKAzIB6pON6NZLnUpsLsczF5dt04yxF78KuyeTyp//btX5VLCOMo+icS29DETf7TzA+RWtcQsvovcF2fODwUu08W2gJrGGHAvKx m9I7uiF6 eAsWMtjkDyeX60VzWJVlGDU2N11Fu2Gtd9vhIUI+sKUnv3SM2B1m0MM7TUZJtdmW0msDfWeKspFybmhpO1k99OZGyr1xzEnBkB7cY5Fq6/JTohxkBv0QWNrK6/x4t8KzQXrlA/oSm/X22roIBd3vM7V8nXzsCzvYsnrXG/+2Ton1fxgy2VIoeTP6ZYpCZmbSOhm006WfxcCnpSkpE3r0u+33pNM6dZADNLcgHslFYgyct+vm1/fE0Xy+6K94M5ALx+OJV771+u/vZOKWcVqgrnPk9wAuZk5J/jd1f7brN8aorwdFRFTINgSgiE12PhPfe2Ep6FIf/25mJNjvLW7rMEs2cNfpp8bbem/XXRgha0iN4g+k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.011650, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On (25/02/01 17:21), Kairui Song wrote: > This seems will cause a huge regression of performance on multi core > systems, this is especially significant as the number of concurrent > tasks increases: > > Test build linux kernel using ZRAM as SWAP (1G memcg): > > Before: > + /usr/bin/time make -s -j48 > 2495.77user 2604.77system 2:12.95elapsed 3836%CPU (0avgtext+0avgdata > 863304maxresident)k > > After: > + /usr/bin/time make -s -j48 > 2403.60user 6676.09system 3:38.22elapsed 4160%CPU (0avgtext+0avgdata > 863276maxresident)k How many CPUs do you have? I assume, preemption gets into way which is sort of expected, to be honest... Using per-CPU compression streams disables preemption and uses CPU exclusively at a price of other tasks not being able to run. I do tend to think that I made a mistake by switching zram to per-CPU compression streams. What preemption model do you use and to what extent do you overload your system? My tests don't show anything unusual (but I don't overload the system) CONFIG_PREEMPT before 1371.96user 156.21system 1:30.91elapsed 1680%CPU (0avgtext+0avgdata 825636maxresident)k 32688inputs+1768416outputs (259major+51539861minor)pagefaults 0swaps after 1372.05user 155.79system 1:30.82elapsed 1682%CPU (0avgtext+0avgdata 825684maxresident)k 32680inputs+1768416outputs (273major+51541815minor)pagefaults 0swaps (I use zram as a block device with ext4 on it.) > `perf lock contention -ab sleep 3` also indicates the big spin lock in > zcomp_stream_get/put is having significant contention: Hmm it's just spin_lock() list first entry spin_unlock() Shouldn't be "a big spin lock", that's very odd. I'm not familiar with perf lock contention, let me take a look.