From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E886C4725D for ; Fri, 19 Jan 2024 06:19:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C05126B0082; Fri, 19 Jan 2024 01:19:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8E1C6B0087; Fri, 19 Jan 2024 01:19:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E04C6B0088; Fri, 19 Jan 2024 01:19:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 855006B0082 for ; Fri, 19 Jan 2024 01:19:09 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AB74DC05A1 for ; Fri, 19 Jan 2024 06:19:08 +0000 (UTC) X-FDA: 81695057976.17.6CF3B97 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf30.hostedemail.com (Postfix) with ESMTP id 5796F80011 for ; Fri, 19 Jan 2024 06:19:06 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=F88oMRl0; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705645147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oyGEoJfWWhkrJMQj59+AKvuJydk3uKpdOg7oDnHTmgU=; b=aUo2bU5vsXC1vZhJeZcJLuDN3EEhtuNFDc/iHGMf4SXSeme1HK+G4oMdQQx4M+7LXkToB4 UuUtuhp1J1k5Jba+GM+Sh/4oiymvUnBrtSyTrxSw18tRwuwVMttq3XIDVytPc6bWQwQlga q2OCAf5sLR0sV83mcaj1A3/h1v7x4+Y= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=F88oMRl0; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705645147; a=rsa-sha256; cv=none; b=mUgTqs1PAZwo4tR6p+603ZuwBb+CToJbBGYb45407BRPPg+jHbNdpRIKwvZuW4fYNDXtww tHkz90d0Ji2tYT9XMyClgjiDarto1WXjzViMP2ndrMj4Xcjc46fLVEu9FalCL6Tcu1/q3C OVMkJ+CvzWpIPM32Qtuq0YPYPd2lLMM= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-29036dc3a63so263610a91.3 for ; Thu, 18 Jan 2024 22:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1705645145; x=1706249945; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=oyGEoJfWWhkrJMQj59+AKvuJydk3uKpdOg7oDnHTmgU=; b=F88oMRl07H2LrxD+ViyzNpn1Wmxdct+MMre1gUirEo5cMPZrUKGw68gQE+G+YwqZoW Vhhm40bsmQY+c0VdAAMrxYgWTfyxDwj+23Gt1M+zVHlSMntbF4/Q1daI1/or/xLByr3B V+JVapkOL1UAbnhIzcdD1YATr1AhCw8VMFZjZdUxmwgd5imHMtH3PTtHnAaXJOQlQWDn Kzh6TjCaZdyO2hDBrwdWaAHO60lW/vhpq7jNBCPpYbnneWjNQAifGFloUdGwSeYjfTp0 RaQ0y6oYVtSk546nx3c8Ph1tVVCeXAnyB/yhyWpiEN+1oJmeHxqWBR7Zc+KbSBDziX67 CpgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705645145; x=1706249945; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oyGEoJfWWhkrJMQj59+AKvuJydk3uKpdOg7oDnHTmgU=; b=Gs8U9N6yfdQtqnx4OHF5J2MzvpLJ9Pbsu4c4OtOB2KGgOPW0389Ubd+fUglEbSGJTd LyW9wxQWel6RCD8/PYnped/zSp8Ise3oJ2Jfc0F1YhAa3vvQMxPzL7Bp2bFNBTzUDe2t BVkeSv5Dgl4Ey41jt0KdJSwQi9/w9P9T02oBzyNZGvw38sDKEbYGpX6CViGC4T2odl2A G1I1/XcKosRhQ/OIzMUjEOh5SOKwhmYHZXBk2u7OqDoU4cHTB7MhRJHTf9SCwvSVRvjz 4+m6/gfoVbBatJuSFhE8nV2Tpdrn77/awja5/5m+Sl/7ZjQlsM0vRELv6WwdWoeBswNA jdwg== X-Gm-Message-State: AOJu0YyY9gfKWdsccJPZ2FwDeP+VXM1ik5L0I+hCIxjZxVsszs326i89 EN9RfCcGG7UnGkbsZ4ADXK/ftCv3eEEBf3TaQ2Rv5LbgphiBf3LGIm0bVDDy/Xw= X-Google-Smtp-Source: AGHT+IFfRluXn5JTOoxujE8j9Z5Gf3y2nTxyilmVQJUOiLRddb1YYFLzTBiz4+Ual1jHvdOxvyRkfg== X-Received: by 2002:a17:90a:1bc8:b0:28e:850a:56d4 with SMTP id r8-20020a17090a1bc800b0028e850a56d4mr1740460pjr.49.1705645144885; Thu, 18 Jan 2024 22:19:04 -0800 (PST) Received: from [10.254.224.1] ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id u8-20020a17090a5e4800b0028feea2cddfsm1133847pji.0.2024.01.18.22.18.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 18 Jan 2024 22:19:04 -0800 (PST) Message-ID: <9b2f8385-735b-4341-b521-a42c9a9cb04c@bytedance.com> Date: Fri, 19 Jan 2024 14:18:54 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/2] RFC: zswap tree use xarray instead of RB tree Content-Language: en-US To: Chris Li Cc: Yosry Ahmed , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, =?UTF-8?B?V2VpIFh177+8?= , Yu Zhao , Greg Thelen , Chun-Tse Shao , =?UTF-8?Q?Suren_Baghdasaryan=EF=BF=BC?= , Brain Geffon , Minchan Kim , Michal Hocko , Mel Gorman , Huang Ying , Nhat Pham , Johannes Weiner , Kairui Song , Zhongkun He , Kemeng Shi , Barry Song , "Matthew Wilcox (Oracle)" , "Liam R. Howlett" , Joel Fernandes References: <20240117-zswap-xarray-v1-0-6daa86c08fae@kernel.org> <7f52ad78-e10b-438a-b380-49451bf6f64f@bytedance.com> <3a1b124d-4a97-4400-9714-0cceac53bd34@bytedance.com> From: Chengming Zhou In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5796F80011 X-Stat-Signature: ctdwy49czd6b13ea7dqfxwz8gductnwm X-HE-Tag: 1705645146-589870 X-HE-Meta: U2FsdGVkX193yyK8djtCNEpW6T6RPA1UWPVwspfBDeRqJTqdk95N4ycpA1PyexbBdtYfxs/G/kW/R0yyY2lSIrJxe+nxBh11dZQ6SIGf3GDzHYaxy2gbIJX5NR93eGX9Fyq1I+vRSvrnV0400k4a8V+ywyA7vkBnDr4MZOHplY3Q6enOvxEvnq21x9xqqz3sdtGZ66CclDFqm8qQMS/hdrOxOVehIPMmWbKS4R7gbK7StUxscf0bepRlPFR1YGJgdEGA2brUkUSh4CvhP1D06LG+4vdOgxfUuT1Kujm8ObuZkj+ujrsILJoPHCrAI+gSoVfoJstA+5UAO8ay8ZQJu8kWnHTswa7Ql9bBzohw6VeEETHdVr1/1ecI570S09ZCt52ljTJIwo1mau0HfWUPsQy5uIOB4Rz1/w3JPxYlvwJ3RKqdBpr6sptpIgut/ILOSEQEN2ZacowHVGtl7CsArRO9jVc4/zxFgVXxJIcSxg259LfKhA7O1ryFSIWPGTwBTDTPIM67kFjIVYwl/pEntkyvo4C8prWkZcMYFezoWStL6fmmSSGYy3l7Mec0SFk/KUa+Wb6l2wKzVFeevqgSOTLbI2sQILmxhAiVQaC3XIFkBf+u1ImjkYao6L5Ye4MuZkaVkuC+ccWnIf6WJK9boQrYXOkMa5+D9hKoERZX9f8dtTNg7LmaZdtmLTwjwP4Icbpu71NEEfZvH/PrQP2sXB+gyFQkXGx3KY0caJro1lvve9hHlqDjntLxc1E+d2XpRU6th1TWbARfA19Apur2HZZv+Rlj2dpCevMaEYMkPk2iWRRW44GCzfisEHWHY/SMuyVRO5JLa8BvkgLl1mSCivddEDg+MyFPZQH+oX+kQpChr/6LDLtqd9feMiJZcyQHvPBHTg6y32rB6fAMhoGj/ct3pm+VaWiPhiCBih4Xj1xtuLsPgt8etq9giWgCqb4RbLVUJu5VTOuFaOuLRrR I5Q1BCRa XaZoQ0ieLgBdDp994QbKTiilnyDu9lWx6pzTMXl/IOe6sgIFKIvXWCLGArenuWw5h3NWMfc/El2S3rG15p4AILd3BqLj7f6vYsvJgvyWH/IvGHCpU/1XEEBcR9pKzspLK5Ila X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/1/19 12:59, Chris Li wrote: > On Wed, Jan 17, 2024 at 11:35 PM Chengming Zhou > wrote: > >>>>> mm-stable zswap-split-tree zswap-xarray >>>>> real 1m10.442s 1m4.157s 1m9.962s >>>>> user 17m48.232s 17m41.477s 17m45.887s >>>>> sys 8m13.517s 5m2.226s 7m59.305s >>>>> >>>>> Looks like the contention of concurrency is still there, I haven't >>>>> look into the code yet, will review it later. >>> >>> Thanks for the quick test. Interesting to see the sys usage drop for >>> the xarray case even with the spin lock. >>> Not sure if the 13 second saving is statistically significant or not. >>> >>> We might need to have both xarray and split trees for the zswap. It is >>> likely removing the spin lock wouldn't be able to make up the 35% >>> difference. That is just my guess. There is only one way to find out. >> >> Yes, I totally agree with this! IMHO, concurrent zswap_store paths still >> have to contend for the xarray spinlock even though we would have converted >> the rb-tree to the xarray structure at last. So I think we should have both. >> >>> >>> BTW, do you have a script I can run to replicate your results? > > Hi Chengming, > > Thanks for your script. > >> >> ``` >> #!/bin/bash >> >> testname="build-kernel-tmpfs" >> cgroup="/sys/fs/cgroup/$testname" >> >> tmpdir="/tmp/vm-scalability-tmp" >> workdir="$tmpdir/$testname" >> >> memory_max="$((2 * 1024 * 1024 * 1024))" >> >> linux_src="/root/zcm/linux-6.6.tar.xz" >> NR_TASK=32 >> >> swapon ~/zcm/swapfile > > How big is your swapfile here? The swapfile is big enough here, I use a 50GB swapfile. > > It seems you have only one swapfile there. That can explain the contention. > Have you tried multiple swapfiles for the same test? > That should reduce the contention without using your patch. Do you mean to have many 64MB swapfiles to swapon at the same time? Maybe it's feasible to test, I'm not sure how swapout will choose. But in our usecase, we normally have only one swapfile. Thanks. > > Chris > >> echo 60 > /proc/sys/vm/swappiness >> >> echo zsmalloc > /sys/module/zswap/parameters/zpool >> echo lz4 > /sys/module/zswap/parameters/compressor >> echo 1 > /sys/module/zswap/parameters/shrinker_enabled >> echo 1 > /sys/module/zswap/parameters/enabled >> >> if ! [ -d $tmpdir ]; then >> mkdir -p $tmpdir >> mount -t tmpfs -o size=100% nodev $tmpdir >> fi >> >> mkdir -p $cgroup >> echo $memory_max > $cgroup/memory.max >> echo $$ > $cgroup/cgroup.procs >> >> rm -rf $workdir >> mkdir -p $workdir >> cd $workdir >> >> tar xvf $linux_src >> cd linux-6.6 >> make -j$NR_TASK clean >> make defconfig >> time make -j$NR_TASK >> ``` >> >>