From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C27C3C47DB7 for ; Fri, 19 Jan 2024 11:12:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41B2F6B0081; Fri, 19 Jan 2024 06:12:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CBB96B0082; Fri, 19 Jan 2024 06:12:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B266B0085; Fri, 19 Jan 2024 06:12:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 143916B0081 for ; Fri, 19 Jan 2024 06:12:55 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 755BBA0D7B for ; Fri, 19 Jan 2024 11:12:54 +0000 (UTC) X-FDA: 81695798268.29.89EFD60 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by imf30.hostedemail.com (Postfix) with ESMTP id 59E6580005 for ; Fri, 19 Jan 2024 11:12:50 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="ZJ/UFeeb"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705662772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tRrZVuJD4eHtLTHC2iNOBHqJ0J2Pnus5kzjkaHOH7qI=; b=C994NmGd9Dbp9sVPl23umFeOSRay/DErtTCfdSGNQKL1vHfwnOSAWg4HIdjtkd/X0ADUQd OUb2j+/JOTAsYSjVORdZAaemMcvsaKsNoJQAmO9TLQ5PRou811IcUhf9Cp2+7IC7P5Oj3h TLQpyaMPrSxP+sDv9ER3eIdeRun0ZrM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="ZJ/UFeeb"; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf30.hostedemail.com: domain of zhouchengming@bytedance.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=zhouchengming@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705662772; a=rsa-sha256; cv=none; b=mopxM9DJKnO3S6Y0TvOLrCSIK5wgsgkQxvLAdIO3FpSff4uxCLO3Uldx/UMyVQDnXzYYg7 vctmYt8xzpbooFgQoogdDio4fwJVGfW7Zp8lDi6ZVML5k2maxdNNvh4eznRl/mv2vNquNA BOR13qJWkMXQThJWkYQtX/L3CqTYBQA= Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3bda741ad7dso146559b6e.1 for ; Fri, 19 Jan 2024 03:12:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1705662769; x=1706267569; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=tRrZVuJD4eHtLTHC2iNOBHqJ0J2Pnus5kzjkaHOH7qI=; b=ZJ/UFeebV895CuWI6i0CfIC00jJPRiypWs4et8WEQG9bMwykYLfeKbP5Fh+fl5+pbD Q5qNP61e13n8sQIBIfdIiaU8aTjaubd+HAC1oYcETzQXXUAisO8VNjZt5/jnJHrGBmpE FbiZe2LnO3ET2YTKnzESLc+ToYmy3V2o4VdvRWLgCE1OP64WJTD4dv7+LAjcWXJ2HghN nCLubheS8V6xGKBdAcbXhjB9LYHMLiQCuqwVdou3iL322TvZqhElRReiLBY+c33tB2kq VDwA9RcuiQqnOA87H+8Ias5iPv5cNyGD9Rj9ljhfV1DkmLyq0JHZ6b2GUAlYBMGcMsIc r9WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705662769; x=1706267569; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tRrZVuJD4eHtLTHC2iNOBHqJ0J2Pnus5kzjkaHOH7qI=; b=ER+3yr97L3PHwSl9mGSSdJaYqX4ilolFAFe456PmKFepYCfVlXD/LhSHcjKUQsd4dR s7mzEr6wu5XfEYODHHDi5r/T2KeiIbEMXw0Nm+FxG/3pmaNO/VEsAd3o80urSRbC+cbl +QOGP6FLy55uBwjmTkjVOjK3Cr6+eK7IeKclxbEgYx6MzFa2pYfq2itRbsPXWkYG/e+X ULrENVcrJqQJsgb9nyuiWz7Se2a4eU0yNPzRb99IsZhzmvh9tjNRahDduPCIkuQIXBpc Tc/bv/8gmVTY8kTmeMI/jknpjALAy5vONVHIit9VVzuH6B0B0x0t0TL2moozejAemV+/ TrCw== X-Gm-Message-State: AOJu0YyDnwOYfwc1cZoHt7NVmzeBeRYls8kvKZZYVNII8z67qjbI4qkT +SAcCyFbj0iiqGn7SvIcIFsdPINUSxZ2nwSgukaWVM/l9/VS6dn5TUBGUnTvrFI= X-Google-Smtp-Source: AGHT+IGXVK2rcveg704hNARA6FvxhB522Ogm0C75g5x3avcuMvfcvjATtou+TLlNgCaI1MTD96C8dg== X-Received: by 2002:a05:6359:6d08:b0:175:d24d:ab7b with SMTP id te8-20020a0563596d0800b00175d24dab7bmr2067379rwb.3.1705662764307; Fri, 19 Jan 2024 03:12:44 -0800 (PST) Received: from [10.254.224.1] ([139.177.225.241]) by smtp.gmail.com with ESMTPSA id j4-20020a17090276c400b001d5af7fbda0sm2809761plt.122.2024.01.19.03.12.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 19 Jan 2024 03:12:43 -0800 (PST) Message-ID: Date: Fri, 19 Jan 2024 19:12:34 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/2] RFC: zswap tree use xarray instead of RB tree Content-Language: en-US To: Chris Li Cc: Yosry Ahmed , Andrew Morton , linux-kernel@vger.kernel.org, linux-mm@kvack.org, =?UTF-8?B?V2VpIFh177+8?= , Yu Zhao , Greg Thelen , Chun-Tse Shao , =?UTF-8?Q?Suren_Baghdasaryan=EF=BF=BC?= , Brain Geffon , Minchan Kim , Michal Hocko , Mel Gorman , Huang Ying , Nhat Pham , Johannes Weiner , Kairui Song , Zhongkun He , Kemeng Shi , Barry Song , "Matthew Wilcox (Oracle)" , "Liam R. Howlett" , Joel Fernandes References: <20240117-zswap-xarray-v1-0-6daa86c08fae@kernel.org> <7f52ad78-e10b-438a-b380-49451bf6f64f@bytedance.com> <3a1b124d-4a97-4400-9714-0cceac53bd34@bytedance.com> <9b2f8385-735b-4341-b521-a42c9a9cb04c@bytedance.com> From: Chengming Zhou In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 59E6580005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: xpkw6gaxpc6ztnmhewyra3n8p4awczne X-HE-Tag: 1705662770-941143 X-HE-Meta: U2FsdGVkX1/4TcCIjvmSt3WYJiExp5CAPN5/7XRNCT4llPznOhNhiZdxcDvQEczDItK5wmj8NmD7qR+9ykpzNXe6LJZhZUFiBYwWhIaZlCYVPFVuU5oQFLv2hNExYNosVE3nKEYrlgDIj9zrSuDNz2/8prX+5VglVi+cdRqXMMktGERKp1OASdeSoeehopdw8ilar24MujCvjJr7R3PjONRyIZz5i0ziWf6Db66oVBk1CaAJ9O6PcEe2rCi/RwdvGqV1WnF2qrLPpvWsuawCd38dovYMXjXXpFb1eiCT/9pp+N8pebxMYhyhGE5vM/9kLjAh64Scvmi0wZkDkX1/J7rgeMv3MKxBbH+spuU0B6YKD/b8Beb92VBn6EmWiGwqN3utnxhpoeqzVuu7HNt2sNrs1XFKcMCbTJ3IeHVURL9cK+5RrfJ1jAtYeMCYB0BIxnvn8U6XrYF8LMSgNx+WQqKmpUNQKGI4MTLnr6Ati3YBZAsJ/YOdBoUkM2ccYeE3Q7wv0EGEoTj3blrU5/XWOtMrqb33Bbd3v+Mg8UvHfpCcVQkPO4UYFEHdDXjQLHkza6jSxWhTGDib7RPSdvrzimbl4UDty3gpcDoVMyNDHmVAYLZch9alK8AAnynKoYq8ks2D533SsY4JvikHB4mQM7dB0ZBANACHWCJo46Snydt24G7Z6w7rniVQ/blhzyCAnBOj54N+Ya601V+ZMVxRtXmi0o5YCBeiuOYGM93OU2p4pN10nCymrR2q/m3jxwfBmOKiIyGWIbexW+lbT96coIjPtpAH173B94MpY6PdJueEMPSSz6WEBAdE+rEfBSmDWroEl72TX0yuVVB7q+b2xh8OnOVSRLnOsO9NfryDqDt/yxSlb8N8U2OHt3piJC0rwI7H72ttpqEAtZYI51ngFeWG+gJyhdtEBe0T4m7PGvCWuv2otByFd1NWxporpptaCAoAoEPxcWnWZKpi3dI L9cIFI6b U1Ri517snLn3IAiXse4pb3RwdLwUlMh/CafEwnEHlbMw/xbTWr0fYm2xxZM7Z26KowXIElKb6l7IHigYFkIe1N6FI6mJBonNgKgliTkaj3/ypOcyYNmRC8xM+c2RVuZvpHTFkdbZAUaBl0QevqAy9jNoE356z9k9X7BPsXxqFtzIAxBc6MlUVmz2zhCpJROFUQ7QRc2lqiQNlhORqBGI0J1FxeXJ0wy/TJ9ayiS0XVnZiFvRPYVV3nf7nNU3jhcaBwwPnabbj+thk4xNqIltzTGdBOewXp/imAOeKbFF11Wfi68/GKATV8f0Vuh091PA78OwLcGjsRJ3AirXG/NXibNu1Z5iQILe9O8cs0eXpHQ5nR2d5lE+T9U93WHnwdn9iQ5C4GLdIGdszCS0w7bSNwEbKn+gCwNQVix54R2FezksiSIIrElQLJiRgVqYmrEBBM0+Nbj4+In7NhsYdApv5TR5vKw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024/1/19 18:26, Chris Li wrote: > On Thu, Jan 18, 2024 at 10:19 PM Chengming Zhou > wrote: >> >> On 2024/1/19 12:59, Chris Li wrote: >>> On Wed, Jan 17, 2024 at 11:35 PM Chengming Zhou >>> wrote: >>> >>>>>>> mm-stable zswap-split-tree zswap-xarray >>>>>>> real 1m10.442s 1m4.157s 1m9.962s >>>>>>> user 17m48.232s 17m41.477s 17m45.887s >>>>>>> sys 8m13.517s 5m2.226s 7m59.305s >>>>>>> >>>>>>> Looks like the contention of concurrency is still there, I haven't >>>>>>> look into the code yet, will review it later. >>>>> >>>>> Thanks for the quick test. Interesting to see the sys usage drop for >>>>> the xarray case even with the spin lock. >>>>> Not sure if the 13 second saving is statistically significant or not. >>>>> >>>>> We might need to have both xarray and split trees for the zswap. It is >>>>> likely removing the spin lock wouldn't be able to make up the 35% >>>>> difference. That is just my guess. There is only one way to find out. >>>> >>>> Yes, I totally agree with this! IMHO, concurrent zswap_store paths still >>>> have to contend for the xarray spinlock even though we would have converted >>>> the rb-tree to the xarray structure at last. So I think we should have both. >>>> >>>>> >>>>> BTW, do you have a script I can run to replicate your results? >>> >>> Hi Chengming, >>> >>> Thanks for your script. >>> >>>> >>>> ``` >>>> #!/bin/bash >>>> >>>> testname="build-kernel-tmpfs" >>>> cgroup="/sys/fs/cgroup/$testname" >>>> >>>> tmpdir="/tmp/vm-scalability-tmp" >>>> workdir="$tmpdir/$testname" >>>> >>>> memory_max="$((2 * 1024 * 1024 * 1024))" >>>> >>>> linux_src="/root/zcm/linux-6.6.tar.xz" >>>> NR_TASK=32 >>>> >>>> swapon ~/zcm/swapfile >>> >>> How big is your swapfile here? >> >> The swapfile is big enough here, I use a 50GB swapfile. > > Thanks, > >> >>> >>> It seems you have only one swapfile there. That can explain the contention. >>> Have you tried multiple swapfiles for the same test? >>> That should reduce the contention without using your patch. >> Do you mean to have many 64MB swapfiles to swapon at the same time? > > 64MB is too small. There are limits to MAX_SWAPFILES. It is less than > (32 - n) swap files. > If you want to use 50G swap space, you can have MAX_SWAPFILES, each > swapfile 50GB / MAX_SWAPFILES. Right. > >> Maybe it's feasible to test, > > Of course it is testable, I am curious to see the test results. > >> I'm not sure how swapout will choose. > > It will rotate through the same priority swap files first. > swapfile.c: get_swap_pages(). > >> But in our usecase, we normally have only one swapfile. > > Is there a good reason why you can't use more than one swapfile? I think no, but it seems an unneeded change/burden to our admin. So I just tested and optimized for the normal case. > One swapfile will not take the full advantage of the existing code. > Even if you split the zswap trees within a swapfile. With only one > swapfile, you will still be having lock contention on "(struct > swap_info_struct).lock". > It is one lock per swapfile. > Using more than one swap file should get you better results. IIUC, we already have the per-cpu swap entry cache to not contend for this lock? And I don't see much hot of this lock in the testing. Thanks.