From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E86AC433F5 for ; Fri, 29 Apr 2022 19:07:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2B8E6B0071; Fri, 29 Apr 2022 15:07:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDB3B6B0072; Fri, 29 Apr 2022 15:07:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA2AB6B0073; Fri, 29 Apr 2022 15:07:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 9D76D6B0071 for ; Fri, 29 Apr 2022 15:07:19 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 6B9F01209AD for ; Fri, 29 Apr 2022 19:07:19 +0000 (UTC) X-FDA: 79410849798.24.2E7E650 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf30.hostedemail.com (Postfix) with ESMTP id E615080070 for ; Fri, 29 Apr 2022 19:07:09 +0000 (UTC) Received: by mail-pg1-f171.google.com with SMTP id t13so7179221pgn.8 for ; Fri, 29 Apr 2022 12:07:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eugz4b299LAJGviRoH6n/zYxtlNfeDtZuX/41skGRSQ=; b=j+KKd8TayNnf0ZoOfvfN/kf4WPOc5GvDmZoLi2SB6fzA/VKCl1lw2NMM0yLP2prH+8 1GGlmGYS3kRII/PnI8s4kpyOmBO8L1wnO242BDLhEPGl61fh/Z7SZUr2GYiVTsJ4a2oF uTyxkvG1m1xfhoeBTHk2xShivqAlv8G8EKWK2j5oSw80Nh2UqZgtjR6nz/IdDM2IkV9m emXkT5Wcy98CaCmLA5aggANfsmvoR3iWeFX73m06sqiXFumta2V7H+rOnqN8HJV8rJzg O5V05KdlxxzBFMdoqejRJWuRYcDUec4ruyOtP8drBT1CUsYAgLciZ7/QOp5oAsNkIhMf SSIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eugz4b299LAJGviRoH6n/zYxtlNfeDtZuX/41skGRSQ=; b=3+znN6FXcCyoDbaDZzrJeLxxDjlH5Ux/LYYalUEIef+Ps3HHN4zXd7x1VaBROR+amr XNI8H/KaFlASacyE36dlxwfIVe3ybjrcl3D9P2mPQwSNp4UA/EQDu7rtUKaqeuDtzrcN jn6o2guaXxXhVvF6WAE7mUiuth8uIUhAXcn2H5d6LRFklU2/ev+wBEu0lGZg3j5w9Skc UeeMjlPz3+Wje2XEIfdSiN55R3Evt2IrfvEx7F1NQlPtXIFGJ+ZJPbKWzASE11DD2LYS rM5dh8+Zs7JEwelngdv/Zp1ngoDZ5+FudKVT1qYiugrPMUU+ti58XvIpJaTUntxlRBSA TboQ== X-Gm-Message-State: AOAM532389I5mLfEqg4FTBLZuRz4SKT5hrZUMxqGmwGJ6+N+3wRHgFEB h6H8gFatvL9UNK8TSasruvgN3F3Is7hTpLEFWhU= X-Google-Smtp-Source: ABdhPJxKktCuHL2P0bMHC+35pvSzkKtvzuFOAm8jlnxebQnU+R/CMxBzVivQLRAqrM3pGd/YYiRpHRxJDtENBSoTfvM= X-Received: by 2002:a05:6a00:228d:b0:50a:934f:e302 with SMTP id f13-20020a056a00228d00b0050a934fe302mr376830pfe.20.1651259238116; Fri, 29 Apr 2022 12:07:18 -0700 (PDT) MIME-Version: 1.0 References: <20220407020953.475626-1-shy828301@gmail.com> <6f7210be7353d1c01dc9f872b2692b83f87f5452.camel@intel.com> In-Reply-To: From: Yang Shi Date: Fri, 29 Apr 2022 12:07:05 -0700 Message-ID: Subject: Re: [PATCH] mm: swap: determine swap device by using page nid To: Aaron Lu Cc: "ying.huang@intel.com" , Michal Hocko , Andrew Morton , Linux MM , Linux Kernel Mailing List , Tim Chen Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: jsu55rr76r7f4w5p8457tibqfqtqy1gz X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: E615080070 X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=j+KKd8Ta; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of shy828301@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=shy828301@gmail.com X-HE-Tag: 1651259229-206582 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Apr 29, 2022 at 3:27 AM Aaron Lu wrote: > > On Fri, Apr 22, 2022 at 10:00:59AM -0700, Yang Shi wrote: > > On Thu, Apr 21, 2022 at 11:24 PM Aaron Lu wrote: > > > > > > On Thu, Apr 21, 2022 at 04:34:09PM +0800, ying.huang@intel.com wrote: > > > > On Thu, 2022-04-21 at 16:17 +0800, Aaron Lu wrote: > > > > > On Thu, Apr 21, 2022 at 03:49:21PM +0800, ying.huang@intel.com wrote: > > > > > > ... ... > > > > > > > > > For swap-in latency, we can use pmbench, which can output latency > > > > > > information. > > > > > > > > > > > > > > > > OK, I'll give pmbench a run, thanks for the suggestion. > > > > > > > > Better to construct a senario with more swapin than swapout. For > > > > example, start a memory eater, then kill it later. > > > > > > What about vm-scalability/case-swapin? > > > https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-swapin > > > > > > I think you are pretty familiar with it but still: > > > 1) it starts $nr_task processes and each mmaps $size/$nr_task area and > > > then consumes the memory, after this, it waits for a signal; > > > 2) start another process to consume $size memory to push the memory in > > > step 1) to swap device; > > > 3) kick processes in step 1) to start accessing their memory, thus > > > trigger swapins. The metric of this testcase is the swapin throughput. > > > > > > I plan to restrict the cgroup's limit to $size. > > > > > > Considering there is only one NVMe drive attached to node 0, I will run > > > the test as described before: > > > 1) bind processes to run on node 0, allocate on node 1 to test the > > > performance when reclaimer's node id is the same as swap device's. > > > 2) bind processes to run on node 1, allocate on node 0 to test the > > > performance when page's node id is the same as swap device's. > > > > > Thanks to Tim, he has found me a server that has a single Optane disk > attached to node 0. > > Let's use task0_mem0 to denote tasks bound to node 0 and memory bound > to node 0 through cgroup cpuset. And for the above swapin case: > when nr_task=1: > task0_mem0 throughput: [571652, 587158, 594316], avg=584375 -> baseline > task0_mem1 throughput: [582944, 583752, 589026], avg=585240 +0.15% > task1_mem0 throughput: [569349, 577459, 581107], avg=575971 -1.4% > task1_mem1 throughput: [564482, 570664, 571466], avg=568870 -2.6% > > task0_mem1 is slightly better than task1_mem0. > > For nr_task=8 or nr_task=16, I also gave it a run and the result is > almost the same for all 4 cases. > > > > Ying and Yang, > > > > > > Let me know what you think about the case used and the way the test is > > > conducted. > > > > Looks fine to me. To measure the latency, you could also try the below > > bpftrace script: > > > > Trying to install bpftrace on an old distro(Ubuntu 16.04) is a real > pain, I gave up... But I managed to get an old bcc installed. Using > the provided funclatency script to profile 30 seconds swap_readpage(), > there is no obvious difference from the histrogram. Thank you so much for the testing. > > So for now, from the existing results, it did't show big difference. > Theoretically, for IO device, when swapping a remote page, using the > remote swap device that is at the same node as the page can reduce the > traffic of the interlink and improve performance. I think this is the > main motivation for this code change? Yes. Given the result it seems better to keep the code as-is. > On swapin time, it's hard to say which node the task will run on anyway > so it's hard to say where to swap is beneficial. >