From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11353C352BE for ; Sat, 18 Apr 2020 09:48:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AA954214D8 for ; Sat, 18 Apr 2020 09:48:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ME8UtMFW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AA954214D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F188A8E0005; Sat, 18 Apr 2020 05:48:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC9868E0003; Sat, 18 Apr 2020 05:48:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB7738E0005; Sat, 18 Apr 2020 05:48:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id BFB938E0003 for ; Sat, 18 Apr 2020 05:48:20 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 6FF13180ACF8F for ; Sat, 18 Apr 2020 09:48:20 +0000 (UTC) X-FDA: 76720500360.04.walk99_2baab4886083d X-HE-Tag: walk99_2baab4886083d X-Filterd-Recvd-Size: 7574 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Sat, 18 Apr 2020 09:48:19 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id g13so3732017wrb.8 for ; Sat, 18 Apr 2020 02:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to; bh=yppnx2RRRH6dXEAoSsBWx8ZDGYXI8EKm8jvsYqKLP1E=; b=ME8UtMFWbT941uR7oSocXnBCpbpKwfoPlneWRsX58TC3vJrZ/9VvbpiTseKmNK01vX R7lTwAvYFwHCidRBF+U+UGw1fadYUhb1qkRRr7wDWQ0C8Z3A7Vt/gvd8q+j94AhSUFB4 EdFgcLTPa/aBeW10TQMn803vIgBPf1Eh+yd+lBf9a74n0KCLzbeyPggtMAXbpe+uIn6D M/lz9isgvD6P78K4Lce/hoBgODW29Y3gjMpUyAqA37DO/FSv2JthSth81Wz8wQGo9Jzi ZeXHCS74SIBJQBvqO/dtHmUdzAxRTlOK2mzdAyz+H4ZoK2+Az1cvAoEAIwxMnS8g+9n9 24tA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to; bh=yppnx2RRRH6dXEAoSsBWx8ZDGYXI8EKm8jvsYqKLP1E=; b=OxOU7X3QjO7S4QxLPsBwEJtc4b3ItpSMBLxRqmgWDx1vfJ28iinBODIeNpO8rhXnek QVmxif08wV0i26WacmwkWkxImQgwgiRUuzyukfwQnUw4kXh0NwVS7ShFtkOIcitQN7YS v746PO5XuJB9JSLbcyrXcCD/h8foCjRKJAqyWL4rqLK9tp7sZTimYgoC2DKIisqo+/7U Jp6qt4013au9SSbtNxJMa2wzG7vdligQJ8zwDrV3v4YQEXuGphD+D1qA3naRJh8ztGkN ih7QqKtuaZuGPUttiVEt35q67MO24DIL9bGeX2YQrt3b9fdr30fB4UKUV3VUNHPEr/NL xoDA== X-Gm-Message-State: AGi0PuanzG0RFSTRkAqT/cLCz+ZgBZnOHkqsI0UPcA5PKB7hOliSXcqO SnLzWezYSmqdDrpOYoz0gOA= X-Google-Smtp-Source: APiQypJheoymwNGaxz1WvTAhfi+zRIqvQi08HK25/tXlituE/0ujV6q8nWaV2xC0KgcR+k5rtdBlbw== X-Received: by 2002:a5d:6850:: with SMTP id o16mr8024101wrw.309.1587203298550; Sat, 18 Apr 2020 02:48:18 -0700 (PDT) Received: from localhost.localdomain ([2a02:2450:10d2:194d:d02f:2f6b:b441:f83]) by smtp.gmail.com with ESMTPSA id p3sm17185593wrx.82.2020.04.18.02.48.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 18 Apr 2020 02:48:17 -0700 (PDT) From: SeongJae Park To: Mel Gorman Cc: SeongJae Park , Mel Gorman , Peter Zijlstra , "Huang, Ying" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Ingo Molnar , Rik van Riel , Daniel Jordan , Tejun Heo , Dave Hansen , Tim Chen , Aubrey Li Subject: Re: Re: Re: Re: Re: [RFC] autonuma: Support to scan page table asynchronously Date: Sat, 18 Apr 2020 11:48:11 +0200 Message-Id: <20200418094811.4757-1-sj38.park@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200417144626.GA3711@suse.de> (raw) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 17 Apr 2020 15:46:26 +0100 Mel Gorman wrote: > On Fri, Apr 17, 2020 at 02:44:37PM +0200, SeongJae Park wrote: > > On Fri, 17 Apr 2020 13:16:29 +0100 Mel Gorman wrote: > > > > > On Fri, Apr 17, 2020 at 12:21:29PM +0200, SeongJae Park wrote: > > > > On Fri, 17 Apr 2020 12:04:17 +0200 Peter Zijlstra wrote: > > > > > > > > > On Fri, Apr 17, 2020 at 09:05:08AM +0200, SeongJae Park wrote: > > > > > > I think the main idea of DAMON[1] might be able to applied here. Have you > > > > > > considered it? > > > > > > > > > > > > [1] https://lore.kernel.org/linux-mm/20200406130938.14066-1-sjpark@amazon.com/ > > > > > > > > > > I've ignored that entire thing after you said the information it > > > > > provides was already available through the PMU. > > > > > > > > Sorry if my answer made you confused. What I wanted to say was that the > > > > fundamental access checking mechanism that DAMON depends on is PTE Accessed bit > > > > for now, but it could be modified to use PMU or other features instead. > > > > > > I would not be inclined to lean towards either approach for NUMA > > > balancing. Fiddling with the accessed bit can have consequences for page > > > aging and residency -- fine for debugging a problem, not to fine for > > > normal usage. I would expect the PMU approach would have high overhead > > > as well as taking over a PMU counter that userspace debugging may expect > > > to be available. > > > > Good point. But, isn't it ok to use Accessed bit as long as PG_Idle and > > PG_Young is adjusted properly? Current DAMON implementation does so, as > > idle_page_tracking also does. > > > > PG_idle and PG_young are page flags that are used by idle tracking. The > accessed bit I'm thinking of is in the PTE bit and it's the PTE that > is changed by page_referenced() during reclaim. So it's not clear how > adjusting the page bits help avoid page aging problems during reclaim. The idle tracking reads and updates not only PG_idle and PG_young, but also the accessed bit. So idle tacking was also required to deal with the interference of the reclaimer logic and thus it introduced PG_young. The commit 33c3fc71c8cf ("mm: introduce idle page tracking") explains this as below: The Young page flag is used to avoid interference with the memory reclaimer. A page's Young flag is set whenever the Access bit of a page table entry pointing to the page is cleared by writing to the bitmap file. If page_referenced() is called on a Young page, it will add 1 to its return value, therefore concealing the fact that the Access bit was cleared. DAMON adjusts PG_young in similar way to avoid the interference. DAMON further adjusts PG_idle so that it does not interfere to page idle tracking mechanism. I think I made you confused by unnecessarily mentioning PG_idle, sorry. > > Maybe your suggestion was to move NUMA balancing to use the PG_idle and > PG_young bits from idle tracking but I'm not sure what that gains us. > This may be because I did not look too closely at DAMON as for debugging > and analysis, the PMU sounded more suitable. > > It's not clear to me how looking at the page or pte bit handling of > DAMON helps us reduce the scanning cost of numa balancing. There may > be some benefit in splitting the address space and rescanning sections > that were referenced but it has similar pitfalls to simply tracking the > highest/lowest address that trapped a NUMA hinting fault. DAMON allows users to know which address ranges have which access frequency. Thus, I think DAMON could be used for detection of hot pages, which will be good migration candidates, instead of the NUMA hinting fault based referenced pages detection. The benefits I expect from this change are better accuracy and less overhead. As we can know not only referenced pages but hot pages, migration will be more effective. In terms of the overhead, DAMON allows users to set the upperbound of the monitoring overhead regardless of the size of the targe address space, unlike the page scanning based mechanisms. Thus, the overhead could be reduced, especially if the target process has large memory footprint. As I'm not deeply understanding AutoNUMA, I might saying something unrealistic or missing some important points. Thanks, SeongJae Park > > -- > Mel Gorman > SUSE Labs