From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F8B0C43334 for ; Tue, 14 Jun 2022 15:30:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26D166B009F; Tue, 14 Jun 2022 11:30:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21D0B6B00A0; Tue, 14 Jun 2022 11:30:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0BD686B00A1; Tue, 14 Jun 2022 11:30:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F13666B009F for ; Tue, 14 Jun 2022 11:30:52 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AC809F48 for ; Tue, 14 Jun 2022 15:30:52 +0000 (UTC) X-FDA: 79577229144.03.845BDF4 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 1A5961400A6 for ; Tue, 14 Jun 2022 15:30:51 +0000 (UTC) Received: by mail-qk1-f172.google.com with SMTP id a184so6629742qkg.5 for ; Tue, 14 Jun 2022 08:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=JhfKtFfiAGo/gAbCmokDRZ06JTnmfd4HUXshARByGug=; b=M8nZEG46t1hnT/izWr93Tj0je2+YgGFfSnaCWsCuN/VMruWBpydUtKCsIkwhO8/oAr 44yX9Vqwii20g0XYTVOzdVl8NCLMvpWMAbETpa+XiEBP71tA71Y/ZLozyiCOj7sX3cKB S70P/xmQHvcX+TNdd/rk2nT4ljAnkHDWw6uWGuu2P8FG/17JmddgzE+l1/GPKswOUOqO 5cyxDEhfrBTusSZDDWvVvmcZfTKVkaRpR0xWNDKjserTNJFunblQQHySYHI0vbHQaNfx WkGhYNAresMrJ5E+A3PHSCBxZZEDwdGKPHmi7Y8yTqG4AAJecHixiE5COJvyFppHXGK6 PjqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=JhfKtFfiAGo/gAbCmokDRZ06JTnmfd4HUXshARByGug=; b=Iq91t1xXf+ncRN9FpvaFKrJ5ZIaQjH612yCjbsrgFiS5auBzxk8eVpyVSU+kokM1hh AGGZ+W40cmjRK/j7ZDn7TizQZHH/G7NeQTn5Xi7wGbyLB5RQ1XKD+a+IgfxWkE49U1is T/P+yAyVQitbIjUxoriQhE7y/IhXZvbMUTvoJf4V8B5vgcTN7Yb0Nxd+CGNKhBNLUCH/ 7AZ9XmOrerjZzbPG6aPxN5vM8oVHKVSs3N3lwBzoSNW7Rf/C1e89r3ZGjvTMvCOsGaxj xN3rTZor2QURMbJj7KiSl9Ka3CR61KnbN5PkN1R55BULHtqlMf75FBtLSWIi5WbFmh58 QRPQ== X-Gm-Message-State: AOAM532NXqxgjizwLIM1Zvuf6nLR4ENVa/uE1s2z+OFNeviKu07bjesl OCXKdfc4ky0RccCP+y186bFj9A== X-Google-Smtp-Source: ABdhPJy4ML+jkJj2AP96ShcZxmI4jgipOk1Aa/fgD3oSjTGB5nbFWpp2Rooz/AFJECQwfZeA0NL0fg== X-Received: by 2002:a37:8701:0:b0:6a5:ff4f:b2bc with SMTP id j1-20020a378701000000b006a5ff4fb2bcmr4507594qkd.584.1655220650781; Tue, 14 Jun 2022 08:30:50 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:2c45]) by smtp.gmail.com with ESMTPSA id i23-20020a05620a151700b006a6f6cdfd94sm9111789qkk.117.2022.06.14.08.30.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 08:30:50 -0700 (PDT) Date: Tue, 14 Jun 2022 11:30:49 -0400 From: Johannes Weiner To: Huang Ying Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Michal Hocko , Rik van Riel , Mel Gorman , Peter Zijlstra , Dave Hansen , Yang Shi , Zi Yan , Wei Xu , osalvador , Shakeel Butt , Zhong Jiang Subject: Re: [PATCH -V3 0/3] memory tiering: hot page selection Message-ID: References: <20220614081635.194014-1-ying.huang@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220614081635.194014-1-ying.huang@intel.com> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655220652; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JhfKtFfiAGo/gAbCmokDRZ06JTnmfd4HUXshARByGug=; b=i1veNb4O60iMYOT7GlkVZuozNs4/rMI+7rvJsq61e5AaQ5F2Q66Qfm11vEWm90IH5YEhRG L+NPVsxeP3V9eX7TwlaNkCqHeeEZEdhPrBNF/JdxZmDrHmn1yER55xchyLtVErQhQ5ys+d LckVJcMQhtavR992S1jWeuTDEHdyeEg= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=M8nZEG46; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf26.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.172 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655220652; a=rsa-sha256; cv=none; b=lfCoHEE4kCaz1c5G2gHl075pGzhPPqRggBb1y8zWHUTIVKn/0WD1jrDJdY6AkRluN6xCYO YRrPzeo/ziVcyY+6ynhmGELvGNPHzZjftDt7ZdBIKRzpqpnXYpN1RStx6sYOeHYXZHNVV8 ghrnoK7ueO/o9u1789FAYKLTvz/A6Wo= X-Stat-Signature: s7qs5cxfojd8e8dfhwqu3z76urhcqdr9 X-Rspamd-Queue-Id: 1A5961400A6 X-Rspam-User: X-Rspamd-Server: rspam05 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=M8nZEG46; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf26.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.172 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-HE-Tag: 1655220651-698105 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Huang, Have you had a chance to look at our hot page detection patch that Hasan has sent out some time ago? [1] It hooks into page reclaim to determine what is and isn't hot. Reclaim is an existing, well-tested mechanism to do just that. It's just 13 lines of code: set active bit on the first hint fault; promote on the second one if the active bit is still set. This promotes only pages hot enough that they can compete with toptier access frequencies. It's not just convenient, it's also essential to link tier promotion rate to page aging. Tiered NUMA balancing is about establishing a global LRU order across two (or more) nodes. LRU promotions *within* a node require multiple LRU cycles with references. LRU promotions *between* nodes must follow the same rules, and be subject to the same aging pressure, or you can get much colder pages promoted into a very hot workingset and wreak havoc. We've hammered this patch quite extensively with several Meta production workloads and it's been working reliably at keeping reasonable promotion rates. @@ -4202,6 +4202,19 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) last_cpupid = page_cpupid_last(page); page_nid = page_to_nid(page); + + /* Only migrate pages that are active on non-toptier node */ + if (numa_promotion_tiered_enabled && + !node_is_toptier(page_nid) && + !PageActive(page)) { + count_vm_numa_event(NUMA_HINT_FAULTS); + if (page_nid == numa_node_id()) + count_vm_numa_event(NUMA_HINT_FAULTS_LOCAL); + mark_page_accessed(page); + pte_unmap_unlock(vmf->pte, vmf->ptl); + goto out; + } + target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid, &flags); pte_unmap_unlock(vmf->pte, vmf->ptl); [1] https://lore.kernel.org/all/20211130003634.35468-1-hasanalmaruf@fb.com/t/#m85b95624622f175ca17a00cc8cc0fc9cc4eeb6d2