From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01E01C433F5 for ; Tue, 10 May 2022 04:43:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F8636B0072; Tue, 10 May 2022 00:43:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 47FE68D0001; Tue, 10 May 2022 00:43:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F98C6B0074; Tue, 10 May 2022 00:43:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 1A5FA6B0072 for ; Tue, 10 May 2022 00:43:40 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id E87D960EE6 for ; Tue, 10 May 2022 04:43:39 +0000 (UTC) X-FDA: 79448590158.11.33C9E52 Received: from mail-vk1-f178.google.com (mail-vk1-f178.google.com [209.85.221.178]) by imf11.hostedemail.com (Postfix) with ESMTP id BC43840094 for ; Tue, 10 May 2022 04:43:34 +0000 (UTC) Received: by mail-vk1-f178.google.com with SMTP id j14so5185482vkp.4 for ; Mon, 09 May 2022 21:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ti1foShj1iTCmaM6cyCkyOU/54zgXNg0Z+MWa8lGpwE=; b=i/lhJAfNJS1T3aWLg3tEmuSGTRIrdIuOhiYgUtbKa7OvB05xOEzz1aidrZX/3Y0oJ0 jqhN1asTxqUWsB4my07mpcD1/nsr6tjEgP8aF5a8CnG/rqVKBKKGb9Qg5ptTaupM42Gv Ha7nTekckik9jppD3d5sCB8pdC3u8cEwJoaIonkZ8VQtZEHMWuJvqn7VuxM3aBnJEWzg piLBIbXG/fmLgpGfP0Uet1w7MdjwkPjDKtWqtdBtuzJcuCBpR7O1enFKsiJQcydn5JJN DVGb8Oj8ZPBqIPmvm8qV1YvxbK2OcNZyzLl+zmcwBh+Xt3DeL2TYek7iJYOdVT60yHFf 09Ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ti1foShj1iTCmaM6cyCkyOU/54zgXNg0Z+MWa8lGpwE=; b=SKwMvGHjaqIIq/IRThaECllpFcyHQOqKZz9acR1k32KmVd6cpwrJt7MUJBzI+CEzb6 3Qy1CESSRJBpaAj/f1/Lct1noGcs932tVi2UAyy7mdFHYLSutUtZYNpAL+J5N8Hf9Z/X Wn+PChKfylvjdt4ueNWP3WQJXTFJF21qhbVG/Aa4M89VVFYDXe0x6AqWdkLkwNECiKHN +3Dia/r2MDPkgcY9diMX83FeesNSVsfY3SEv5qqfxXhr2RDpU8EYnFJT3jUfacDlP1ye DxKAZBBfHEDCVDyU+31mzrVmtckYnqogr05juCFKqbMLSzXMPJ89niStxFqrCIoncO9Y L8gg== X-Gm-Message-State: AOAM532UxyWphcOF/KP6jemvXj+goi2MIhu6ULBnndYCBfgcFS/RnVIs iTP+d4j2jB8dXTW2SaxAda7xa7y33b5HoNU/nZn3JQ== X-Google-Smtp-Source: ABdhPJzfgbaBZyzipWD/FN7OG6uaepNf8E6TvE5ShAOCeemGClqd6HGC02rUdA1hA/BcNZcdHcrfI4VtLYULlmzhdnU= X-Received: by 2002:a1f:9b85:0:b0:32d:4d56:cf64 with SMTP id d127-20020a1f9b85000000b0032d4d56cf64mr11173892vke.31.1652157818664; Mon, 09 May 2022 21:43:38 -0700 (PDT) MIME-Version: 1.0 References: <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal> <87mtfygoxs.fsf@nvdebian.thelocal> <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com> <52541497-c097-5a51-4718-feed13660255@intel.com> In-Reply-To: From: Wei Xu Date: Mon, 9 May 2022 21:43:27 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Dave Hansen Cc: Alistair Popple , Davidlohr Bueso , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Yang Shi , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BC43840094 X-Stat-Signature: 15rbzs5i51txig99wj8he4j33usapo7p Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="i/lhJAfN"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of weixugc@google.com designates 209.85.221.178 as permitted sender) smtp.mailfrom=weixugc@google.com X-Rspam-User: X-HE-Tag: 1652157814-639220 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, May 5, 2022 at 7:24 AM Dave Hansen wrote: > > On 5/4/22 23:35, Wei Xu wrote: > > On Wed, May 4, 2022 at 10:02 AM Dave Hansen wrote: > >> That means a lot of page table and EPT walks to map those linear > >> addresses back to physical. That adds to the inefficiency. > > > > That's true if the tracking is purely based on physical pages. For > > hot page tracking from PEBS, we can consider tracking in > > virtual/linear addresses. We don't need to maintain the history for > > all linear page addresses nor for an indefinite amount of time. After > > all, we just need to identify pages accessed frequently recently and > > promote them. > > Except that you don't want to promote on *every* access. That might > lead to too much churn. Certainly. We should use the PMU events to help build the page heatmap in software and select the hottest pages to promote accordingly. > You're also assuming that all accesses to a physical page are via a > single linear address, which ignores shared memory mapped at different > linear addresses. Our (maybe wrong) assumption has been that shared > memory is important enough to manage that it can't be ignored. Shared memory is important. Special handling will be needed to better support such pages for linear address based hot page tracking. > >> In the end, you get big PEBS buffers with lots of irrelevant data that > >> needs significant post-processing to make sense of it. > > > > I am curious about what are "lots of irrelevant data" if PEBS data is > > filtered on data sources (e.g. DRAM vs PMEM) by hardware. If we need > > to have different policies for the pages from the same data source, > > then I agree that the software has to do a lot of filtering work. > > Perhaps "irrelevant" was a bad term to use. I meant that you can't just > take the PEBS data and act directly on it. It has to be post-processed > and you will see things in there like lots of adjacent accesses to a > page. Those additional accesses can be interesting but at some point > you have all the weight you need to promote the page and the _rest_ are > irrelevant. That's right. The software has to do the post-processing work to build the page heatmap with what the existing hardware can provide. > >> The folks at Intel that tried this really struggled to take this mess and turn it into a successful hot-page tracking. > >> > >> Maybe someone else will find a better way to do it, but we tried and > >> gave up. > > > > It might be challenging to use PEBS as the only and universal hot page > > tracking hardware mechanism. For example, there are challenges to use > > PEBS to sample KVM guest accesses from the host. > > Yep, agreed. This aspect of the hardware is very painful at the moment. > > > On the other hand, PEBS with hardware-based data source filtering can > > be a useful mechanism to improve hot page tracking in conjunction > > with other techniques. > > Rather than "can", I'd say: "might". Backing up to what I said originally: > > > So, in practice, these events (PEBS) weren't very useful > > for driving memory tiering. > > By "driving" I really meant solely driving. Like, can PEBS be used as > the one and only mechanism? We couldn't make it work. But, the > hardware _is_ sitting there mostly unused. It might be great to augment > what is there, and nobody should be discouraged from looking at it again. I think we are on the same page.