From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 275ADC433F5 for ; Thu, 5 May 2022 06:35:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FDF46B0071; Thu, 5 May 2022 02:35:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2AE1A6B0073; Thu, 5 May 2022 02:35:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1752D6B0074; Thu, 5 May 2022 02:35:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0781B6B0071 for ; Thu, 5 May 2022 02:35:27 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id D304780E54 for ; Thu, 5 May 2022 06:35:26 +0000 (UTC) X-FDA: 79430727852.28.61D86D7 Received: from mail-vs1-f53.google.com (mail-vs1-f53.google.com [209.85.217.53]) by imf10.hostedemail.com (Postfix) with ESMTP id D6DB0C0088 for ; Thu, 5 May 2022 06:35:09 +0000 (UTC) Received: by mail-vs1-f53.google.com with SMTP id c62so3306284vsc.10 for ; Wed, 04 May 2022 23:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4bztzCmwlVS2QcT4CrwLAm2LJYy0mvP8yzB7o5GzUlE=; b=VAwbnBGTaJd6CJdGKIY38ODe/I0OSZG3i+j+HGdcVT0DkKaZqnFssiWzfzn+ARvpQu dvaH2jDGTkpqey5QsaZ89a3EGLC+YJSCRuyxJ1EuSnmgWMQ4I3Yd8uQsGh8YmzTeJSGv LWLbrwZIxlxVtZNLES/wezGhRQHZrvUcTuD1K1eS5Z3rE0RoB92Fm5/4FntJaUMND2GZ kyk8uzMJJ4kKfGoZmY9fVXY4HSPfWn7xVjX9cKGPKImthjYk9qrybX8HYuiGW/wVyvp/ FPazVDw3DE3/oSqqLHe02RbelyDOpBN3DVCVkUtSw16cC08mfEx3KPlxVMONUeA4+vUJ Qp5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4bztzCmwlVS2QcT4CrwLAm2LJYy0mvP8yzB7o5GzUlE=; b=A8foOG97KMT/CFFgGomgnlEQ2/Utmryx19rrI5tFGOsj97sJydk3LK5UjF9zVZh391 I/am680swIlack51jLpQ/C2QBR/cRA9uUCF9s9ZHOhZAqZC3avhENRVFmIgAzfLZTUSi bVFvsrAMdtP6SljcebWAiwW4l1uN/ZXyoDfIKEz2JD5r54oug1HkRkVIH7iKNhwJAwym CBzh6ux1ZJhEGwsWt2GJ5sAlipf9x9whQwBf2zrp+dVJ9Ce2MIoMj8xKVcdY+//GbWUw erzT7Vy7iuxqwjRAOEGv6loarNAZ+y5g8JhiVUlOembCTi0lYW4sS4D5xIPQTzagPDJx ihUg== X-Gm-Message-State: AOAM531yG3s3g4JSWQsgWOV79Cnz3czHlCg1tY4Rz0D7LeopSvTgp7h0 D0Cml5CZ7UhE73JTCSD3cyOGYVk94JxFtMwgNOz2mA== X-Google-Smtp-Source: ABdhPJyuv+aiZuvubpfM0EsZUwPFAaIYXVBDKpW+Szkd7a/nvSf+KIq0OFlREKrCHj2NU6EZD8olW9vfDC1yhXwI+1k= X-Received: by 2002:a67:2fc6:0:b0:32d:7195:32e9 with SMTP id v189-20020a672fc6000000b0032d719532e9mr1741557vsv.44.1651732525536; Wed, 04 May 2022 23:35:25 -0700 (PDT) MIME-Version: 1.0 References: <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal> <87mtfygoxs.fsf@nvdebian.thelocal> <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com> <52541497-c097-5a51-4718-feed13660255@intel.com> In-Reply-To: <52541497-c097-5a51-4718-feed13660255@intel.com> From: Wei Xu Date: Wed, 4 May 2022 23:35:14 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Dave Hansen Cc: Alistair Popple , Davidlohr Bueso , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Yang Shi , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D6DB0C0088 X-Stat-Signature: hjqdua76gpo7maq49x5qrs5jpjhmsmrz X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=VAwbnBGT; spf=pass (imf10.hostedemail.com: domain of weixugc@google.com designates 209.85.217.53 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1651732509-386021 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 4, 2022 at 10:02 AM Dave Hansen wrote: > > On 5/3/22 18:31, Wei Xu wrote: > >> Well, x86 CPUs have performance monitoring hardware that can > >> theoretically collect physical access information too. But, this > >> performance monitoring hardware wasn't designed for this specific use > >> case in mind. So, in practice, these events (PEBS) weren't very useful > >> for driving memory tiering. > > The PEBS events without any filtering might not be useful for memory > > tiering, but the PEBS events with hardware-based data source filtering > > can be useful in driving promotions in memory tiering. Certainly, > > because these events are not designed for this specific use case in > > mind, there are inefficiencies using them for memory tiering, e.g. > > instead of just getting a heat counter for each hot page, we can get > > events repeatedly on the hot pages. > > Also, I believe the addresses that come out of the PEBS events are > virtual addresses (Data Linear Addresses according to the SDM). If the > events are written from a KVM guest, you get guest linear addresses. > > That means a lot of page table and EPT walks to map those linear > addresses back to physical. That adds to the inefficiency. That's true if the tracking is purely based on physical pages. For hot page tracking from PEBS, we can consider tracking in virtual/linear addresses. We don't need to maintain the history for all linear page addresses nor for an indefinite amount of time. After all, we just need to identify pages accessed frequently recently and promote them. > In the end, you get big PEBS buffers with lots of irrelevant data that > needs significant post-processing to make sense of it. I am curious about what are "lots of irrelevant data" if PEBS data is filtered on data sources (e.g. DRAM vs PMEM) by hardware. If we need to have different policies for the pages from the same data source, then I agree that the software has to do a lot of filtering work. > The folks at Intel that tried this really struggled to take this mess and turn it into a successful hot-page tracking. > > Maybe someone else will find a better way to do it, but we tried and > gave up. It might be challenging to use PEBS as the only and universal hot page tracking hardware mechanism. For example, there are challenges to use PEBS to sample KVM guest accesses from the host. On the other hand, PEBS with hardware-based data source filtering can be a useful mechanism to improve hot page tracking in conjunction with other techniques.