From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0CA0CA9EB9 for ; Wed, 23 Oct 2019 23:12:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 74F9E2064B for ; Wed, 23 Oct 2019 23:12:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LPKaA53L" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74F9E2064B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0B9586B0005; Wed, 23 Oct 2019 19:12:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 069BB6B0006; Wed, 23 Oct 2019 19:12:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC0C36B0007; Wed, 23 Oct 2019 19:12:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id C9D896B0005 for ; Wed, 23 Oct 2019 19:12:20 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 5D00F181AEF1F for ; Wed, 23 Oct 2019 23:12:20 +0000 (UTC) X-FDA: 76076600040.25.shock54_317961637ef3a X-HE-Tag: shock54_317961637ef3a X-Filterd-Recvd-Size: 7025 Received: from mail-qt1-f171.google.com (mail-qt1-f171.google.com [209.85.160.171]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Wed, 23 Oct 2019 23:12:19 +0000 (UTC) Received: by mail-qt1-f171.google.com with SMTP id w14so34795508qto.9 for ; Wed, 23 Oct 2019 16:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6chDs+IgEJKamu+cJb66Y7u49Uq6uTH7s8nRkQlDdp4=; b=LPKaA53L94ItgBQ7cRZRd0OF0ksHyc+ImxDvEoxDh+IWg0ADUvqw/DCa45By3Onsik gEU3iQy9d0ZpW+jKvEhPB/dns4TYCrDRKf0CiWDDwH5SKMn8XbAbBPGiBn+/jWiyLdHV awdW1hPhMkH/78y45KV6mVxeOLr1oGi9oABZLlyG7YxD8RpylW8NaZFy2c18py4LFwrr mgLjOB3reVQsqTcuG1jtcwbqv+gBtAHK4fwACqARXqd892/pvyR2RP3+YIgeqONkcrAy 32DsNnCm4eybLA6NFqb4tqohrizzvQFkrEqxQ/0NCElAzRP796aEz92xJmbwSlNVwvTS wIOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6chDs+IgEJKamu+cJb66Y7u49Uq6uTH7s8nRkQlDdp4=; b=DHdYR7cnd97TL3wUJS7PejxN1e3FeXUvVyZ3M2zDK2hgX1BBFX9rAjnxu6v3U2Ut9/ 3FspDPaossCK2beGVb+sqG8b0S5XrJVpi9DDhOPpZVLtDPK51AlvQqCOr/ae6GiIS3zQ 93z4b6G5aw+0mQhZxK5NVHXTRl/qc0VmMQpsQBuUoKzlmj72NUiO/4ZKPBIPI7wnAMxU 2ioSqHQgszsvyJcGIAfzuajEZ8R/+CnGqged+MaYqN+6raFN13bpbw5B6PjzSHn4Sh2k i1s2tSEyd/upO5OTq/zh4TRzkMsjhPI4IjeUM1RVuh5J9Qgf0GRDHfD5AaTFYTrWg0kz sKEw== X-Gm-Message-State: APjAAAXizIUhLHlyrXDR/0G4EEEPhqQVbfCCuILlQh3ZdwAxP7RNbtqt 6a+T6i1+ZUwoGxlaZcV1PO73TOB0XOx52BGMYtq9 X-Google-Smtp-Source: APXvYqw9jjyjfdvDyweAefKCm5wL44bLbMEPax372VuFPYTjsi7cw8qlwb8rL02E3il5G17GqHSSzKN2WhUwQdETlsU= X-Received: by 2002:aed:3f57:: with SMTP id q23mr1240067qtf.116.1571872338418; Wed, 23 Oct 2019 16:12:18 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jonathan Adams Date: Wed, 23 Oct 2019 16:11:39 -0700 Message-ID: Subject: Re: [RFC] Memory Tiering To: Dave Hansen Cc: Linux-MM , LKML , "Williams, Dan J" , "Verma, Vishal L" , Wu Fengguang , Huang Ying Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: 1 On Wed, Oct 16, 2019 at 1:05 PM Dave Hansen wrote: > > The memory hierarchy is getting more complicated and the kernel is > playing an increasing role in managing the different tiers. A few > different groups of folks described "migration" optimizations they were > doing in this area at LSF/MM earlier this year. One of the questions > folks asked was why autonuma wasn't being used. > > At Intel, the primary new tier that we're looking at is persistent > memory (PMEM). We'd like to be able to use "persistent memory" > *without* using its persistence properties, treating it as slightly > slower DRAM. Keith Busch has some patches to use NUMA migration to > automatically migrate DRAM->PMEM instead of discarding it near the end > of the reclaim process. Huang Ying has some patches which use a > modified autonuma to migrate frequently-used data *back* from PMEM->DRAM. > > We've tried to do this all generically so that it is not tied to > persistent memory and can be applied to any memory types in lots of > topologies. > > We've been running this code in various forms for the past few months, > comparing it to pure DRAM and hardware-based caching. The initial > results are encouraging and we thought others might want to take a look > at the code or run their own experiments. We're expecting to post the > individual patches soon. But, until then, the code is available here: > > https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git > > and is tagged with "tiering-0.2", aka. d8e31e81b1dca9. Hi Dave, Thanks for sharing this git link and information on your approach. This interesting, and lines up somewhat with the approach Google has been investigating. As we discussed at LSF/MM[1] and Linux Plumbers[2], we're working on an approach which integrates with our proactive reclaim work, with a similar attitude to PMEM (use it as "slightly slower" DRAM, ignoring its persistence). The prototype we have has a similar basic structure to what you're doing here and Yang Shi's patchset from March[3] (separate NUMA nodes for PMEM), but relies on a fair amount of kernel changes to control allocations from the NUMA nodes, and uses a similar "is_far" NUMA flag to Yang Shi's approach. We're working on redesigning to reduce the scope of kernel changes and to remove the "is_far" special handling; we still haven't refined down to a final approach, but one basic part we want to keep from the prototype is proactively pushing PMEM data back to DRAM when we've noticed it's in use. If we look at a two-socket system: A: DRAM & CPU node for socket 0 B: PMEM node for socket 0 C: DRAM & CPU node for socket 1 D: PMEM node for socket 1 instead of the unidirectional approach your patches go for: A is marked as "in reclaim, push pages to" B C is marked as "in reclaim, push pages to" D B & D have no markings we would have a bidirectional attachment: A is marked "move cold pages to" B B is marked "move hot pages to" A C is marked "move cold pages to" D D is marked "move hot pages to" C By using autonuma for moving PMEM pages back to DRAM, you avoid needing the B->A & D->C links, at the cost of migrating the pages back synchronously at pagefault time (assuming my understanding of how autonuma works is accurate). Our approach still lets you have multiple levels of hierarchy for a given socket (you could imaging an "E" node with the same relation to "B" as "B" has to "A"), but doesn't make it easy to represent (say) an "E" which was equally close to all sockets (which I could imagine for something like remote memory on GenZ or what-have-you), since there wouldn't be a single back link; there would need to be something like your autonuma support to achieve that. Does that make sense? Thanks, - Jonathan [1] Shakeel's talk, I can't find a link at the moment. The basic kstaled/kreclaimd approach we built upon is talked about in https://blog.acolyer.org/2019/05/22/sw-far-memory/ and the linked ASPLOS paper [2] https://linuxplumbersconf.org/event/4/contributions/561/; slides at https://linuxplumbersconf.org/event/4/contributions/561/attachments/363/596/Persistent_Memory_as_Memory.pdf [3] https://lkml.org/lkml/2019/3/23/10