From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D78D5C433EF for ; Thu, 30 Jun 2022 00:57:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0FA28E0001; Wed, 29 Jun 2022 20:57:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC0E86B0073; Wed, 29 Jun 2022 20:57:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DAE288E0001; Wed, 29 Jun 2022 20:57:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CC7E46B0072 for ; Wed, 29 Jun 2022 20:57:46 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A289C35402 for ; Thu, 30 Jun 2022 00:57:46 +0000 (UTC) X-FDA: 79633089732.07.5FD8855 Received: from mail-ot1-f54.google.com (mail-ot1-f54.google.com [209.85.210.54]) by imf03.hostedemail.com (Postfix) with ESMTP id 4CEE62002F for ; Thu, 30 Jun 2022 00:57:46 +0000 (UTC) Received: by mail-ot1-f54.google.com with SMTP id cb12-20020a056830618c00b00616b871cef3so11199958otb.5 for ; Wed, 29 Jun 2022 17:57:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=SKqfiDiYuvfDETXVnoeeg5pQ6AZyqtVKR2qDMSZMeUk=; b=MbC6Tw1nfwmIzq9H1rKp+n/UqqIX05yDgOOlpIZdHXKKpP/zyZ6fwpSvAzJ8lYcYlZ YIqyVrj6+1G7upZyavcalRFk921/82SZDL0tu+kbZDdrp3N3BRkw2+YCNvyBvYkqe+Z6 a49VJNtdaXVVd5x52SPZ4zs+wyBDcXV32Mmbq5HeQQJV3ErI64ZXkiDtII1g+jJ+dMkh Tgz7TMKzDmDwv9e8+fFx5f8nNR4YDVRJaQKq2XzRQd3G7WQCzRc96Qj/lG0FDtU/wMqL PNqTzGaTuRcp8DiuT8CHwAo16HJuuG3v1fKzsv/9AtubCRHbETL8X6/SxKqE7UptDZuZ MWsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=SKqfiDiYuvfDETXVnoeeg5pQ6AZyqtVKR2qDMSZMeUk=; b=xk6DJJ4SdALpwaWKiNMTR0T+iA4d+6HVue0DwcEuUy1mqpPD+IJmk8gcz+caFUZwrL SyKJPn6gYggZr8bHowaNIDqGdRS42SVUZqwy7H6+5TXDH795mzhxD4L8KMv3f+xpr9lj awrz61Ir4Q198UmMVQXgTnSE1Od47nahu1O0wTHOGUcJS5+hTXxZcii10Fn0Wn5JcSD+ 7BAGuyzu4JSYB5SJs1LNFOt2mn566UzW6CBpXEmLUnmuEm7vUSQmd7mbnjSmFjOSO70t BxPbjdBhikof4y6steYzJEinB7bHEHWbipaU+a6uGrwXvxfZnaZM7cWhNIZWt4USYzVQ ODBg== X-Gm-Message-State: AJIora9rZhNKUKjzgha7YEhFA04FZvZ+/ZsYnqNcxvYPYXkk+lW/Jw6O RjRbi2ag/By9IVBFYDNeXWfk4z5AbHG5Rd80dow= X-Google-Smtp-Source: AGRyM1uuqNmU1rx2B7OEGgbZcMQT3ivFsBg7sH7fuoVOlpvED76IbLTkh316ZjKbIN9vW844cpjPQkNBZoXKO5IHV/4= X-Received: by 2002:a05:6830:d13:b0:618:b519:5407 with SMTP id bu19-20020a0568300d1300b00618b5195407mr201343otb.219.1656550665471; Wed, 29 Jun 2022 17:57:45 -0700 (PDT) MIME-Version: 1.0 References: <20220622082513.467538-1-aneesh.kumar@linux.ibm.com> <20220622082513.467538-12-aneesh.kumar@linux.ibm.com> In-Reply-To: <20220622082513.467538-12-aneesh.kumar@linux.ibm.com> From: Souptick Joarder Date: Thu, 30 Jun 2022 06:27:34 +0530 Message-ID: Subject: Re: [PATCH v7 11/12] mm/demotion: Add documentation for memory tiering To: "Aneesh Kumar K.V" Cc: Linux-MM , Andrew Morton , Wei Xu , Huang Ying , Yang Shi , Davidlohr Bueso , Tim C Chen , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Jagdish Gediya Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656550666; a=rsa-sha256; cv=none; b=452VAnCJYQxm/0MZVcZnWBSuMoqgP0vgaKX3D/HXMzL2sDFFta8wIZV/QlBrHQjP1olEfe Dbjqiw5RSrGhzgaeGzNNQGio9kVynDZXLOUP5WIebXF6nKqydLF6boxFaQ5dT81J3wq7Ci Wm1eBJ+DKKs0y/gAS5WukJglBgjG5sc= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MbC6Tw1n; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of jrdr.linux@gmail.com designates 209.85.210.54 as permitted sender) smtp.mailfrom=jrdr.linux@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656550666; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SKqfiDiYuvfDETXVnoeeg5pQ6AZyqtVKR2qDMSZMeUk=; b=HwYQqWTqkVgyYWFfHeSqJ8CPiaWvBAtwRD2i8NXKZQYxBuB0D9mXD+K9IGxOu4GEpR8tto GDvqwNzEXNuJaz0SxFoexrrCjqanfH6UhA6F78I+zjumqPqaVLx6UNwEm6vOdZkt2QkAth uGkENPDfXoMbzfl3KUz87FAfY43NiBc= X-Rspamd-Queue-Id: 4CEE62002F X-Rspam-User: Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=MbC6Tw1n; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf03.hostedemail.com: domain of jrdr.linux@gmail.com designates 209.85.210.54 as permitted sender) smtp.mailfrom=jrdr.linux@gmail.com X-Rspamd-Server: rspam02 X-Stat-Signature: afb7m8jg8xaye1xpwfeu4676kebpyya5 X-HE-Tag: 1656550666-841905 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 22, 2022 at 2:04 PM Aneesh Kumar K.V wrote: > > From: Jagdish Gediya > > All N_MEMORY nodes are divided into 3 memoty tiers with tier ID value s /memoty/ memory > MEMORY_TIER_HBM_GPU, MEMORY_TIER_DRAM and MEMORY_TIER_PMEM. By default, > all nodes are assigned to default memory tier. I think adding the default memory tier name will be helpful. > > Demotion path for all N_MEMORY nodes is prepared based on the tier ID value > of memory tiers. > > This patch adds documention for memory tiering introduction, its sysfs > interfaces and how demotion is performed based on memory tiers. > > Suggested-by: Wei Xu > Signed-off-by: Jagdish Gediya > Signed-off-by: Aneesh Kumar K.V > --- > Documentation/admin-guide/mm/index.rst | 1 + > .../admin-guide/mm/memory-tiering.rst | 182 ++++++++++++++++++ > 2 files changed, 183 insertions(+) > create mode 100644 Documentation/admin-guide/mm/memory-tiering.rst > > diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-guide/mm/index.rst > index c21b5823f126..3f211cbca8c3 100644 > --- a/Documentation/admin-guide/mm/index.rst > +++ b/Documentation/admin-guide/mm/index.rst > @@ -32,6 +32,7 @@ the Linux memory management. > idle_page_tracking > ksm > memory-hotplug > + memory-tiering > nommu-mmap > numa_memory_policy > numaperf > diff --git a/Documentation/admin-guide/mm/memory-tiering.rst b/Documentation/admin-guide/mm/memory-tiering.rst > new file mode 100644 > index 000000000000..142c36651f5d > --- /dev/null > +++ b/Documentation/admin-guide/mm/memory-tiering.rst > @@ -0,0 +1,182 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +.. _admin_guide_memory_tiering: > + > +=========== > +Memory tiers > +============ > + > +This document describes explicit memory tiering support along with > +demotion based on memory tiers. > + > +Introduction > +============ > + > +Many systems have multiple types of memory devices e.g. GPU, DRAM and > +PMEM. The memory subsystem of these systems can be called a memory > +tiering system because the performance of the different types of > +memory is different. Memory tiers are defined based on the hardware > +capabilities of memory nodes. Each memory tier is assigned a tier ID > +value that determines the memory tier position in demotion order. > + > +The memory tier assignment of each node is independent of each > +other. Moving a node from one tier to another tier doesn't affect > +the tier assignment of any other node. > + > +Memory tiers are used to build the demotion targets for nodes. A node > +can demote its pages to any node of any lower tiers. > + > +Memory tier rank > +================= > + > +Memory nodes are divided into 3 types of memory tiers with tier ID > +value as shown based on their hardware characteristics. > + > + > +MEMORY_TIER_HBM_GPU > +MEMORY_TIER_DRAM > +MEMORY_TIER_PMEM > + > +Memory tiers initialization and (re)assignments > +=============================================== > + > +By default, all nodes are assigned to the memory tier with the default tier ID > +DEFAULT_MEMORY_TIER which is 200 (MEMORY_TIER_DRAM). The memory tier of > +the memory node can be either modified through sysfs or from the driver. On > +hotplug, the memory tier with default tier ID is assigned to the memory node. > + > + > +Sysfs interfaces > +================ > + > +Nodes belonging to specific tier can be read from, > +/sys/devices/system/memtier/memtierN/nodelist (Read-Only) > + > +Where N is 0 - 2. > + > +Example 1: > +For a system where Node 0 is CPU + DRAM nodes, Node 1 is HBM node, > +node 2 is a PMEM node an ideal tier layout will be > + > +$ cat /sys/devices/system/memtier/memtier0/nodelist > +1 > +$ cat /sys/devices/system/memtier/memtier1/nodelist > +0 > +$ cat /sys/devices/system/memtier/memtier2/nodelist > +2 > + > +Example 2: > +For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM > +nodes. > + > +$ cat /sys/devices/system/memtier/memtier0/nodelist > +cat: /sys/devices/system/memtier/memtier0/nodelist: No such file or > +directory > +$ cat /sys/devices/system/memtier/memtier1/nodelist > +0-1 > +$ cat /sys/devices/system/memtier/memtier2/nodelist > +2-3 > + > +Default memory tier can be read from, > +/sys/devices/system/memtier/default_tier (Read-Only) > + > +e.g. > +$ cat /sys/devices/system/memtier/default_tier > +memtier200 > + > +Max memory tier ID supported can be read from, > +/sys/devices/system/memtier/max_tier (Read-Only) > + > +e.g. > +$ cat /sys/devices/system/memtier/max_tier > +400 > + > +Individual node's memory tier can be read of set using, > +/sys/devices/system/node/nodeN/memtier (Read-Write) > + > +where N = node id > + > +When this interface is written, Node is moved from the old memory tier > +to new memory tier and demotion targets for all N_MEMORY nodes are > +built again. > + > +For example 1 mentioned above, > +$ cat /sys/devices/system/node/node0/memtier > +1 > +$ cat /sys/devices/system/node/node1/memtier > +0 > +$ cat /sys/devices/system/node/node2/memtier > +2 > + > +Additional memory tiers can be created by writing a tier ID value to this file. > +This results in a new memory tier creation and moving the specific NUMA node to > +that memory tier. > + > +Demotion > +======== > + > +In a system with DRAM and persistent memory, once DRAM > +fills up, reclaim will start and some of the DRAM contents will be > +thrown out even if there is a space in persistent memory. > +Consequently, allocations will, at some point, start falling over to the slower > +persistent memory. > + > +That has two nasty properties. First, the newer allocations can end up in > +the slower persistent memory. Second, reclaimed data in DRAM are just > +discarded even if there are gobs of space in persistent memory that could > +be used. > + > +Instead of a page being discarded during reclaim, it can be moved to > +persistent memory. Allowing page migration during reclaim enables > +these systems to migrate pages from fast(higher) tiers to slow(lower) > +tiers when the fast(higher) tier is under pressure. > + > + > +Enable/Disable demotion > +----------------------- > + > +By default demotion is disabled, it can be enabled/disabled using > +below sysfs interface, > + > +$ echo 0/1 or false/true > /sys/kernel/mm/numa/demotion_enabled > + > +preferred and allowed demotion nodes > +------------------------------------ > + > +Preferred nodes for a specific N_MEMORY node are the best nodes > +from the next possible lower memory tier. Allowed nodes for any > +node are all the nodes available in all possible lower memory > +tiers. > + > +Example: > + > +For a system where Node 0 & 1 are CPU + DRAM nodes, node 2 & 3 are PMEM > +nodes, > + > +node distances: > +node 0 1 2 3 > + 0 10 20 30 40 > + 1 20 10 40 30 > + 2 30 40 10 40 > + 3 40 30 40 10 > + > +memory_tiers[0] = > +memory_tiers[1] = 0-1 > +memory_tiers[2] = 2-3 > + > +node_demotion[0].preferred = 2 > +node_demotion[0].allowed = 2, 3 > +node_demotion[1].preferred = 3 > +node_demotion[1].allowed = 3, 2 > +node_demotion[2].preferred = > +node_demotion[2].allowed = > +node_demotion[3].preferred = > +node_demotion[3].allowed = > + > +Memory allocation for demotion > +------------------------------ > + > +If a page needs to be demoted from any node, the kernel 1st tries > +to allocate a new page from the node's preferred node and fallbacks to > +node's allowed targets in allocation fallback order. > + > -- > 2.36.1 > >