From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5A6FC43334 for ; Mon, 13 Jun 2022 15:50:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 453688D0171; Mon, 13 Jun 2022 11:50:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B47B6B02A4; Mon, 13 Jun 2022 11:50:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22F2A8D0171; Mon, 13 Jun 2022 11:50:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0D9746B02A3 for ; Mon, 13 Jun 2022 11:50:59 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id DE0956049E for ; Mon, 13 Jun 2022 15:50:58 +0000 (UTC) X-FDA: 79573650996.03.DC828A1 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) by imf08.hostedemail.com (Postfix) with ESMTP id 5594A160089 for ; Mon, 13 Jun 2022 15:50:58 +0000 (UTC) Received: by mail-qk1-f177.google.com with SMTP id 15so4319788qki.6 for ; Mon, 13 Jun 2022 08:50:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=H0T+Ju0FY0ctldM76CfR3ifPomNMCFqKjz28sSDHzOo=; b=CRP/awNTFqsFKDaZFcnWqTMwOfEC1FNH1yVjUL6uKl/vBwqyravouD90Jv5QMA7cm0 qW1ubwVfAbdEKIgaIf3TK6U+mPjYw2RqW26CSKTQ6fC0Wqi238uRtSPqXUhgp/c8YXWC +03V571nkBn5BKJFt7qABBhZxie8tWshu/fvvWYkfhJZytIailOGrLJr/gwJui+pfRc+ Qgca9fqO+EGc5a20FwGUwksH4Razm9mopNFgEWMomWdAzJ4KtEMQd9dBOwqaMQyrSkHq NX8JdM6Y3zmdpzIr8nLxJDt6mLdZtiG5mXqVQWfsIl1ollke0vCuWEAf0iyLrFdVg+d+ zLjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=H0T+Ju0FY0ctldM76CfR3ifPomNMCFqKjz28sSDHzOo=; b=he4g9TEURT5nUrbkziUIImcghZpJuLfRcZOUfHnrzxK3DbIMelqKUxw0iuMnMb8+SB 8v2NF65X4u7DajFcpRDpLBmUmsZPEPk/3wcE5TNwJB3K4c+7im1QDlaC4alzJOJCFGjZ fZ/6Y7Cixi5m+elridRtGo8/FXh9aVcLXfWPe+K8e5GnTe+MET6N1Ud8v4XfKVYxdx9d 4+Ma14ovry7u9M5UlB2x4a1ydUZtHYqHexGK/q9Bi34rvIoURHK1+DzMr7FHZMPxZ7Cu yT9ol7xIuuFxt/uRPq6vItDEFgMG/cnLvOJFG2zBJ5LyLbIJrfvMZ88zrKIlxxha6YqZ 4ZJQ== X-Gm-Message-State: AOAM530mWONawJAecVCjJzJ/Gzx1jeGWkwpS/2ddgV+IWEPXNW4jLBZr iwwcTnKPavZgXIjsx1ynwP1hMg== X-Google-Smtp-Source: ABdhPJz6KlDsQy5A8f6VgC3uqf5VsQ54EDxPFTNE5iUKh8DjJ8U/91+syvnkllo6PVOdof4Zs7pmGQ== X-Received: by 2002:a05:620a:4e9:b0:6a7:8357:303d with SMTP id b9-20020a05620a04e900b006a78357303dmr467386qkh.105.1655135457443; Mon, 13 Jun 2022 08:50:57 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:2c45]) by smtp.gmail.com with ESMTPSA id j4-20020a05620a0a4400b006a684c87f4bsm6839593qka.68.2022.06.13.08.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jun 2022 08:50:57 -0700 (PDT) Date: Mon, 13 Jun 2022 11:50:56 -0400 From: Johannes Weiner To: Aneesh Kumar K V Cc: Jonathan Cameron , linux-mm@kvack.org, akpm@linux-foundation.org, Wei Xu , Huang Ying , Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes Subject: Re: [PATCH v5 1/9] mm/demotion: Add support for explicit memory tiers Message-ID: References: <02ee2c97-3bca-8eb6-97d8-1f8743619453@linux.ibm.com> <20220609152243.00000332@Huawei.com> <20220610105708.0000679b@Huawei.com> <4297bd21-e984-9d78-2bca-e70c11749a72@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4297bd21-e984-9d78-2bca-e70c11749a72@linux.ibm.com> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655135458; a=rsa-sha256; cv=none; b=ip6s/0Q+YcM4FMtBr5Eb6tPrvg97LkHOnYq9rmhaFUxT4Bp61EkOeOkfwUWKhBWG3h8RoC GLyFznjlxdE5KouqfN2X69kWwE09d/Kg2bJo23uUq5xFmUeab5GtOw16Vs4w6suhGoSgyQ uTUC3wh4e+7+ikygEHXDZLST2fBUAdo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655135458; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=H0T+Ju0FY0ctldM76CfR3ifPomNMCFqKjz28sSDHzOo=; b=y5a5ItnoSvEJS63Vzc3vHauwfFGnkBdwuULqu/eyeP4LV/BL+F+HCL7LOJgL1ki/M81FfA 96xgIFRZvxBjeTiAHXNEPKsaX0atGOE2NHYx4U9yPTbUqgK1aX7xOZjmmyugJ7kOgOKInQ mBgTaExO7r5EKo1KvpxOtZro0gBhZqI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="CRP/awNT"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.177 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b="CRP/awNT"; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf08.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.177 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5594A160089 X-Stat-Signature: m6rdmy75z79d6j5di9hsdtzxrtcnput3 X-HE-Tag: 1655135458-802901 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 13, 2022 at 07:53:03PM +0530, Aneesh Kumar K V wrote: > On 6/13/22 7:35 PM, Johannes Weiner wrote: > > On Fri, Jun 10, 2022 at 10:57:08AM +0100, Jonathan Cameron wrote: > > > I'm not sure completely read only is flexible enough (though mostly RO is fine) > > > as we keep sketching out cases where any attempt to do things automatically > > > does the wrong thing and where we need to add an extra tier to get > > > everything to work. Short of having a lot of tiers I'm not sure how > > > we could have the default work well. Maybe a lot of "tiers" is fine > > > though perhaps we need to rename them if going this way and then they > > > don't really work as current concept of tier. > > > > > > Imagine a system with subtle difference between different memories such > > > as 10% latency increase for same bandwidth. To get an advantage from > > > demoting to such a tier will require really stable usage and long > > > run times. Whilst you could design a demotion scheme that takes that > > > into account, I think we are a long way from that today. > > > > Good point: there can be a clear hardware difference, but it's a > > policy choice whether the MM should treat them as one or two tiers. > > > > What do you think of a per-driver/per-device (overridable) distance > > number, combined with a configurable distance cutoff for what > > constitutes separate tiers. E.g. cutoff=20 means two devices with > > distances of 10 and 20 respectively would be in the same tier, devices > > with 10 and 100 would be in separate ones. The kernel then generates > > and populates the tiers based on distances and grouping cutoff, and > > populates the memtier directory tree and nodemasks in sysfs. > > > > Right now core/generic code doesn't get involved in building tiers. It just > defines three tiers where drivers could place the respective devices they > manage. The above suggestion would imply we are moving quite a lot of policy > decision logic into the generic code?. No. The driver still chooses its own number, just from a wider range. The only policy in generic code is the distance cutoff for which devices are grouped into tiers together. > At some point, we will have to depend on more attributes other than > distance(may be HMAT?) and each driver should have the flexibility to place > the device it is managing in a specific tier? By then we may decide to > support more than 3 static tiers which the core kernel currently does. If we start with a larger possible range of "distance" values right away, we can still let the drivers ballpark into 3 tiers for now (100, 200, 300). But it will be easier to take additional metrics into account later and fine tune accordingly (120, 260, 90 etc.) without having to update all the other drivers as well. > If the kernel still can't make the right decision, userspace could rearrange > them in any order using rank values. Without something like rank, if > userspace needs to fix things up, it gets hard with device > hotplugging. ie, the userspace policy could be that any new PMEM tier device > that is hotplugged, park it with a very low-rank value and hence lowest in > demotion order by default. (echo 10 > > /sys/devices/system/memtier/memtier2/rank) . After that userspace could > selectively move the new devices to the correct memory tier? I had touched on this in the other email. This doesn't work if two drivers that should have separate policies collide into the same tier - which is very likely with just 3 tiers. So it seems to me the main usecase for having a rank tunable falls apart rather quickly until tiers are spaced out more widely. And it does so at the cost of an, IMO, tricky to understand interface. In the other email I had suggested the ability to override not just the per-device distance, but also the driver default for new devices to handle the hotplug situation. This should be less policy than before. Driver default and per-device distances (both overridable) combined with one tunable to set the range of distances that get grouped into tiers. With these parameters alone, you can generate an ordered list of tiers and their devices. The tier numbers make sense, and no rank is needed. Do you still need the ability to move nodes by writing nodemasks? I don't think so. Assuming you would never want to have an actually slower device in a higher tier than a faster device, the only time you'd want to move a device is when the device's distance value is wrong. So you override that (until you update to a fixed kernel).