From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A974C43460 for ; Wed, 5 May 2021 22:30:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF448611AD for ; Wed, 5 May 2021 22:30:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF448611AD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 511676B006C; Wed, 5 May 2021 18:30:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49A456B006E; Wed, 5 May 2021 18:30:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ED666B0070; Wed, 5 May 2021 18:30:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 0D2AC6B006C for ; Wed, 5 May 2021 18:30:48 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C7F508249980 for ; Wed, 5 May 2021 22:30:47 +0000 (UTC) X-FDA: 78108623334.36.C81143E Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf06.hostedemail.com (Postfix) with ESMTP id C8492C0007C3 for ; Wed, 5 May 2021 22:30:48 +0000 (UTC) Received: by mail-qk1-f175.google.com with SMTP id a22so2658787qkl.10 for ; Wed, 05 May 2021 15:30:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=SPq92WwUkkbUZi8i20jXBtq5YFPc01M81BIgdGcCXik=; b=1XsoNbpEqhTeEDcH7JxyK1q4zEs5Rwp9aoUuhagED0iCyc41POI1c/VVIoc67o9OjE a3Jb3Siz5tq48y7cWLCSTOm33rfApdoNOrKkwEdQt0YnzYm9fi3qMnjHZ3BYKCT02QiX 0VphxMM7zC+tWpB5PSY6bUR2xGMZqvPoif6p5qHIecAcllv23Hk710cDnh+RoL9WJ6eP t+IXCXBHlMZdWEcUZsCJCGvilm+OZYcCTYYD44m6NbH0DXuOHl+1cMO9wrTg+OW7f3cW hIjacaScsm+ZUwPQ70PCCQD2ycyzWsb2o1ZH9gcOP0dFIhTmI7U4y3yVboLv2Dnp+c23 16ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=SPq92WwUkkbUZi8i20jXBtq5YFPc01M81BIgdGcCXik=; b=k6+8Vil0/hoiOlv9FUpHXcynKxR3mD/GMSFCAHVM4PmzpQeW+B9CqbnI1LK+e7Cnau pU1xbVNIcXoMUobUHFzK31Pvi1eotX5XKFG4hu00YuPkDr4fMKQv/LFi+Ipdz2m0jlE9 FyxH6J2j+l3BvNVfLqRpVdL964MPebY/hljao6ksY5k50NliwfQFswDm7zHUwfz7Teh1 QiLzyRYwlfvG68EMUhISuJUAx5R6pOtITE6E61DMTRAzDFUgNPM1YhD/gDb4leOmapFj opUE8oWS/p8UQ7H8+FjaChzVsptJyqixWEa0FtnEqWGJViWOhwy7IA+Y5xecPxcL7rNb auAw== X-Gm-Message-State: AOAM531tfSuRO+ltZ9TSlJhviRGejRgIKVzvPZKbkpA3+NUSouP05JfT JXXscjx4KGynv5klOvRj02kPcQ== X-Google-Smtp-Source: ABdhPJw0WA4KlmeJNSng9tMnOiLQgmBjymemgeDCJXg5r1D/Nvlzw9wZF5IVlF2lDIYsU+9hXHXTlA== X-Received: by 2002:a05:620a:4043:: with SMTP id i3mr1024866qko.380.1620253846537; Wed, 05 May 2021 15:30:46 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:ab71]) by smtp.gmail.com with ESMTPSA id w7sm515484qtn.91.2021.05.05.15.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 May 2021 15:30:45 -0700 (PDT) Date: Wed, 5 May 2021 18:30:44 -0400 From: Johannes Weiner To: Tejun Heo Cc: Abel Wu , akpm@linux-foundation.org, lizefan.x@bytedance.com, corbet@lwn.net, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 2/3] cgroup/cpuset: introduce cpuset.mems.migration Message-ID: References: <20210426065946.40491-1-wuyun.abel@bytedance.com> <20210426065946.40491-3-wuyun.abel@bytedance.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: C8492C0007C3 X-Stat-Signature: b91hp9bchzcrn97n1pd9cemwfpfmnts1 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=cmpxchg-org.20150623.gappssmtp.com header.s=20150623 header.b=1XsoNbpE; spf=pass (imf06.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.175 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org Received-SPF: none (cmpxchg.org>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=mail-qk1-f175.google.com; client-ip=209.85.222.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620253848-199150 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Apr 27, 2021 at 10:43:31AM -0400, Tejun Heo wrote: > Hello, >=20 > On Mon, Apr 26, 2021 at 02:59:45PM +0800, Abel Wu wrote: > > When a NUMA node is assigned to numa-service, the workload > > on that node needs to be moved away fast and complete. The > > main aspects we cared about on the eviction are as follows: > >=20 > > a) it should complete soon enough so that numa-services > > won=E2=80=99t wait too long to hurt user experience > > b) the workloads to be evicted could have massive usage on > > memory, and migrating such amount of memory may lead to > > a sudden severe performance drop lasting tens of seconds > > that some certain workloads may not afford > > c) the impact of the eviction should be limited within the > > source and destination nodes > > d) cgroup interface is preferred > >=20 > > So we come to a thought that: > >=20 > > 1) fire up numa-services without waiting for memory migration > > 2) memory migration can be done asynchronously by using spare > > memory bandwidth > >=20 > > AutoNUMA seems to be a solution, but its scope is global which > > violates c&d. And cpuset.memory_migrate performs in a synchronous >=20 > I don't think d) in itself is a valid requirement. How does it violate = c)? >=20 > > fashion which breaks a&b. So a mixture of them, the new cgroup2 > > interface cpuset.mems.migration, is introduced. > >=20 > > The new cpuset.mems.migration supports three modes: > >=20 > > - "none" mode, meaning migration disabled > > - "sync" mode, which is exactly the same as the cgroup v1 > > interface cpuset.memory_migrate > > - "lazy" mode, when walking through all the pages, unlike > > cpuset.memory_migrate, it only sets pages to protnone, > > and numa faults triggered by later touch will handle the > > movement. >=20 > cpuset is already involved in NUMA allocation but it always felt like > something bolted on - it's weird to have cpu to NUMA node settings at g= lobal > level and then to have possibly conflicting direct NUMA configuration v= ia > cpuset. My preference would be putting as much configuration as possibl= e on > the mm / autonuma side and let cpuset's node confinements further restr= ict > their operations rather than cpuset having its own set of policy > configurations. >=20 > Johannes, what are your thoughts? This is basically a cgroup interface for the existing MPOL_MF_LAZY / MPOL_F_MOF flags, which are per task (set_mempolicy()) and per-vma (mbind()) scope respectively. They're not per-node, so cannot be cgroupified through cpuset's node restrictions alone, and I understand why a cgroup interface could be convenient. On the other hand, this is not really about configuring a shared resource. Rather it's using cgroups to set an arbitrary task parameter on a bunch of tasks simultaneously. It's the SIMD type usecase of cgroup1 that we tried to get away from in cgroup2, simply because it's so unbounded in scope. There are *a lot* of possible task parameters, and we could add a lot of kernel interfaces that boil down to css_task_iter and setting or clearing a task flag. So I'm also thinking this cgroup interface isn't desirable. If you want to control numa policies of tasks from the outside, it's probably best to extend the numa syscall interface to work on pids. And then use cgroup.procs to cgroupify the operation from userspace. Or extend the NUMA interface to make the system-wide default behavior configurable, so that you can set MPOL_F_MOF in there (without having to enable autonuma). But yeah, cgroups doesn't seem like the right place to do this. Thanks