From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05E8EC43460 for ; Wed, 28 Apr 2021 07:24:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6DAE4613EA for ; Wed, 28 Apr 2021 07:24:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6DAE4613EA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 399586B0036; Wed, 28 Apr 2021 03:24:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3480A6B006E; Wed, 28 Apr 2021 03:24:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 210E36B0070; Wed, 28 Apr 2021 03:24:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 042546B0036 for ; Wed, 28 Apr 2021 03:24:50 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B1223440E for ; Wed, 28 Apr 2021 07:24:50 +0000 (UTC) X-FDA: 78080938740.29.ECBD84B Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf18.hostedemail.com (Postfix) with ESMTP id 7B8682000241 for ; Wed, 28 Apr 2021 07:24:51 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id l10-20020a17090a850ab0290155b06f6267so2418726pjn.5 for ; Wed, 28 Apr 2021 00:24:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=ocOiwhnINfr+vKnvSHIfg9Nbg66LG1Qzz5IHcV/tGpI=; b=eVN+NK23/ESRrBYM2lxx0JJdpsKDZ22tgKcqU4ZQ5DFwQMOPQ935OCwLL4eRBcpt6b HxY//w+DZIIaMWpUPH0oGLw+/OhMrGPu2xNL9RubfJ+do06n4Pr9hmubQyF6wksegxHI SzBNqNn/9rVxc0B+YiHhTFWCvQRyGNvaPm3/R6+1Lx6QOsEQzQyUP9Ix1O5XFrv8+59a DV+QdW1aWQ/po8Y377Yl3+PbWy8EY3IQEC6z7wh5BB55uwIDQW/lbzaZGc3kjFBRLTJE SZoq9XsQy7+XmDFdzb/xmHW04Iu8A410Ai1Fq4A4ST8qN2msmz6C+2phGOV76KgvnIaR nBZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=ocOiwhnINfr+vKnvSHIfg9Nbg66LG1Qzz5IHcV/tGpI=; b=AyD46sVbcZcv3uI88Vdg/vAi9OLhmkFCmuuAL0B5969hQLjgiSRZ5kRK6isoHbUo3B rUZG7vqxPexW9SNK52tsBhumKOTxVh6rSt+zRDBMjrCHoz6axfqji3UAXpizyxjIG+oY 6fZbizCTE5IeOogx83cBYNyGhTxTt3230R9epVy5lFdzB7EF0dV805katVlLddE6YdnX IkktkhFAoSsqf48K7kq0g0Q/ReO/pYLmAr4rw5OaJsSkrWYl1N9G06laD08Z9gx/OoQz dscvuNk4HtYDVezvw7BD+/KIpejhvj09eOmPUuD+MUYDL80a5beIJ0c4GxkrZUXQqwL9 2A3w== X-Gm-Message-State: AOAM531fb1Egx5nrvL9CvPgBiYNB/klJDDbFgRii4Mq8nlvIO2D5WXL9 pnVq8OebTwK5mUSBaCKsoenpUg== X-Google-Smtp-Source: ABdhPJxcIEGDPedhqMOT2+LOr0kK7lINj+wlYhZkeKnm1k35EsmrDc1yUxcpXF/9LFTdEwFCz5quoQ== X-Received: by 2002:a17:902:e892:b029:ec:d257:b8a2 with SMTP id w18-20020a170902e892b02900ecd257b8a2mr29159002plg.15.1619594688274; Wed, 28 Apr 2021 00:24:48 -0700 (PDT) Received: from [10.94.59.238] ([139.177.225.240]) by smtp.gmail.com with ESMTPSA id f10sm4152090pju.27.2021.04.28.00.24.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 28 Apr 2021 00:24:47 -0700 (PDT) Subject: Re: [PATCH 2/3] cgroup/cpuset: introduce cpuset.mems.migration To: Tejun Heo Cc: akpm@linux-foundation.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, cgroups@vger.kernel.org, linux-mm@kvack.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org References: <20210426065946.40491-1-wuyun.abel@bytedance.com> <20210426065946.40491-3-wuyun.abel@bytedance.com> From: Abel Wu Message-ID: <67632345-9c30-9e87-f9b2-ba4e18b5ae91@bytedance.com> Date: Wed, 28 Apr 2021 15:24:42 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Rspamd-Queue-Id: 7B8682000241 X-Stat-Signature: rjd7sfsqe3m5dheykhomhc73txbxjemn X-Rspamd-Server: rspam02 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf18; identity=mailfrom; envelope-from=""; helo=mail-pj1-f51.google.com; client-ip=209.85.216.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619594691-423146 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello Tejun, thanks for your review, On 4/27/21 10:43 PM, Tejun Heo wrote: > Hello, >=20 > On Mon, Apr 26, 2021 at 02:59:45PM +0800, Abel Wu wrote: >> When a NUMA node is assigned to numa-service, the workload >> on that node needs to be moved away fast and complete. The >> main aspects we cared about on the eviction are as follows: >> >> a) it should complete soon enough so that numa-services >> won=E2=80=99t wait too long to hurt user experience >> b) the workloads to be evicted could have massive usage on >> memory, and migrating such amount of memory may lead to >> a sudden severe performance drop lasting tens of seconds >> that some certain workloads may not afford >> c) the impact of the eviction should be limited within the >> source and destination nodes >> d) cgroup interface is preferred >> >> So we come to a thought that: >> >> 1) fire up numa-services without waiting for memory migration >> 2) memory migration can be done asynchronously by using spare >> memory bandwidth >> >> AutoNUMA seems to be a solution, but its scope is global which >> violates c&d. And cpuset.memory_migrate performs in a synchronous >=20 > I don't think d) in itself is a valid requirement. How does it violate = c)? Yes, d) is more like a preference, since we operate in cgroup level. Process/thread level interfaces are also acceptable. AutoNUMA violates c) in its global effect that not only the source and destination nodes, the processes running on other nodes would also suffer from unwanted overhead due to numa faults. And besides the global effect, one-shot mode migration is expected in this scenario, like cpuset.memory_migrate, rather than autonuma's periodic behavior. >=20 >> fashion which breaks a&b. So a mixture of them, the new cgroup2 >> interface cpuset.mems.migration, is introduced. >> >> The new cpuset.mems.migration supports three modes: >> >> - "none" mode, meaning migration disabled >> - "sync" mode, which is exactly the same as the cgroup v1 >> interface cpuset.memory_migrate >> - "lazy" mode, when walking through all the pages, unlike >> cpuset.memory_migrate, it only sets pages to protnone, >> and numa faults triggered by later touch will handle the >> movement. >=20 > cpuset is already involved in NUMA allocation but it always felt like > something bolted on - it's weird to have cpu to NUMA node settings at g= lobal > level and then to have possibly conflicting direct NUMA configuration v= ia > cpuset. My preference would be putting as much configuration as possibl= e on > the mm / autonuma side and let cpuset's node confinements further restr= ict > their operations rather than cpuset having its own set of policy > configurations. Such conflicting configuration exists in our system in order to reduce TCO, and yet we haven't found a proper way to get rid of it. Say a numa-service occupies the whole memory of a node but still leave some cpus free. The free cpus may be assigned to some services, the ones that are not memory latency sensitive and are forbidden to use local memory. Thoughts? Thanks, Abel >=20 > Johannes, what are your thoughts? >=20 > Thanks. >=20