From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21B22C433EF for ; Tue, 12 Jul 2022 15:08:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6AAB16B0157; Tue, 12 Jul 2022 11:08:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6353F6B0158; Tue, 12 Jul 2022 11:08:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D3F7940063; Tue, 12 Jul 2022 11:08:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3EB676B0157 for ; Tue, 12 Jul 2022 11:08:11 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 14EED120B3E for ; Tue, 12 Jul 2022 15:08:11 +0000 (UTC) X-FDA: 79678778382.26.8BEDCA3 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf20.hostedemail.com (Postfix) with ESMTP id 8755C1C00B1 for ; Tue, 12 Jul 2022 15:08:09 +0000 (UTC) Received: by mail-pf1-f169.google.com with SMTP id w185so7718675pfb.4 for ; Tue, 12 Jul 2022 08:08:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=C6szfa/YFuVli7Zl8D3g07FtZCaBqKl0sr0C3s801fw=; b=wvGh1LaGBDR/G4joma/N0cvbJnBEIUQnqZL1xqzrAx2II/seQanvVrR3VC/+z7Zjld zVpL3S1Sbap6Qr3EziCQwuz57jjlFSyqAIG//PrF57ckKsOjYnNA74IslrVRV8nnnvRN 3wVRUsNjq8E8LfWdZImw+6t4uy2qdxEczsZkafnLIRX6/tVcYH4YdvtrreW/yTwotczk 0N6jnX5GZaEEOxdbD89BmI2kjEa6YY8NeG+QQlMPkGzOPlGhGFtT53OgEj7W2CYR3byM LQOFNKZQ1nRsJMbeAYgWlG0zgQMstkobCqEuCJ2MiVopQxFfrxJf8bduX6B6jN534ryq abmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=C6szfa/YFuVli7Zl8D3g07FtZCaBqKl0sr0C3s801fw=; b=5dXtViDi/CUW+jUdjeTOEKaizZ7x/pXQ7waoVS2vu1SCN+gk7IPOsLHPW0U2dNAYM8 0aR9A0EgxUw2gvulUP63xUw0wgMlTLZN6uwcH5HgTfF7CjEelj2XqY/+3drJCjbkeu1y yuNUpa1niKaAADm3A0rjJMHv4mXUysUP6Fxorlo3r5wNdvURQ1MLcQ4z5RY1L0seTGN8 A7u/KC2bzxh8tlKN7Ktaocg6hN/myfezxoXhjtsWF9j1m5LNpycX7xc6V0e8UkjZyPdv NMfILwPXlxxQbIZBwnDx6ggz2FN+4KQ+kScp5MP/0hcVLPHsGaZB2HAbvK2PAivIIz0+ 8Aqw== X-Gm-Message-State: AJIora9b2qhvBzgns754AeH8RQDfyC29b/ngN8qPNMaMrtT2BXJASMkL Qck7+sHwx0uzptaZDCaBUUMvNUYJ1n3iKXAz X-Google-Smtp-Source: AGRyM1uF2yZ0fhTWiRIzaCuJhrMoYd2YKnjhN3avJPfSwtQ8ASEEXv5DvqmJeYAEZYa99COktBhysA== X-Received: by 2002:a63:9752:0:b0:3c6:5a7a:5bd6 with SMTP id d18-20020a639752000000b003c65a7a5bd6mr21070507pgo.390.1657638073175; Tue, 12 Jul 2022 08:01:13 -0700 (PDT) Received: from [10.4.113.6] ([139.177.225.234]) by smtp.gmail.com with ESMTPSA id l1-20020a170902f68100b0016bfbd99f64sm6892899plg.118.2022.07.12.08.00.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 12 Jul 2022 08:01:12 -0700 (PDT) Message-ID: <6f6a2257-3b60-e312-3ee3-fb08b972dbf2@bytedance.com> Date: Tue, 12 Jul 2022 23:00:55 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Subject: Re: [PATCH v2 0/5] mm, oom: Introduce per numa node oom for CONSTRAINT_{MEMORY_POLICY,CPUSET} Content-Language: en-US To: Michal Hocko Cc: Gang Li , akpm@linux-foundation.org, surenb@google.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, viro@zeniv.linux.org.uk, ebiederm@xmission.com, keescook@chromium.org, rostedt@goodmis.org, mingo@redhat.com, peterz@infradead.org, acme@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, david@redhat.com, imbrenda@linux.ibm.com, adobriyan@gmail.com, yang.yang29@zte.com.cn, brauner@kernel.org, stephen.s.brennan@oracle.com, zhengqi.arch@bytedance.com, haolee.swjtu@gmail.com, xu.xin16@zte.com.cn, Liam.Howlett@oracle.com, ohoono.kwon@samsung.com, peterx@redhat.com, arnd@arndb.de, shy828301@gmail.com, alex.sierra@amd.com, xianting.tian@linux.alibaba.com, willy@infradead.org, ccross@google.com, vbabka@suse.cz, sujiaxun@uniontech.com, sfr@canb.auug.org.au, vasily.averin@linux.dev, mgorman@suse.de, vvghjk1234@gmail.com, tglx@linutronix.de, luto@kernel.org, bigeasy@linutronix.de, fenghua.yu@intel.com, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-perf-users@vger.kernel.org, hezhongkun.hzk@bytedance.com References: <20220708082129.80115-1-ligang.bdlg@bytedance.com> <41ae31a7-6998-be88-858c-744e31a76b2a@bytedance.com> From: Abel Wu In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657638489; a=rsa-sha256; cv=none; b=oCHNUO/z4OJCuQj+uYuEKgrbmr8VJNt/7PmimdPXWBT6a/DksaHbXn27XCz7H3cxWIoAnR ZkI8j9mh3RvY0sDLcLArk1rx3eNkD/7j9xXE1YPY0gX4rPl8irec+0CszVFQQgs4tyDIcY RLUPn/d+CLBSA5JNtUooojKtF+PqGCA= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=wvGh1LaG; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of wuyun.abel@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=wuyun.abel@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657638489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=C6szfa/YFuVli7Zl8D3g07FtZCaBqKl0sr0C3s801fw=; b=K9EbIf6V90Mp8HcZD3NLZVJilcTx6aSpDAvy1x/AYQHE9+vyenX/B4aNWVaEKpsvsoQcYV 8VJS3skkgOkXpw4wwePFG52u6c+9lQFmtNRXAecpz8YvcQRIFpb111DGpB1DclN/wmw+aE zfnIwS9abx8rmqKbZMDIvGWKaUiMnKc= X-Rspam-User: X-Stat-Signature: gm1hoqc96o6xiox6cyzwny65dafkmw4p X-Rspamd-Queue-Id: 8755C1C00B1 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=wvGh1LaG; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of wuyun.abel@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=wuyun.abel@bytedance.com X-Rspamd-Server: rspam03 X-HE-Tag: 1657638489-66490 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 7/12/22 9:35 PM, Michal Hocko Wrote: > On Tue 12-07-22 19:12:18, Abel Wu wrote: > [...] >> I was just going through the mail list and happen to see this. There >> is another usecase for us about per-numa memory usage. >> >> Say we have several important latency-critical services sitting inside >> different NUMA nodes without intersection. The need for memory of these >> LC services varies, so the free memory of each node is also different. >> Then we launch several background containers without cpuset constrains >> to eat the left resources. Now the problem is that there doesn't seem >> like a proper memory policy available to balance the usage between the >> nodes, which could lead to memory-heavy LC services suffer from high >> memory pressure and fails to meet the SLOs. > > I do agree that cpusets would be rather clumsy if usable at all in a > scenario when you are trying to mix NUMA bound workloads with those > that do not have any NUMA proferences. Could you be more specific about > requirements here though? Yes, these LC services are highly sensitive to memory access latency and bandwidth, so they are provisioned by NUMA node granule to meet their performance requirements. While on the other hand, they usually do not make full use of cpu/mem resources which increases the TCO of our IDCs, so we have to co-locate them with background tasks. Some of these LC services are memory-bound but leave much of cpu's capacity unused. In this case we hope the co-located background tasks to consume some leftover without introducing obvious mm overhead to the LC services. > > Let's say you run those latency critical services with "simple" memory > policies and mix them with the other workload without any policies in > place so they compete over memory. It is not really clear to me how can > you achieve any reasonable QoS in such an environment. Your latency > critical servises will be more constrained than the non-critical ones > yet they are more demanding AFAIU. Yes, the QoS over memory is the biggest block in the way (the other resources are relatively easier). For now, we hacked a new mpol to achieve weighted-interleave behavior to balance the memory usage across NUMA nodes, and only set memcg protections to the LC services. If the memory pressure is still high, the background tasks will be killed. Ideas? Thanks! Abel