From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F63CC2D0EA for ; Thu, 9 Apr 2020 09:25:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 59C402084D for ; Thu, 9 Apr 2020 09:25:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59C402084D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-vserver.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D268B8E000D; Thu, 9 Apr 2020 05:25:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD8308E0006; Thu, 9 Apr 2020 05:25:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BED288E000D; Thu, 9 Apr 2020 05:25:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0254.hostedemail.com [216.40.44.254]) by kanga.kvack.org (Postfix) with ESMTP id B05BA8E0006 for ; Thu, 9 Apr 2020 05:25:29 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 81D63801F384 for ; Thu, 9 Apr 2020 09:25:29 +0000 (UTC) X-FDA: 76687783578.21.song11_2f9d63facf713 X-HE-Tag: song11_2f9d63facf713 X-Filterd-Recvd-Size: 3567 Received: from smtprelay.restena.lu (smtprelay.restena.lu [158.64.1.62]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Thu, 9 Apr 2020 09:25:27 +0000 (UTC) Received: from hemera.lan.sysophe.eu (unknown [IPv6:2001:a18:1:12::4]) by smtprelay.restena.lu (Postfix) with ESMTPS id 5007D40CD4; Thu, 9 Apr 2020 11:25:22 +0200 (CEST) Date: Thu, 9 Apr 2020 11:25:05 +0200 From: Bruno =?UTF-8?B?UHLDqW1vbnQ=?= To: cgroups@vger.kernel.org, linux-mm@kvack.org Cc: Johannes Weiner , Michal Hocko , Vladimir Davydov Subject: Memory CG and 5.1 to 5.6 uprade slows backup Message-ID: <20200409112505.2e1fc150@hemera.lan.sysophe.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Upgrading from 5.1 kernel to 5.6 kernel on a production system using cgroups (v2) and having backup process in a memory.high=2G cgroup sees backup being highly throttled (there are about 1.5T to be backuped). Most memory usage in that cgroup is for file cache. Here are the memory details for the cgroup: memory.current:2147225600 memory.events:low 0 memory.events:high 423774 memory.events:max 31131 memory.events:oom 0 memory.events:oom_kill 0 memory.events.local:low 0 memory.events.local:high 423774 memory.events.local:max 31131 memory.events.local:oom 0 memory.events.local:oom_kill 0 memory.high:2147483648 memory.low:33554432 memory.max:2415919104 memory.min:0 memory.oom.group:0 memory.pressure:some avg10=90.42 avg60=72.59 avg300=78.30 total=298252577711 memory.pressure:full avg10=90.32 avg60=72.53 avg300=78.24 total=295658626500 memory.stat:anon 10887168 memory.stat:file 2062102528 memory.stat:kernel_stack 73728 memory.stat:slab 76148736 memory.stat:sock 360448 memory.stat:shmem 0 memory.stat:file_mapped 12029952 memory.stat:file_dirty 946176 memory.stat:file_writeback 405504 memory.stat:anon_thp 0 memory.stat:inactive_anon 0 memory.stat:active_anon 10121216 memory.stat:inactive_file 1954959360 memory.stat:active_file 106418176 memory.stat:unevictable 0 memory.stat:slab_reclaimable 75247616 memory.stat:slab_unreclaimable 901120 memory.stat:pgfault 8651676 memory.stat:pgmajfault 2013 memory.stat:workingset_refault 8670651 memory.stat:workingset_activate 409200 memory.stat:workingset_nodereclaim 62040 memory.stat:pgrefill 1513537 memory.stat:pgscan 47519855 memory.stat:pgsteal 44933838 memory.stat:pgactivate 7986 memory.stat:pgdeactivate 1480623 memory.stat:pglazyfree 0 memory.stat:pglazyfreed 0 memory.stat:thp_fault_alloc 0 memory.stat:thp_collapse_alloc 0 Numbers that change most are pgscan/pgsteal Regularly the backup process seems to be blocked for about 2s, but not within a syscall according to strace. Is there a way to tell kernel that this cgroup should not be throttled and its inactive file cache given up (rather quickly). The aim here is to avoid backup from killing production task file cache but not starving it. If there is some useful info missing, please tell (eventually adding how I can obtain it). On a side note, I liked v1's mode of soft/hard memory limit where the memory amount between soft and hard could be used if system has enough free memory. For v2 the difference between high and max seems almost of no use. A cgroup parameter for impacting RO file cache differently than anonymous memory or otherwise dirty memory would be great too. Thanks, Bruno