From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECCA7C2BA2B for ; Thu, 9 Apr 2020 15:40:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4BCA20784 for ; Thu, 9 Apr 2020 15:40:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4BCA20784 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-vserver.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 618838E000D; Thu, 9 Apr 2020 11:40:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C9A08E0006; Thu, 9 Apr 2020 11:40:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DFBE8E000D; Thu, 9 Apr 2020 11:40:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 416F28E0006 for ; Thu, 9 Apr 2020 11:40:45 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 160F2989E for ; Thu, 9 Apr 2020 15:40:45 +0000 (UTC) X-FDA: 76688729250.19.cough45_7adc90f775210 X-HE-Tag: cough45_7adc90f775210 X-Filterd-Recvd-Size: 2878 Received: from smtprelay.restena.lu (smtprelay.restena.lu [158.64.1.62]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Thu, 9 Apr 2020 15:40:44 +0000 (UTC) Received: from hemera.lan.sysophe.eu (unknown [IPv6:2001:a18:1:12::4]) by smtprelay.restena.lu (Postfix) with ESMTPS id AE53F40FB0; Thu, 9 Apr 2020 17:40:42 +0200 (CEST) Date: Thu, 9 Apr 2020 17:40:42 +0200 From: Bruno =?UTF-8?B?UHLDqW1vbnQ=?= To: Chris Down Cc: Michal Hocko , cgroups@vger.kernel.org, linux-mm@kvack.org, Johannes Weiner , Vladimir Davydov Subject: Re: Memory CG and 5.1 to 5.6 uprade slows backup Message-ID: <20200409174042.2a3389ba@hemera.lan.sysophe.eu> In-Reply-To: <20200409152417.GB1040020@chrisdown.name> References: <20200409112505.2e1fc150@hemera.lan.sysophe.eu> <20200409094615.GE18386@dhcp22.suse.cz> <20200409121733.1a5ba17c@hemera.lan.sysophe.eu> <20200409103400.GF18386@dhcp22.suse.cz> <20200409170926.182354c3@hemera.lan.sysophe.eu> <20200409152417.GB1040020@chrisdown.name> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 9 Apr 2020 16:24:17 +0100 wrote: > Bruno Pr=C3=A9mont writes: > >Could it be that cache is being prevented from being reclaimed by a task > >in another cgroup? > > > >e.g. > > cgroup/system/backup > > first reads $files (reads each once) > > cgroup/workload/bla > > second&more reads $files > > > >Would $files remain associated to cgroup/system/backup and not > >reclaimed there instead of being reassigned to cgroup/workload/bla? =20 >=20 > Yes, that's entirely possible. The first cgroup to fault in the pages is= =20 > charged for the memory. Other cgroups may use them, but they are not acco= unted=20 > for as part of that other cgroup. They may also still be "active" as a re= sult=20 > of use by another cgroup. But the memory would then be 'active' in the original cgroup? which is not the case here I feel. If the remain inactive-unreclaimable in the first cgroup due to use in another cgroup that would be at least surprising. Doubling the high value helped (but for how long?), back with memory.current around memory.high nut no throttling yet. But from increase until now memory.pressure is small/zero. Capturing=20 memory.stat:pgscan 47519855 memory.stat:pgsteal 44933838 over time for Michal and will report back later this evening. When seen stuck backup was reading a multi-GiB file with open(, O_NOATIME) while (read()) { transform and write to network } close() thus plain sequential file read through file cache (and for this backup run, only files not in use by anyone else, or some being just appended to by others). Bruno