From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F9FCC433E1 for ; Tue, 14 Jul 2020 15:32:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4869E221EF for ; Tue, 14 Jul 2020 15:32:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TSj2JWpT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4869E221EF Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B85096B0006; Tue, 14 Jul 2020 11:32:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B36808D0001; Tue, 14 Jul 2020 11:32:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A233A6B000C; Tue, 14 Jul 2020 11:32:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0219.hostedemail.com [216.40.44.219]) by kanga.kvack.org (Postfix) with ESMTP id 8A32A6B0006 for ; Tue, 14 Jul 2020 11:32:26 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2B40D1E04 for ; Tue, 14 Jul 2020 15:32:26 +0000 (UTC) X-FDA: 77037073092.07.frogs10_3c1468626ef2 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id B803A1803F9CC for ; Tue, 14 Jul 2020 15:32:23 +0000 (UTC) X-HE-Tag: frogs10_3c1468626ef2 X-Filterd-Recvd-Size: 7020 Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Tue, 14 Jul 2020 15:32:23 +0000 (UTC) Received: by mail-lf1-f66.google.com with SMTP id y13so12000045lfe.9 for ; Tue, 14 Jul 2020 08:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oqbV9Nb2FFz6+EoRL68dxT5M8+4XjqDj51J15tcXDqg=; b=TSj2JWpT7atvmen6t6ki58YuhCN+eCwS79p1FUpE9N5QRc+YVPKIylEYF0XkIn4ojQ n9suCH/mse8LpARQjWFyAmnQQG0mvE33JDrl17LlBdjqrYyEL9XR6TR6HiAFx+zpPWIr b5XktYTaXhyiCxoBVFk5GQL7BaHSXNlwa1350f0SSaFmwxjcwNbrhA45tedwUJmdGTOM R2aqL+w9fA84Yr2PcgKKOSYcl0XcukNRMlHnUtQ0XFG4iff9z3cKFx2hUNLQa24maz5n zDrxLrKus91Kvqk2I+ooprMV7o9sVg5RMk4V30ubIx6uafPYDXZBqYldm7a2fS0PVExk ABPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oqbV9Nb2FFz6+EoRL68dxT5M8+4XjqDj51J15tcXDqg=; b=R4Con5Pcu8wXFJ91PkRa6qFSkVnKHd9CfYM8x73gAPAQWSOGbGi4Y6O8h+6Ul+SaJw qGNDj0WXcaf0ABNqhrZzgD72H4amfVPjAKP6uoLYGjo0sTZy1E6LWeAdb+pzpl8PEjh0 xEeoo9DzhC2nIa5RnfYYZyPbVNKRQ/AvUGq9FTfsz5K1Vw5pERztI3f4mvmGgj+w8xVv +Vxp5bkeewt2yLDiG/6dJjpp7MhE+4+45tyALYcis8arsRLC962L8zY1uww5Po2VZYpW aUTeIaQSFmvH2g/nxX/6jvea6/LVszQUvd8++XHAV2JCfA7LSlotkGc5nrTRuKHEPZVo EefQ== X-Gm-Message-State: AOAM531JhzujCNxK7IbvStohrc3tdyxmJEkbqiqX3koRu3dfr3PEvbhp HHLDiWYebnzBlFKZqZ4PUGji7bHwqnydrigudHm0Ng== X-Google-Smtp-Source: ABdhPJyshvzFFUz5zSXNdPHetv2lNNpNn7NtKemJ5lODpGTU4sd+e4L4n9wEfQ3JXg5wqbW0yojfA047VaG29mbX/Bg= X-Received: by 2002:a05:6512:3f1:: with SMTP id n17mr2470898lfq.125.1594740741307; Tue, 14 Jul 2020 08:32:21 -0700 (PDT) MIME-Version: 1.0 References: <20200709194718.189231-1-guro@fb.com> <20200710122917.GB3022@dhcp22.suse.cz> <20200710184205.GB350256@carbon.dhcp.thefacebook.com> <20200714084123.GG24642@dhcp22.suse.cz> In-Reply-To: <20200714084123.GG24642@dhcp22.suse.cz> From: Shakeel Butt Date: Tue, 14 Jul 2020 08:32:09 -0700 Message-ID: Subject: Re: [PATCH] mm: memcontrol: avoid workload stalls when lowering memory.high To: Michal Hocko Cc: Roman Gushchin , Andrew Morton , Johannes Weiner , Linux MM , Kernel Team , LKML , Domas Mituzas , Tejun Heo , Chris Down Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: B803A1803F9CC X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Jul 14, 2020 at 1:41 AM Michal Hocko wrote: > > On Fri 10-07-20 12:19:37, Shakeel Butt wrote: > > On Fri, Jul 10, 2020 at 11:42 AM Roman Gushchin wrote: > > > > > > On Fri, Jul 10, 2020 at 07:12:22AM -0700, Shakeel Butt wrote: > > > > On Fri, Jul 10, 2020 at 5:29 AM Michal Hocko wrote: > > > > > > > > > > On Thu 09-07-20 12:47:18, Roman Gushchin wrote: > > > > > > Memory.high limit is implemented in a way such that the kernel > > > > > > penalizes all threads which are allocating a memory over the limit. > > > > > > Forcing all threads into the synchronous reclaim and adding some > > > > > > artificial delays allows to slow down the memory consumption and > > > > > > potentially give some time for userspace oom handlers/resource control > > > > > > agents to react. > > > > > > > > > > > > It works nicely if the memory usage is hitting the limit from below, > > > > > > however it works sub-optimal if a user adjusts memory.high to a value > > > > > > way below the current memory usage. It basically forces all workload > > > > > > threads (doing any memory allocations) into the synchronous reclaim > > > > > > and sleep. This makes the workload completely unresponsive for > > > > > > a long period of time and can also lead to a system-wide contention on > > > > > > lru locks. It can happen even if the workload is not actually tight on > > > > > > memory and has, for example, a ton of cold pagecache. > > > > > > > > > > > > In the current implementation writing to memory.high causes an atomic > > > > > > update of page counter's high value followed by an attempt to reclaim > > > > > > enough memory to fit into the new limit. To fix the problem described > > > > > > above, all we need is to change the order of execution: try to push > > > > > > the memory usage under the limit first, and only then set the new > > > > > > high limit. > > > > > > > > > > Shakeel would this help with your pro-active reclaim usecase? It would > > > > > require to reset the high limit right after the reclaim returns which is > > > > > quite ugly but it would at least not require a completely new interface. > > > > > You would simply do > > > > > high = current - to_reclaim > > > > > echo $high > memory.high > > > > > echo infinity > memory.high # To prevent direct reclaim > > > > > # allocation stalls > > > > > > > > > > > > > This will reduce the chance of stalls but the interface is still > > > > non-delegatable i.e. applications can not change their own memory.high > > > > for the use-cases like application controlled proactive reclaim and > > > > uswapd. > > > > > > Can you, please, elaborate a bit more on this? I didn't understand > > > why. > > > > > > > Sure. Do we want memory.high a CFTYPE_NS_DELEGATABLE type file? I > > don't think so otherwise any job on a system can change their > > memory.high and can adversely impact the isolation and memory > > scheduling of the system. > > Is this really the case? There should always be a parent cgroup that > overrides the setting. Can you explain a bit more? I don't see any requirement of having a layer of cgroup between root and the job cgroup. Internally we schedule jobs as top level cgroups. There do exist jobs which are a combination of other jobs and there we do use an additional layer of cgroup (similar to pods running multiple containers in kubernetes). Surely we can add a layer for all the jobs but it comes with an overhead and at scale that overhead is not negligible. > Also you can always set the hard limit if you do > not want to add another layer of cgroup in the hierarchy before > delegation. Or am I missing something? > Yes, we can set memory.max though it has different oom semantics and not really a replacement for memory.high.