From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.4 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84C6EC2BB1D for ; Tue, 17 Mar 2020 18:26:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4205F20714 for ; Tue, 17 Mar 2020 18:26:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ia3p2hdK" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4205F20714 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C79BE6B0005; Tue, 17 Mar 2020 14:26:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C50716B0006; Tue, 17 Mar 2020 14:26:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B66206B0007; Tue, 17 Mar 2020 14:26:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 9C61E6B0005 for ; Tue, 17 Mar 2020 14:26:05 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 78173180AD81D for ; Tue, 17 Mar 2020 18:26:05 +0000 (UTC) X-FDA: 76605683490.11.tent82_523b4cabaf455 X-HE-Tag: tent82_523b4cabaf455 X-Filterd-Recvd-Size: 5411 Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Tue, 17 Mar 2020 18:26:04 +0000 (UTC) Received: by mail-io1-f41.google.com with SMTP id q9so2183999iod.4 for ; Tue, 17 Mar 2020 11:26:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Fyw/0w4c822Iz8zK/btqki5MNWC2a7i6nqG8rTs3u5M=; b=ia3p2hdK/bH2uAcKuAtM7Vy5dqhj2+AYw5LfWCnQvi47yfSN8Qq3N/LJzxzq8efZxv oRY/Z4WABXRwFzLP6QndyyhvjUrECiBJ+th+ofyqm4dmeY1BgAKvEzPaomlU/rpTYXOA PIVq2rBnbG2V+SM3t9G6g+8gzOR6bVGb5NN/A/xgCc7pobdG/3YCo8Uz2Dp+ApzQk6bU Fh5Cg31ZeMUosMb2tbzeEYZNDWEeLyK/9cdFHSRGavB6zEuo+dhz3Ung5ZBCOe6UuqfC FPwtBcZdKoUG/BPDU//j5wUSVKVUS9nv0X+LjlILaMAcV9IjGeFxFe0A93ciOBGGL5FR KCeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Fyw/0w4c822Iz8zK/btqki5MNWC2a7i6nqG8rTs3u5M=; b=jG5M1wfu5H/LLg6KRSunHCFdROv2Yze+0oO8gaQyPDQC06/F/eI+p1wIu9lJZHhiDS OAsffhZmT1W1Ds1iiDtx0zHeeTO0F9OqtU0csBlwOJaqjMBmnEVglrYtSIbrKzyTDBGn TlUJS9Iu35zyTuVfSLmPNNn1z5DDzrkTUl2gqXIacpXt6jjxX0zjfguDVljk0uMGJdHS TTrvvc1QgCvKHSLznWj9A6a5xCpX82T/SVMQapnGwwv548OijE2PpTVGjTcpbAQTD9qS WFg4p9y5PM6XNFqJiE2AlM89FVVXbojR8iTgwiOIhgPb5OjngZSF1aRvZ923SKUvLwbd kAFg== X-Gm-Message-State: ANhLgQ2UtJRFrnUMOQb5yIzU0tmKKjXDQy25/1hw9W0m6bivP8uDTJO8 W6rBzG1aAUXKMkFHBdw6vlSEdP+8MNCDH2w8PW81 X-Google-Smtp-Source: ADFU+vu9BJTFp0PT0YUtvfkYoLZmrBS0vRz8OFdOnp46l9u7QhhVKzeQUd/fcRn/1DqCeyMgp7eVPGDcZMKvCpW0fGI= X-Received: by 2002:a02:304a:: with SMTP id q71mr607149jaq.123.1584469563999; Tue, 17 Mar 2020 11:26:03 -0700 (PDT) MIME-Version: 1.0 References: <20200310221938.GF8447@dhcp22.suse.cz> In-Reply-To: From: Robert Kolchmeyer Date: Tue, 17 Mar 2020 11:25:52 -0700 Message-ID: Subject: Re: [patch] mm, oom: make a last minute check to prevent unnecessary memcg oom kills To: David Rientjes Cc: Michal Hocko , Andrew Morton , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ami Fischman Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Mar 10, 2020 at 3:54 PM David Rientjes wrote: > > Robert, could you elaborate on the user-visible effects of this issue that > caused it to initially get reported? > Ami (now cc'ed) knows more, but here is my understanding. The use case involves a Docker container running multiple processes. The container has a memory limit set. The container contains two long-lived, important processes p1 and p2, and some arbitrary, dynamic number of usually ephemeral processes p3,...,pn. These processes are structured in a hierarchy that looks like p1->p2->[p3,...,pn]; p1 is a parent of p2, and p2 is the parent for all of the ephemeral processes p3,...,pn. Since p1 and p2 are long-lived and important, the user does not want p1 and p2 to be oom-killed. However, p3,...,pn are expected to use a lot of memory, and it's ok for those processes to be oom-killed. If the user sets oom_score_adj on p1 and p2 to make them very unlikely to be oom-killed, p3,...,pn will inherit the oom_score_adj value, which is bad. Additionally, setting oom_score_adj on p3,...,pn is tricky, since processes in the Docker container (specifically p1 and p2) don't have permissions to set oom_score_adj on p3,...,pn. The ephemeral nature of p3,...,pn also makes setting oom_score_adj on them tricky after they launch. So, the user hopes that when one of p3,...,pn triggers an oom condition in the Docker container, the oom killer will almost always kill processes from p3,...,pn (and not kill p1 or p2, which are both important and unlikely to trigger an oom condition). The issue of more processes being killed than are strictly necessary is resulting in p1 or p2 being killed much more frequently when one of p3,...,pn triggers an oom condition, and p1 or p2 being killed is very disruptive for the user (my understanding is that p1 or p2 going down with high frequency results in significant unhealthiness in the user's service). The change proposed here has not been run in a production system, and so I don't think anyone has data that conclusively demonstrates that this change will solve the user's problem. But, from observations made in their production system, the user is confident that addressing this aggressive oom killing will solve their problem, and we have data that shows this change does considerably reduce the frequency of aggressive oom killing (from 61/100 oom killing events down to 0/100 with this change). Hope this gives a bit more context. Thanks, -Robert