From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88706C33CA2 for ; Thu, 9 Jan 2020 21:25:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 56C892080D for ; Thu, 9 Jan 2020 21:25:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56C892080D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0F8148E0005; Thu, 9 Jan 2020 16:25:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 05A088E0001; Thu, 9 Jan 2020 16:25:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E8ACE8E0005; Thu, 9 Jan 2020 16:25:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0061.hostedemail.com [216.40.44.61]) by kanga.kvack.org (Postfix) with ESMTP id CE87C8E0001 for ; Thu, 9 Jan 2020 16:25:20 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 8CA3052D9 for ; Thu, 9 Jan 2020 21:25:20 +0000 (UTC) X-FDA: 76359376800.11.crowd96_117e2f6cc9d4c X-HE-Tag: crowd96_117e2f6cc9d4c X-Filterd-Recvd-Size: 4372 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Thu, 9 Jan 2020 21:25:20 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id p17so4440283wma.1 for ; Thu, 09 Jan 2020 13:25:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=M/fC6s3aJPS96C3A0a7p8hYvGSefECWe4S/GhKz+rM4=; b=bE08I0PGafmla8tRJthhENCz2ohm5qsYjpW6560d70KyM44nKoV/+5JKZC7ZKXHNWf xzg7OfEV+4DXKtQvfVA9HfBG0zGpXabExFtWgSGsS5BmyBKRSZ7qc8YLc++Ub/kyNulu zak91cQK7xZCDD/31Ry8HtymtmuqoLKfjKzTy7kUy0mdk1FQ3+EkN2m5F7sG0trwOzJO 6dtEiSVvZaYFAYauYDAMiRmztYX7yWXeqf73AfbkDChZIUthLjvXT2UT5Zh4I3yEs2a7 LwJDndIiZCPi6/rbrs2JYehZ9EUv7nlUr/TK9/cOxCxAqDtYdICSFselgX3HNO0Z94Mb yncg== X-Gm-Message-State: APjAAAUiKT8i36azpD0UpO+SdgXjqSMw5AEjavkZh6Umfj4fqquk70h8 ol1hgB7WhpY2pvcXONdfv0Y= X-Google-Smtp-Source: APXvYqw8M4bievOHJLVqpOPRe4k3DoDwDFFbBTUI/8frm+mCFsybwOP5RYcBie/CzQOSQy250fLBLQ== X-Received: by 2002:a1c:1dcd:: with SMTP id d196mr19577wmd.106.1578605118956; Thu, 09 Jan 2020 13:25:18 -0800 (PST) Received: from localhost (ip-37-188-146-105.eurotel.cz. [37.188.146.105]) by smtp.gmail.com with ESMTPSA id b21sm4124385wmd.37.2020.01.09.13.25.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2020 13:25:17 -0800 (PST) Date: Thu, 9 Jan 2020 22:25:16 +0100 From: Michal Hocko To: Pavel Machek Cc: kernel list , Andrew Morton , linux-mm@kvack.org, akpm@linux-foundation.org Subject: Re: OOM killer not nearly agressive enough? Message-ID: <20200109212516.GA23620@dhcp22.suse.cz> References: <20200107204412.GA29562@amd> <20200109115633.GR4951@dhcp22.suse.cz> <20200109210307.GA1553@duo.ucw.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200109210307.GA1553@duo.ucw.cz> User-Agent: Mutt/1.12.2 (2019-09-21) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000487, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 09-01-20 22:03:07, Pavel Machek wrote: > On Thu 2020-01-09 12:56:33, Michal Hocko wrote: > > On Tue 07-01-20 21:44:12, Pavel Machek wrote: > > > Hi! > > > > > > I updated my userspace to x86-64, and now chromium likes to eat all > > > the memory and bring the system to standstill. > > > > > > Unfortunately, OOM killer does not react: > > > > > > I'm now running "ps aux", and it prints one line every 20 seconds or > > > more. Do we agree that is "unusable" system? I attempted to do kill > > > from other session. > > > > Does sysrq+f help? > > May try that next time. > > > > Do we agree that OOM killer should have reacted way sooner? > > > > This is impossible to answer without knowing what was going on at the > > time. Was the system threshing over page cache/swap? In other words, is > > the system completely out of memory or refaulting the working set all > > the time because it doesn't fit into memory? > > Swap was full, so "completely out of memory", I guess. Chromium does > that fairly often :-(. The oom heuristic is based on the reclaim failure. If the reclaim makes some progress then the oom killer is not hit. Have a look at should_reclaim_retry for more details. > > > Is there something I can tweak to make it behave more reasonably? > > > > PSI based early OOM killing might help. See https://github.com/facebookincubator/oomd > > Um. Before doing that... is there some knob somewhere saying "hey > oomkiller, one hour to recover machine is a bit too much, can you > please react sooner"? No, there is nothing like that. > PSI is completely different system, but I guess > I should attempt to tweak the existing one first... PSI is measuring the cost of the allocation (among other things) and that can give you some idea on how much time is spent to get memory. Userspace can implement a policy based on that and act. The kernel oom killer is the last resort when there is really no memory to allocate. -- Michal Hocko SUSE Labs