From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A259C282DD for ; Thu, 9 Jan 2020 21:59:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 41B4E20656 for ; Thu, 9 Jan 2020 21:59:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 41B4E20656 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A06618E0005; Thu, 9 Jan 2020 16:59:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B6508E0001; Thu, 9 Jan 2020 16:59:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CC1F8E0005; Thu, 9 Jan 2020 16:59:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id 784C88E0001 for ; Thu, 9 Jan 2020 16:59:02 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id 21B9C180AD801 for ; Thu, 9 Jan 2020 21:59:02 +0000 (UTC) X-FDA: 76359461724.03.pump83_14a087e6d3e22 X-HE-Tag: pump83_14a087e6d3e22 X-Filterd-Recvd-Size: 4882 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Thu, 9 Jan 2020 21:59:01 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id z7so8994779wrl.13 for ; Thu, 09 Jan 2020 13:59:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=kDXENq3pxHwGmYQcpY44U06RWM91Gjk8kdFmevm8mzA=; b=qouax02md8fxzchYr+n70ifZEUIuM8tDAsTqyUvmNVTNWnrdY6jPMro/mNoOtEVmoL uWFaA6NZ9qzpezJJSC17LhjpLSGbQaf2LsSxfq1Z4EBWjEGxbrvPzrwXj/Vn914zt7FZ D8JBQmqNjgKJPHSrLOIJqV9BXBvg8WD4Kui1zzKs+XFG2MKMa3CXsCZQ3Q6yFan9Ngzz c4sj0MXj13gP+zxOBSlMM6drRSPwNF175Kc/d88CeNlbNsMpDiZ5v2HJr9Ed/8chIGS9 tZ/yu5B4R3GtCl6CqM3BJW3ds/1IEt81c9SdVkxrnmKSBIPBeSqT6kVwZf3Vvy27/2+M dzfg== X-Gm-Message-State: APjAAAUut+5XXBvC0N6u6/VSNBXK6qMLF2gJ8WVKZurnUqyuclF9rN47 a+Ex74MjU5moqp7cBK/dRTSskpIm X-Google-Smtp-Source: APXvYqwxykAeg3qT/6OmUUL8ATOvGkTLMyDiwE8iGhHAiZGVPFqAjT/O1ElElTdyAQ7hDmpmYDah0g== X-Received: by 2002:adf:e683:: with SMTP id r3mr12837384wrm.38.1578607140530; Thu, 09 Jan 2020 13:59:00 -0800 (PST) Received: from localhost (ip-37-188-146-105.eurotel.cz. [37.188.146.105]) by smtp.gmail.com with ESMTPSA id k7sm4068539wmi.19.2020.01.09.13.58.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2020 13:58:58 -0800 (PST) Date: Thu, 9 Jan 2020 22:58:58 +0100 From: Michal Hocko To: Vito Caputo Cc: Pavel Machek , kernel list , Andrew Morton , linux-mm@kvack.org, akpm@linux-foundation.org Subject: Re: OOM killer not nearly agressive enough? Message-ID: <20200109215858.GD23620@dhcp22.suse.cz> References: <20200107204412.GA29562@amd> <20200109115633.GR4951@dhcp22.suse.cz> <20200109210307.GA1553@duo.ucw.cz> <20200109214604.nfzsksyv3okj3ec2@shells.gnugeneration.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200109214604.nfzsksyv3okj3ec2@shells.gnugeneration.com> User-Agent: Mutt/1.12.2 (2019-09-21) X-Bogosity: Ham, tests=bogofilter, spamicity=0.017227, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu 09-01-20 13:46:04, Vito Caputo wrote: > On Thu, Jan 09, 2020 at 10:03:07PM +0100, Pavel Machek wrote: > > On Thu 2020-01-09 12:56:33, Michal Hocko wrote: > > > On Tue 07-01-20 21:44:12, Pavel Machek wrote: > > > > Hi! > > > > > > > > I updated my userspace to x86-64, and now chromium likes to eat all > > > > the memory and bring the system to standstill. > > > > > > > > Unfortunately, OOM killer does not react: > > > > > > > > I'm now running "ps aux", and it prints one line every 20 seconds or > > > > more. Do we agree that is "unusable" system? I attempted to do kill > > > > from other session. > > > > > > Does sysrq+f help? > > > > May try that next time. > > > > > > Do we agree that OOM killer should have reacted way sooner? > > > > > > This is impossible to answer without knowing what was going on at the > > > time. Was the system threshing over page cache/swap? In other words, is > > > the system completely out of memory or refaulting the working set all > > > the time because it doesn't fit into memory? > > > > Swap was full, so "completely out of memory", I guess. Chromium does > > that fairly often :-(. > > > > Have you considered restricting its memory limits a la `ulimit -m`? The kernel ignores RLIMIT_RSS. Unless the browser takes it into consideration then I do not see how that would help. > I've taken to running browsers in nspawn containers for general > isolation improvements, but this also makes it easy to set cgroup > resource limits like memcg. i.e. --property MemoryMax=2G Yes, this should help to isolate the problem. > This prevents the browser from bogging down the entire system, but it > doesn't prevent thrashing before FF OOMs within its control group. > > I do feel there's a problem with the kernel's reclaim algorithm, it > seems far too willing to evict file-backed pages that are recently in > use. It is true that the memory reclaim is quite page cache reclaim biased unless there is very small amount of the page cache. Page cache refault is considered during the reclaim but I am afraid that there are still corner cases where the workload might end up threshing. Be it on the page cache or the anonymous memory depending on the workload. Anyway getting data from real workloads is always good so that we can think on improving existing heuristics. -- Michal Hocko SUSE Labs