From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98929C282DD for ; Wed, 8 Jan 2020 21:14:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40B9C20643 for ; Wed, 8 Jan 2020 21:14:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=colorremedies-com.20150623.gappssmtp.com header.i=@colorremedies-com.20150623.gappssmtp.com header.b="yRNZ2AaY" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40B9C20643 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=colorremedies.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A5038E0005; Wed, 8 Jan 2020 16:14:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 954B38E0001; Wed, 8 Jan 2020 16:14:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8926B8E0005; Wed, 8 Jan 2020 16:14:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 73D738E0001 for ; Wed, 8 Jan 2020 16:14:40 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with SMTP id 1B56D181AEF09 for ; Wed, 8 Jan 2020 21:14:40 +0000 (UTC) X-FDA: 76355721120.29.soup37_b90acbf9ad37 X-HE-Tag: soup37_b90acbf9ad37 X-Filterd-Recvd-Size: 5748 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Wed, 8 Jan 2020 21:14:39 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id d73so487566wmd.1 for ; Wed, 08 Jan 2020 13:14:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=colorremedies-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=p+r62Lm9Jjg8iEPJwShFAal+snZF8PnrTvJGtCIAo8I=; b=yRNZ2AaY+zMoRXcQoN2R0OvIae4KZz01Gp3GBASxkCob9bez8Dh4H5BhJty7oQi7rf FqMEZVuy4MvSPb8h3TakMrPHw/Q1+2HWsJ2NZFBqEJ+qVZNaM9Q44++ivIivExVRubmX 0w6p0GFB8oHLHMnPlnHP1uSQZUn1XVOV4Jvok9/fJ+tH/E4QdQk4/9qkMdLrataRjQe1 XJ75ANp09mXZACTS6g0mHOsHxbcqAkFHYbQIfIsg/3XctlFV+ko2gQqvJL4byLT8tcOX /DHmLcCGAlgbl0EKBL6MKlHQmEG62VWMOnRua1c98lHS9TX18izFydc87EjB3lf1+0Y+ 3S2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=p+r62Lm9Jjg8iEPJwShFAal+snZF8PnrTvJGtCIAo8I=; b=hRrNbDeCREYie6FbrqczN/gEi7TCsYX0BFejXI4UuKPgRSUo/0FDeOnh0J2b5qfZSf 4Jv0V89PmYju8YIABKYDqC65wQwlietGmhaWlrwO5oYT3vxYytjrjS+t3fNF/UInWQaQ IXOlDKJrzyB2DJSYPvLzYZrrdZe1QbcQUZQmzyFVMJkueunDXk2wujhGAz0NQ3y+hk5z DMFX9DaN6RJUhAg1NOUMZk08Ym02nC4qr0sZlQoCA7UXHUl+oHdDuyXUI4WCYK56Gdej jFyHRiWgLBCKHULfk8u82Zw8MDaiUaronjlFBcsJ8a3jCwelIs9aeDDBRPD9OPijxXew 1ssQ== X-Gm-Message-State: APjAAAV3vjQWAI/Cy62K/oDEkaFvLTsIRlOi96rxfAULPlB2gMDTCEmX H3Mkgk1cBew4FfIgJirrwTrcq/dY/dMl05hhhDxuUA== X-Google-Smtp-Source: APXvYqzEvQfm0+0SAiT/qsDY69Iy4vrfrPYqNnaPpi072suWeisT6wC+sMbtXPpdBg/fP9P57qmwRZCtKeNZK71Qets= X-Received: by 2002:a1c:61c1:: with SMTP id v184mr661794wmb.160.1578518078000; Wed, 08 Jan 2020 13:14:38 -0800 (PST) MIME-Version: 1.0 References: <20200107205824.GM32178@dhcp22.suse.cz> <20200108092501.GO32178@dhcp22.suse.cz> In-Reply-To: <20200108092501.GO32178@dhcp22.suse.cz> From: Chris Murphy Date: Wed, 8 Jan 2020 14:14:22 -0700 Message-ID: Subject: Re: user space unresponsive, followup: lsf/mm congestion To: Michal Hocko Cc: linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 8, 2020 at 2:25 AM Michal Hocko wrote: > > On Tue 07-01-20 14:25:46, Chris Murphy wrote: > > On Tue, Jan 7, 2020 at 1:58 PM Michal Hocko wrote: > [...] > > > Btw. from a quick look at the sysrq output there seems to be quite a lot > > > of tasks (more than 1k) running on the system. Only handful of them > > > belong to the compilation. kswapd is busy and 13 processes in direct > > > reclaim all swapping out to the disk. > > > > There might be many dozens of tabs in Firefox with nothing loaded in > > them, trying to keep the testing more real world (a compile while > > browsing) rather than being too deferential to the compile. That does > > clutter the sysrq+t but it doesn't change the outcome of the central > > culprit which is the ninja compile, which by default does n+2 jobs > > where n is the number of virtual CPUs. > > How much memory does the compile process eat? By default it sets jobs to numcpus+2, which is 10. But each job variably has two processes, and each process's memory requirement varies a ton, few M to over 1G. In the first 20 minutes, about 13000 processes have started and stopped. I've updated the bug, attaching kernel messages and /proc/vmstate in 1s increments, although quite often during the build multiple seconds of sampling were just skipped as the system was under too much pressure. > If you know that the compilation process is too disruptive wrt. > memory/cpu consumption then you can use cgroups (memory and cpu > controllers) to throttle that consumption and protect the rest of the > system. The compilation process will take much more time of course and > the explicit configuration is obviously less comfortable than out of the > box auto configuration but the kernel simply doesn't have information to > prioritize resources. Yes but this isn't scalable for regular users who just follow an upstream's build instructions. > I do agree that the oom detection could be improved to detect a heavy > threshing - be it on page cache or swapin/out - and kill something > rather than leave the system struggling in a highly unproductive state. > This is far from trivial because what is productive is not something > kernel can tell easily as it depends on the workload. As mentioned > elsewhere userspace is likely much better suited to define that policy > and PSI seems to be a good indicator. And even user space doesn't know what resources are required in advance. The user can guess this has been estimated incorrectly, force power off, start over by passing a lower number of jobs or whatever. As for PSI, from oomd folks it sounds like swap is a requirement. And yet, because of the poor performance of swapping, quite a lot of users don't have any swap. It's also mixed in server environments to have swap, and rare in cloud environments to have swap. So if there's a hard requirement on swap existing, PSI isn't a universal solution. Thanks, -- Chris Murphy