From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.4 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2F3CC4727C for ; Tue, 22 Sep 2020 16:30:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 450B523A1B for ; Tue, 22 Sep 2020 16:30:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="k9V3cN/r" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 450B523A1B Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BA2606B0099; Tue, 22 Sep 2020 12:30:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADEAC6B009B; Tue, 22 Sep 2020 12:30:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93A2F6B009D; Tue, 22 Sep 2020 12:30:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0220.hostedemail.com [216.40.44.220]) by kanga.kvack.org (Postfix) with ESMTP id 6D5686B0099 for ; Tue, 22 Sep 2020 12:30:03 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 26B158249980 for ; Tue, 22 Sep 2020 16:30:03 +0000 (UTC) X-FDA: 77291234286.19.love32_4110fc72714f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id CC2AB1ACC22 for ; Tue, 22 Sep 2020 16:30:01 +0000 (UTC) X-HE-Tag: love32_4110fc72714f X-Filterd-Recvd-Size: 4148 Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) by imf32.hostedemail.com (Postfix) with ESMTP for ; Tue, 22 Sep 2020 16:30:01 +0000 (UTC) Received: by mail-lf1-f66.google.com with SMTP id y2so18701627lfy.10 for ; Tue, 22 Sep 2020 09:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+Bx8gekgm2SEOvt6u33teukzzHIcXbMrSd4fJ3huDS4=; b=k9V3cN/rbuswExLp67fruMpypgqAsZd+NisBoTQvS+xzFFj+/mdrPU/GcSAmSlXgsR 3I01CvoEm2O8zkQOEw8J0Y2KZKyD5QSMRhtcVQcUQx3drK5Lje0kTDvU8eFnJx5PDa4k E3blL8kFwcTLssklMilM9R2qGHtef2b1DVXhkyhjS1/juNyd4ISZXq+c9sOb9tzJkcWB glwO4W3iGR7SJ/isjmflyTnALmC+fYbI8+eCeDFKbH+cH3GxyHG+CYsrDi6RUgNpH58+ CW7rbZ/pM5i0JcOz+7yRy0O0rbNXrDqHgw9pSTahiSPmSRVewNxhaYn//AF8o2roPDyH BE3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+Bx8gekgm2SEOvt6u33teukzzHIcXbMrSd4fJ3huDS4=; b=LcDVfMLgDWIzl9JFe/2GKE730mhTVPwoHbXg0JiAqgQRSXvfZPdwufm5CBkQBpC1Iq ZZQ+fVx0M/cpWIXdV2tEz6znoAeq+mwSIZYsD39X3lesvH2tPeY0LF68ufKIkY7ytNbS 2TRcfd9W44fH3fVFa83driMoXMwEJvIia1Fe+NMHpGdYeRHKTrzdQs4L7jWBLLFJ/+Zq qbE4kcb4VEYVldb6LkvwBUKZspZkkqvYZ4RqUeNTuw1OwhyQgeWy3auIbGkOKcXNwHpk DcA8/73F+jW6TTNAYbBL9hKRx8Ig/uXbEfGr3Uo9Qhx/8wC2EkAGuCNWWnl5TfBEiVYo 6gFA== X-Gm-Message-State: AOAM5321vg1Wz3nIMyDVBxt1Vws7HGkrEq8/Z+GgI95Qj342OD1SUbkg 1ly73WxmLyNFZdzIZzR0AhwKIxch0NOjz2Xa82QVq+PdvB0= X-Google-Smtp-Source: ABdhPJyirm0v+rO7vqImS4UsI+KLO3mtBjQY6yfGaAUPyv3UQn89NbK8UvFvw9pzcyMGvD4manAbbc7DR8AwSP6QVPU= X-Received: by 2002:a19:4084:: with SMTP id n126mr1793874lfa.54.1600792199501; Tue, 22 Sep 2020 09:29:59 -0700 (PDT) MIME-Version: 1.0 References: <20200922111202.GY12990@dhcp22.suse.cz> <20200922151654.GA12990@dhcp22.suse.cz> In-Reply-To: <20200922151654.GA12990@dhcp22.suse.cz> From: Shakeel Butt Date: Tue, 22 Sep 2020 09:29:48 -0700 Message-ID: Subject: Re: Machine lockups on extreme memory pressure To: Michal Hocko Cc: Johannes Weiner , Linux MM , Andrew Morton , Roman Gushchin , LKML , Greg Thelen Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Sep 22, 2020 at 8:16 AM Michal Hocko wrote: > > On Tue 22-09-20 06:37:02, Shakeel Butt wrote: > [...] > > > I would recommend to focus on tracking down the who is blocking the > > > further progress. > > > > I was able to find the CPU next in line for the list_lock from the > > dump. I don't think anyone is blocking the progress as such but more > > like the spinlock in the irq context is starving the spinlock in the > > process context. This is a high traffic machine and there are tens of > > thousands of potential network ACKs on the queue. > > So there is a forward progress but it is too slow to have any reasonable > progress in userspace? Yes. > > > I talked about this problem with Johannes at LPC 2019 and I think we > > talked about two potential solutions. First was to somehow give memory > > reserves to oomd and second was in-kernel PSI based oom-killer. I am > > not sure the first one will work in this situation but the second one > > might help. > > Why does your oomd depend on memory allocation? > It does not but I think my concern was the potential allocations during syscalls. Anyways, what do you think of the in-kernel PSI based oom-kill trigger. I think Johannes had a prototype as well.