From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FB35C47094 for ; Mon, 7 Jun 2021 20:32:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 33FE361153 for ; Mon, 7 Jun 2021 20:32:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33FE361153 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C94BF6B006C; Mon, 7 Jun 2021 16:32:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C458E6B006E; Mon, 7 Jun 2021 16:32:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ABEAE6B0070; Mon, 7 Jun 2021 16:32:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id 79BDF6B006C for ; Mon, 7 Jun 2021 16:32:13 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1A698180AD802 for ; Mon, 7 Jun 2021 20:32:13 +0000 (UTC) X-FDA: 78228074946.25.AE7EBCB Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf13.hostedemail.com (Postfix) with ESMTP id BD32CE012EF2 for ; Mon, 7 Jun 2021 20:31:58 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out1.suse.de (Postfix) with ESMTP id 88AFC219C1; Mon, 7 Jun 2021 19:36:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1623094607; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dMYKPJur+bblV8fB0rrtyqM/Pi0SpR71lsEX2LI8jjQ=; b=HZllzxztpoQLS++JYoiC3sW9cqieN0oZjMAL+7D5NxhhLT9rRvvzQ8jNo5AoqGHa3t970W SVc4F7bUolKY8B38Q57jAY1a/S8VGTd6L4e3rVnUojkYcjagDfpI9Gkg89sMllCfH7L8fA VF8I2jSzhJB/1pYkm2jmbKFtuli080M= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 57D5CA3BD1; Mon, 7 Jun 2021 19:36:47 +0000 (UTC) Date: Mon, 7 Jun 2021 21:36:46 +0200 From: Michal Hocko To: Waiman Long Cc: Shakeel Butt , Aaron Tomlin , Linux MM , Andrew Morton , Vlastimil Babka , LKML Subject: Re: [RFC PATCH] mm/oom_kill: allow oom kill allocating task for non-global case Message-ID: References: <20210607163103.632681-1-atomlin@redhat.com> <6d23ce58-4c4b-116a-6d74-c2cf4947492b@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <6d23ce58-4c4b-116a-6d74-c2cf4947492b@redhat.com> Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=HZllzxzt; spf=temperror (imf13.hostedemail.com: error in processing during lookup of mhocko@suse.com: DNS error) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com X-Stat-Signature: 7errxpb6pz5edt65hrtecfzy3hp1kx55 X-Rspamd-Queue-Id: BD32CE012EF2 X-Rspamd-Server: rspam06 X-HE-Tag: 1623097918-346074 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 07-06-21 15:18:38, Waiman Long wrote: > On 6/7/21 3:04 PM, Michal Hocko wrote: > > On Mon 07-06-21 14:51:05, Waiman Long wrote: > > > On 6/7/21 2:43 PM, Shakeel Butt wrote: > > > > On Mon, Jun 7, 2021 at 9:45 AM Waiman Long wro= te: > > > > > On 6/7/21 12:31 PM, Aaron Tomlin wrote: > > > > > > At the present time, in the context of memcg OOM, even when > > > > > > sysctl_oom_kill_allocating_task is enabled/or set, the "alloc= ating" > > > > > > task cannot be selected, as a target for the OOM killer. > > > > > >=20 > > > > > > This patch removes the restriction entirely. > > > > > >=20 > > > > > > Signed-off-by: Aaron Tomlin > > > > > > --- > > > > > > mm/oom_kill.c | 6 +++--- > > > > > > 1 file changed, 3 insertions(+), 3 deletions(-) > > > > > >=20 > > > > > > diff --git a/mm/oom_kill.c b/mm/oom_kill.c > > > > > > index eefd3f5fde46..3bae33e2d9c2 100644 > > > > > > --- a/mm/oom_kill.c > > > > > > +++ b/mm/oom_kill.c > > > > > > @@ -1089,9 +1089,9 @@ bool out_of_memory(struct oom_control *= oc) > > > > > > oc->nodemask =3D NULL; > > > > > > check_panic_on_oom(oc); > > > > > >=20 > > > > > > - if (!is_memcg_oom(oc) && sysctl_oom_kill_allocating_tas= k && > > > > > > - current->mm && !oom_unkillable_task(current) && > > > > > > - oom_cpuset_eligible(current, oc) && > > > > > > + if (sysctl_oom_kill_allocating_task && current->mm && > > > > > > + !oom_unkillable_task(current) && > > > > > > + oom_cpuset_eligible(current, oc) && > > > > > > current->signal->oom_score_adj !=3D OOM_SCORE_ADJ= _MIN) { > > > > > > get_task_struct(current); > > > > > > oc->chosen =3D current; > > > > > To provide more context for this patch, we are actually seeing = that in a > > > > > customer report about OOM happened in a container where the dom= inating > > > > > task used up most of the memory and it happened to be the task = that > > > > > triggered the OOM with the result that no killable process coul= d be > > > > > found. > > > > Why was there no killable process? What about the process allocat= ing > > > > the memory or is this remote memcg charging? > > > It is because the other processes have a oom_adjust_score of -1000.= So they > > > are non-killable. Anyway, they don't consume that much memory and k= illing > > > them won't free up that much. > > >=20 > > > The other process that uses most of the memory is the one that trig= ger the > > > OOM kill in the first place because the memory limit has been reach= ed in new > > > memory allocation. Based on the current logic, this process cannot = be killed > > > at all even if we set the oom_kill_allocating_task to 1 if the OOM = happens > > > only within the memcg context, not in a global OOM situation. This = patch is > > > to allow this process to be killed under this circumstance. > > Do you have the oom report? I do not see why the allocating task hasn= 't > > been chosen. >=20 > A partial OOM report below: Do you happen to have the full report? > [ 8221.433608] memory: usage 21280kB, limit 204800kB, failcnt 49116 > =A0 : > [ 8227.239769] [ pid ]=A0=A0 uid=A0 tgid total_vm=A0=A0=A0=A0=A0 rss pg= tables_bytes swapents oom_score_adj name > [ 8227.242495] [1611298]=A0=A0=A0=A0 0 1611298=A0=A0=A0 35869=A0=A0=A0=A0= =A0 635 167936=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0 -1000 conmo= n > [ 8227.242518] [1702509]=A0=A0=A0=A0 0 1702509=A0=A0=A0 35869=A0=A0=A0=A0= =A0 701 176128=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0 -1000 conmo= n > [ 8227.242522] [1703345] 1001050000 1703294=A0=A0 183440=A0=A0=A0=A0=A0= =A0=A0 0 2125824=A0=A0=A0=A0=A0=A0=A0 0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 999= node > [ 8227.242706] Out of memory and no killable processes... > [ 8227.242731] node invoked oom-killer: gfp_mask=3D0x6000c0(GFP_KERNEL)= , nodemask=3D(null), order=3D0, oom_score_adj=3D999 > [ 8227.242732] node cpuset=3Dcrio-b8ac7e23f7b520c0365461defb66738231918= 243586e287bfb9e206bb3a0227a.scope mems_allowed=3D0-1 >=20 > So in this case, node cannot kill itself and no other processes are > available to be killed. The process is clearly listed as eligible so the oom killer should find it and if it hasn't then this should be investigated. Which kernel is this? --=20 Michal Hocko SUSE Labs