From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.4 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07924C433E0 for ; Sun, 12 Jul 2020 18:34:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9CFDE2067D for ; Sun, 12 Jul 2020 18:34:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="j9AjgHBD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CFDE2067D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF3938D0003; Sun, 12 Jul 2020 14:34:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA4838D0002; Sun, 12 Jul 2020 14:34:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C92C48D0003; Sun, 12 Jul 2020 14:34:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id B03F28D0002 for ; Sun, 12 Jul 2020 14:34:24 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3728A181AC9CB for ; Sun, 12 Jul 2020 18:34:24 +0000 (UTC) X-FDA: 77030274048.30.hose49_4b0991126ee1 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id F32E1180B3C8E for ; Sun, 12 Jul 2020 18:34:23 +0000 (UTC) X-HE-Tag: hose49_4b0991126ee1 X-Filterd-Recvd-Size: 7275 Received: from mail-lf1-f68.google.com (mail-lf1-f68.google.com [209.85.167.68]) by imf12.hostedemail.com (Postfix) with ESMTP for ; Sun, 12 Jul 2020 18:34:23 +0000 (UTC) Received: by mail-lf1-f68.google.com with SMTP id g139so6422077lfd.10 for ; Sun, 12 Jul 2020 11:34:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=IO9uKm8bvdgEjXJqj5Rrs1xcnUatMvk40LVf7RvbX1E=; b=j9AjgHBDK8GhxGwrcFrgpH75XSyVbuNfHykEx4D4mLydcgD/rn7GbBfs1u5ekG2XOP rBa7qpU1iHuVcWsWMIRn7fxyBREUsOaBW/HBn0hRaqYEmT1Lp+z0JecsqXPvA+hKXKVl v2yBsHzTJgFCEACderYqPlp4GsvJHID7YtykwUWRvBI+w9q2TiHt63YIoEmwOaRd5+09 T5sBBkxwaK4lXsk6SYK81iiVCS6QO7NZqHKUm5PMG9Kvtqd7AFw3ENSx8k/PrJgPH7+H ibHqQj2ZOIIUdUhcijI56PQmDDiRoA46ppkUqgEu9YbLVxe/fLhErxtqw5u40U1KDBPC YGuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=IO9uKm8bvdgEjXJqj5Rrs1xcnUatMvk40LVf7RvbX1E=; b=bao9ENVBxIXC8ZHRiR7jRz30wa8U7aQJjkART0ElSkbPMo4JogLvz619nV06n2qmBt cARVXERmOF6L+2llgUukqRFIIuZ0vDuoOotwbXITrDXBYXz4oYgIStAmetKf1hrwXz4f NqB2Kv40SlnC/2qmerP6bW+MaoAcUjtzJKGKXXHk1B9beDduKmYGe084BPma11fqL3a/ fkftphutUv7sRy/YSiyGNLj3mqEkUALBEb1bnPi/VYFfAahcjSTxqsXAeLfAPdpPCF7Z 3g3+LMR7vAC4mAx3hDboIoq/S4U8Lci7GMkQwphs1vc8xrBoiPraQJUENKmj19S5Wte7 cb2w== X-Gm-Message-State: AOAM531DcZcTAZFV4YlWuLhBH7aJTh4v1XCEk8VXN9Dok5a8Z+cU/tw0 u9i7pKE+B5NehUmqKgYx64VH/r07nmlZFMXQq/706A== X-Google-Smtp-Source: ABdhPJyD3BLm7rKT28e/lXdMw50zAGeCsTW4qgPGdvr71CB1pTn9a1l/8uedaLfOGd2pmua+XpcP+k3dqyADBddg7Zk= X-Received: by 2002:a05:6512:74f:: with SMTP id c15mr40253704lfs.26.1594578861881; Sun, 12 Jul 2020 11:34:21 -0700 (PDT) MIME-Version: 1.0 References: <1594396651-9931-1-git-send-email-laoar.shao@gmail.com> In-Reply-To: <1594396651-9931-1-git-send-email-laoar.shao@gmail.com> From: Naresh Kamboju Date: Mon, 13 Jul 2020 00:04:10 +0530 Message-ID: Subject: Re: [PATCH v3] mm, oom: make the calculation of oom badness more accurate To: Yafang Shao Cc: Michal Hocko , Qian Cai , rientjes@google.com, Andrew Morton , linux-mm , Stephen Rothwell , Linux-Next Mailing List , open list , lkft-triage@lists.linaro.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: F32E1180B3C8E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 10 Jul 2020 at 21:28, Yafang Shao wrote: > > Recently we found an issue on our production environment that when memcg > oom is triggered the oom killer doesn't chose the process with largest > resident memory but chose the first scanned process. Note that all > processes in this memcg have the same oom_score_adj, so the oom killer > should chose the process with largest resident memory. > > Bellow is part of the oom info, which is enough to analyze this issue. > [7516987.983223] memory: usage 16777216kB, limit 16777216kB, failcnt 5284= 3037 > [7516987.983224] memory+swap: usage 16777216kB, limit 9007199254740988kB,= failcnt 0 > [7516987.983225] kmem: usage 301464kB, limit 9007199254740988kB, failcnt = 0 [...] > [7516987.984221] oom-kill:constraint=3DCONSTRAINT_MEMCG,nodemask=3D(null)= ,cpuset=3D3aa16c9482ae3a6f6b78bda68a55d32c87c99b985e0f11331cddf05af6c4d753,= mems_allowed=3D0-1,oom_memcg=3D/kubepods/podf1c273d3-9b36-11ea-b3df-246e969= 3c184,task_memcg=3D/kubepods/podf1c273d3-9b36-11ea-b3df-246e9693c184/1f246a= 3eeea8f70bf91141eeaf1805346a666e225f823906485ea0b6c37dfc3d,task=3Dpause,pid= =3D5740,uid=3D0 > [7516987.984254] Memory cgroup out of memory: Killed process 5740 (pause)= total-vm:1028kB, anon-rss:4kB, file-rss:0kB, shmem-rss:0kB > [7516988.092344] oom_reaper: reaped process 5740 (pause), now anon-rss:0k= B, file-rss:0kB, shmem-rss:0kB > > We can find that the first scanned process 5740 (pause) was killed, but i= ts > rss is only one page. That is because, when we calculate the oom badness = in > oom_badness(), we always ignore the negtive point and convert all of thes= e > negtive points to 1. Now as oom_score_adj of all the processes in this > targeted memcg have the same value -998, the points of these processes ar= e > all negtive value. As a result, the first scanned process will be killed. > > The oom_socre_adj (-998) in this memcg is set by kubelet, because it is a > a Guaranteed pod, which has higher priority to prevent from being killed = by > system oom. > > To fix this issue, we should make the calculation of oom point more > accurate. We can achieve it by convert the chosen_point from 'unsigned > long' to 'long'. > > [cai@lca.pw: reported a issue in the previous version] > [mhocko@suse.com: fixed the issue reported by Cai] > [mhocko@suse.com: add the comment in proc_oom_score()] > Signed-off-by: Yafang Shao > Acked-by: Michal Hocko > Cc: David Rientjes > Cc: Qian Cai > > --- > v2 -> v3: > - fix the type of variable 'point' in oom_evaluate_task() > - initialize oom_control->chosen_points in select_bad_process() per Micha= l > - update the comment in proc_oom_score() per Michal > > Signed-off-by: Yafang Shao Tested-by: Naresh Kamboju I have noticed kernel panic with v2 patch while running LTP mm test suite. [ 63.451494] Out of memory and no killable processes... [ 63.456633] Kernel panic - not syncing: System is deadlocked on memory Then I have removed the v2 patch and applied this below v3 patch and re-tes= ted. No regression noticed with v3 patch while running LTP mm on x86_64 and arm. OTOH, oom01 test case started with 100 iterations but runltp got killed after the 6th iteration [3]. I think this is expected. test steps: - cd /opt/ltp - ./runltp -s oom01 -I 100 || true [ 209.052842] Out of memory: Killed process 519 (runltp) total-vm:10244kB, anon-rss:904kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:60kB oom_score_adj:0 [ 209.066782] oom_reaper: reaped process 519 (runltp), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB /lava-1558245/0/tests/0_prep-tmp-disk/run.sh: line 21: 519 Killed ./runltp -s oom01 -I 100 > --- > fs/proc/base.c | 11 ++++++++++- > include/linux/oom.h | 4 ++-- > mm/oom_kill.c | 22 ++++++++++------------ > 3 files changed, 22 insertions(+), 15 deletions(-) Reference test jobs, [1] https://lkft.validation.linaro.org/scheduler/job/1558246#L9189 [2] https://lkft.validation.linaro.org/scheduler/job/1558247#L17213 [3] https://lkft.validation.linaro.org/scheduler/job/1558245#L1407