From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A8DEC77B73 for ; Wed, 31 May 2023 08:43:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F1466B0072; Wed, 31 May 2023 04:43:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 57A1A8E0001; Wed, 31 May 2023 04:43:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 41C1D6B0075; Wed, 31 May 2023 04:43:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2BBF76B0072 for ; Wed, 31 May 2023 04:43:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F206A120286 for ; Wed, 31 May 2023 08:42:59 +0000 (UTC) X-FDA: 80849910078.03.2150107 Received: from smtpbgjp3.qq.com (smtpbgjp3.qq.com [54.92.39.34]) by imf03.hostedemail.com (Postfix) with ESMTP id 27D1B2000F for ; Wed, 31 May 2023 08:42:56 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of gouhao@uniontech.com designates 54.92.39.34 as permitted sender) smtp.mailfrom=gouhao@uniontech.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685522578; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=IBGCKyOUW9m2Ng6+okX7GD+L6w+P6XDG7H1txZZ/wds=; b=J71ZcshqGm+kvK1379dzNf98v3WoMaB8jSap16bFZGI0AgkYjB7ZFl1nMWd6w6hlcZ3H54 /18kL2NrNxPaAJw739a9/AZYX4xmD6yW6/htolc3BhrVxaeabVI+08tWgkhcJRCO40rvWf u3Ekhhe5YJ4Z8kRSwz/TBCAo6qGR5gA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685522578; a=rsa-sha256; cv=none; b=FamFTpWiF9DmyfmSRX75tFPTcAYZp9g1lBMGFUEnDMNObYY/4ejze5z4ONaMOccBoZpMEA YeVcAhSZ5YN//TqxEl+g5CB/UBeMJ/7WaGgeIstsV7ebnpL+GgN0XuLWQjmF9kmaHIE8Qp mJAL7UNpNComed2Bp1dhJKHJeRvfd1w= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none; spf=pass (imf03.hostedemail.com: domain of gouhao@uniontech.com designates 54.92.39.34 as permitted sender) smtp.mailfrom=gouhao@uniontech.com; dmarc=none X-QQ-mid: bizesmtp88t1685522565t2l3mhjk Received: from [10.7.13.54] ( [113.200.76.118]) by bizesmtp.qq.com (ESMTP) with id ; Wed, 31 May 2023 16:42:43 +0800 (CST) X-QQ-SSF: 01400000000000C0G000000A0000000 X-QQ-FEAT: q+EIYT+FhZpO2RFB20DsmFECDaJGO91x8w+wNl4XDHHAajtQ9f5E8NOLnnpy8 k+i7Z+2QMoNh2nL4KZUC63TwRvq6JjM5hs93tZuCF2LaUSmmK61Y1KZLfrKdgm2TAOSL+GY M3ahetdg9PHC/IAE7uEvKMx9/GeLtxbxCdXj/WLJth5+J05O/0KGxaWZo3L5WsWqkjRImAW uz6WVKPUlUjmuZFJTZgqwUC1pQSm/eil6vl5sBsrGglBBtDkq5ACjVzdD10VudVpBTogCL/ HomMQapV5Ae5dsyhU6Ap3d+ONnQ5/2kSEqBQrli1WQ2s9ZcnIHAUTp+7zN4Svsq/Qgt8L14 lsuRZZ5JjFC0xPwEZpZrBcKVyAHOlucSC2hN4nLqmYU7qTdekrRiJt4E6D7dg== X-QQ-GoodBg: 1 X-BIZMAIL-ID: 7957098482953656911 Message-ID: Date: Wed, 31 May 2023 16:42:43 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Content-Language: en-US From: Gou Hao Subject: Question about oom-killer To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-QQ-SENDSIZE: 520 Feedback-ID: bizesmtp:uniontech.com:qybglogicsvrgz:qybglogicsvrgz7a-0 X-Stat-Signature: 4qcekizdcgaq7qxyw451u8xfxnd5rs9g X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 27D1B2000F X-Rspam-User: X-HE-Tag: 1685522576-144637 X-HE-Meta: U2FsdGVkX18o1QSQirsNsey5zw8I0NAPhZJEEEaCMMsaO3Q3k4Of7vB+r2G9mQY0fNZe1JzfUJ9DNiEO+kwFFJWxrLmdv1kVZFSZH4CaS/Blf6+xj0QJHNBdshvGYCub8ONN3AdHUDRU8sjjB3Sn6fs2rT5m7Bj5+mY99F70ikdL/zYHAyrCHKkBn19zTJAjXniLB+BgwhOkz6wWUAQ1SdRaJTnBbXSPv2oI2Uv/6LAh0ee2TpBhxfaQV2UImyBjoThM+JKDdNYxrnZlh2DCw86FgTXizc13uihsO4er23aRtPvTO1h/mDLL2ahbynLJvm62g/ezerfCr/rY+Gdpp6MDukivLGaXPHGjpSJn5SFs4wrHIoc8FTrEH+ATJGUWh27NIZyaJBoApmPCMftJZHVUZFpoUO0lQnv2WFiX3WRUUndgsWrCeVL0yt2BAYon3fy3HGB5KMHpq/vh7NCEeW5TxbEX7gxsgSruf0ANTVSEYlDib/+yE9I34zaT14WUZwsqNaE5ZBC9RLpA9ae8MzbIBZ9kmZHQzylsRHnjW7L6dONdH3B5GWxLv4OjwKDQSUJKvuWw5xTNuNKuVnH3UWLS6d2XFCZ8XbOIWVVrQo5EvhxmLCvnD2sp3efLkTI1mL1mDrp/peayhwmfPgxISC8vw40se1N5+hAbExarVy2IHaC2M3khG5V0+LisVZtBDIY3vp5WuQZHCC9Qg9S8vCsnmmVse66RCiIAXOtrqcvIm12USHyPWY9gMqXdvlkeuHPTGAqVPsKool75sF+WhCUacWwKiMvxRcuO3m2twM7fcdynKBrMzcZ6z3OZl6sWqQq6w+tM3UpZDoACB5Xiv1LRs0/5WgCX6dKulGmPDqd/uKOhUWWf0+c67cuNXv9aICxiHiDg5YBwrfyrqF2pUI8rDqa9EfaKIT4KXnRd6CNW8rj1pwsiOR7T6CDab2ARex3RuDs/PER6TvX9imn NvWDrhA6 BGraJDDgtQ6MzAc8DpsS1puGkO+n3L/pBXiZfuUZTYJpUbVSVfrFFSPGU23XMr7SpKqhId+WO5dTHzGBd8JxDXXwoDBGkOzKYPiyxgG8Cpvrns02CpnytAggBOeqIBRa+Pc+QVDfYTSpgntWmNxzGrz1j9ftdNqCVj5MjWNxeuy3k5F5QR05tawh6O6FlN1q2bAkNRHahH9SOGanzRz7lHQhxKX6rqwyLs+Q/VyAIEVeSzJLkWCbwPug3Ng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hello everyone, Recently, my kernel restarted while I was running ltp-oom02(It allocates memory infinitely in a loop, testing whether the oom-killer works properly ). log: ``` [480156.950100] Tasks state (memory values in pages): [480156.950101] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name [480156.950302] [   2578]    81  2578      523        0 393216        6          -900 dbus-daemon [480156.950309] [   2648]   172  2596     2435        0 393216        5             0 rtkit-daemon [480156.950322] [   5256]     0  2826    25411        0 589824        0             0 DetectThread [480156.950328] [   5404]     0  5404      412        2 393216       64         -1000 sshd [480156.950357] [  10518]     0 10518     2586        0 393216       10             0 at-spi2-registr [480156.950361] [  10553]     0 10551    10543        0 458752        9             0 QXcbEventQueue [480156.950365] [  10867]     0 10567    17579        0 589824       16             0 QXcbEventQueue [480156.950370] [  10928]     0 10921     6999        0 458752       17             0 QXcbEventQueue [480156.950390] [  11882]     0 11811     7377        0 458752       10             0 QXcbEventQueue [480156.950394] [  12052]     0 12052     5823        0 458752       21             0 fcitx [480156.950404] [  12115]     0 12114    11678        0 524288       21             0 QXcbEventQueue [480156.950408] [ 101558]     0 101558     3549        0 393216        0             0 runltp [480156.950486] [1068864]     0 1068864      771        6 327680       85         -1000 systemd-udevd [480156.950552] [1035639]     0 1035639       52        0 393216       14         -1000 oom02 [480156.950556] [1035640]     0 1035640       52        0 393216       23         -1000 oom02 [480156.950561] [1036065]     0 1036065      493       60 393216        0          -250 systemd-journal [480156.950565] [1036087]     0 1036073  6258739  3543942 37814272        0             0 oom02 [480156.950572] Out of memory and no killable processes... [480156.950575] Kernel panic - not syncing: System is deadlocked on memory ``` oom02-1036073 has been already killed before crash. log: ``` [480152.242506] [1035177]     0 1035177     4773       20 393216      115             0 sssd_nss [480152.242510] [1035376]     0 1035376    25500      391 589824      602             0 tuned [480152.242514] [1035639]     0 1035639       52        0 393216       14         -1000 oom02 [480152.242517] [1035640]     0 1035640       52        0 393216       19         -1000 oom02 [480152.242522] [1036065]     0 1036065      493      114 393216       62          -250 systemd-journal [480152.242525] [1036073]     0 1036073  6258739  3540314 37814272      104             0 oom02 [480152.242529] Out of memory: Kill process 1036073 (oom02) score 755 or sacrifice child [480152.243869] Killed process 1036073 (oom02) total-vm:400559296kB, anon-rss:226578368kB, file-rss:1728kB, shmem-rss:0kB [480152.365804] oom_reaper: reaped process 1036073 (oom02), now anon-rss:226594048kB, file-rss:0kB, shmem-rss:0kB ``` but its memory can not be reclaimed.I add trace-log to oom_reaper code in kernel, I found that there is a large range vma in the memory that cannot be reclaimed, and the vma has the  `VM_LOCKED` flag, so cannot be reclaimed immediately. ```log       oom_reaper-57    [007] ....   126.063581: __oom_reap_task_mm: gh: vma is anon:1048691, range=65536       oom_reaper-57    [007] ....   126.063581: __oom_reap_task_mm: gh: vma is anon:1048691, range=196608       oom_reaper-57    [007] ....   126.063582: __oom_reap_task_mm: gh: vma continue: 1056883, range:3221225472       oom_reaper-57    [007] ....   126.063583: __oom_reap_task_mm: gh: vma is anon:112, range=65536       oom_reaper-57    [007] ....   126.063584: __oom_reap_task_mm: gh: vma is anon:1048691, range=8388608 ``` `vma continue: 1056883, range:3221225472` is the memory that can not reclaims. 1057883(0x102073) is vma->vm_flags, it has VM_LOCKED` flag oom02 created `nr_cpu` threads and used mmap to allocate memory. mmap will merge continuous vma into one, so as long as one thread is still running, the entire vma will not be released. In extreme cases, crashes may occur due to the lack of memory reclamation. I'm not sure if this is a kernel's bug ? -- thanks, Gou Hao