From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A6A2C07E94 for ; Fri, 4 Jun 2021 11:37:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 73A816141C for ; Fri, 4 Jun 2021 11:37:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 73A816141C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=molgen.mpg.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E89806B0036; Fri, 4 Jun 2021 07:37:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E11D16B006C; Fri, 4 Jun 2021 07:37:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB21D6B006E; Fri, 4 Jun 2021 07:37:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 968C16B006C for ; Fri, 4 Jun 2021 07:37:07 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 38F34F04C for ; Fri, 4 Jun 2021 11:37:07 +0000 (UTC) X-FDA: 78215840094.22.469EBC1 Received: from mx1.molgen.mpg.de (mx3.molgen.mpg.de [141.14.17.11]) by imf27.hostedemail.com (Postfix) with ESMTP id 8EF8A8019893 for ; Fri, 4 Jun 2021 11:36:45 +0000 (UTC) Received: from [192.168.0.3] (ip5f5ae886.dynamic.kabel-deutschland.de [95.90.232.134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id 9623961E646D4; Fri, 4 Jun 2021 13:36:41 +0200 (CEST) To: Andrew Morton Cc: linux-mm@kvack.org, it+linux-mm@molgen.mpg.de From: Paul Menzel Subject: Cannot allocate memory despite buff/cache non-zero Message-ID: Date: Fri, 4 Jun 2021 13:36:41 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------DD9ED5143E034E68FC600E4D" Content-Language: en-US X-Stat-Signature: 93z5fq73no3i1wpjuk5co4ysh7mqacan X-Rspamd-Queue-Id: 8EF8A8019893 X-Rspamd-Server: rspam02 Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf27.hostedemail.com: domain of pmenzel@molgen.mpg.de designates 141.14.17.11 as permitted sender) smtp.mailfrom=pmenzel@molgen.mpg.de X-HE-Tag: 1622806605-819961 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a multi-part message in MIME format. --------------DD9ED5143E034E68FC600E4D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Dear Linux folks, On a 1 TB RAM compute server with Linux 5.10.24 and memory=20 overcommitting disabled, we ran into a situation where processes like=20 SSH couldn=E2=80=99t allocate memory anymore. $ more /proc/cmdline BOOT_IMAGE=3D/boot/bzImage-5.10.24.mx64.375 root=3DLABEL=3Droot ro=20 crashkernel=3D256M console=3DttyS0,115200n8 console=3Dtty0 init=3D/bin/sy= stemd=20 audit=3D0 random.trust_cpu=3Don 2021-06-03T22:00:28+02:00 godsavethequeen sshd[89163]:=20 pam_systemd(sshd:session): Failed to create session: Unit=20 session-25654.scope not found. 2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]: error:=20 do_exec_no_pty: fork: Cannot allocate memory 2021-06-03T22:00:29+02:00 godsavethequeen sshd[89163]:=20 pam_unix(sshd:session): session closed for user root 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error: fork:=20 Cannot allocate memory 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:=20 ssh_msg_send: write: Broken pipe 2021-06-03T22:01:41+02:00 godsavethequeen sshd[1834]: error:=20 send_rexec_state: ssh_msg_send failed $ free -h total used free shared buff/cache=20 available Mem: 1.0T 606G 2.6G 2.2M 395G=20 391G Swap: 0B 0B 0B Looking at this, I would have expected, that the pages(?) in buff/cache=20 would be moved/deleted to make memory available. Looking at `/proc/meminfo` (attached): MemTotal: 1052411824 kB MemFree: 2709976 kB MemAvailable: 410847908 kB [=E2=80=A6] CommitLimit: 1052411824 kB Committed_AS: 1052455260 kB [=E2=80=A6] Committed_AS is greater than the commit limit (total memory). Is such behavior expected? Kind regards, Paul --------------DD9ED5143E034E68FC600E4D Content-Type: text/plain; charset=UTF-8; name="20210603-godsavethequeen-meminfo.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="20210603-godsavethequeen-meminfo.txt" TWVtVG90YWw6ICAgICAgIDEwNTI0MTE4MjQga0IKTWVtRnJlZTogICAgICAgICAyNzA5OTc2 IGtCCk1lbUF2YWlsYWJsZTogICA0MTA4NDc5MDgga0IKQnVmZmVyczogICAgICAgICAgICAz MjEyIGtCCkNhY2hlZDogICAgICAgICA0MTEwODM3ODgga0IKU3dhcENhY2hlZDogICAgICAg ICAgICAwIGtCCkFjdGl2ZTogICAgICAgICAzMDMxNzU4MjQga0IKSW5hY3RpdmU6ICAgICAg IDc0MDA4MDEwMCBrQgpBY3RpdmUoYW5vbik6ICAgICAgIDE0NDgga0IKSW5hY3RpdmUoYW5v bik6IDYzMjE2OTcyNCBrQgpBY3RpdmUoZmlsZSk6ICAgMzAzMTc0Mzc2IGtCCkluYWN0aXZl KGZpbGUpOiAxMDc5MTAzNzYga0IKVW5ldmljdGFibGU6ICAgICAgICAgICAwIGtCCk1sb2Nr ZWQ6ICAgICAgICAgICAgICAgMCBrQgpTd2FwVG90YWw6ICAgICAgICAgICAgIDAga0IKU3dh cEZyZWU6ICAgICAgICAgICAgICAwIGtCCkRpcnR5OiAgICAgICAgICAgICAgICAgMCBrQgpX cml0ZWJhY2s6ICAgICAgICAgICAgIDAga0IKQW5vblBhZ2VzOiAgICAgIDYzMjE2MTk0OCBr QgpNYXBwZWQ6ICAgICAgICAgICA1MDg4ODAga0IKU2htZW06ICAgICAgICAgICAgICAyMjQ4 IGtCCktSZWNsYWltYWJsZTogICAgMjM1NzQxNiBrQgpTbGFiOiAgICAgICAgICAgIDMxMTky NDAga0IKU1JlY2xhaW1hYmxlOiAgICAyMzU3NDE2IGtCClNVbnJlY2xhaW06ICAgICAgIDc2 MTgyNCBrQgpLZXJuZWxTdGFjazogICAgICAgNTA1MTIga0IKUGFnZVRhYmxlczogICAgICAx NzE1MzcyIGtCCk5GU19VbnN0YWJsZTogICAgICAgICAgMCBrQgpCb3VuY2U6ICAgICAgICAg ICAgICAgIDAga0IKV3JpdGViYWNrVG1wOiAgICAgICAgICAwIGtCCkNvbW1pdExpbWl0OiAg ICAxMDUyNDExODI0IGtCCkNvbW1pdHRlZF9BUzogICAxMDUyNDU1MjYwIGtCClZtYWxsb2NU b3RhbDogICAzNDM1OTczODM2NyBrQgpWbWFsbG9jVXNlZDogICAgICA0ODA1MDgga0IKVm1h bGxvY0NodW5rOiAgICAgICAgICAwIGtCClBlcmNwdTogICAgICAgICAgIDQ2MDgwMCBrQgpB bm9uSHVnZVBhZ2VzOiAgNTA5Mzg0NzA0IGtCClNobWVtSHVnZVBhZ2VzOiAgICAgICAgMCBr QgpTaG1lbVBtZE1hcHBlZDogICAgICAgIDAga0IKRmlsZUh1Z2VQYWdlczogICAgICAgICAw IGtCCkZpbGVQbWRNYXBwZWQ6ICAgICAgICAgMCBrQgpIdWdlUGFnZXNfVG90YWw6ICAgICAg IDAKSHVnZVBhZ2VzX0ZyZWU6ICAgICAgICAwCkh1Z2VQYWdlc19Sc3ZkOiAgICAgICAgMApI dWdlUGFnZXNfU3VycDogICAgICAgIDAKSHVnZXBhZ2VzaXplOiAgICAgICAyMDQ4IGtCCkh1 Z2V0bGI6ICAgICAgICAgICAgICAgMCBrQgpEaXJlY3RNYXA0azogICAgIDU5OTUzNDQga0IK RGlyZWN0TWFwMk06ICAgIDQ4MzU5MDE0NCBrQgpEaXJlY3RNYXAxRzogICAgNTc5ODYyNTI4 IGtCCg== --------------DD9ED5143E034E68FC600E4D--