From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 422CEC3DA64 for ; Thu, 1 Aug 2024 04:54:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BCE016B007B; Thu, 1 Aug 2024 00:54:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B57926B0082; Thu, 1 Aug 2024 00:54:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1E946B0083; Thu, 1 Aug 2024 00:54:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 8130D6B007B for ; Thu, 1 Aug 2024 00:54:52 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F38578086F for ; Thu, 1 Aug 2024 04:54:51 +0000 (UTC) X-FDA: 82402461582.16.C70A93F Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf11.hostedemail.com (Postfix) with ESMTP id 293594000D for ; Thu, 1 Aug 2024 04:54:49 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ESrnt2Fb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722488044; a=rsa-sha256; cv=none; b=3sz/rF1zkfstNO8Xo4y+TLSOqlxqP+n1fN+/EPiUMbDitnYa3ley0dA76T4EAtjqjPPYhJ 3hzsr5t/sT0eF0Gbo66nOy5mWtPT1EkxiRJ82cnEuvbySVNUimZFXV3rpZO+k0QDGqOfpW AUB1oPThZ3g2WMGuOJK5FDl/mivxO/4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ESrnt2Fb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf11.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=ioworker0@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722488044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=sGqa7xCvNcLqwg533sIrs11rG2Jt0FPP2Ws8eQJ23PA=; b=zK6e28AfjJ8m3K3F/7zh1UNt96a4YqU6cPSB4VnFATxZOJq0dQirMNMuU0A6PWr2cd/mHv ncPMh4+XgG+uuBfepPorcE/14O2d7srvp5PKPE10tBTLfh4Hxlo4aACUDcnzvugARCoMc8 OZGlFQXXT9whDFGecq91MxvmXhSVOfc= Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-6c5bcb8e8edso4951152a12.2 for ; Wed, 31 Jul 2024 21:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722488089; x=1723092889; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=sGqa7xCvNcLqwg533sIrs11rG2Jt0FPP2Ws8eQJ23PA=; b=ESrnt2Fbq3vGv9iaAIQwR5QfDbJ2Yz3eE5HX6OjgsOxP9V/iPj1wt5gfiZZnFCzbVO fDPaHvg5r0+1NycYqx955zyOz07z44eXehz4T+o+Oe+RiC1ljHON6AYGDwCLjAnZczW8 jHAHMrhQBkZ86GuZFEv7JFMshca8sAKS43503CTlfbwaeE5UopjhM1DbJFkL9lPFmhz0 guZIMS+Nbg0Tf761BYdINJD/lk856ioQGbSwzwN8jJ/hIM1jXcH/YHVhuoKtlEY+aR1F 3xzmKPDijhV4/X6mcs1WA1XpTvWegJttfUIKdpo/KXl5MTLkrY86TIrwnSJJqhZtCazc bn1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722488089; x=1723092889; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sGqa7xCvNcLqwg533sIrs11rG2Jt0FPP2Ws8eQJ23PA=; b=GjSjRN+gHBpYYV+tLkM2ievcwDeKeZEwFgLweDc4UzsRBvSe44yxI9RxJdYUgUnXDJ xTCl2RRlz0FylqCGcMzHAX38MUCh6LRDYqm6086cT5vyzPAmPEXJX1ib/H2K0Gh/tb8J q+i6hoxgCWfbVLFjkxeXleqSMjxmqnFzh9OXgUvoQZ7Lzcy1SGUxH6zrdvVKDrK+J1mi IyK7+4MetfCxmogNZ5FZ8CKAnDHH2pEJhCR6Rfbh5XXQv5lW/A8ZQqmWYq4unrwV0xA9 t+fd1yW9UOV/LOXoxIj9Zam90MH208/QD4BFl6rB9pyFL3uL2ShqafkN4S3YqiIi0eB7 g2RA== X-Forwarded-Encrypted: i=1; AJvYcCVhMb/wSu3ICMjyLIThBQInoF85ouzQ9IUUPJEnHZTfBQgESgsbN76W1yyDltcZ7lHmZINVGhOB87AB+RbVx8Tl6rg= X-Gm-Message-State: AOJu0YzwdSuq2Y7/EqS8/KakK4NCGvCEK9oqN6kjAvX2qC0ZUFq63x32 PtuvMLDfHC3FPDC3qPKuX0+y+iQ4HzxyaGA1fw9pfEk7xLiD6K4s X-Google-Smtp-Source: AGHT+IE9qi7gQazGX1QrRlMTjcR7jxSkAIG8xrW3RgBdWGxQ+Ko7k1VdDttRLTUsPb8iBba6+WwfSg== X-Received: by 2002:a05:6a21:458b:b0:1c3:ce0f:bfb2 with SMTP id adf61e73a8af0-1c68cf82b39mr1202292637.23.1722488088574; Wed, 31 Jul 2024 21:54:48 -0700 (PDT) Received: from EBJ9932692.tcent.cn ([43.129.202.66]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1fed7ff3d64sm128325595ad.299.2024.07.31.21.54.42 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 31 Jul 2024 21:54:48 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: 21cnbao@gmail.com, ryan.roberts@arm.com, david@redhat.com, shy828301@gmail.com, ziy@nvidia.com, libang.li@antgroup.com, baolin.wang@linux.alibaba.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Lance Yang Subject: [BUG] mm/cgroupv2: memory.min may lead to an OOM error Date: Thu, 1 Aug 2024 12:54:27 +0800 Message-ID: <20240801045430.48694-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.45.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 293594000D X-Stat-Signature: ud6djj8serzo63c9ydd3jwsfsoaymeud X-Rspam-User: X-HE-Tag: 1722488089-243968 X-HE-Meta: U2FsdGVkX1/WE9gR20YRTLFydikpKGbLD6cg/DmRweEwEwbh5Q/j5yIvZ5pdlFQAOBt91gjIhI3ByOHU4gX5PLDch9FFsDkzTQiEGnV0rQlCOywxyKx3ifRkvdtIG/rqiUnmlVl/R/tYgJ8UqT5nZ3GTtO/SrTQ83W9UteASQDk3q136d74IwbUbDoWnpCHwFoehMmDB/QHxNJuAGCSxu9NSHQEw3fGr5d/hsESfdgkZERGy1A8TJIc5mqkdUwf2WS7Hu9nFASU8Vl5iDtJMKf1vXAjaJ9A3xKGctDUGni2UmqCtuzvXlzk7UnqfE3marOpqv2dnhaD+xaa5/w7UsQx9n/w+zmDHPvsVb8dDen7e8hzshmpYfnoaOzZ8vy3/ZJW9/OTBpAZTy922vel5jPS6+QA5mJocY4S6VQ4aSaP6Ufgo128nr3aFfJKI3yF+qRc3G2X2ZZhbZ4NIY8E3ldvcOF655xHJZSQ4Z8zxDWqeyMsSH8/hAhaJM/MZijsouggiLjUI5ZglP+wncdWCFjEYGD1UqOwxr2IRIMgROqXGYsfMra6DUw38ylRNywRnT6WZ9sFAd4bc04FIL5iNuxNqquKlRe1YZR0tErYQCVaMpNs5Jk7g3nOPu5JJPYyaMVblAwaZ0nKVk0aaYBzr2bLodoNnyehvWSLvDEgiON+ysMIfZA0EI3Ywh3lOnp0G4IJJ5kqLJRRJG+anq5S/MEfmQoTZaq8DYvxt4pUEVf3DGsRenLIFldRyrx3itD0G3WjPAu+/DxW7OJQ1876h7sNOySHNAl+iry4GnM5MCsTEEqEILTX58htM17F/9D1ZIfmdv7Smee7ghWpXaZ9EYLzitYEIIr5uR3WKk4v4JxCEMwH6AUcuKWHbbrlYPteAc/W0lt3lgxlW7SyAUotp4dMHE5ivwusxfy5eBNWQKxgwJcCLliSj2C8grshib9d1Ah+YW7EJLiqvkl2cgJ2 k9MD+dvD tayMohaGSX7BodHwziES9EpymcadZ/4M2WKpzBYwWFq9m8ffyvEGVsiKppfs4Z0o0Dq53jxFooRegT4vKKDiCD5+jcsTjxHok8Kw+e7PTgtkGhU6bnKI4WbLmuhG3sLKjBbuS6sC9CtugWbsg1PkUdBBMcnY3TVW3na8MjO0Bgrsv69H3y+KszHeBv/V2Fo5nw2Ez9ye/CguDZxes/1KBtU5AalbKduf8uZVv3LH1UvJwVuI4zaroXWfeUhDiri6uCTyy/jmYhK8m6Q0zVlmLk14I79Bv8P+/h7rScmA/gvbxTKNEdtrDicbZwMc2O/aG1acIRIJYUQTWqSIRaxlqnSnbqUvwtBnoD5EcSu6MbqVhfrEfiOhERo+TzPWJnr5g1zBAXySFePNQPEE4rbgDtYTtf8F0TmmLpZMYKkkHYlZmGpD5ixmxwoPiiA7hNh9t0Saa X-Bogosity: Ham, tests=bogofilter, spamicity=0.185919, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi all, It's possible to encounter an OOM error if both parent and child cgroups are configured such that memory.min and memory.max are set to the same values, as is practice in Kubernetes. Hmm... I'm not sure that whether this behavior is a bug or an expected aspect of the kernel design. To reproduce the bug, we can follow these command-based steps: 1. Check Kernel Version and OS release: ``` $ uname -r 6.10.0-rc5+ $ cat /etc/os-release PRETTY_NAME="Ubuntu 24.04 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="" SUPPORT_URL="" BUG_REPORT_URL="" PRIVACY_POLICY_URL="" UBUNTU_CODENAME=noble LOGO=ubuntu-logo ``` 2. Navigate to the cgroup v2 filesystem, create a test cgroup, and set memory settings: ``` $ cd /sys/fs/cgroup/ $ stat -fc %T /sys/fs/cgroup cgroup2fs $ mkdir test $ echo "+memory" > cgroup.subtree_control $ mkdir test/test-child $ echo 1073741824 > memory.max $ echo 1073741824 > memory.min $ cat memory.max 1073741824 $ cat memory.min 1073741824 $ cat memory.low 0 $ cat memory.high max ``` 3. Set up and check memory settings in the child cgroup: ``` $ cd test-child $ echo 1073741824 > memory.max $ echo 1073741824 > memory.min $ cat memory.max 1073741824 $ cat memory.min 1073741824 $ cat memory.low 0 $ cat memory.high max ``` 4. Add process to the child cgroup and verify: ``` $ echo $$ > cgroup.procs $ cat cgroup.procs 1131 1320 $ ps -ef|grep 1131 root 1131 1014 0 10:45 pts/0 00:00:00 -bash root 1321 1131 99 11:06 pts/0 00:00:00 ps -ef root 1322 1131 0 11:06 pts/0 00:00:00 grep --color=auto 1131 ``` 5. Attempt to create a large file using dd and observe the process being killed: ``` $ dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200 Killed ``` 6. Check kernel messages related to the OOM event: ``` $ dmesg ... [ 1341.112388] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=/,mems_allowed=0,oom_memcg=/test,task_memcg=/test/test-child,task=dd,pid=1324,uid=0 [ 1341.112418] Memory cgroup out of memory: Killed process 1324 (dd) total-vm:15548kB, anon-rss:10240kB, file-rss:1764kB, shmem-rss:0kB, UID:0 pgtables:76kB oom_score_adj:0 ``` 7. Reduce the `memory.min` setting in the child cgroup and attempt the same large file creation, and then this issue is resolved. ``` # echo 107374182 > memory.min # dd if=/dev/zero of=/tmp/2gbfile bs=10M count=200 200+0 records in 200+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 1.8713 s, 1.1 GB/s ``` Thanks, Lance