From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2183C47258 for ; Fri, 12 Jan 2024 19:04:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B88B6B0083; Fri, 12 Jan 2024 14:04:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 068586B0085; Fri, 12 Jan 2024 14:04:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E726A6B0087; Fri, 12 Jan 2024 14:04:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D48B46B0083 for ; Fri, 12 Jan 2024 14:04:21 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 73E59160E40 for ; Fri, 12 Jan 2024 19:04:21 +0000 (UTC) X-FDA: 81671584722.28.D9706CF Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf04.hostedemail.com (Postfix) with ESMTP id DBC8D40021 for ; Fri, 12 Jan 2024 19:04:18 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ATArYHx0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of shakeelb@google.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705086258; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eeDNfRNz+QBqCGpkSmD12YuuNM7kkG4W5cbld1c18MA=; b=yLHRS+yEwz26RgntItsx8RgRaYk11bvxcujQ7W67W5AjGadBLxGgynlcHGOklkcyWTvm4b /HPXvu0QIWQmPlW/2SjoUCkbETYjPXiFYlJcmClsi667dx81J2PwXhOpzLqOqHYV4CUCEt cM+Nc0zwzbNIwPztRdd3yKGI9oi1AWw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ATArYHx0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of shakeelb@google.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=shakeelb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705086258; a=rsa-sha256; cv=none; b=4BElct7Oc6mApDSWWY15ULsKt15rTmF41QnB27DaxVL+7xxybpjf1dKvC3WA73OyxOFq2o FIRvNN8Wy8UC74Gzw2W5xGzh9wriHp1Cn/X4Qnk+CqtQlwLD5rvto5+I9ZBQfZCm2iER2p RIJson8rSYVd64ZIuyK+TbV17NFbA4w= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-1d47fae33e0so14745ad.0 for ; Fri, 12 Jan 2024 11:04:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1705086258; x=1705691058; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=eeDNfRNz+QBqCGpkSmD12YuuNM7kkG4W5cbld1c18MA=; b=ATArYHx0zkFDwTUVbmu1RoVk8JM6O/6ON/twfUtNukqi8s6WS/uYC58b3EO6R/BXCs 04O/+jyYukInJFatmJw+jwqGoY+lVsBR76twOguBecKytvXclqQXhSs+vE9OHeRchAfm ve7o2MVty2WG+lns3OwmknIlTd+9NMysaejjHJYMXmfDygeoDyRPbjHwHA2uui4gr3Dn YIkC4RNVC29m/xhveRAb5imTwnTzs3DjggeOvRE6JvrFc1DEorq1bVicpU1LYBseyReH jGkHpVkRCk1l3aH3Z5MlOwThCrSSuGJKtV8ZWiIocwj//fc5mYapzMvofr43/fOTIL+o RxLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705086258; x=1705691058; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eeDNfRNz+QBqCGpkSmD12YuuNM7kkG4W5cbld1c18MA=; b=NVtOof7yUzMxxig9mDZqmyAEJ03qlYppFJTo8z0HYvzueQez7hEIQ+GAEb3/F6YyE0 4RT+xXLRMaA4agYRoaU40FTlMb3Jx9kvRScAZEZa4GlLMM2wZEYCafNQwTL60T7LuUXt GGafHvsySzo0P1Kvqit2hQaR4AT6AP4WD6iGbkQSCVyrB5QMaTcrtcF+6uOgia73gj+t rd2uZRJX05uEGANhDabITehmW7Hgp8+LPYxFPXekBdHln86tpxdnWTTS1Jz3PLS3DygG l8wVQPpGNHx10iN6375yze9XvhSsned+u56iMoLQy/jHOikLufwSwTk8537M/M0WbRt7 gcTg== X-Gm-Message-State: AOJu0YxCHG79lITJwaTvH4hH2x6tMvgNKzB1rSjMGctXwb5Fw+XDheal UCBPovGlYtg04QthlHizlqqDTCch5hHeHWu5BUkpGQL11TCB X-Google-Smtp-Source: AGHT+IHmDtIiBSAvIHSBY0fw1ypkyv9G7sKc3LzyfT2UTtNc9WEqtghWjTCXS9f8DCtoFJq1xC6QuTVb3XSU99Lf5Rw= X-Received: by 2002:a17:902:d4c5:b0:1d0:7052:aa20 with SMTP id o5-20020a170902d4c500b001d07052aa20mr451697plg.12.1705086257476; Fri, 12 Jan 2024 11:04:17 -0800 (PST) MIME-Version: 1.0 References: <20240111132902.389862-1-hannes@cmpxchg.org> <20240111192807.GA424308@cmpxchg.org> In-Reply-To: <20240111192807.GA424308@cmpxchg.org> From: Shakeel Butt Date: Fri, 12 Jan 2024 11:04:06 -0800 Message-ID: Subject: Re: [PATCH] mm: memcontrol: don't throttle dying tasks on memory.high To: Johannes Weiner Cc: Roman Gushchin , Andrew Morton , Michal Hocko , Muchun Song , Tejun Heo , Dan Schatzberg , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: 3uo1mqnkhygt7pnycdxh4j73d37cwbpp X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DBC8D40021 X-HE-Tag: 1705086258-401537 X-HE-Meta: U2FsdGVkX19ibRzJCmf98+PvdKemM5t+4+LgCTOlQuKerjVrvjnPBMa2OH7213FpmfrmESZZ6dsNSaVbY5SSCSkX+fF4EUZL7U9eIGpRQmE58Hf1IDDdiWDDHzz00SpwNmjxrT2hXu6XK5Pj6SKI/kMZXbkidGYnM6hXUN+I3wfLD1IpKMDPxDbczyI1MqgBpZEPTbjj6N/LDrdEJomuEM0cMCrRxduIFBN1kzU2QZnH2p7ULrc7TiYRjEAtHJUSJmPAsaNAQXwLvcvQCUSgxcSTQFQFuDfk+SfgToA6axZzGrv5WrO2ZL6W1TKnNHuHrSmM8gMPQSn9qHKuc225eeGHnYUwxf7cqNGnnMocpqp5tTnmiXlIvDuPR7uRg+tS+3Nha1TDJo8aNKbrwSH15XDBM3YzHz7jxS0cXiW8TzWTM0MF5ObzMxhvv3r9bA/DKC4gPgWxryCjoCTAiZORvT2ISoCv8R+7wS3AcCzI9+O1Sq/wZjNNJ9E0V5hqgesLrsFyYGAGVDOsgyAB8rpit7ZED3dIhU3N7RJ1nZCzywybaiAjKOU5lpHTc//u1fgg4sEKjiTwN8n29gQOLwOxYArMczHnOqh5WolZPfNLeTnvWZ+yNCC5CW3cqXOkphqOYEUTTtdxMeONi8BCPtOCkj7yjAkJgDwMlL6Na7KCssyvzlnlKGDyZb5Njr87UzcEMYsJ/tLkDJHvTMaq5Gbbr5Owm4dFBB+q59tovqTtFexIRDdQGpBOpj5Czn+0HsuI76bnfWVttcG6K0U24DCvhvJKo5GF12o6fvAFogfLch2Xz2mm38emTqvJic0BKhthyoYNGCUulseLOukReXRhZ6e+Pweo/YbDrxHxKaYu6Livmsn3TnDOfL9uoKC5l0AEAcauFeNaEFQBjRazOe1O5W+1V5/QM/s5vKv8ndn0piP5aF7L1HsAKvr143oompyfyLPS9MgYRZBitumLU2b KxIKELc1 tXomcp8rbuRqELciRSupTpBwDWzQ1vxyx4NE86lyIqbP8WFXWnjnXjDHuALtpbwtPjWv7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Jan 11, 2024 at 11:28=E2=80=AFAM Johannes Weiner wrote: > [...] > > From 6124a13cb073f5ff06b9c1309505bc937d65d6e5 Mon Sep 17 00:00:00 2001 > From: Johannes Weiner > Date: Thu, 11 Jan 2024 07:18:47 -0500 > Subject: [PATCH] mm: memcontrol: don't throttle dying tasks on memory.hig= h > > While investigating hosts with high cgroup memory pressures, Tejun > found culprit zombie tasks that had were holding on to a lot of > memory, had SIGKILL pending, but were stuck in memory.high reclaim. > > In the past, we used to always force-charge allocations from tasks > that were exiting in order to accelerate them dying and freeing up > their rss. This changed for memory.max in a4ebf1b6ca1e ("memcg: > prohibit unconditional exceeding the limit of dying tasks"); it noted > that this can cause (userspace inducable) containment failures, so it > added a mandatory reclaim and OOM kill cycle before forcing charges. > At the time, memory.high enforcement was handled in the userspace > return path, which isn't reached by dying tasks, and so memory.high > was still never enforced by dying tasks. > > When c9afe31ec443 ("memcg: synchronously enforce memory.high for large > overcharges") added synchronous reclaim for memory.high, it added > unconditional memory.high enforcement for dying tasks as well. The > callstack shows that this path is where the zombie is stuck in. > > We need to accelerate dying tasks getting past memory.high, but we > cannot do it quite the same way as we do for memory.max: memory.max is > enforced strictly, and tasks aren't allowed to move past it without > FIRST reclaiming and OOM killing if necessary. This ensures very small > levels of excess. With memory.high, though, enforcement happens lazily > after the charge, and OOM killing is never triggered. A lot of > concurrent threads could have pushed, or could actively be pushing, > the cgroup into excess. The dying task will enter reclaim on every > allocation attempt, with little hope of restoring balance. > > To fix this, skip synchronous memory.high enforcement on dying tasks > altogether again. Update memory.high path documentation while at it. > > Fixes: c9afe31ec443 ("memcg: synchronously enforce memory.high for large = overcharges") > Reported-by: Tejun Heo > Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt I am wondering if you have seen or suspected a similar issue but for remote memcg charging. For example pageout on a global reclaim which has to allocate buffers for some other memcg.