From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB9DDCCA473 for ; Fri, 24 Jun 2022 08:04:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17ECB8E01EF; Fri, 24 Jun 2022 04:04:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 108678E01E7; Fri, 24 Jun 2022 04:04:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC3C08E01EF; Fri, 24 Jun 2022 04:04:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D6F438E01E7 for ; Fri, 24 Jun 2022 04:04:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id A235860C8C for ; Fri, 24 Jun 2022 08:04:49 +0000 (UTC) X-FDA: 79612393098.06.8A1F030 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf29.hostedemail.com (Postfix) with ESMTP id 530A5120024 for ; Fri, 24 Jun 2022 08:04:49 +0000 (UTC) Received: by mail-ed1-f41.google.com with SMTP id ej4so2305054edb.7 for ; Fri, 24 Jun 2022 01:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=LhrZ1UxjIpVC2XBdksQBTmD7BAYYNmGWhWbEjzEkz2c=; b=IT/2QY1S3wcM8FS/v0FfdNQ+N0HXcgQBGNh20MRJf8eVnqxcRW0P4dlyNLO+vVZ8g/ 6Q+9dkqSDlXiGXgDpZQ0Bpt/FxnTiQwhmLrYF2pPF4xCysoOJkrd6xdjVCkn+Uvy+ffB quJO025nVRDKuN9elrZ48yacmrwHG6j8fQ5ySbuBDjbc6A7DHjXpkjjTouvrDvdQmO8c ms0eTCTSdv2gdgpHBUxImXnUGSIc33A3tu+afJmcDei235oxk6m1CWDn/CTQ5PlKsxAC SPK5To3d1dYAUUDB5YpwfCNc5A+uFc9Q9nF0XhuNgx0cPr6s4OJUdGvNRl1Rtwlr7tA1 Xb0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=LhrZ1UxjIpVC2XBdksQBTmD7BAYYNmGWhWbEjzEkz2c=; b=yxhTdDN0VLbgJqU7RPDJbUSEeKpxq8l8UmjSvZGxdr/3/1rGMHIrUUsdlA0REaNiZm dzXcbgIkh1WB9lb8clr0cWBs5ZU8DMKkiKX3px6L/UQkyBdqNs6ug/BO4qi1nuGMFMyW sH7seYcXwq3ip/qi7ZFOtF85DfN6PAvY1gYVWnITYMfDHtZ4ClDsmbU9N3XwM482zm5U 9no7dIZbWa3YJMyziPMRA9+Mo5S337/c/tRO04e/4YBBOCDIeFLAaqAK8/0UlIzKXW3Z gvQtVPL1mavs1kAy9UJgeIw6HOlMQIzyhEpdIpi1/Xmj/piyc3zwMMDJbCYb68dvRGLD Y/lg== X-Gm-Message-State: AJIora9a5hFPOldk79BMYRsIJ6gdlWkeqs4Ic994H0GAX3faE3DMK53B dnfvPuOZhlIPbPK+ir3S3Ws= X-Google-Smtp-Source: AGRyM1vntYWPcpiFR2W5uxDMzVDamojnDeNXHGRVhdRGjoQWJzpXxx+4TF3fYwBFNAdDWENhklW23w== X-Received: by 2002:aa7:d29a:0:b0:435:705f:1319 with SMTP id w26-20020aa7d29a000000b00435705f1319mr15758286edq.54.1656057887845; Fri, 24 Jun 2022 01:04:47 -0700 (PDT) Received: from able.fritz.box (p57b0bd9f.dip0.t-ipconnect.de. [87.176.189.159]) by smtp.gmail.com with ESMTPSA id c19-20020a170906155300b006fea43db5c1sm697779ejd.21.2022.06.24.01.04.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 24 Jun 2022 01:04:47 -0700 (PDT) From: "=?UTF-8?q?Christian=20K=C3=B6nig?=" X-Google-Original-From: =?UTF-8?q?Christian=20K=C3=B6nig?= To: linux-media@vger.kernel.org, linux-kernel@vger.kernel.org, intel-gfx@lists.freedesktop.org, amd-gfx@lists.freedesktop.org, nouveau@lists.freedesktop.org, linux-tegra@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, dri-devel@lists.freedesktop.org Cc: mhocko@suse.com Subject: [RFC] Per file OOM-badness / RSS once more Date: Fri, 24 Jun 2022 10:04:30 +0200 Message-Id: <20220624080444.7619-1-christian.koenig@amd.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="IT/2QY1S"; spf=pass (imf29.hostedemail.com: domain of ckoenig.leichtzumerken@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ckoenig.leichtzumerken@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656057889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=LhrZ1UxjIpVC2XBdksQBTmD7BAYYNmGWhWbEjzEkz2c=; b=BjMYjBvi2BMCxN/ZIYWMPxr012JHH8P4GjOyjzl4Dr4QSfYKnyerNFwGDzaULk8YsWRLc3 qDu6Y8/N/tOLpcnc0/7ApdAb8h3F83cvdMP250Sij2337eB32LJgzEvGT4CLsGEPuqZjkm xqfRIjzGQcjKb5jMNOZCc7w35vmv8TI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656057889; a=rsa-sha256; cv=none; b=B+jwTbjU++92iVGO34eajLpTSI2pwORjssu1AiWLjzjIfd1xgsKecz6zaOCv9jFk+KNaYG qQSYhy1uzyDyTGCXQ0Q9n9LgQa7pdWkjp4ZQOOxAXp12/02+mtJG/b2/PPZ0ncYQwxz1gn 4iWaxvT4wdEbJuYEq0bbJmDnwf+PDtw= X-Stat-Signature: 781gng55p56t34mzn7t8stwe66o443fg X-Rspamd-Queue-Id: 530A5120024 X-Rspam-User: Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="IT/2QY1S"; spf=pass (imf29.hostedemail.com: domain of ckoenig.leichtzumerken@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=ckoenig.leichtzumerken@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam02 X-HE-Tag: 1656057889-513290 X-Bogosity: Ham, tests=bogofilter, spamicity=0.063237, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello everyone, To summarize the issue I'm trying to address here: Processes can allocate resources through a file descriptor without being held responsible for it. I'm not explaining all the details again. See here for a more deeply description of the problem: https://lwn.net/ml/linux-kernel/20220531100007.174649-1-christian.koenig@amd.com/ With this iteration I'm trying to address a bunch of the comments Michal Hocko (thanks a lot for that) gave as well as giving some new ideas. Changes made so far: 1. Renamed the callback into file_rss(). This is at least a start to better describe what this is all about. I've been going back and forth over the naming here, if you have any better idea please speak up. 2. Cleanups, e.g. now providing a helper function in the fs layer to sum up all the pages allocated by the files in a file descriptor table. 3. Using the actual number of allocated pages for the shmem implementation instead of just the size. I also tried to ignore shmem files which are part of tmpfs, cause that has a separate accounting/limitation approach. 4. The OOM killer now prints the memory of the killed process including the per file pages which makes the whole things much more comprehensible. 5. I've added the per file pages to the different reports in RSS in procfs. This has the interesting effect that tools like top suddenly give a much more accurate overview of the memory use as well. This of course increases the overhead of gathering those information quite a bit and I'm not sure how feasible that is for up-streaming. On the other hand this once more clearly shows that we need to do something about this issue. Another rather interesting observation is that multiple subsystems (shmem, tmpfs, ttm) came up with the same workaround of limiting the memory which can be allocated through them to 50% of the whole system memory. Unfortunately that isn't the same 50% and it doesn't apply everywhere, so you can still easily crash the box. Ideas and/or comments are really welcome. Thanks, Christian.