From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FD1DC433F5 for ; Mon, 18 Oct 2021 13:33:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0D61F61352 for ; Mon, 18 Oct 2021 13:33:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0D61F61352 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 821AD6B006C; Mon, 18 Oct 2021 09:33:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D193900002; Mon, 18 Oct 2021 09:33:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6BFBD6B0072; Mon, 18 Oct 2021 09:33:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 56CBA6B006C for ; Mon, 18 Oct 2021 09:33:32 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D891D1809AF1C for ; Mon, 18 Oct 2021 13:33:31 +0000 (UTC) X-FDA: 78709650222.10.323F806 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf05.hostedemail.com (Postfix) with ESMTP id 72BED5081DD7 for ; Mon, 18 Oct 2021 13:33:27 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 258F21FD6D; Mon, 18 Oct 2021 13:33:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1634564010; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YoCaAIU3gS5Go5DLCN1kv6dXHZiM0iQ6w6RrFGGICso=; b=XmgPTBlbQwJY3r+TZtpbKUocEfQeYD958gxtDPq6oJ1jojMVSpnjhZulUshuSUa25saqEP redaJ3+UFAYoa8FwoqWco0w/CkkO92hkKFZE8UtdZxPTx1LUvS1UkorkJ4jmDW3wvD3xrP MUYOayUfMVAdj3iQs/U9nXSIRCmILGY= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id 751B1A3BC8; Mon, 18 Oct 2021 13:33:29 +0000 (UTC) Date: Mon, 18 Oct 2021 15:33:25 +0200 From: Michal Hocko To: Mina Almasry Cc: Roman Gushchin , Shakeel Butt , Greg Thelen , Johannes Weiner , Hugh Dickins , Tejun Heo , Linux-MM , "open list:FILESYSTEMS (VFS and infrastructure)" , cgroups@vger.kernel.org, riel@surriel.com Subject: Re: [RFC Proposal] Deterministic memcg charging for shared memory Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 72BED5081DD7 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=XmgPTBlb; spf=pass (imf05.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com X-Stat-Signature: jnix6qsykm6wm6mm6rxhmhcyuhyye6gm X-Rspamd-Server: rspam05 X-HE-Tag: 1634564007-738430 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.023734, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 13-10-21 12:23:19, Mina Almasry wrote: > Below is a proposal for deterministic charging of shared memory. > Please take a look and let me know if there are any major concerns: >=20 > Problem: > Currently shared memory is charged to the memcg of the allocating > process. This makes memory usage of processes accessing shared memory > a bit unpredictable since whichever process accesses the memory first > will get charged. We have a number of use cases where our userspace > would like deterministic charging of shared memory: >=20 > 1. System services allocating memory for client jobs: > We have services (namely a network access service[1]) that provide > functionality for clients running on the machine and allocate memory > to carry out these services. The memory usage of these services > depends on the number of jobs running on the machine and the nature of > the requests made to the service, which makes the memory usage of > these services hard to predict and thus hard to limit via memory.max. > These system services would like a way to allocate memory and instruct > the kernel to charge this memory to the client=E2=80=99s memcg. >=20 > 2. Shared filesystem between subtasks of a large job > Our infrastructure has large meta jobs such as kubernetes which spawn > multiple subtasks which share a tmpfs mount. These jobs and its > subtasks use that tmpfs mount for various purposes such as data > sharing or persistent data between the subtask restarts. In kubernetes > terminology, the meta job is similar to pods and subtasks are > containers under pods. We want the shared memory to be > deterministically charged to the kubernetes's pod and independent to > the lifetime of containers under the pod. >=20 > 3. Shared libraries and language runtimes shared between independent jo= bs. > We=E2=80=99d like to optimize memory usage on the machine by sharing li= braries > and language runtimes of many of the processes running on our machines > in separate memcgs. This produces a side effect that one job may be > unlucky to be the first to access many of the libraries and may get > oom killed as all the cached files get charged to it. >=20 > Design: > My rough proposal to solve this problem is to simply add a > =E2=80=98memcg=3D/path/to/memcg=E2=80=99 mount option for filesystems (= namely tmpfs): > directing all the memory of the file system to be =E2=80=98remote charg= ed=E2=80=99 to > cgroup provided by that memcg=3D option. Could you be more specific about how this matches the above mentioned usecases? What would/should happen if the target memcg doesn't or stop existing under remote charger feet? > Caveats: > 1. One complication to address is the behavior when the target memcg > hits its memory.max limit because of remote charging. In this case the > oom-killer will be invoked, but the oom-killer may not find anything > to kill in the target memcg being charged. In this case, I propose > simply failing the remote charge which will cause the process > executing the remote charge to get an ENOMEM This will be documented > behavior of remote charging. Say you are in a page fault (#PF) path. If you just return ENOMEM then you will get a system wide OOM killer via pagefault_out_of_memory. This is very likely not something you want, right? Even if we remove this behavior, which is another story, then the #PF wouldn't have other ways than keep retrying which doesn't really look great either. The only "reasonable" way I can see right now is kill the remote charging task. That might result in some other problems though. > 2. I would like to provide an initial implementation that adds this > support for tmpfs, while leaving the implementation generic enough for > myself or others to extend to more filesystems where they find the > feature useful. How do you envision other filesystems would implement that? Should the information be persisted in some way? I didn't have time to give this a lot of thought and more questions will likely come. My initial reaction is that this will open a lot of interesting corner cases which will be hard to deal with. --=20 Michal Hocko SUSE Labs