From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BAC8C4332F for ; Mon, 28 Nov 2022 18:07:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D55E86B0073; Mon, 28 Nov 2022 13:07:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D05E96B0074; Mon, 28 Nov 2022 13:07:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCEBD6B0075; Mon, 28 Nov 2022 13:07:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A9FCA6B0073 for ; Mon, 28 Nov 2022 13:07:32 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E61E71207EB for ; Mon, 28 Nov 2022 18:07:31 +0000 (UTC) X-FDA: 80183633502.23.89DB51D Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) by imf01.hostedemail.com (Postfix) with ESMTP id 1A0A940023 for ; Mon, 28 Nov 2022 18:07:27 +0000 (UTC) Received: by mail-qv1-f42.google.com with SMTP id i12so7951144qvs.2 for ; Mon, 28 Nov 2022 10:07:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=qOn1PUQxMJCGp3yaVSqPelmstrqLulCIcemPhpNVzdU=; b=4tQKhAmlgqJFeuomcyGn+GyBJTF3TXNZND3NJQrXoNmD8007Bxyc2BZx8oFt4hfIHz DcHStdfL2bgNWjZTK0ib7Ohf9zqFJ+82s5fozbOWW3QWW289UEPRstHix3b0fcK1+fOF ud9ZTqyADxWT2iqEUic/48O1fS33VMxPFrU3i1iO79Mwue4ofOjSA5Yhg7qSYgKtqIYq c0XG+rJLlfayL++XpOTdm2AUjwy3zr8n5NSnhkwYqQPKPXDvpOjCuAvDzEVYTP13jtUi c80FdjP5USS2wz5z/YixX3LluJBQGexKDWu0K9Qraubay+EYw+V5uZ1Iy3idEWE+bCZM Fl8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qOn1PUQxMJCGp3yaVSqPelmstrqLulCIcemPhpNVzdU=; b=TZ1PxfkZ6zJYZW+UIr1UJ5Nor8N+dDU+JdXPd7JKTsgNcurpgC2rnv8xF2CluIVlQE 9z33aCr1UaZAy8aEYt8TxpbrzLKzAOJ8WpNkHejKM+R2nVuT4cex4x6LfWHM+8c34xGE KC5bgVTWt3A0MhRghEL/vnEDPgUngOqv/9rU67U6j4Y9B2npieMcFXCORQvvnS97s2/B KYOBnSrtH+o2HHldN5FxaZRIT4Ad7OPJeh06VLnSPNjfWuH5W2RCeLSqFspqrf//o3xa 9WjncmsCsldfX56hPI1XQr5sUo9ho7e/gVfAbJJTAm9dULIN+sHTGAMC7MjHgAijiUO4 MLgw== X-Gm-Message-State: ANoB5pmDXhntpdnZKfUjnZci2von6Ox1fHsy+HPb7Cwcf/Gv2q+5UyFd uHo8TG2r7lSsLjRU+1+y08WN4Q== X-Google-Smtp-Source: AA0mqf6DaYR4aiOKw2PMYmb+XewvGkHvB6VF27wWN2x2o6DHCI1w5vwLSnOS5O+2XZOdfSZK+r2GVg== X-Received: by 2002:a0c:ff28:0:b0:4b8:6953:aed6 with SMTP id x8-20020a0cff28000000b004b86953aed6mr34287361qvt.47.1669658847054; Mon, 28 Nov 2022 10:07:27 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-9175-2920-760a-79fa.res6.spectrum.com. [2603:7000:c01:2716:9175:2920:760a:79fa]) by smtp.gmail.com with ESMTPSA id u19-20020a37ab13000000b006e99290e83fsm8607554qke.107.2022.11.28.10.07.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:07:26 -0800 (PST) Date: Mon, 28 Nov 2022 13:07:25 -0500 From: Johannes Weiner To: Ivan Babrou Cc: Linux MM , Linux Kernel Network Developers , linux-kernel , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Eric Dumazet , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Subject: Re: Low TCP throughput due to vmpressure with swap enabled Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669658848; a=rsa-sha256; cv=none; b=nmS6bRrn8BlCBzAxhE3MikiyNKwGHQ3M0u3Vn8bVcGzt68bbLPLoUjXZ4D7bYuo3TspmrB leMPj9fCECYxiQc83vbaTeD5MYQ5uF9BPfmFJMpHBc/6wRK6E3J8ncwmQD6jsDEckro9Ae ytzuKz71nwXbz4TKpfSyWwoUfxwoXpQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=4tQKhAml; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.42 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669658848; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qOn1PUQxMJCGp3yaVSqPelmstrqLulCIcemPhpNVzdU=; b=5oOwpbXxz17+M6zfnKnZk7DXDCK7UDk8spjXo81pNrM7sSt35Nh65hnpZafmOMWOKwRS4N n2DzwQFy1ffQ6B6iKwP0bGE3/z+MOA94qdcCdJTr/EhYquu1NlyeVjFOyheJt3hXie9H7N esYLOzXvT/bAfyH35S7tocy0gJ4r9jU= X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=4tQKhAml; spf=pass (imf01.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.42 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org X-Stat-Signature: bxkr4t76mip1tfy31q7t83bnumdd9ith X-Rspamd-Queue-Id: 1A0A940023 X-Rspamd-Server: rspam12 X-HE-Tag: 1669658847-644809 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Nov 22, 2022 at 05:28:24PM -0800, Ivan Babrou wrote: > On Tue, Nov 22, 2022 at 2:11 PM Ivan Babrou wrote: > > > > On Tue, Nov 22, 2022 at 12:05 PM Johannes Weiner wrote: > > > > > > On Mon, Nov 21, 2022 at 04:53:43PM -0800, Ivan Babrou wrote: > > > > Hello, > > > > > > > > We have observed a negative TCP throughput behavior from the following commit: > > > > > > > > * 8e8ae645249b mm: memcontrol: hook up vmpressure to socket pressure > > > > > > > > It landed back in 2016 in v4.5, so it's not exactly a new issue. > > > > > > > > The crux of the issue is that in some cases with swap present the > > > > workload can be unfairly throttled in terms of TCP throughput. > > > > > > Thanks for the detailed analysis, Ivan. > > > > > > Originally, we pushed back on sockets only when regular page reclaim > > > had completely failed and we were about to OOM. This patch was an > > > attempt to be smarter about it and equalize pressure more smoothly > > > between socket memory, file cache, anonymous pages. > > > > > > After a recent discussion with Shakeel, I'm no longer quite sure the > > > kernel is the right place to attempt this sort of balancing. It kind > > > of depends on the workload which type of memory is more imporant. And > > > your report shows that vmpressure is a flawed mechanism to implement > > > this, anyway. > > > > > > So I'm thinking we should delete the vmpressure thing, and go back to > > > socket throttling only if an OOM is imminent. This is in line with > > > what we do at the system level: sockets get throttled only after > > > reclaim fails and we hit hard limits. It's then up to the users and > > > sysadmin to allocate a reasonable amount of buffers given the overall > > > memory budget. > > > > > > Cgroup accounting, limiting and OOM enforcement is still there for the > > > socket buffers, so misbehaving groups will be contained either way. > > > > > > What do you think? Something like the below patch? > > > > The idea sounds very reasonable to me. I can't really speak for the > > patch contents with any sort of authority, but it looks ok to my > > non-expert eyes. > > > > There were some conflicts when cherry-picking this into v5.15. I think > > the only real one was for the "!sc->proactive" condition not being > > present there. For the rest I just accepted the incoming change. > > > > I'm going to be away from my work computer until December 5th, but > > I'll try to expedite my backported patch to a production machine today > > to confirm that it makes the difference. If I can get some approvals > > on my internal PRs, I should be able to provide the results by EOD > > tomorrow. > > I tried the patch and something isn't right here. Thanks for giving it a sping. > With the patch applied I'm capped at ~120MB/s, which is a symptom of a > clamped window. > > I can't find any sockets with memcg->socket_pressure = 1, but at the > same time I only see the following rcv_ssthresh assigned to sockets: Hm, I don't see how socket accounting would alter the network behavior other than through socket_pressure=1. How do you look for that flag? If you haven't yet done something comparable, can you try with tracing to rule out sampling errors? diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 066166aebbef..134b623bee6a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7211,6 +7211,7 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, goto success; } memcg->socket_pressure = 1; + trace_printk("skmem charge failed nr_pages=%u gfp=%pGg\n", nr_pages, &gfp_mask); if (gfp_mask & __GFP_NOFAIL) { try_charge(memcg, gfp_mask, nr_pages); goto success;