From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F6B5C4332F for ; Wed, 23 Nov 2022 01:28:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF5066B0071; Tue, 22 Nov 2022 20:28:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA5486B0073; Tue, 22 Nov 2022 20:28:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6D0B8E0001; Tue, 22 Nov 2022 20:28:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 97C9C6B0071 for ; Tue, 22 Nov 2022 20:28:37 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6610B1403AA for ; Wed, 23 Nov 2022 01:28:37 +0000 (UTC) X-FDA: 80162972274.21.332B918 Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf12.hostedemail.com (Postfix) with ESMTP id 0730F4000B for ; Wed, 23 Nov 2022 01:28:35 +0000 (UTC) Received: by mail-yb1-f178.google.com with SMTP id d128so850115ybf.10 for ; Tue, 22 Nov 2022 17:28:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=M42sWyNVtSXf63+/nFxRglHm/7sJT4os0b4YITaV5ns=; b=EEDxxsC8axWN6EfK6HuKM4nKLyz6jyKaF6MQb03199u9fI9ncH2Pzi0Pkz2Rmfaln3 0LcVHqzeUlYiBsH/6c+Cd8NT5OeaxAlFYOn2KKOOs0oao+6e3/WwTuRD4LxatCOIGywm VjfLjjAHKCoOCfpD6aq+RuMP4XmMlWcW/Y98o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=M42sWyNVtSXf63+/nFxRglHm/7sJT4os0b4YITaV5ns=; b=jT3DcFLL/iSn0A9z8qAMYgMi0/HzObyE4+tP6WiGUflSKJlO4fDyA9gOBqb0gLLRJW UMa+t0UBEZ4G1tqowW81bRqf6DREDlAIureP/fIGYQtCwf6qhc3qwzvB+nOG8HUrN6RI c7gFahclniYr9bEH7UwA/uFDBuXz1daoqSyzNdatxQ8wZnS40m1b5kLYtEZWx9z2kMcQ Bv7D2mXqWTCNa29qoW9gZPu500wCGiiQ2ZqW2OPG/ntZNrzkk41RqHA+05H1NfMfC3Ap zo/Y3yo4Vpqa9Hmxi1zEl7VROJBfwcnYTuafVNdo8kXIZmgaT5T2KRj6i1bkZCqzCp8g PnKQ== X-Gm-Message-State: ANoB5pk+L4pQgT15vtS0GUyCL28dPY5bb5k+wbdR0RHColMTxb2VOWFv SG+fGP7DuuVaQOIqZAmXh5fOGEs305n1tbJTB4jYnw== X-Google-Smtp-Source: AA0mqf6JztjXsKrOZAvtzlYQyGBcMnqFAAI03f5vVpHLgvM2RaEycI/1qnyVsvINe/s4LO7o2SeAM+HqA9xXEvxUoQ4= X-Received: by 2002:a25:a0d4:0:b0:6ea:3fec:4adb with SMTP id i20-20020a25a0d4000000b006ea3fec4adbmr15397598ybm.305.1669166915244; Tue, 22 Nov 2022 17:28:35 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Ivan Babrou Date: Tue, 22 Nov 2022 17:28:24 -0800 Message-ID: Subject: Re: Low TCP throughput due to vmpressure with swap enabled To: Johannes Weiner Cc: Linux MM , Linux Kernel Network Developers , linux-kernel , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Eric Dumazet , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669166916; a=rsa-sha256; cv=none; b=E3wtVq3ZA5nS53Cc3ACLsH7b7WhbjdcagKlpr3UPBzkrNZR+Iw5srVf2875tcex+wDqQmr Daxn6aHqqobpZhkTyQ8JPGrk50H1gU/0OU7EjT1R8ohaORBx54xST7yF4l/n5KUGAaOD4N xEfXl2EzIgcYhlQDW/CLK+ScJuznNME= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=EEDxxsC8; spf=pass (imf12.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=ivan@cloudflare.com; dmarc=pass (policy=reject) header.from=cloudflare.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669166916; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=M42sWyNVtSXf63+/nFxRglHm/7sJT4os0b4YITaV5ns=; b=HLL2iSjGoBzqCbz7J6kS6H/KJq91GKlCw52JFaJWT/kBeDHT8xy+w74QaZEqqSxnr6gwx3 se8j6DD9p+XKah+z0360pKiLrTuga2BC7Rsjh9H/bR8vXR3V0PiFzLVStPAMcnM9wLF2jz QXXarZ2cR8aiZ/7TAZ44jIDVrYnN6qg= X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0730F4000B Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=EEDxxsC8; spf=pass (imf12.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=ivan@cloudflare.com; dmarc=pass (policy=reject) header.from=cloudflare.com X-Rspam-User: X-Stat-Signature: eojjuxe7913am4k9neh8cda8u1ngb9ya X-HE-Tag: 1669166915-451107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Nov 22, 2022 at 2:11 PM Ivan Babrou wrote: > > On Tue, Nov 22, 2022 at 12:05 PM Johannes Weiner wrote: > > > > On Mon, Nov 21, 2022 at 04:53:43PM -0800, Ivan Babrou wrote: > > > Hello, > > > > > > We have observed a negative TCP throughput behavior from the following commit: > > > > > > * 8e8ae645249b mm: memcontrol: hook up vmpressure to socket pressure > > > > > > It landed back in 2016 in v4.5, so it's not exactly a new issue. > > > > > > The crux of the issue is that in some cases with swap present the > > > workload can be unfairly throttled in terms of TCP throughput. > > > > Thanks for the detailed analysis, Ivan. > > > > Originally, we pushed back on sockets only when regular page reclaim > > had completely failed and we were about to OOM. This patch was an > > attempt to be smarter about it and equalize pressure more smoothly > > between socket memory, file cache, anonymous pages. > > > > After a recent discussion with Shakeel, I'm no longer quite sure the > > kernel is the right place to attempt this sort of balancing. It kind > > of depends on the workload which type of memory is more imporant. And > > your report shows that vmpressure is a flawed mechanism to implement > > this, anyway. > > > > So I'm thinking we should delete the vmpressure thing, and go back to > > socket throttling only if an OOM is imminent. This is in line with > > what we do at the system level: sockets get throttled only after > > reclaim fails and we hit hard limits. It's then up to the users and > > sysadmin to allocate a reasonable amount of buffers given the overall > > memory budget. > > > > Cgroup accounting, limiting and OOM enforcement is still there for the > > socket buffers, so misbehaving groups will be contained either way. > > > > What do you think? Something like the below patch? > > The idea sounds very reasonable to me. I can't really speak for the > patch contents with any sort of authority, but it looks ok to my > non-expert eyes. > > There were some conflicts when cherry-picking this into v5.15. I think > the only real one was for the "!sc->proactive" condition not being > present there. For the rest I just accepted the incoming change. > > I'm going to be away from my work computer until December 5th, but > I'll try to expedite my backported patch to a production machine today > to confirm that it makes the difference. If I can get some approvals > on my internal PRs, I should be able to provide the results by EOD > tomorrow. I tried the patch and something isn't right here. With the patch applied I'm capped at ~120MB/s, which is a symptom of a clamped window. I can't find any sockets with memcg->socket_pressure = 1, but at the same time I only see the following rcv_ssthresh assigned to sockets: $ sudo ss -tim dport 6443 | fgrep rcv_ssthresh | sed 's/.*rcv_ssthresh://' | awk '{ print $1 }' | sort -n | uniq -c | sort -n | tail 1 64076 181 65495 1456 5792 16531 64088 * 64088 is the default value * 5792 is 4 * advmss (clamped) Compare this to a machine without the patch but with cgroup.memory=nosocket in cmdline: $ sudo ss -tim dport 6443 | fgrep rcv_ssthresh | sed 's/.*rcv_ssthresh://' | awk '{ print $1 }' | sort -n | uniq -c | sort -n | tail 8 2806862 8 3777338 8 72776 8 86068 10 2024018 12 3777354 23 91172 29 66984 101 65495 5439 64088 There aren't any clamped sockets here and there are many different rcv_ssthresh values.