From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F9FCC4708E for ; Tue, 6 Dec 2022 00:51:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A89A88E0002; Mon, 5 Dec 2022 19:51:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3A1A8E0001; Mon, 5 Dec 2022 19:51:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9017F8E0002; Mon, 5 Dec 2022 19:51:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 7FDF68E0001 for ; Mon, 5 Dec 2022 19:51:01 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4C5E780323 for ; Tue, 6 Dec 2022 00:51:01 +0000 (UTC) X-FDA: 80210051922.23.14DAE3E Received: from mail-il1-f176.google.com (mail-il1-f176.google.com [209.85.166.176]) by imf01.hostedemail.com (Postfix) with ESMTP id 3ED314000A for ; Tue, 6 Dec 2022 00:50:58 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=EJHHbJiu; dmarc=pass (policy=reject) header.from=cloudflare.com; spf=pass (imf01.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.166.176 as permitted sender) smtp.mailfrom=ivan@cloudflare.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670287858; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t7EllS7EycwJQKQ7E4+tjHuL0D77O7aeDFmbgKj4v/I=; b=fE61R1lPXlPkG6C1i2mKKmuAPdQbHQMPqZ+Q0XQwl+JJBSWEUSKfcC352wq8ss3P8qJ6wT Ejh6i6YzB4WZPKab/y7OR1+NEhusMfi9JUXUFgmBzJZMjdgY/rMVL+f5UNHxhT3V79K//8 pxiNCu3QOhZKzn5daf6mMag5Nh9bwCo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=cloudflare.com header.s=google header.b=EJHHbJiu; dmarc=pass (policy=reject) header.from=cloudflare.com; spf=pass (imf01.hostedemail.com: domain of ivan@cloudflare.com designates 209.85.166.176 as permitted sender) smtp.mailfrom=ivan@cloudflare.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670287858; a=rsa-sha256; cv=none; b=1jJIPd6PSpqQjMrEXTWV2VhLIZ9LZARJV1iizSpWofSylimJRc6O/0XuwvlEW+I3Z7ZSGO f6vvVylQptqA/TCEdg06PWojoRU+bAAaBOWZo1lMIgw6TlMCUsYqrTSDJFVngwESmUgW3t uoTMuBQUffSbI2rkZc+o917Yz5t9no0= Received: by mail-il1-f176.google.com with SMTP id y2so4083931ily.5 for ; Mon, 05 Dec 2022 16:50:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloudflare.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=t7EllS7EycwJQKQ7E4+tjHuL0D77O7aeDFmbgKj4v/I=; b=EJHHbJiu3ulNrc8DJW8/qrbdCa4J7MJZMd/Wlf1oS4r/IF6DcNFguC+jw1hlPEuHSt 1fOLkP0UpyyFpW4Q9rsWCBOlASujjlxifFP4fjd0u2AmzLWhQNCpmJupByUKqb5zod0I Qtbp43Ngn3R8VJj6ir32hJg1qc9AIYu/Xw60E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=t7EllS7EycwJQKQ7E4+tjHuL0D77O7aeDFmbgKj4v/I=; b=XkDOYOQbfz1mT7OFE10C4lBSbGR8O1w/k/U5CluuSKkBkfmYC6eZqKgthOpaCNJh8R sKzIb0uo2da61CB1mBDUliWkmm/Ok983frzBTPDRqY+XtQ1Pr82O6XJ0+rpfMxhfzP81 y+ws7JCP6zqkYqrSKdmsK996sHXFJFWFPMEriger05nHebul+36S8vca0odj2AlMFCDP Vn3wHPkTQU2hPtOaklXIEHN1zGhJvT8k49qRm+AzRWFNJdYyGKErp7v7TPmMfbpOBakI N9S/r/0UqLemHF8vY4PpnXHwBlog85owWm+OXTOgvRgt6Y0z9CjLltdfxYu+qvk/7yBv yWrg== X-Gm-Message-State: ANoB5pnw9t5/+XptBFutT4L0MvtLvxzsE7KdnlDuMHIL7+UdV+2dLfZD 6N8Nh+FTqKCDdwb5qe6FmsSI4OEg7RagJpjmwDTJTg== X-Google-Smtp-Source: AA0mqf6xcsiga75ZuTjF0H61IWsLJpXbIrrZk8NtyASGXEnkKaFjtvYsf+OGcArSzU5Rc/FkigeEW6O9vU8MXXcRfxg= X-Received: by 2002:a92:db42:0:b0:2fa:b6c0:80fd with SMTP id w2-20020a92db42000000b002fab6c080fdmr14932425ilq.164.1670287857252; Mon, 05 Dec 2022 16:50:57 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Ivan Babrou Date: Mon, 5 Dec 2022 16:50:46 -0800 Message-ID: Subject: Re: Low TCP throughput due to vmpressure with swap enabled To: Johannes Weiner Cc: Linux MM , Linux Kernel Network Developers , linux-kernel , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Eric Dumazet , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 3ED314000A X-Stat-Signature: riou7urx8ha6rzm1zwursnst4adzand3 X-Spamd-Result: default: False [-2.90 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.166.176:from]; BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DMARC_POLICY_ALLOW(0.00)[cloudflare.com,reject]; RCPT_COUNT_TWELVE(0.00)[17]; DKIM_TRACE(0.00)[cloudflare.com:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; R_DKIM_ALLOW(0.00)[cloudflare.com:s=google]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[] X-HE-Tag: 1670287858-653255 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 5, 2022 at 3:57 PM Ivan Babrou wrote: > > On Mon, Nov 28, 2022 at 10:07 AM Johannes Weiner wrote: > > > > On Tue, Nov 22, 2022 at 05:28:24PM -0800, Ivan Babrou wrote: > > > On Tue, Nov 22, 2022 at 2:11 PM Ivan Babrou wrote: > > > > > > > > On Tue, Nov 22, 2022 at 12:05 PM Johannes Weiner wrote: > > > > > > > > > > On Mon, Nov 21, 2022 at 04:53:43PM -0800, Ivan Babrou wrote: > > > > > > Hello, > > > > > > > > > > > > We have observed a negative TCP throughput behavior from the following commit: > > > > > > > > > > > > * 8e8ae645249b mm: memcontrol: hook up vmpressure to socket pressure > > > > > > > > > > > > It landed back in 2016 in v4.5, so it's not exactly a new issue. > > > > > > > > > > > > The crux of the issue is that in some cases with swap present the > > > > > > workload can be unfairly throttled in terms of TCP throughput. > > > > > > > > > > Thanks for the detailed analysis, Ivan. > > > > > > > > > > Originally, we pushed back on sockets only when regular page reclaim > > > > > had completely failed and we were about to OOM. This patch was an > > > > > attempt to be smarter about it and equalize pressure more smoothly > > > > > between socket memory, file cache, anonymous pages. > > > > > > > > > > After a recent discussion with Shakeel, I'm no longer quite sure the > > > > > kernel is the right place to attempt this sort of balancing. It kind > > > > > of depends on the workload which type of memory is more imporant. And > > > > > your report shows that vmpressure is a flawed mechanism to implement > > > > > this, anyway. > > > > > > > > > > So I'm thinking we should delete the vmpressure thing, and go back to > > > > > socket throttling only if an OOM is imminent. This is in line with > > > > > what we do at the system level: sockets get throttled only after > > > > > reclaim fails and we hit hard limits. It's then up to the users and > > > > > sysadmin to allocate a reasonable amount of buffers given the overall > > > > > memory budget. > > > > > > > > > > Cgroup accounting, limiting and OOM enforcement is still there for the > > > > > socket buffers, so misbehaving groups will be contained either way. > > > > > > > > > > What do you think? Something like the below patch? > > > > > > > > The idea sounds very reasonable to me. I can't really speak for the > > > > patch contents with any sort of authority, but it looks ok to my > > > > non-expert eyes. > > > > > > > > There were some conflicts when cherry-picking this into v5.15. I think > > > > the only real one was for the "!sc->proactive" condition not being > > > > present there. For the rest I just accepted the incoming change. > > > > > > > > I'm going to be away from my work computer until December 5th, but > > > > I'll try to expedite my backported patch to a production machine today > > > > to confirm that it makes the difference. If I can get some approvals > > > > on my internal PRs, I should be able to provide the results by EOD > > > > tomorrow. > > > > > > I tried the patch and something isn't right here. > > > > Thanks for giving it a sping. > > > > > With the patch applied I'm capped at ~120MB/s, which is a symptom of a > > > clamped window. > > > > > > I can't find any sockets with memcg->socket_pressure = 1, but at the > > > same time I only see the following rcv_ssthresh assigned to sockets: > > > > Hm, I don't see how socket accounting would alter the network behavior > > other than through socket_pressure=1. > > > > How do you look for that flag? If you haven't yet done something > > comparable, can you try with tracing to rule out sampling errors? > > Apologies for a delayed reply, I took a week off away from computers. > > I looked with bpftrace (from my bash_history): > > $ sudo bpftrace -e 'kprobe:tcp_try_rmem_schedule { @sk[cpu] = arg0; } > kretprobe:tcp_try_rmem_schedule { $arg = @sk[cpu]; if ($arg) { $sk = > (struct sock *) $arg; $id = $sk->sk_memcg->css.cgroup->kn->id; > $socket_pressure = $sk->sk_memcg->socket_pressure; if ($id == 21379) { > printf("id = %d, socket_pressure = %d\n", $id, $socket_pressure); } } > }' > > I tried your patch on top of v6.1-rc8 (where it produced no conflicts) > in my vm and it still gave me low numbers and nothing in > /sys/kernel/debug/tracing/trace. To be extra sure, I changed it from > trace_printk to just printk and it still didn't show up in dmesg, even > with constant low throughput: > > ivan@vm:~$ curl -o /dev/null https://sim.cfperf.net/cached-assets/zero-5g.bin > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 14 4768M 14 685M 0 0 12.9M 0 0:06:08 0:00:52 0:05:16 13.0M > > I still saw clamped rcv_ssthresh: > > $ sudo ss -tinm dport 443 > State Recv-Q Send-Q > Local Address:Port > Peer Address:Port Process > ESTAB 0 0 > 10.2.0.15:35800 > 162.159.136.82:443 > skmem:(r0,rb2577228,t0,tb46080,f0,w0,o0,bl0,d0) cubic rto:201 > rtt:0.42/0.09 ato:40 mss:1460 pmtu:1500 rcvmss:1440 advmss:1460 > cwnd:10 bytes_sent:12948 bytes_acked:12949 bytes_received:2915062731 > segs_out:506592 segs_in:2025111 data_segs_out:351 data_segs_in:2024911 > send 278095238bps lastsnd:824 lastrcv:154 lastack:154 pacing_rate > 556190472bps delivery_rate 47868848bps delivered:352 app_limited > busy:147ms rcv_rtt:0.011 rcv_space:82199 rcv_ssthresh:5840 > minrtt:0.059 snd_wnd:65535 tcp-ulp-tls rxconf: none txconf: none > > I also tried with my detection program for ebpf_exporter (fexit based version): > > * https://github.com/cloudflare/ebpf_exporter/pull/172/files > > Which also showed signs of a clamped window: > > # HELP ebpf_exporter_tcp_window_clamps_total Number of times that TCP > window was clamped to a low value > # TYPE ebpf_exporter_tcp_window_clamps_total counter > ebpf_exporter_tcp_window_clamps_total 53887 > > In fact, I can replicate this with just curl to a public URL and fio running, I sprinkled some more printk around to get to the bottom of this: static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) { if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->socket_pressure) { printk("socket pressure[1]: %lu", memcg->socket_pressure); return true; } do { if (memcg->socket_pressure) { printk("socket pressure[2]: %lu", memcg->socket_pressure); return true; } } while ((memcg = parent_mem_cgroup(memcg))); return false; } And now I can see plenty of this: [ 108.156707][ T5175] socket pressure[2]: 4294673429 [ 108.157050][ T5175] socket pressure[2]: 4294673429 [ 108.157301][ T5175] socket pressure[2]: 4294673429 [ 108.157581][ T5175] socket pressure[2]: 4294673429 [ 108.157874][ T5175] socket pressure[2]: 4294673429 [ 108.158254][ T5175] socket pressure[2]: 4294673429 I think the first result below is to blame: $ rg '.->socket_pressure' mm mm/memcontrol.c 5280: memcg->socket_pressure = jiffies; 7198: memcg->socket_pressure = 0; 7201: memcg->socket_pressure = 1; 7211: memcg->socket_pressure = 0; 7215: memcg->socket_pressure = 1; While we set socket_pressure to either zero or one in mem_cgroup_charge_skmem, it is still initialized to jiffies on memcg creation. Zero seems like a more appropriate starting point. With that change I see it working as expected with no TCP speed bumps. My ebpf_exporter program also looks happy and reports zero clamps in my brief testing. Since it's not "socket pressure[1]" in dmesg output, then it's probably one of the parent cgroups that is not getting charged for socket memory that is reporting memory pressure. I also think we should downgrade socket_pressure from "unsigned long" to "bool", as it only holds zero and one now.