From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE6FFC4332F for ; Tue, 22 Nov 2022 18:24:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D6828E0001; Tue, 22 Nov 2022 13:24:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 75FDE6B0073; Tue, 22 Nov 2022 13:24:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 600B58E0001; Tue, 22 Nov 2022 13:24:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4C62F6B0071 for ; Tue, 22 Nov 2022 13:24:01 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 22CB61C5F3E for ; Tue, 22 Nov 2022 18:24:01 +0000 (UTC) X-FDA: 80161902282.09.58FAE4C Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) by imf20.hostedemail.com (Postfix) with ESMTP id A95A21C0011 for ; Tue, 22 Nov 2022 18:24:00 +0000 (UTC) Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-3938dc90ab0so133619047b3.4 for ; Tue, 22 Nov 2022 10:24:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WQLDoQzOOOLSeL4BW8P5dyKRJbU6yXunhkxiL7lS7JY=; b=CDJdz0q15FCrFgZ2RgOE7aoOkCnIE0qqjNfpiyaFukahYrSsxZ4fDxJsk+IKeVDOLo fTIZ884+FEiiSCjX4m/RkJvUVv/DQEsLzT8tR6MQJj/DZLHa5vYBkFxjl3a61gRXpZoj rpC2JE+T9pLyf+7khG2HYb3MmtiSv8Hc+Sqy3T0KWu28zVIJIVLhqo1hgYEx0lvQmB8o hBW6p4z3/wlIQ2VX6+WDew8PHl2O8jidBSbLv71s0dt5scFC2DINgLvjBDp/1D7ww7a/ HrKpH89edAqH8LukVTkAMdhBlxmVVkYaKernLE5kg3YAKjOaPWyPaIo0MEZmOcq1MqXj Jk+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WQLDoQzOOOLSeL4BW8P5dyKRJbU6yXunhkxiL7lS7JY=; b=DYVZtkyaSHJeoh3UVv7DxxO+9Cgs6uaTecz2PeWQLKSaCqtYyH1EsHqwuTDzEURl/D 8lzRQLElqXQjOyFFUHmADYz2d5qvf+zMdl4kbdUsQ+QnK+1T7O/Suh4vUdTEv2reyPv/ 7QF3Flem672TL2IV/XBgu+kz+nbnmGkH/EcWK0fW41bmO0qO2cKzFtK+CLFVj7kiMXCf H7ErszWdkUb+M8XN6ObCnxAbBS8xqWiFHtf1A7wffG+mL4yBLeXECLCkczKDJCKO/vFC gVoUnL0VSsf60T81D8Fb1O5/Algrrq1flFMqzlCz6j19/rviJJOHpCmXbflZD3OFIsAS W3kw== X-Gm-Message-State: ANoB5pkDQYan7XAqT7MGnoRD98dq0uYBGvUWYqrRA0ZnKY5UMy2ooDDN NKqzeTzxs400c7GLBeTIItyVO0vKAGmWSCKDfmwDWA== X-Google-Smtp-Source: AA0mqf6LXCkwUEsGG9yFNI5WKVis3+ohpo5D3S2bFziOk7I40KkoxIKh/CgI5DYlRVN1wovpZt5CgwqvQjG2/bPGkQI= X-Received: by 2002:a81:7cd6:0:b0:357:6958:372a with SMTP id x205-20020a817cd6000000b003576958372amr5414093ywc.255.1669141439602; Tue, 22 Nov 2022 10:23:59 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Eric Dumazet Date: Tue, 22 Nov 2022 10:23:48 -0800 Message-ID: Subject: Re: Low TCP throughput due to vmpressure with swap enabled To: Ivan Babrou Cc: Linux MM , Linux Kernel Network Developers , linux-kernel , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1669141440; a=rsa-sha256; cv=none; b=546alpCKnfDe4DnetoWr3vhM6nHKRWithNpC1YY8+OFegpkF1TtftwUbNFReDN+pjAtlrx UTQgWn+iGvgI9Jvx5opLidFoOkVoorBYjwQ6Ij569eKUaRxwuA/kXPSOKTmXosqzojthTU Qyqj9CEOPHiacDcDcEsgotEOpGH48/Y= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CDJdz0q1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of edumazet@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=edumazet@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1669141440; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WQLDoQzOOOLSeL4BW8P5dyKRJbU6yXunhkxiL7lS7JY=; b=i5tlpMX6W4/qtQoWDKbO4AxpepjsJDaNtuX3kxZyaSu6tRMEunLiXnnxmNA0mwhbSTtLQr rVqu+qSlV3vaQuJIa+hQlAmE7Ffwe88lxh5nkYJhVG0Lh8s9XQEyqyHAJG3S8verowleMb 9SFmaV53N195ctMYOzB+MjzcifWwG1Q= X-Stat-Signature: ujoo98x1qetgiw9go11s1q3yn3c8hnpm X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A95A21C0011 Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=CDJdz0q1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of edumazet@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=edumazet@google.com X-Rspam-User: X-HE-Tag: 1669141440-628351 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Nov 22, 2022 at 10:11 AM Ivan Babrou wrote: > > On Tue, Nov 22, 2022 at 10:01 AM Eric Dumazet wrote: > > > > On Mon, Nov 21, 2022 at 4:53 PM Ivan Babrou wrote: > > > > > > Hello, > > > > > > We have observed a negative TCP throughput behavior from the following commit: > > > > > > * 8e8ae645249b mm: memcontrol: hook up vmpressure to socket pressure > > > > > > It landed back in 2016 in v4.5, so it's not exactly a new issue. > > > > > > The crux of the issue is that in some cases with swap present the > > > workload can be unfairly throttled in terms of TCP throughput. > > > > I guess defining 'fairness' in such a scenario is nearly impossible. > > > > Have you tried changing /proc/sys/net/ipv4/tcp_rmem (and/or tcp_wmem) ? > > Defaults are quite conservative. > > Yes, our max sizes are much higher than the defaults. I don't see how > it matters though. The issue is that the kernel clamps rcv_sshtrehsh > at 4 x advmss. There are some places (eg tcp_clamp_window) where we have this additional condition : sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0) So I was suggesting maybe to add a similar condition to tcp_try_rmem_schedule() Then adjust tcp_rmem for your needs. No matter how much TCP memory you end up using, the > kernel will clamp based on responsiveness to memory reclaim, which in > turn depends on swap presence. We're seeing it in production with tens > of thousands of sockets and high max tcp_rmem and I'm able to > replicate the same issue in my vm with the default sysctl values. > > > If for your workload you want to ensure a minimum amount of memory per > > TCP socket, > > that might be good enough. > > That's not my goal at all. We don't have a problem with TCP memory > consumption. Our issue is low throughput because vmpressure() thinks > that the cgroup is memory constrained when it most definitely is not. OK, then I will stop commenting I guess :)