From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71B82C352A1 for ; Tue, 6 Dec 2022 20:51:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C3568E0003; Tue, 6 Dec 2022 15:51:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 073FF8E0001; Tue, 6 Dec 2022 15:51:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E58038E0003; Tue, 6 Dec 2022 15:51:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D65458E0001 for ; Tue, 6 Dec 2022 15:51:28 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9E3B0160DF7 for ; Tue, 6 Dec 2022 20:51:28 +0000 (UTC) X-FDA: 80213077056.24.23759B8 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf14.hostedemail.com (Postfix) with ESMTP id 16DF1100007 for ; Tue, 6 Dec 2022 20:51:26 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=ew9hSEgO; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.218.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670359887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MhNNBuU3zhWHic7xb6ZOwSKSyK5OVL6db6+6pmE+Q3M=; b=F/2YNuXY8h4MKiTC7uvaT17895IK01KVJJ5tz86tZLUE2G+ONjWRAd6b2jSzCjkpFIDNHa tXAwa4YiIyi43218R8h8lLbv883FUzlfaOzb1ZPFD0JIpnmAFGiKFWzKDvq7zfGkCOWFq6 KfA/FGJPCTVLl4keM3SMefkYo8bSkaI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=ew9hSEgO; spf=pass (imf14.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.218.43 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org; dmarc=pass (policy=none) header.from=cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670359887; a=rsa-sha256; cv=none; b=QwyotQsz0O9cbWYA7gPu9kJWHndqYgcZPwJGjhwE6IBYxORcMtGQiRXp5c8knbvfXDEn6g S/UUfsfmrNcEtMU6caHHfy8KxQE6bMRHwVuVCbQJkPDxBd9084QPWVB77tvbnHDo6B7NHl 5Ht3hdKy5gfFrFssQfUw853x66scorU= Received: by mail-ej1-f43.google.com with SMTP id n20so9434385ejh.0 for ; Tue, 06 Dec 2022 12:51:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MhNNBuU3zhWHic7xb6ZOwSKSyK5OVL6db6+6pmE+Q3M=; b=ew9hSEgOp7zxS5LHySeYnEzHgmaD+v212Cb4Fd4PwkYBPj5pH62iZWOulKd0ei+VIF hG4aPrHYGGD2WjWCwTeiUbs4eJpgiBWAD11E9kH3ukMi930ryVJILwhbma6Z6EzrMM05 mLbvFWbLYmMuszo96rfDjQSt3bGw11HgQ5uhXrEPsPukevWYu9Se1bCczpK+/vleLV5K AGPm4VSO4i06JLwAL1yP1lr+XAgD8QZhihWIylphT7Vy0PwUMDgG1DNbLcgxqTsHQe/q icM80qePTJzH/mDe7I8K1001EPbcfyl2/qvXn1BCmCuF8HDKBXbdkhqjrLXdTuledxeF tJ2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MhNNBuU3zhWHic7xb6ZOwSKSyK5OVL6db6+6pmE+Q3M=; b=EM8bPPlUGAZ+NryliYp3JcPnJEVozxwMn/hP5sXxAH7koVCkBiqR71jSilLrxiiY6Z ldViLXtX6A0yFth5z5ywN5e/vVRs2LoSr3mkOmmCD1aB+sL4NJqYULEWfkkgv5csbjof 1l9Htp8uutN2N0oBn+C5tnBVXOl9+7TDW3s0uhcHEOFMQ3mMCtrjtM+Utr3kabaiznwf wg6F+fXNDpng/DDdCqNJOxw3nn2s9kkYa596KC+NCBU0JKMBv3RsEZ+5idkSU8A3tC0y 185Pe7f6PKcuIrMtPSArqf0HhscL29y8E3so/P8/JtC4TVddnJAqidSld5r2c0w1c7rS Q97A== X-Gm-Message-State: ANoB5pkopSOgxKm8sF4xNGGkl8kAeRax7ixP92HvBawBXItAl15tZGK7 6UwkI7DGBYrD0vx6GKSljoiOTg== X-Google-Smtp-Source: AA0mqf5xxsOgNUB2YZXbcbbtJVAVE6QcQm9i9oyuo3Txh3iXPhEwR4ypb+UnCsY8aq8y04JS3YdgNA== X-Received: by 2002:a17:906:fa11:b0:7c0:d94c:f9f with SMTP id lo17-20020a170906fa1100b007c0d94c0f9fmr12177439ejb.542.1670359885503; Tue, 06 Dec 2022 12:51:25 -0800 (PST) Received: from localhost ([2a02:8070:6387:ab20:15aa:3c87:c206:d15e]) by smtp.gmail.com with ESMTPSA id fi22-20020a1709073ad600b007c0d4d3a0c1sm4576925ejc.32.2022.12.06.12.51.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Dec 2022 12:51:25 -0800 (PST) Date: Tue, 6 Dec 2022 21:51:01 +0100 From: Johannes Weiner To: Eric Dumazet Cc: Ivan Babrou , Linux MM , Linux Kernel Network Developers , linux-kernel , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Subject: Re: Low TCP throughput due to vmpressure with swap enabled Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-2.90 / 9.00]; BAYES_HAM(-6.00)[99.99%]; SORBS_IRL_BL(3.00)[209.85.218.43:from]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; BAD_REP_POLICIES(0.10)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_TWELVE(0.00)[17]; DMARC_POLICY_ALLOW(0.00)[cmpxchg.org,none]; DKIM_TRACE(0.00)[cmpxchg-org.20210112.gappssmtp.com:+]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; R_DKIM_ALLOW(0.00)[cmpxchg-org.20210112.gappssmtp.com:s=20210112]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 16DF1100007 X-Stat-Signature: pt6ygm4w5sft61dh1qe3p8pcrgesdgoe X-HE-Tag: 1670359886-102799 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 06, 2022 at 08:13:50PM +0100, Eric Dumazet wrote: > On Tue, Dec 6, 2022 at 8:00 PM Johannes Weiner wrote: > > @@ -1701,10 +1701,10 @@ void mem_cgroup_sk_alloc(struct sock *sk); > > void mem_cgroup_sk_free(struct sock *sk); > > static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) > > { > > - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->tcpmem_pressure) > > + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->socket_pressure) > > && READ_ONCE(memcg->socket_pressure)) > > > return true; > > do { > > - if (time_before(jiffies, READ_ONCE(memcg->socket_pressure))) > > + if (memcg->socket_pressure) > > if (READ_ONCE(...)) Good point, I'll add those. > > @@ -7195,10 +7194,10 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, > > struct page_counter *fail; > > > > if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) { > > - memcg->tcpmem_pressure = 0; > > Orthogonal to your patch, but: > > Maybe avoid touching this cache line too often and use READ/WRITE_ONCE() ? > > if (READ_ONCE(memcg->socket_pressure)) > WRITE_ONCE(memcg->socket_pressure, false); Ah, that's a good idea. I think it'll be fine in the failure case, since that's associated with OOM and total performance breakdown anyway. But certainly, in the common case of the charge succeeding, we should not keep hammering false into that variable over and over. How about the delta below? I also flipped the branches around to keep the common path at the first indentation level, hopefully making that a bit clearer too. Thanks for taking a look, Eric! diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ef1c388be5b3..13ae10116895 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1701,10 +1701,11 @@ void mem_cgroup_sk_alloc(struct sock *sk); void mem_cgroup_sk_free(struct sock *sk); static inline bool mem_cgroup_under_socket_pressure(struct mem_cgroup *memcg) { - if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && memcg->socket_pressure) + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && + READ_ONCE(memcg->socket_pressure)) return true; do { - if (memcg->socket_pressure) + if (READ_ONCE(memcg->socket_pressure)) return true; } while ((memcg = parent_mem_cgroup(memcg))); return false; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0d4b9dbe775a..96c4ec0f11ca 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7193,31 +7193,29 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) { struct page_counter *fail; - if (page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) { - memcg->socket_pressure = false; - return true; + if (!page_counter_try_charge(&memcg->tcpmem, nr_pages, &fail)) { + WRITE_ONCE(memcg->socket_pressure, true); + if (gfp_mask & __GFP_FAIL) { + page_counter_charge(&memcg->tcpmem, nr_pages); + return true; + } + return false; } - memcg->socket_pressure = true; + if (unlikely(READ_ONCE(memcg->socket_pressure))) + WRITE_ONCE(memcg->socket_pressure, false); + } + + if (try_charge(memcg, gfp_mask & ~__GFP_NOFAIL, nr_pages) < 0) { + WRITE_ONCE(memcg->socket_pressure, true); if (gfp_mask & __GFP_NOFAIL) { - page_counter_charge(&memcg->tcpmem, nr_pages); + try_charge(memcg, gfp_mask, nr_pages); + mod_memcg_state(memcg, MEMCG_SOCK, nr_pages); return true; } return false; } - - if (try_charge(memcg, gfp_mask & ~__GFP_NOFAIL, nr_pages) == 0) { - memcg->socket_pressure = false; - goto success; - } - memcg->socket_pressure = true; - if (gfp_mask & __GFP_NOFAIL) { - try_charge(memcg, gfp_mask, nr_pages); - goto success; - } - - return false; - -success: + if (unlikely(READ_ONCE(memcg->socket_pressure))) + WRITE_ONCE(memcg->socket_pressure, false); mod_memcg_state(memcg, MEMCG_SOCK, nr_pages); return true; }