From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C80FC32771 for ; Wed, 17 Aug 2022 17:14:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DABE58D0001; Wed, 17 Aug 2022 13:13:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D5BA46B0074; Wed, 17 Aug 2022 13:13:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C230D8D0001; Wed, 17 Aug 2022 13:13:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B2B466B0073 for ; Wed, 17 Aug 2022 13:13:59 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 859CD121721 for ; Wed, 17 Aug 2022 17:13:59 +0000 (UTC) X-FDA: 79809732198.21.1330226 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf24.hostedemail.com (Postfix) with ESMTP id 01EEA180082 for ; Wed, 17 Aug 2022 17:13:57 +0000 (UTC) Received: by mail-qk1-f171.google.com with SMTP id i7so714557qka.13 for ; Wed, 17 Aug 2022 10:13:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc; bh=aN3vJ2rM0Hc11j+m8iw80ADUWQ38gpksEtAyFV9ecwc=; b=Cv+aYyMKycJyWptvlaesV64u0spW8xgqi87m6gWliE2zm+TlFIJRBz4eef1VJh4su5 2xKgis8HA+lsKsAeI2kU2YR+KP3VaxwEcex4Mxcof6JmcmUuRjEniNMV0zATu+OOBW7L mvu2T7fDSz/AUAM43nuCqWYUchFzewT5cUgdhGCVmQYV/uXi8/3iF5UrC9ASGtCUy2c6 Ik1JekdH7bUFcNdRE38RgVYtwzuEs6pd65rCt5yn9ZZObw1t1+hJdchdKR1bD6AfraT+ R9t9ejemuhDNp3LEgtKtikAGixilncS/OhUs2PEOFTHC4niIjw4yqoUBrEucXy/4eAYK Kl6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc; bh=aN3vJ2rM0Hc11j+m8iw80ADUWQ38gpksEtAyFV9ecwc=; b=2q5SY6MAZQf07DPYwI0KeEi+85gYaYqFD0i2M58dvtpFsw+58dDEw5KwlGxUxQMpLy IT3K2Wbmg/mfc9e8XNVhwl6eLb+LA7lMqMWrCJVOq9bsnlkIJh8HM1jlxWVepMq5pYwY cwCCjp1FDOctQp1J4Zl6u0xXt5UpiwkUb3rPBhExdERWyV4HG8aY11KJ1MG0WJ1dxgAx YhHqKsvg7+OIWEW427eiXHaohEjpXU6zHcvtJ9DbddFhSNR+QDg3xqRepJieLsSay2yF XVvv87k/sCdYelEqYXB4FArbxTumhOFmc5ixriDFka0uPhQ/3I3wTZFCcIRPuykEH2oj ccdg== X-Gm-Message-State: ACgBeo1Y1gL4TqWZseDvk6OolI9UbhNcolNh6km140uCe0+mJMa0Bgh1 VA6vgaix1SEV6JSMBRBr4LnZCA== X-Google-Smtp-Source: AA6agR5uymcSZsYdlIY7eiAbG0a6+f9+a1QeHjqeo8s6Oz95BxxN/uSL0IXX1JzDHxabpZhGjvPTpg== X-Received: by 2002:a05:620a:28c1:b0:6bb:5deb:a888 with SMTP id l1-20020a05620a28c100b006bb5deba888mr7109059qkp.485.1660756437198; Wed, 17 Aug 2022 10:13:57 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::39e8]) by smtp.gmail.com with ESMTPSA id y6-20020ae9f406000000b006aee5df383csm3679070qkl.134.2022.08.17.10.13.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Aug 2022 10:13:56 -0700 (PDT) Date: Wed, 17 Aug 2022 13:13:56 -0400 From: Johannes Weiner To: =?utf-8?Q?Gra=C5=BEvydas?= Ignotas Cc: Wei Wang , Shakeel Butt , Michal Hocko , Roman Gushchin , linux-mm@kvack.org, cgroups@vger.kernel.org Subject: Re: UDP rx packet loss in a cgroup with a memory limit Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660756438; a=rsa-sha256; cv=none; b=p+cr3KGG+R4qETwtivGS7d+aV1jYgHsZdYi7NhwIMT8rjJA13V+5lqyclJGIfyCBBFfJbu L942jdRtm8kbRctE1Ws6JOLED3hkcWzqfmDI3RzCJl89cNtv3tfBEMOviiY9C/C1jaoVhY g2/dNsq44zYBicB/gTJRt2nSLIv67Hk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=Cv+aYyMK; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.171 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660756438; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aN3vJ2rM0Hc11j+m8iw80ADUWQ38gpksEtAyFV9ecwc=; b=3go/CTOlm95UY9ukSnKRfYJai/hPYC0xR6cs8eujgs3nFJ5M/KXAyU4g8C/MhtsfJRvJ7f eIQVVWs78aCb8n7+VqXWF1DlX5BHKIxVkb4SOHaLC4sSMDFVsrSiKMC/qv5fvX+BzID65P sQAycBsmbGIk55xR6bfZ9G0kjPn69Ek= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 01EEA180082 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=cmpxchg-org.20210112.gappssmtp.com header.s=20210112 header.b=Cv+aYyMK; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf24.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.222.171 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org X-Stat-Signature: rdxuejqn9go7dowpzq1uquswd5c6mj8x X-HE-Tag: 1660756437-705034 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Aug 17, 2022 at 07:50:13PM +0300, Gražvydas Ignotas wrote: > On Tue, Aug 16, 2022 at 9:52 PM Gražvydas Ignotas wrote: > > Basically, when there is git activity in the container with a memory > > limit, other processes in the same container start to suffer (very) > > occasional network issues (mostly DNS lookup failures). > > ok I've traced this and it's failing in try_charge_memcg(), which > doesn't seem to be trying too hard because it's called from irq > context. > > Here is the backtrace: > > ? fib_validate_source+0xb4/0x100 > ? ip_route_input_slow+0xa11/0xb70 > mem_cgroup_charge_skmem+0x4b/0xf0 > __sk_mem_raise_allocated+0x17f/0x3e0 > __udp_enqueue_schedule_skb+0x220/0x270 > udp_queue_rcv_one_skb+0x330/0x5e0 > udp_unicast_rcv_skb+0x75/0x90 > __udp4_lib_rcv+0x1ba/0xca0 > ? ip_rcv_finish_core.constprop.0+0x63/0x490 > ip_protocol_deliver_rcu+0xd6/0x230 > ip_local_deliver_finish+0x73/0xa0 > __netif_receive_skb_one_core+0x8b/0xa0 > process_backlog+0x8e/0x120 > __napi_poll+0x2c/0x160 > net_rx_action+0x2a2/0x360 > ? rebalance_domains+0xeb/0x3b0 > __do_softirq+0xeb/0x2eb > __irq_exit_rcu+0xb9/0x110 > sysvec_apic_timer_interrupt+0xa2/0xd0 > > > Calling mem_cgroup_print_oom_meminfo() in such a case reveals: > > memory: usage 7812476kB, limit 7812500kB, failcnt 775198 > swap: usage 0kB, limit 0kB, failcnt 0 > Memory cgroup stats for > /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f4f0e9_fb95_4f2d_8443_e6a78f235c9a.slice/docker-9e7cad93b2e0774d49148474989b41fe6d67a5985d059d08d9d64495f1539a81.scope: > anon 348016640 > file 7502163968 > kernel 146997248 > kernel_stack 327680 > pagetables 2224128 > percpu 0 > sock 4096 > vmalloc 0 > shmem 0 > zswap 0 > zswapped 0 > file_mapped 112041984 > file_dirty 1181028352 > file_writeback 2686976 > swapcached 0 > anon_thp 44040192 > file_thp 0 > shmem_thp 0 > inactive_anon 350756864 > active_anon 36864 > inactive_file 3614003200 > active_file 3888070656 > unevictable 0 > slab_reclaimable 143692600 > slab_unreclaimable 545120 > slab 144237720 > workingset_refault_anon 0 > workingset_refault_file 2318 > workingset_activate_anon 0 > workingset_activate_file 2318 > workingset_restore_anon 0 > workingset_restore_file 0 > workingset_nodereclaim 0 > pgfault 334152 > pgmajfault 1238 > pgrefill 3400 > pgscan 819608 > pgsteal 791005 > pgactivate 949122 > pgdeactivate 1694 > pglazyfree 0 > pglazyfreed 0 > zswpin 0 > zswpout 0 > thp_fault_alloc 709 > thp_collapse_alloc 0 > > So it basically renders UDP inoperable because of disk cache. I hope > this is not the intended behavior. Naturally booting with > cgroup.memory=nosocket solves this issue. This is most likely a regression caused by this patch: commit 4b1327be9fe57443295ae86fe0fcf24a18469e9f Author: Wei Wang Date: Tue Aug 17 12:40:03 2021 -0700 net-memcg: pass in gfp_t mask to mem_cgroup_charge_skmem() Add gfp_t mask as an input parameter to mem_cgroup_charge_skmem(), to give more control to the networking stack and enable it to change memcg charging behavior. In the future, the networking stack may decide to avoid oom-kills when fallbacks are more appropriate. One behavior change in mem_cgroup_charge_skmem() by this patch is to avoid force charging by default and let the caller decide when and if force charging is needed through the presence or absence of __GFP_NOFAIL. Signed-off-by: Wei Wang Reviewed-by: Shakeel Butt Signed-off-by: David S. Miller We never used to fail these allocations. Cgroups don't have a kswapd-style watermark reclaimer, so the network relied on force-charging and leaving reclaim to allocations that can block. Now it seems network packets could just fail indefinitely. The changelog is a bit terse given how drastic the behavior change is. Wei, Shakeel, can you fill in why this was changed? Can we revert this for the time being?