From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA558C3F6B0 for ; Wed, 17 Aug 2022 20:12:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D2CE78D0002; Wed, 17 Aug 2022 16:12:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDAE68D0001; Wed, 17 Aug 2022 16:12:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7B698D0002; Wed, 17 Aug 2022 16:12:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A45058D0001 for ; Wed, 17 Aug 2022 16:12:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 72A9581765 for ; Wed, 17 Aug 2022 20:12:46 +0000 (UTC) X-FDA: 79810182732.19.26DC455 Received: from mail-wr1-f54.google.com (mail-wr1-f54.google.com [209.85.221.54]) by imf25.hostedemail.com (Postfix) with ESMTP id 0D262A008A for ; Wed, 17 Aug 2022 20:12:45 +0000 (UTC) Received: by mail-wr1-f54.google.com with SMTP id u14so2881029wrq.9 for ; Wed, 17 Aug 2022 13:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc; bh=AMde7Yzl8mmvlqIunu6+T9Ocr3m9Ip7KzH42Q+pGJAc=; b=D1m+Xmnga/AgU/CL4ihU3qMGfrmtpQeV9FfQ2kfDyU8emY0rugHv/OgbWjgQdINtVu 5uy0KyJjMS8pyt1qAQdE3/WL4zS15brolv7oMsCPLDt8mwY1Hrqea6iZMXlQzg4fP83m gGMTLcJClIQedXLzq9cgxYoSSwUd7v1Y64S95id/u3TA/Xb7aZIQt+qD5wEilawlJ8/T 4by8HvmwytA5lQHl7VmR0akFS+42S1Q0zvQNeW8SPxuV7kaJFxbF9sjJuwRNNeW+nHtq 8Uadm+/KpP0uyEOKawD3ObcGAC0xBbjGMKTjT1LijEN90qzcPo4JM2pJ4W0/n0Mu5/P+ Y4VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc; bh=AMde7Yzl8mmvlqIunu6+T9Ocr3m9Ip7KzH42Q+pGJAc=; b=0TZ6tFJ9ILZK7h1onbGwjmMkI2YjZML79VYvaQozgG00vzTyA2WCrw7U3+BNVOL2gR 90sCtNMz05xH28j+FUM8dvuCzjeFz+SuyOlVw+vCze5N+VWDtE69PTKaeL8IB8mKNwnk zauihAKLdrkiJQm1nv4NiOO+nLzMVpc/48Lf3wD3JdxasfAEbFxn8h25m9gQZ1g3vNLT aXbCTnAu8KpIHOwck7K7SRRrdJx52jW5/j6vCd4SJnMn6MdngkHBcEP/RI2C0ytrpjJm JRZNwl/y9FYrZRUDhtjonRSLpZi2wDBPK16E+IG7e3ZZmk/Mm8AeFtVvRpdFIFNseLhF hb4g== X-Gm-Message-State: ACgBeo3oJSJEOKm4AkRrsVzUAGFKh35TjWRzSdYvo3FEXrq5sAyJY2PE 2LOdqmggipTUo7WeSAluAWxYNA1xMTbiNxePuCE= X-Google-Smtp-Source: AA6agR4PCb4y2KFpr/E5wBDzEavpP4S6HHd9PKD2tzwt3c7eFPathFgrBEhQnKgLEnGAvl70WXokEKyQURRqVW+w1yI= X-Received: by 2002:a05:6000:811:b0:220:6262:ac66 with SMTP id bt17-20020a056000081100b002206262ac66mr14541307wrb.529.1660767164592; Wed, 17 Aug 2022 13:12:44 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Gra=C5=BEvydas_Ignotas?= Date: Wed, 17 Aug 2022 23:12:33 +0300 Message-ID: Subject: Re: UDP rx packet loss in a cgroup with a memory limit To: Wei Wang Cc: Shakeel Butt , Johannes Weiner , Eric Dumazet , netdev , Michal Hocko , Roman Gushchin , Linux MM , Cgroups Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1660767166; a=rsa-sha256; cv=none; b=gXgWeu3+SvQRKTm4wf/xpmFiZfTl4pHaYQT9/L6hhbCUN3CbRtwpq3Dlje2KzL5Rsnxxr7 dEkotAWon5OuHeOgUXFaw6Z6O6aKVZBsxOyjF5ztlcELpsjfP0WXivDonBQHQZKfG5AtO1 +WBamZJoNtH5Lbwr/qcu2fq2AsW7I6I= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D1m+Xmng; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of notasas@gmail.com designates 209.85.221.54 as permitted sender) smtp.mailfrom=notasas@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1660767166; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AMde7Yzl8mmvlqIunu6+T9Ocr3m9Ip7KzH42Q+pGJAc=; b=eumDXmDkDKJxBeBWE4RXxXq81aY0+VDnQc6oSwlrtuDSkZYAJQ0OjwIe2pf/jvxwBFLu54 YgsfQuRiW2cojarQZzmcZYfinnPlhfAoeJxJDLWvxeqkht3y5duRgWgwaIQxEWID2de2Tz y/ZlH46Fk2f8ZPpQEnB6NNQT3xVjwiU= X-Stat-Signature: uc5mnohbjgh8ma7k4owukj9rnb4hmhsp X-Rspamd-Queue-Id: 0D262A008A Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=D1m+Xmng; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf25.hostedemail.com: domain of notasas@gmail.com designates 209.85.221.54 as permitted sender) smtp.mailfrom=notasas@gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1660767165-445914 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Aug 17, 2022 at 9:16 PM Wei Wang wrote: > > On Wed, Aug 17, 2022 at 10:37 AM Shakeel Butt wrote: > > > > + Eric and netdev > > > > On Wed, Aug 17, 2022 at 10:13 AM Johannes Weiner wrote: > > > > > > This is most likely a regression caused by this patch: > > > > > > commit 4b1327be9fe57443295ae86fe0fcf24a18469e9f > > > Author: Wei Wang > > > Date: Tue Aug 17 12:40:03 2021 -0700 > > > > > > net-memcg: pass in gfp_t mask to mem_cgroup_charge_skmem() > > > > > > Add gfp_t mask as an input parameter to mem_cgroup_charge_skmem(), > > > to give more control to the networking stack and enable it to change > > > memcg charging behavior. In the future, the networking stack may decide > > > to avoid oom-kills when fallbacks are more appropriate. > > > > > > One behavior change in mem_cgroup_charge_skmem() by this patch is to > > > avoid force charging by default and let the caller decide when and if > > > force charging is needed through the presence or absence of > > > __GFP_NOFAIL. > > > > > > Signed-off-by: Wei Wang > > > Reviewed-by: Shakeel Butt > > > Signed-off-by: David S. Miller > > > > > > We never used to fail these allocations. Cgroups don't have a > > > kswapd-style watermark reclaimer, so the network relied on > > > force-charging and leaving reclaim to allocations that can block. > > > Now it seems network packets could just fail indefinitely. > > > > > > The changelog is a bit terse given how drastic the behavior change > > > is. Wei, Shakeel, can you fill in why this was changed? Can we revert > > > this for the time being? > > > > Does reverting the patch fix the issue? However I don't think it will. > > > > Please note that we still have the force charging as before this > > patch. Previously when mem_cgroup_charge_skmem() force charges, it > > returns false and __sk_mem_raise_allocated takes suppress_allocation > > code path. Based on some heuristics, it may allow it or it may > > uncharge and return failure. > > The force charging logic in __sk_mem_raise_allocated only gets > considered on tx path for STREAM socket. So it probably does not take > effect on UDP path. And, that logic is NOT being altered in the above > patch. > So specifically for UDP receive path, what happens in > __sk_mem_raise_allocated() BEFORE the above patch is: > - mem_cgroup_charge_skmem() gets called: > - try_charge() with GFP_NOWAIT gets called and failed > - try_charge() with __GFP_NOFAIL > - return false > - goto suppress_allocation: > - mem_cgroup_uncharge_skmem() gets called > - return 0 (which means failure) > > AFTER the above patch, what happens in __sk_mem_raise_allocated() is: > - mem_cgroup_charge_skmem() gets called: > - try_charge() with GFP_NOWAIT gets called and failed > - return false > - goto suppress_allocation: > - We no longer calls mem_cgroup_uncharge_skmem() > - return 0 > > So I agree with Shakeel, that this change shouldn't alter the behavior > of the above call path in such a situation. > But do let us know if reverting this change has any effect on your test. The problem is still there (the kernel wasn't compiling after revert, had to adjust another seemingly unrelated callsite). It's hard to tell if it's better or worse since it happens so randomly. > > > > > The given patch has not changed any heuristic. It has only changed > > when forced charging happens. After the path the initial call > > mem_cgroup_charge_skmem() can fail and we take suppress_allocation > > code path and if heuristics allow, we force charge with __GFP_NOFAIL.