From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FFA8C64E7B for ; Mon, 30 Nov 2020 22:57:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8D6342076C for ; Mon, 30 Nov 2020 22:57:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KgRqSq+m" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8D6342076C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B79378D0003; Mon, 30 Nov 2020 17:57:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B2B6A8D0001; Mon, 30 Nov 2020 17:57:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F2898D0003; Mon, 30 Nov 2020 17:57:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0148.hostedemail.com [216.40.44.148]) by kanga.kvack.org (Postfix) with ESMTP id 8722A8D0001 for ; Mon, 30 Nov 2020 17:57:36 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4A0EA8249980 for ; Mon, 30 Nov 2020 22:57:36 +0000 (UTC) X-FDA: 77542598112.05.moon06_04095e5273a5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id 31003180257FC for ; Mon, 30 Nov 2020 22:57:36 +0000 (UTC) X-HE-Tag: moon06_04095e5273a5 X-Filterd-Recvd-Size: 6791 Received: from mail-ej1-f65.google.com (mail-ej1-f65.google.com [209.85.218.65]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Mon, 30 Nov 2020 22:57:35 +0000 (UTC) Received: by mail-ej1-f65.google.com with SMTP id f23so5268724ejt.8 for ; Mon, 30 Nov 2020 14:57:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VPiFGQkiqWAFPJXQUHUTo1ORgX89o0bCN92vawl5snc=; b=KgRqSq+mb9FeUCYdNxy5lVl/TeDwcTk3FhwJ04jq285EVz2Qrm56pUpjxUUcl0JlG4 CoH8pcNtTuSGqyF3oELr/YGrlWhw9WbyxstlzH0jtLC86akm5F3/ni8pfQaehPgcfoiZ PPazu2lIKQbNgn6v6gqY+jsCJCHsmC6YL63W2FKz9gFWcDMyfL+MsNT1O3um3pLlO9Sh HR2JAcX7kimDAyyYSn7pgVlEHzSQ5QOTiBQljDm2W3qTzkfPn4Rdog+b0xW/VhGK5DLV pMKnzA0Mj6wVzkiOkEhfbCJX5MVQbBb/vz612jFpTIzpiXYLd9s05ff8MOPRIwiWKm5g W4AA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VPiFGQkiqWAFPJXQUHUTo1ORgX89o0bCN92vawl5snc=; b=QGEeYXZFQkyMPwKjg+Hg5e7oIr+x5Oysee+//VLsewzRkve3JaTZWODrdx3YAiCzya JFDM/YvogXaNCpRCxVVeGFHtioMt6p7BqSbAhNv3vVAxJiSQhjkYCg8nJCSPcCH2hOIm VivKoFeRnAKnlz4FBq1U/0LNmz/jym5Eo8OMcny5U40LbS2HWR7Hj3Tm4G7jlcRR8W52 6ChZjHQMDX6Rv2Lyv1l6XlqPb1AmCfabyu50gpJG4vfV8FBtHfCy5/HDIeglLpfIHSBG f0+bvqp+f1xNs+aThvdVkVc8xI/lCSoudaQ4pVLn6Xq/6v4Dvx6CtCqt7ZCqh3oEoTQ8 cEww== X-Gm-Message-State: AOAM532xPHY+Ol6BUIGX2kTRgJbwq1N+HLccFNlJ2TDH4CBNvlAHTWi6 VyjzU/pYgR+LeZStl0X64gNm5N+p7zgLS4Xe/hQ= X-Google-Smtp-Source: ABdhPJyWHDxtxgVzsyEmdHRSHw5yyUKB4fq2DqJ7mDYsDO9Aj5qtpcI56NKuYH3RYmkkGXJ8dyZ7JDU9ssNrPdNWtCE= X-Received: by 2002:a17:906:24c3:: with SMTP id f3mr81171ejb.238.1606777054774; Mon, 30 Nov 2020 14:57:34 -0800 (PST) MIME-Version: 1.0 References: <20201130184514.551950-1-shy828301@gmail.com> <20201130200936.GA1354703@carbon.DHCP.thefacebook.com> <20201130225255.GF840171@carbon.dhcp.thefacebook.com> In-Reply-To: <20201130225255.GF840171@carbon.dhcp.thefacebook.com> From: Yang Shi Date: Mon, 30 Nov 2020 14:57:23 -0800 Message-ID: Subject: Re: [PATCH] mm: list_lru: hold nlru lock to avoid reading transient negative nr_items To: Roman Gushchin Cc: Vladimir Davydov , Kirill Tkhai , Shakeel Butt , Andrew Morton , Linux MM , Linux Kernel Mailing List Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Nov 30, 2020 at 2:53 PM Roman Gushchin wrote: > > On Mon, Nov 30, 2020 at 12:57:47PM -0800, Yang Shi wrote: > > On Mon, Nov 30, 2020 at 12:09 PM Roman Gushchin wrote: > > > > > > On Mon, Nov 30, 2020 at 10:45:14AM -0800, Yang Shi wrote: > > > > When investigating a slab cache bloat problem, significant amount of > > > > negative dentry cache was seen, but confusingly they neither got shrunk > > > > by reclaimer (the host has very tight memory) nor be shrunk by dropping > > > > cache. The vmcore shows there are over 14M negative dentry objects on lru, > > > > but tracing result shows they were even not scanned at all. The further > > > > investigation shows the memcg's vfs shrinker_map bit is not set. So the > > > > reclaimer or dropping cache just skip calling vfs shrinker. So we have > > > > to reboot the hosts to get the memory back. > > > > > > > > I didn't manage to come up with a reproducer in test environment, and the > > > > problem can't be reproduced after rebooting. But it seems there is race > > > > between shrinker map bit clear and reparenting by code inspection. The > > > > hypothesis is elaborated as below. > > > > > > > > The memcg hierarchy on our production environment looks like: > > > > root > > > > / \ > > > > system user > > > > > > > > The main workloads are running under user slice's children, and it creates > > > > and removes memcg frequently. So reparenting happens very often under user > > > > slice, but no task is under user slice directly. > > > > > > > > So with the frequent reparenting and tight memory pressure, the below > > > > hypothetical race condition may happen: > > > > > > > > CPU A CPU B CPU C > > > > reparent > > > > dst->nr_items == 0 > > > > shrinker: > > > > total_objects == 0 > > > > add src->nr_items to dst > > > > set_bit > > > > retrun SHRINK_EMPTY > > > > clear_bit > > > > list_lru_del() > > > > reparent again > > > > dst->nr_items may go negative > > > > due to current list_lru_del() > > > > on CPU C > > > > The second run of shrinker: > > > > read nr_items without any > > > > synchronization, so it may > > > > see intermediate negative > > > > nr_items then total_objects > > > > may return 0 conincidently > > > > > > > > keep the bit cleared > > > > dst->nr_items != 0 > > > > skip set_bit > > > > add scr->nr_item to dst > > Btw, I think I have a simpler explanation: > > A (0 objects) > | > B (N objects) > > Let's say the reparenting races with the deletion of a single slab object. > list_lru_del() can see parent's lru list and substract 1 from nr_items == 0, > setting A's nr_items to -1 (the item is actually still in B's list). > > memcg_drain_list_lru_node() will check !dst->nr_items && src->nr_items > !-1 && N => 0 and not set the bit. But now we have (N-1) objects in A's list > and the shrinker bit not set. Yes, this is the exact race I elaborated in the commit log. > > My proposed fix should resolve it. Alternatively, we maybe can check if > dst->nr_items <= 0 and only then set the bit, but it seems to be an unnecessary > optimization. Yes, I think "src->nr_items != 0" is good enough. > > Thanks!