From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94EA8C32771 for ; Tue, 28 Jan 2020 00:46:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5708C20678 for ; Tue, 28 Jan 2020 00:46:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X4ON06sk" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5708C20678 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 06E956B0007; Mon, 27 Jan 2020 19:46:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 01E2C6B0008; Mon, 27 Jan 2020 19:46:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9CBD6B000A; Mon, 27 Jan 2020 19:46:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0222.hostedemail.com [216.40.44.222]) by kanga.kvack.org (Postfix) with ESMTP id D0D9D6B0007 for ; Mon, 27 Jan 2020 19:46:28 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with SMTP id A447D180AD802 for ; Tue, 28 Jan 2020 00:46:28 +0000 (UTC) X-FDA: 76425202056.16.tent59_388494d28843f X-HE-Tag: tent59_388494d28843f X-Filterd-Recvd-Size: 4949 Received: from mail-ot1-f67.google.com (mail-ot1-f67.google.com [209.85.210.67]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 28 Jan 2020 00:46:28 +0000 (UTC) Received: by mail-ot1-f67.google.com with SMTP id i6so6523724otr.7 for ; Mon, 27 Jan 2020 16:46:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CAMZ8Aw/a/sq7vYMNuydKAQ+vHD5b05Zm+cMKCnvvbE=; b=X4ON06sk4JCxWSzl6IyivPOV6QC6kyFa7EI/UrPdBL9l5GEVsV7qtSSKimkqt3CbtK WspgLPX8kJTVWlG6PCRAq632TiDYSewzIDQbh79DQU0hRMDb6ExTuIKNR45b9PI16NnX /g04KWqGgguKcPK8yEJ8h/t9kwND19nNLnUTrwza4YTdzPmf7/dTsT3B67JqSVR7xAPk 83D4pomEorKigx8pWW/nUo97P9A2t83O+WrAzxRYRqhDgyyE3qjPhM/7p52bYIDiHu+9 cQq4YA6ByW/C333B6qdpiE1ozu0nfBINO0G8I3a0/vZo8Gnig3BwvmXJv/JgT4LpxpSN h8ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CAMZ8Aw/a/sq7vYMNuydKAQ+vHD5b05Zm+cMKCnvvbE=; b=qBqJzOWWKCael6xaQ3z6qNFx3GsLjSxrQ0UOl3fKP0AM2C6NJbjRpryyboy7C/IJwN CgAL71MSW4R8dtnX5chcYLli2PnkWxC/D1EFAt8I75AWbkWJ4zwxT2CHeR9NgIJqQz5s 4PGXVQO8zpk8wV+yPrj5pP80NN3vPXlcNXaeAkCMzaP4/cpEOwff2G7TVt6AYn3QjmGu EbVRU5Ujh0nQVVjjST4PdfvUhVvdxjG0c55Sr4w+jk1i1nojN4aazwrIOOS96B0N6VqY f/Eba51PO250sPrfa7niUJowD44ceJfQqz+A5WOBc5p/tvs19Z+ASXFI//eGd9dAIA9c PFKw== X-Gm-Message-State: APjAAAVadmhnCDmEElLh0yVphC3mOFb7bkTztgEMslApRdSYoa+kmtpR BmpvDSRj0eVm0SQ3WsBvLAhMTXb38m4HRZ48HcY= X-Google-Smtp-Source: APXvYqzsZxz1kv92CbfpsiaZOUVTkIIwHUagpMF6gc9UEUNm/CAy7Iuj+SOwdeBhMsBZMV94Ijn46VAg+xm3ctypNb0= X-Received: by 2002:a9d:62c7:: with SMTP id z7mr14168249otk.189.1580172387271; Mon, 27 Jan 2020 16:46:27 -0800 (PST) MIME-Version: 1.0 References: <20200109225646.22983-1-xiyou.wangcong@gmail.com> <20200110073822.GC29802@dhcp22.suse.cz> <20200121090048.GG29276@dhcp22.suse.cz> <20200127144931.GM1183@dhcp22.suse.cz> In-Reply-To: <20200127144931.GM1183@dhcp22.suse.cz> From: Cong Wang Date: Mon, 27 Jan 2020 16:46:16 -0800 Message-ID: Subject: Re: [PATCH] mm: avoid blocking lock_page() in kcompactd To: Michal Hocko Cc: LKML , Andrew Morton , linux-mm , Mel Gorman , Vlastimil Babka Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jan 27, 2020 at 6:49 AM Michal Hocko wrote: > > On Sun 26-01-20 11:53:55, Cong Wang wrote: > > On Tue, Jan 21, 2020 at 1:00 AM Michal Hocko wrote: > > > > > > On Mon 20-01-20 14:48:05, Cong Wang wrote: > > > > It got stuck somewhere along the call path of mem_cgroup_try_charge(), > > > > and the trace events of mm_vmscan_lru_shrink_inactive() indicates this > > > > too: > > > > > > So it seems that you are condending on the page lock. It is really > > > unexpected that the reclaim would take that long though. Please try to > > > enable more vmscan tracepoints to see where the time is spent. > > > > Sorry for the delay. I have been trying to collect more data in one shot. > > > > This is a a typical round of the loop after enabling all vmscan tracepoints: > > > > <...>-455450 [007] .... 4048595.842992: > > mm_vmscan_memcg_reclaim_begin: order=0 may_writepage=1 > > gfp_flags=GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE > > classzone_idx=4 > > <...>-455450 [007] .... 4048595.843012: > > mm_vmscan_memcg_reclaim_end: nr_reclaimed=0 > > This doesn't tell us much though. This reclaim round has taken close to > no time. See timestamps. > > > The whole trace output is huge (33M), I can provide it on request. > > Focus on reclaim rounds that take a long time and see where it gets you. I reviewed the tracing output with my eyes, they all took little time. But of course I can't review all of them given the size is huge. For me, it seems that the loop happens in its caller, something like: retry: mm_vmscan_memcg_reclaim_begin(); ... mm_vmscan_memcg_reclaim_end(); goto retry; So, I think we should focus on try_charge()? More interestingly, the margin of that memcg is 0: $ sudo cat /sys/fs/cgroup/memory/system.slice/osqueryd.service/memory.usage_in_bytes 262144000 $ sudo cat /sys/fs/cgroup/memory/system.slice/osqueryd.service/memory.limit_in_bytes 262144000 Thanks!