From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.1 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53518C433E2 for ; Thu, 16 Jul 2020 14:14:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 10A75207DD for ; Thu, 16 Jul 2020 14:14:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mSUV7Ne4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10A75207DD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B18786B0087; Thu, 16 Jul 2020 10:14:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEC246B0088; Thu, 16 Jul 2020 10:14:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A05838D001F; Thu, 16 Jul 2020 10:14:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0176.hostedemail.com [216.40.44.176]) by kanga.kvack.org (Postfix) with ESMTP id 8B0446B0087 for ; Thu, 16 Jul 2020 10:14:55 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 42DE9181AEF07 for ; Thu, 16 Jul 2020 14:14:55 +0000 (UTC) X-FDA: 77044135350.15.brush48_5e1770a26f02 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id 2635818027468 for ; Thu, 16 Jul 2020 14:11:25 +0000 (UTC) X-HE-Tag: brush48_5e1770a26f02 X-Filterd-Recvd-Size: 6330 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Thu, 16 Jul 2020 14:11:24 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id q74so6173031iod.1 for ; Thu, 16 Jul 2020 07:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZwX/mhlAD3zr833qIWvUPD76zWFR7r5RHDAo4CaqDJs=; b=mSUV7Ne41+i4WmO2Vm8ptf3P9RJODM2Y9Mg+O0LU/7IyBGkaVo1SQrKWvgj1Aqnoy8 g08NJT8igdEaGYT6EwoRxBGeK4NsqhiRzrMZ+xgito13FEBBjxlN5v15bCpgv6DKnkkG I10aEZB/G9JJ5HXxoPIO1KVUCT5+ZC5u7WRj6wXuOBwVXYuO3XCXM0XI3GnwJ6bhUPqc TvUNrXNCEPqbBByGmnniW7ylJGm2aUQEArKr9pbDkCj/RCbqTXdx5deU0iC+/OFcvqIx MD+8n1h3b7ddIhzb3mMpul58aqCsG4fODqusChCjuq8/vBxZnkDCtxt4sbpsciw72B66 EH+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZwX/mhlAD3zr833qIWvUPD76zWFR7r5RHDAo4CaqDJs=; b=QAvljKQMKIFCySZjpfkTBt389/Bg/lMS45a/h5iHowEspPIXMiGAWWb1/s5uIhMNK7 /eJp5IQ5CqEtQh6lx4NzIxO9Q+Igy779bX3s6RtcpRd56B8Xy/eaUl1d1vWSPJzBZ7a/ qo5ZKIf3H0oZvtD1aUn/Duv5JdImrori1yAW8YvI6i3kSbo/rQZ7TAP0ToiHyAad9hWp 0cCsxbXBo1RozheKSbWihr1TygasS6Qei5jo/f6jHJwIl9ObUkidN4LJQampTPtHytTd XuMzRKOoCpUg10BSIFYVQwHSwXm66FGGhyoWcep35NLsM02zDTQ1UyauH/k1BJcHY2C4 wphQ== X-Gm-Message-State: AOAM531jnOXhu9MlFImO4jFws2zRoI2qGlSHBisEGgqGZ13OyAtOYc9L vSQA2aYt9Jo93OilnuicWglcE+tWBHpeenN9DHs= X-Google-Smtp-Source: ABdhPJy91dfh0GT+dqKF3C2rWnAEox2zlavfLO3RPZaRa7Nk/LRvW3ITiol1O0WjbC5f8lyH21Ai23FETBwlxZ8cts0= X-Received: by 2002:a02:c888:: with SMTP id m8mr5117977jao.114.1594908683752; Thu, 16 Jul 2020 07:11:23 -0700 (PDT) MIME-Version: 1.0 References: <1594429136-20002-1-git-send-email-alex.shi@linux.alibaba.com> In-Reply-To: <1594429136-20002-1-git-send-email-alex.shi@linux.alibaba.com> From: Alexander Duyck Date: Thu, 16 Jul 2020 07:11:12 -0700 Message-ID: Subject: Re: [PATCH v16 00/22] per memcg lru_lock To: Alex Shi Cc: Andrew Morton , Mel Gorman , Tejun Heo , Hugh Dickins , Konstantin Khlebnikov , Daniel Jordan , Yang Shi , Matthew Wilcox , Johannes Weiner , kbuild test robot , linux-mm , LKML , cgroups@vger.kernel.org, Shakeel Butt , Joonsoo Kim , Wei Yang , "Kirill A. Shutemov" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 2635818027468 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jul 10, 2020 at 5:59 PM Alex Shi wrote: > > The new version which bases on v5.8-rc4. Add 2 more patchs: > 'mm/thp: remove code path which never got into' > 'mm/thp: add tail pages into lru anyway in split_huge_page()' > and modified 'mm/mlock: reorder isolation sequence during munlock' > > Current lru_lock is one for each of node, pgdat->lru_lock, that guard for > lru lists, but now we had moved the lru lists into memcg for long time. Still > using per node lru_lock is clearly unscalable, pages on each of memcgs have > to compete each others for a whole lru_lock. This patchset try to use per > lruvec/memcg lru_lock to repleace per node lru lock to guard lru lists, make > it scalable for memcgs and get performance gain. > > Currently lru_lock still guards both lru list and page's lru bit, that's ok. > but if we want to use specific lruvec lock on the page, we need to pin down > the page's lruvec/memcg during locking. Just taking lruvec lock first may be > undermined by the page's memcg charge/migration. To fix this problem, we could > take out the page's lru bit clear and use it as pin down action to block the > memcg changes. That's the reason for new atomic func TestClearPageLRU. > So now isolating a page need both actions: TestClearPageLRU and hold the > lru_lock. > > The typical usage of this is isolate_migratepages_block() in compaction.c > we have to take lru bit before lru lock, that serialized the page isolation > in memcg page charge/migration which will change page's lruvec and new > lru_lock in it. > > The above solution suggested by Johannes Weiner, and based on his new memcg > charge path, then have this patchset. (Hugh Dickins tested and contributed much > code from compaction fix to general code polish, thanks a lot!). > > The patchset includes 3 parts: > 1, some code cleanup and minimum optimization as a preparation. > 2, use TestCleanPageLRU as page isolation's precondition > 3, replace per node lru_lock with per memcg per node lru_lock > > Following Daniel Jordan's suggestion, I have run 208 'dd' with on 104 > containers on a 2s * 26cores * HT box with a modefied case: > https://git.kernel.org/pub/scm/linux/kernel/git/wfg/vm-scalability.git/tree/case-lru-file-readtwice > With this patchset, the readtwice performance increased about 80% > in concurrent containers. > > Thanks Hugh Dickins and Konstantin Khlebnikov, they both brought this > idea 8 years ago, and others who give comments as well: Daniel Jordan, > Mel Gorman, Shakeel Butt, Matthew Wilcox etc. > > Thanks for Testing support from Intel 0day and Rong Chen, Fengguang Wu, > and Yun Wang. Hugh Dickins also shared his kbuild-swap case. Thanks! Hi Alex, I think I am seeing a regression with this patch set when I run the will-it-scale/page_fault3 test. Specifically the processes result is dropping from 56371083 to 43127382 when I apply these patches. I haven't had a chance to bisect and figure out what is causing it, and wanted to let you know in case you are aware of anything specific that may be causing this. Thanks. - Alex