From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E876ECE58C for ; Mon, 14 Oct 2019 18:31:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0EE9B20659 for ; Mon, 14 Oct 2019 18:31:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0EE9B20659 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AFE6A8E0005; Mon, 14 Oct 2019 14:31:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A87BE8E0001; Mon, 14 Oct 2019 14:31:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 975728E0005; Mon, 14 Oct 2019 14:31:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0079.hostedemail.com [216.40.44.79]) by kanga.kvack.org (Postfix) with ESMTP id 6EF6C8E0001 for ; Mon, 14 Oct 2019 14:31:11 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with SMTP id E6C5D80465FE for ; Mon, 14 Oct 2019 18:31:10 +0000 (UTC) X-FDA: 76043232300.03.shock19_7eb0af4e2e540 X-HE-Tag: shock19_7eb0af4e2e540 X-Filterd-Recvd-Size: 3356 Received: from mx1.suse.de (mx2.suse.de [195.135.220.15]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Mon, 14 Oct 2019 18:31:10 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id C86EABB6A; Mon, 14 Oct 2019 18:31:08 +0000 (UTC) Date: Mon, 14 Oct 2019 20:31:07 +0200 From: Michal Hocko To: Tim Chen Cc: Dave Hansen , Honglei Wang , Johannes Weiner , linux-mm@kvack.org, Andrew Morton Subject: Re: memcgroup lruvec_lru_size scaling issue Message-ID: <20191014183107.GO317@dhcp22.suse.cz> References: <20191014173723.GM317@dhcp22.suse.cz> <0da86744-be11-06a8-2b38-5525dfe9d21e@intel.com> <20191014175918.GN317@dhcp22.suse.cz> <40748407-eafc-e08b-5777-1cbf892fcc52@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <40748407-eafc-e08b-5777-1cbf892fcc52@linux.intel.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon 14-10-19 11:06:24, Tim Chen wrote: > On 10/14/19 10:59 AM, Michal Hocko wrote: > > On Mon 14-10-19 10:49:49, Dave Hansen wrote: > >> On 10/14/19 10:37 AM, Michal Hocko wrote: > >>>> for_each_possible_cpu(cpu) > >>>> x += per_cpu(pn->lruvec_stat_local->count[idx], cpu); > >>>> > >>>> It is costly looping through all the cpus to get the lru vec size info. > >>>> And doing this on our workload with 96 cpu threads and 500 mem cgroups > >>>> makes things much worse. We might end up having 96 cpus * 500 cgroups * 2 (main) LRUs pagevecs, > >>>> which is a lot of data structures to be running through all the time. > >>> Why does the number of cgroup matter? > >> > >> I was thinking purely of the cache footprint. If it's reading > >> pn->lruvec_stat_local->count[idx] is three separate cachelines, so 192 > >> bytes of cache *96 CPUs = 18k of data, mostly read-only. 1 cgroup would > >> be 18k of data for the whole system and the caching would be pretty > >> efficient and all 18k would probably survive a tight page fault loop in > >> the L1. 500 cgroups would be ~90k of data per CPU thread which doesn't > >> fit in the L1 and probably wouldn't survive a tight page fault loop if > >> both logical threads were banging on different cgroups. > >> > >> It's just a theory, but it's why I noted the number of cgroups when I > >> initially saw this show up in profiles. > > > > Yes, the cache traffic might be really high but I still find it a bit > > surprising that it makes such a large footprint because this should be > > mostly called from slow paths (reclaim) and the real work done should > > just be larger - at least that's my intuition which might be quite off > > here. How much is that 25% of the system time in the total time btw? > > > > About 7% of total cpu cycles. OK, 7% on a highly direct reclaim bound workload sounds more realistic. Please put all that to the changelog. It would be also great to see whether that really scales with the number of cgroups if that is easy to check with your benchmark. -- Michal Hocko SUSE Labs