From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59043C54E94 for ; Wed, 25 Jan 2023 07:44:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C99526B0073; Wed, 25 Jan 2023 02:44:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C491F6B0075; Wed, 25 Jan 2023 02:44:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC29D6B007D; Wed, 25 Jan 2023 02:44:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 989F96B0073 for ; Wed, 25 Jan 2023 02:44:27 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 550AA120B78 for ; Wed, 25 Jan 2023 07:44:27 +0000 (UTC) X-FDA: 80392533774.21.FFD3780 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id 1E90A120007 for ; Wed, 25 Jan 2023 07:44:24 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ecETXQha; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632665; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BkeMwblj9hneaBAEGr4WY7HayLEL2jAXC1+f00C91Gc=; b=7jQ2aC7yUwibNRYUt4fBD8eZhDJz8D3un1G+AQRduIcZP8dHziAdIkmNhdN6bMEbGZXpD/ HIYzEr4tuRQMGGODldWTncpRPfwzwEUUMCQxqXHSXtS1lupRI+msx1PsAkLkvuHwePUMqs kEakLOUFg31HPgepz6e8k+Y35KGSIVA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ecETXQha; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632665; a=rsa-sha256; cv=none; b=YgmBaYMe6/8SwLeWC0R0Wq1ws950Q34ulyK4d6rULg2ba1dMes8lID7pC17bpLyqENWJw0 Q+3S9t1DCZKL0Xq+zlxWQHqjd7DHdGWYLDYEE9oGUnFR+kGXoJXac1kTwypDZqAUrMMD4w sd+G8BFFFnI/pqRW475GAIrnq840G24= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632664; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BkeMwblj9hneaBAEGr4WY7HayLEL2jAXC1+f00C91Gc=; b=ecETXQha7bGH2bR2B1N0BH8X1pQUXfLZ4I8+4UaPB33qDQsHmssB0pul9kyW1idN8dnSSK 6DPqAA9X7zf0BqvwgrIOHHWHn8IXbebSkh0nVMrCUejl5UJFFa/f5IeoOkRMPyje9qkpiO xxUtIsB0edwvRtVrZQhzNi1SVZImVB4= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-67-4XZD8rH8PyuFhtU51PUtpg-1; Wed, 25 Jan 2023 02:44:23 -0500 X-MC-Unique: 4XZD8rH8PyuFhtU51PUtpg-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-15fad335c1dso6688451fac.15 for ; Tue, 24 Jan 2023 23:44:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BkeMwblj9hneaBAEGr4WY7HayLEL2jAXC1+f00C91Gc=; b=QPnWKFJXC8N4xklOzqE7rWjZbCPnyi+tf0dBZAaJuSoAzyQAFRuqSidOwJZcHhm01q Etm3pGPJv3JJ8SnsnAB5KQrWZjC6TYOHckNH1Kkzc8uFXuDwgq6JOQVLaAE0El1b9E/Q g+Xhi38vdc0w5cmYTMcQFtv1RovDIGI3vmziOshpo/hiUfRscncjEW0n5cDU91P46V4x ojBVqrvGZ93e8V9OQMxTbxWCwn8uPr878ETUnsF9T9JEVwhpewPB+87QHtFv1KnmpBM4 Cpinzq+7NZGNlDNASmmZsXVGMT1wf7YGUGt58vEnlhJFmEWBHuK38y8BPc3k7AQTuqVM WXcQ== X-Gm-Message-State: AFqh2koxBgvQSTCUaVyeC7xeSTXOUp7bK9fK0Z4I01NUPCJ4ynsztOuK opnIW0N6RFQEwdq6uWwK50y9MZTZQoRY7LoO2p94rb8WzAxB7rS3476vWUMbPsHL4xC4gQVTYXE xRbRIYjdSwrc= X-Received: by 2002:a05:6870:4288:b0:15e:e50c:1813 with SMTP id y8-20020a056870428800b0015ee50c1813mr17604041oah.55.1674632662615; Tue, 24 Jan 2023 23:44:22 -0800 (PST) X-Google-Smtp-Source: AMrXdXsmOOC0na3goHYTCuYUR80h3zsrixqTPIdufe5WuJ2AFLGjawvqUoIdakZTwJYrZcPAxvXVeg== X-Received: by 2002:a05:6870:4288:b0:15e:e50c:1813 with SMTP id y8-20020a056870428800b0015ee50c1813mr17604031oah.55.1674632662379; Tue, 24 Jan 2023 23:44:22 -0800 (PST) Received: from ?IPv6:2804:1b3:a800:14fa:9361:c141:6c70:c877? ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id p9-20020a4ad449000000b004fb2935d0e7sm1574351oos.36.2023.01.24.23.44.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:44:21 -0800 (PST) Message-ID: <958969c204e1041dead005d1c801cf3c54ab86f1.camel@redhat.com> Subject: Re: [PATCH v1 0/3] Avoid scheduling cache draining to isolated cpus From: Leonardo =?ISO-8859-1?Q?Br=E1s?= To: Michal Hocko Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Phil Auld , Marcelo Tosatti , linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Date: Wed, 25 Jan 2023 04:44:12 -0300 In-Reply-To: References: <20221102020243.522358-1-leobras@redhat.com> <07810c49ef326b26c971008fb03adf9dc533a178.camel@redhat.com> <0183b60e79cda3a0f992d14b4db5a818cd096e33.camel@redhat.com> <3c4ae3bb70d92340d9aaaa1856928476641a8533.camel@redhat.com> <4a4a6c73f3776d65f70f7ca92eb26fc90ed3d51a.camel@redhat.com> User-Agent: Evolution 3.46.2 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 1E90A120007 X-Stat-Signature: dj3rtgfgbbs37h5yuakxzyr8z894pubs X-Rspam-User: X-HE-Tag: 1674632664-652717 X-HE-Meta: U2FsdGVkX1/t3zfIuWyo79GrxAy1NAmaiBOg37RMYg64CsBROIDJMN44ZoOQ9OWISu+0c8YzKkREYE4bRErAbCCWjDriemNQkWrz5979ODpwcpII+uwc24AcqZOmZbsnzppXk6VvcWlIcFbuJjtOJNdsfhEOpDniJhgeEf0juJ3xOb6MNUegJBD6yC0kZkhraSvbQS6BozlU4nBFtksAn7jZxgq8MB8bUF8oIqi8H/JJwAzUe1e9/WbaXQTM/fySACrNT8G5biXeazPZS2OpKR8fX8vs1CxbHD0mTgOgNjaNLKJuopPsL9cwRkp/ohvgHUKOS6SQn03+LSGNLxiJc/+THVis3aZ/6LWrIX8GdSDciSs1eqUXgN4rLw91eIGdMNpTrQ+CQskoqmRzwepWgKYLcUbGjrchDGiVIEKOcq86XGWkepgk1aJkp5uGkLsbRRObvOhLkWJV3hptzm9EcejAbJ3Y02tHWzE7syKuqNbRL9swThOjMjf8GwMAAlwGg5Zva2v+KLDsdG/eRp4qjlq6B/2eMN5J64L1Cj5tDGIOmmi1jMYbFgV5Fl8h8peHr1Q4f23aSnFyESr61JR6eR6/DYD0FGKTiY1W1/NGzS/r2bfXebIUS8utgW+mUra+VdJSKeudQy9KHooqsW3HJahJDHYNQwyYdztyiIo99QvzUHhdjh0BzcVAwjXcE5D8TwzwUv4OTjoGKXkcMo6K7gBg+QaPA/dMcaW5OViyNng3jWBIwCR2gyzSQYjpn44RsW9aWG348bA+KWKEbl0tbF2STsLFzeK9qsQnI/Yj/KTOaCYaVaCV2pHsTzBSETKPyaoICMIzPjDXymmryIBAP8Cl7oHPaRxh3inoWRqGf+a1g47SMuLC0zAUcLCmHH50tvD/MyUSp5YKNTxT/8vcoh7j8d3mRFAXjGcmVKtTUcZHNB8x0iNEmtLp9VP4J3gOIQgL0L9wQBlgo7E/bys IgJRcooc kdAWm68pFvTYnce/+09bXoFDYUsdafWrKjChzfnLaWrhqqk+OUcdqyh0F8BeBvhuQrWBTbrlvEQu2WGL6yMmFmvPSigp8nO6r8WJxgd4SLEFK7TNSl09QLc4NJW+zwSfKyvO5IhS3cRBbyH7OHP5gfFOtaG1atYIkBljUJWKNRlw5yDZqQl5vJ+hWTqKM6WdjkHt4bqvcPX4w9+cq2UPa8pW1RP+JTUrbITTi1YjfYrVuRY6pSfGcu9DHUAPtCCv3ca/f6j8XFLZJ4ueJOvl1ylgz1n0m8+YVc6M206mB++BrpLtgwZ6mnW94cd/Qxc2CSSLeKXW+b3C7KTBbGQxtSd1bCXpaH+qZhhqVq4vkLnUi6q910N53xVWuQL8ulTq47KRJGoypVvQ+fxt5oFWJ5kNSLO1ChnBsijjVzfkcyoJlhFTyjRQc3+CvM6RnT/68cReRNOc6r5Ay3pRbX53NHN7YYc5Mq41ykJOp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 2022-11-09 at 09:05 +0100, Michal Hocko wrote: > On Tue 08-11-22 20:09:25, Leonardo Br=C3=A1s wrote: > [...] > > > Yes, with a notable difference that with your spin lock option there = is > > > still a chance that the remote draining could influence the isolated = CPU > > > workload throug that said spinlock. If there is no pcp cache for that > > > cpu being used then there is no potential interaction at all. > >=20 > > I see.=20 > > But the slow path is slow for some reason, right? > > Does not it make use of any locks also? So on normal operation there co= uld be a > > potentially larger impact than a spinlock, even though there would be n= o > > scheduled draining. >=20 > Well, for the regular (try_charge) path that is essentially page_counter_= try_charge > which boils down to atomic_long_add_return of the memcg counter + all > parents up the hierarchy and high memory limit evaluation (essentially 2 > atomic_reads for the memcg + all parents up the hierchy). That is not > whole of a lot - especially when the memcg hierarchy is not very deep. >=20 > Per cpu batch amortizes those per hierarchy updates as well as atomic > operations + cache lines bouncing on updates. >=20 > On the other hand spinlock would do the unconditional atomic updates as > well and even much more on CONFIG_RT. A plus is that the update will be > mostly local so cache line bouncing shouldn't be terrible. Unless > somebody heavily triggers pcp cache draining but this shouldn't be all > that common (e.g. when a memcg triggers its limit. >=20 > All that being said, I am still not convinced that the pcp cache bypass > for isolated CPUs would make a dramatic difference. Especially in the > context of workloads that tend to run on isolated CPUs and rarely enter > kernel. > =20 > > > It is true true that appart from user > > > space memory which can be under full control of the userspace there a= re > > > kernel allocations which can be done on behalf of the process and tho= se > > > could be charged to memcg as well. So I can imagine the pcp cache cou= ld > > > be populated even if the process is not faulting anything in during R= T > > > sensitive phase. > >=20 > > Humm, I think I will apply the change and do a comparative testing with > > upstream. This should bring good comparison results. >=20 > That would be certainly appreciated! > ( > > > > On the other hand, compared to how it works now now, this should be= a more > > > > controllable way of introducing latency than a scheduled cache drai= n. > > > >=20 > > > > Your suggestion on no-stocks/caches in isolated CPUs would be great= for > > > > predictability, but I am almost sure the cost in overall performanc= e would not > > > > be fine. > > >=20 > > > It is hard to estimate the overhead without measuring that. Do you th= ink > > > you can give it a try? If the performance is not really acceptable > > > (which I would be really surprised) then we can think of a more compl= ex > > > solution. > >=20 > > Sure, I can try that. > > Do you suggest any specific workload that happens to stress the percpu = cache > > usage, with usual drains and so? Maybe I will also try with synthetic w= orloads > > also. >=20 > I really think you want to test it on the isolcpu aware workload. > Artificial benchmark are not all that useful in this context. Hello Michael, I just sent a v2 for this patchset with a lot of changes. https://lore.kernel.org/lkml/20230125073502.743446-1-leobras@redhat.com/ I have tried to gather some data on the performance numbers as suggested, b= ut I got carried away and the cover letter ended up too big. I hope it's not too= much trouble. Best regards, Leo