From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 065A4C433E1 for ; Thu, 16 Jul 2020 05:18:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 94EC820775 for ; Thu, 16 Jul 2020 05:18:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=amacapital-net.20150623.gappssmtp.com header.i=@amacapital-net.20150623.gappssmtp.com header.b="SzLH2bWv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 94EC820775 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amacapital.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A07AE8D0002; Thu, 16 Jul 2020 01:18:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B8F28D0001; Thu, 16 Jul 2020 01:18:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87EC88D0002; Thu, 16 Jul 2020 01:18:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id 6FC128D0001 for ; Thu, 16 Jul 2020 01:18:28 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EA32D180AD802 for ; Thu, 16 Jul 2020 05:18:27 +0000 (UTC) X-FDA: 77042783454.26.salt79_21173c126eff Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id B05261804B667 for ; Thu, 16 Jul 2020 05:18:27 +0000 (UTC) X-HE-Tag: salt79_21173c126eff X-Filterd-Recvd-Size: 8953 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Thu, 16 Jul 2020 05:18:27 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id a14so3139297pfi.2 for ; Wed, 15 Jul 2020 22:18:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amacapital-net.20150623.gappssmtp.com; s=20150623; h=content-transfer-encoding:from:mime-version:subject:date:message-id :references:cc:in-reply-to:to; bh=pjSAQ6Btld2fycHWHEXEy5l6TD0W6bB4dFccT/gpVM8=; b=SzLH2bWvof9kr76xot8dGve40BlKMean/0Ynh3Mixvp/iYGI6jslyiT1H/1n9Qj1ow Mh6ujatjCTcxc2s25k7iA9sn8U89yBeRY8ebObRWZj6HYuZXFeSDTBb/EWPHPkSP4FNU WwDuuMXK/rQCsP66Mjz/JMKuDuyf2EqCTf0wV7Jeao4eNmnGi3HcAtNuOo1DPvCl3PrS G8xS6Xv4Ca2XuthzckdOpVp0+F0+/78sRIME7/YVN8R9H8xTFBhlymC5jiziD9Jh1/PT 1dEIfLgzq81IlVYg4rVZj6jYicZCgwuZlDfimKz4575Fr0aZwG6NbpNsQCvhpVa8sGpY J2Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:date:message-id:references:cc:in-reply-to:to; bh=pjSAQ6Btld2fycHWHEXEy5l6TD0W6bB4dFccT/gpVM8=; b=fOcVuQxmPPIPoArvPip2ql7+eWEp5/SOmh3zsOF4M/fUYQmQ0m3NwB2rncwu9PcHCz J5gy52KORjWJNNNVu7n5HBlozFiMk3ouqIlI7Kx+1gwQj6a1pV6hsJyB1SBIsD3fF+aH oe5uX59Sph9/rudODBAUa9HCd09M5/F4jUnLK1dzBPN0Z3P5NctYQEkYoin+HDN7Uxmr WSIJMtzkn4dM8YkSGeXvjt71BJUBVoZMKxjtrJG2WSwHDYDAD6HSkHvRZurq3mzAp02v D1EzSRXln3tJ5TvEeCDCWVgGAE618OLPZ20CcxKff3AWgGY24Os0kwBpMKz/NHCLfgbG zgrg== X-Gm-Message-State: AOAM531c+c0fUqbDd6KFrlRe/Dz4rQ4Zu5F+HfT7m5nIdF/Rm6O1XUuG YWr/zX9LcVbmyrJoegq1x2ysNw== X-Google-Smtp-Source: ABdhPJz36lMtXy7jtJjFqq3zOAV60PvUS/Hlk91z+f9BEuTeTezdLYpRcGCyT7+UAXDv2D+6odC4ig== X-Received: by 2002:a63:444b:: with SMTP id t11mr2886208pgk.134.1594876706172; Wed, 15 Jul 2020 22:18:26 -0700 (PDT) Received: from ?IPv6:2601:646:c200:1ef2:c8f2:5437:b9af:674c? ([2601:646:c200:1ef2:c8f2:5437:b9af:674c]) by smtp.gmail.com with ESMTPSA id k7sm3623730pgh.46.2020.07.15.22.18.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 15 Jul 2020 22:18:25 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Andy Lutomirski Mime-Version: 1.0 (1.0) Subject: Re: [RFC PATCH 4/7] x86: use exit_lazy_tlb rather than membarrier_mm_sync_core_before_usermode Date: Wed, 15 Jul 2020 22:18:20 -0700 Message-Id: References: <1594868476.6k5kvx8684.astroid@bobo.none> Cc: Mathieu Desnoyers , Anton Blanchard , Arnd Bergmann , linux-arch , linux-kernel , linux-mm , linuxppc-dev , Andy Lutomirski , Peter Zijlstra , x86 In-Reply-To: <1594868476.6k5kvx8684.astroid@bobo.none> To: Nicholas Piggin X-Mailer: iPhone Mail (17F80) X-Rspamd-Queue-Id: B05261804B667 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > On Jul 15, 2020, at 9:15 PM, Nicholas Piggin wrote: >=20 > =EF=BB=BFExcerpts from Mathieu Desnoyers's message of July 14, 2020 12:13 a= m: >> ----- On Jul 13, 2020, at 9:47 AM, Nicholas Piggin npiggin@gmail.com wrot= e: >>=20 >>> Excerpts from Nicholas Piggin's message of July 13, 2020 2:45 pm: >>>> Excerpts from Andy Lutomirski's message of July 11, 2020 3:04 am: >>>>> Also, as it stands, I can easily see in_irq() ceasing to promise to >>>>> serialize. There are older kernels for which it does not promise to >>>>> serialize. And I have plans to make it stop serializing in the >>>>> nearish future. >>>>=20 >>>> You mean x86's return from interrupt? Sounds fun... you'll konw where t= o >>>> update the membarrier sync code, at least :) >>>=20 >>> Oh, I should actually say Mathieu recently clarified a return from >>> interrupt doesn't fundamentally need to serialize in order to support >>> membarrier sync core. >>=20 >> Clarification to your statement: >>=20 >> Return from interrupt to kernel code does not need to be context serializ= ing >> as long as kernel serializes before returning to user-space. >>=20 >> However, return from interrupt to user-space needs to be context serializ= ing. >=20 > Hmm, I'm not sure it's enough even with the sync in the exit_lazy_tlb > in the right places. >=20 > A kernel thread does a use_mm, then it blocks and the user process with > the same mm runs on that CPU, and then it calls into the kernel, blocks, > the kernel thread runs again, another CPU issues a membarrier which does > not IPI this one because it's running a kthread, and then the kthread > switches back to the user process (still without having unused the mm), > and then the user process returns from syscall without having done a=20 > core synchronising instruction. >=20 > The cause of the problem is you want to avoid IPI'ing kthreads. Why? > I'm guessing it really only matters as an optimisation in case of idle > threads. Idle thread is easy (well, easier) because it won't use_mm, so=20= > you could check for rq->curr =3D=3D rq->idle in your loop (in a suitable=20= > sched accessor function). >=20 > But... I'm not really liking this subtlety in the scheduler for all this=20= > (the scheduler still needs the barriers when switching out of idle). >=20 > Can it be improved somehow? Let me forget x86 core sync problem for now > (that _may_ be a bit harder), and step back and look at what we're doing. > The memory barrier case would actually suffer from the same problem as > core sync, because in the same situation it has no implicit mmdrop in > the scheduler switch code either. >=20 > So what are we doing with membarrier? We want any activity caused by the=20= > set of CPUs/threads specified that can be observed by this thread before=20= > calling membarrier is appropriately fenced from activity that can be=20 > observed to happen after the call returns. >=20 > CPU0 CPU1 > 1. user stuff > a. membarrier() 2. enter kernel > b. read rq->curr 3. rq->curr switched to kthread > c. is kthread, skip IPI 4. switch_to kthread > d. return to user 5. rq->curr switched to user thread > 6. switch_to user thread > 7. exit kernel > 8. more user stuff >=20 > As far as I can see, the problem is CPU1 might reorder step 5 and step > 8, so you have mmdrop of lazy mm be a mb after step 6. >=20 > But why? The membarrier call only cares that there is a full barrier > between 1 and 8, right? Which it will get from the previous context > switch to the kthread. >=20 > I must say the memory barrier comments in membarrier could be improved > a bit (unless I'm missing where the main comment is). It's fine to know > what barriers pair with one another, but we need to know which exact > memory accesses it is ordering >=20 > /* > * Matches memory barriers around rq->curr modification in > * scheduler. > */ >=20 > Sure, but it doesn't say what else is being ordered. I think it's just > the user memory accesses, but would be nice to make that a bit more > explicit. If we had such comments then we might know this case is safe. >=20 > I think the funny powerpc barrier is a similar case of this. If we > ever see remote_rq->curr->flags & PF_KTHREAD, then we _know_ that > CPU has or will have issued a memory barrier between running user > code. >=20 > So AFAIKS all this membarrier stuff in kernel/sched/core.c could > just go away. Except x86 because thread switch doesn't imply core > sync, so CPU1 between 1 and 8 may never issue a core sync instruction > the same way a context switch must be a full mb. >=20 > Before getting to x86 -- Am I right, or way off track here? I find it hard to believe that this is x86 only. Why would thread switch imp= ly core sync on any architecture? Is x86 unique in having a stupid expensiv= e core sync that is heavier than smp_mb()? But I=E2=80=99m wondering if all this deferred sync stuff is wrong. In the b= rave new world of io_uring and such, perhaps kernel access matter too. Heck= , even: int a[2]; Thread A: a[0] =3D 1; a[1] =3D 2: Thread B: write(fd, a, sizeof(a)); Doesn=E2=80=99t do what thread A is expecting. Admittedly this particular e= xample is nonsense, but maybe there are sensible cases that matter to someon= e. =E2=80=94Andy >=20 > Thanks, > Nick