From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 476C5C54EAA for ; Mon, 23 Jan 2023 07:36:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EE116B0071; Mon, 23 Jan 2023 02:36:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 29E838E0001; Mon, 23 Jan 2023 02:36:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13F876B0073; Mon, 23 Jan 2023 02:36:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 018B36B0071 for ; Mon, 23 Jan 2023 02:36:01 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B6F721C5F33 for ; Mon, 23 Jan 2023 07:36:01 +0000 (UTC) X-FDA: 80385254922.08.6C07B12 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf30.hostedemail.com (Postfix) with ESMTP id D362F80014 for ; Mon, 23 Jan 2023 07:35:59 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=B93OkJvi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nadav.amit@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674459359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wgOELi+0GLLFDRqtM9K/SOPPwlEPk5gkUlBy27WnbHg=; b=kHTRqI02wK/r8o5TMRvrcAYBe1eUYgrEnecQDaNtVfj5YaLO6GiXE/lqvR8pRb51/4xkCR nWiZyDFeoN0g0kc2M6GDAomJeKIJRX5EiA6R9l0POgP0ojsgTROhGIhZN1spK4FDUwgk7j htUOC8jjt6AcRkoOf29jY2IGU7tCraE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=B93OkJvi; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of nadav.amit@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=nadav.amit@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674459359; a=rsa-sha256; cv=none; b=kiWAA947FLO/Mq0lke4ggwvxGcYc+F1hcv6t/sIJoRfMstjX2dQNvtSfODeWyXgMkwsJM7 LlB2ynatPN0jF/67HvlhoMI1RhRDkq51D/RJOuNQQ0uev2tyLxh1Hv6EyPrwJ+KQM/DfvO ZjA7VgfN3M7zoupq+P1AzeazonSk/CM= Received: by mail-wr1-f41.google.com with SMTP id d14so6196464wrr.9 for ; Sun, 22 Jan 2023 23:35:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=wgOELi+0GLLFDRqtM9K/SOPPwlEPk5gkUlBy27WnbHg=; b=B93OkJviOA6e8TpHPeqQjlji4hoLV71Kb4swdlcDUj2ruZmkuPLm+wtIcOaMSikxGX zGf75x2K1pBgYlohzbaY80LlAsl0apssgjasj69sed7Rcqkz+f35/psiVWCp/ES777M0 F/ZYCM25CgrWbYi83KDrTCg3KaYpKfVpg7B4CBm+eLPuKj8ufmjPB+NCVDY6vPSVqnLH EqaTHJM879e2ZQFQ6cxN0IY8hwwuxguNcZd5sN/P8Uu7XGSZdlaxOA5+ywT76M9yWBXy ArCVGU8yTsV597fRPIGL9OSSflkdbWlCSLRPwY4D7FJHb7oPiyXWI4q7zO9ZCGH5Ev6X MxSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wgOELi+0GLLFDRqtM9K/SOPPwlEPk5gkUlBy27WnbHg=; b=qBQl3qz+CwQ6nw1zaB5PT6k9a07yihyPEKHgyxYdXEHtIYoNps+SYIF2sb1b5pGJUd 10UvXh408/XDWInibj5cOnsdmkBcTO90p+VzUCFX8PIrSp9fGFrzuJPiLvHqsQvV6jOt RVcZvGgqOEilgURoA+PT53+zR3PgJXEyy0MaVPaYPqCy2YPIKbxjgZoUCDdzbMnYWPQo GHaSawoPlLIBOREMUUfCoeN+tke81Ehes7+Z1Rcp3HYvJ55BZ/Hg7a2Ud5RpbtTVtM5Z UkKm3IugVD70ivS7tj95i+967CMn6D4VtfSSupENILZKLKWpv4sp+3fUxH3xLzSu74ww o0dw== X-Gm-Message-State: AFqh2kqqWfGPgUxuguKVu5p4qtgOosDUlh9Lj7c1gJ7BDsiqw1KlaLXA FRcOBzDifgwCaLofifRkpAY= X-Google-Smtp-Source: AMrXdXuqSx4yAhQaR73tNlp4h3VYBOLpcs4hjb1mTAIQerCXFQaQnHcLGQeXEz6WGulurruLqj7JGg== X-Received: by 2002:a05:6000:1708:b0:2bd:daf1:9e4c with SMTP id n8-20020a056000170800b002bddaf19e4cmr22685079wrc.43.1674459358027; Sun, 22 Jan 2023 23:35:58 -0800 (PST) Received: from [192.168.86.94] ([77.137.66.37]) by smtp.gmail.com with ESMTPSA id e17-20020adfe7d1000000b0024cb961b6aesm4138922wrn.104.2023.01.22.23.35.56 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 22 Jan 2023 23:35:57 -0800 (PST) Message-ID: Date: Mon, 23 Jan 2023 09:35:55 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.7.0 Subject: Re: [PATCH v6 2/5] lazy tlb: allow lazy tlb mm refcounting to be configurable Content-Language: en-US To: Nicholas Piggin , Andrew Morton Cc: Andy Lutomirski , Linus Torvalds , linux-arch , linux-mm , linuxppc-dev@lists.ozlabs.org References: <20230118080011.2258375-1-npiggin@gmail.com> <20230118080011.2258375-3-npiggin@gmail.com> From: Nadav Amit In-Reply-To: <20230118080011.2258375-3-npiggin@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D362F80014 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: x3uoiypdsspwf3wx37utyaozj4ektcbj X-HE-Tag: 1674459359-696314 X-HE-Meta: U2FsdGVkX19Q3o2s+e0jLgWr0rG/i/W9YIR2rZW0B0cEocC+p4iracLRKtL5clXz9pQ1beJqbpDAR2AeXOwFzImjBblSYYo4Z3QrUeuVw8hi7MiaXjn8xoulyf9NFtJlBsixar3CZCkHuqyF1REOBX/xE7fo9zmqTzEYDf85PD6iDfFez7HttLAl8tGWgorWTyhoXPtQXnr8ffXdn6OmZudlPD5pEdqYNatzIivcUkErsa6dpou1n6PAQTU1YO618JggDR5oLOIyRR9i3b3Te+4JHdiQXGu3E4BhXJO/USk85znO4dA0TUHK4u9WHbiPbCidkRLv4KHz3l19DYhoVPhunFOGGRZe6alxavAGB1c8PNf7lUxZoKGvkLU5BtyUJZvapTvdFztK1j2K2kHhSI5TA1h9PTOqxK/MuLWfr18xipDmT6IbOK4dIVORHS3crMBNGE7RAId2g6dJqr5CKXFrde103eAnP+iRYPLiPYgdTVFeXS6Xn0FMxdVrnmhfzC4UpGlafYBh716TV+5ppy0i37xcx1JVw7w5BuMmD+bD3iFjrMRa6Z3f5/GkafFUT9npv48LPlzv82eByobV8eVfOZp34yEh/u/c0Z5zmK1YYwBT79JD9CcLnlGNwVYWzC8MsBrZYg1C9/IQAyRz1yRrFA+Bhve/6goXK3xWdLAxRU3IHO9l2P5HxDpT4rQ5kVOy47pPm74n2q55NIFoVpamqN0coDyQvNtFUHMjcI7yyvJ6O3eSH4NbW3Aw4ZF4NUinY6rPiLwEF2M94kospNVFToyaVqt4FaC4yjIoYL+wOL1YnGQT10nbJqL2cSsLSqqVB7pYegSjyjFq6WIC5RXRtRTK4fVuWYg5v8JtnUeUlY2523L7+8Zj0rqJor3qEc2V8MDvgCerNUOVMkNMqkiD5zOfd9+xrfm3On0oRUjfs//ncULk0zHJ1zRoFAyASQQDmYtasEyYBQ7LHdV lAm433pb INkSyrd91eKSPVdNJkPHxVXNtTTmjOcJRxfH3xx4gf1dvgrfGFuEEFATtdUpALU8sLtTN8acpQpcGk7rEOhwFDoFnWjp+b9MGDvwLy7Dk/P4huCt55IWhSmdOzkS/L9QXVf+Z/hLwJt4obX5sxjEbrLimNN+7IjNbI/HKk96Z4ciww351GqF6roSzyIcpFmosAhchrGrBI3tJ7Z5W4KkoWHuVly5U+vUxRZYpV5zTgCwqi6/BI1bnr9EnAe+J1f/8oRutCYYGCwiMcMw7XgJ0Z7JBIEtDF9pMPn3xJMXhz48+Euk5D5K+rWXrE8f4Q50lzVVJSBaxB7D718/JdL7g1tb6dk85lGk2Jg8Epx90QsBghCimJvmYFDmpljZLe706fLtAyuzgkjeLz7LDT/q1MGeke014/Gg1lDB+EJZVV96ZJVfibq+YEql2UaxBAocurNN9awZvbcaP2uM2FrU2UnmX4mqIXTHOcG+mqEBJ8N/FSMndM44XFZnymjRmYI7RyXr0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 1/18/23 10:00 AM, Nicholas Piggin wrote: > Add CONFIG_MMU_TLB_REFCOUNT which enables refcounting of the lazy tlb mm > when it is context switched. This can be disabled by architectures that > don't require this refcounting if they clean up lazy tlb mms when the > last refcount is dropped. Currently this is always enabled, which is > what existing code does, so the patch is effectively a no-op. > > Rename rq->prev_mm to rq->prev_lazy_mm, because that's what it is. > > Signed-off-by: Nicholas Piggin > --- > Documentation/mm/active_mm.rst | 6 ++++++ > arch/Kconfig | 17 +++++++++++++++++ > include/linux/sched/mm.h | 18 +++++++++++++++--- > kernel/sched/core.c | 22 ++++++++++++++++++---- > kernel/sched/sched.h | 4 +++- > 5 files changed, 59 insertions(+), 8 deletions(-) > > diff --git a/Documentation/mm/active_mm.rst b/Documentation/mm/active_mm.rst > index 6f8269c284ed..2b0d08332400 100644 > --- a/Documentation/mm/active_mm.rst > +++ b/Documentation/mm/active_mm.rst > @@ -4,6 +4,12 @@ > Active MM > ========= > > +Note, the mm_count refcount may no longer include the "lazy" users > +(running tasks with ->active_mm == mm && ->mm == NULL) on kernels > +with CONFIG_MMU_LAZY_TLB_REFCOUNT=n. Taking and releasing these lazy > +references must be done with mmgrab_lazy_tlb() and mmdrop_lazy_tlb() > +helpers which abstracts this config option. > + > :: > > List: linux-kernel > diff --git a/arch/Kconfig b/arch/Kconfig > index 12e3ddabac9d..b07d36f08fea 100644 > --- a/arch/Kconfig > +++ b/arch/Kconfig > @@ -465,6 +465,23 @@ config ARCH_WANT_IRQS_OFF_ACTIVATE_MM > irqs disabled over activate_mm. Architectures that do IPI based TLB > shootdowns should enable this. > > +# Use normal mm refcounting for MMU_LAZY_TLB kernel thread references. > +# MMU_LAZY_TLB_REFCOUNT=n can improve the scalability of context switching > +# to/from kernel threads when the same mm is running on a lot of CPUs (a large > +# multi-threaded application), by reducing contention on the mm refcount. > +# > +# This can be disabled if the architecture ensures no CPUs are using an mm as a > +# "lazy tlb" beyond its final refcount (i.e., by the time __mmdrop frees the mm > +# or its kernel page tables). This could be arranged by arch_exit_mmap(), or > +# final exit(2) TLB flush, for example. > +# > +# To implement this, an arch *must*: > +# Ensure the _lazy_tlb variants of mmgrab/mmdrop are used when dropping the > +# lazy reference of a kthread's ->active_mm (non-arch code has been converted > +# already). > +config MMU_LAZY_TLB_REFCOUNT > + def_bool y > + > config ARCH_HAVE_NMI_SAFE_CMPXCHG > bool > > diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h > index 5376caf6fcf3..68bbe8d90c2e 100644 > --- a/include/linux/sched/mm.h > +++ b/include/linux/sched/mm.h > @@ -82,17 +82,29 @@ static inline void mmdrop_sched(struct mm_struct *mm) > /* Helpers for lazy TLB mm refcounting */ > static inline void mmgrab_lazy_tlb(struct mm_struct *mm) > { > - mmgrab(mm); > + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) > + mmgrab(mm); > } > > static inline void mmdrop_lazy_tlb(struct mm_struct *mm) > { > - mmdrop(mm); > + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) { > + mmdrop(mm); > + } else { > + /* > + * mmdrop_lazy_tlb must provide a full memory barrier, see the > + * membarrier comment finish_task_switch which relies on this. > + */ > + smp_mb(); > + } > } Considering the fact that mmdrop_lazy_tlb() replaced mmdrop() in various locations in which smp_mb() was not required, this comment might be confusing. IOW, for the cases in most cases where mmdrop_lazy_tlb() replaced mmdrop(), this comment was irrelevant, and therefore it now becomes confusing. I am not sure the include the smp_mb() here instead of "open-coding" it helps. > > static inline void mmdrop_lazy_tlb_sched(struct mm_struct *mm) > { > - mmdrop_sched(mm); > + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_REFCOUNT)) > + mmdrop_sched(mm); > + else > + smp_mb(); // see above > } Wrong style of comment.