From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25E15FA3741 for ; Fri, 28 Oct 2022 02:14:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 747716B0072; Thu, 27 Oct 2022 22:14:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F8CF6B0073; Thu, 27 Oct 2022 22:14:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BF316B0074; Thu, 27 Oct 2022 22:14:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4C1906B0072 for ; Thu, 27 Oct 2022 22:14:47 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 27A0F1A04F9 for ; Fri, 28 Oct 2022 02:14:47 +0000 (UTC) X-FDA: 80068739814.22.6F8982F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 6802B80005 for ; Fri, 28 Oct 2022 02:14:45 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2BAF523A; Thu, 27 Oct 2022 19:14:48 -0700 (PDT) Received: from [192.168.0.146] (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AD67A3F445; Thu, 27 Oct 2022 19:14:32 -0700 (PDT) Message-ID: <8a3ade4c-1714-5ffd-ed57-02ab0509725b@arm.com> Date: Fri, 28 Oct 2022 07:44:29 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: [PATCH v4 2/2] arm64: support batched/deferred tlb shootdown during page reclamation Content-Language: en-US To: Barry Song <21cnbao@gmail.com>, Punit Agrawal Cc: Yicong Yang , yangyicong@hisilicon.com, corbet@lwn.net, peterz@infradead.org, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, x86@kernel.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, akpm@linux-foundation.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, wangkefeng.wang@huawei.com, xhao@linux.alibaba.com, prime.zeng@hisilicon.com, Barry Song , Nadav Amit , Mel Gorman , catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org References: <20220921084302.43631-1-yangyicong@huawei.com> <20220921084302.43631-3-yangyicong@huawei.com> <168eac93-a6ee-0b2e-12bb-4222eff24561@arm.com> <8e391962-4e3a-5a56-64b4-78e8637e3b8c@huawei.com> <87o7tx5oyx.fsf@stealth> From: Anshuman Khandual In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666923285; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X/IAeVLm3MaiCSxL9tGQ8bcbfh04dB74CIDrNC+8Edk=; b=SPCpLi6XGawNC/QpBh2GV5Yurd+ZTkl+TK6a/25k8I/w6pl0hGwXu3dUmuqoKbgOA1yWCa QNWGpFVjTTXSKams5ILQ+ZavmF+/v2vmuXk7dvf+5HQRto11y3PBd4gi1WcHHMr5T8UiwY wsPp3OjkP3Ju+67Ed8qkomYCsJuTxa8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666923285; a=rsa-sha256; cv=none; b=X1nt0bsbm+5VIP5xODb3XC+CcYlc1qyg7irvGNXGFfY7vYbB3X/DICxoZ/1KwFYuK57T0A clp1P/d/5zoxrvQLUGZOmL3dnX01c0Y/Zq3iWuKTQBfJASrV2bJXW2hFii3Yk7cMI/l78g UER3MB+k3QZkdIRXxmt1eePfTh7cpQI= X-Stat-Signature: qrjnk4yfrx9cyh7aepihzwtrqsh6jg7d X-Rspamd-Queue-Id: 6802B80005 Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of anshuman.khandual@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=anshuman.khandual@arm.com; dmarc=pass (policy=none) header.from=arm.com X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1666923285-490692 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/28/22 03:25, Barry Song wrote: > On Fri, Oct 28, 2022 at 3:19 AM Punit Agrawal > wrote: >> >> [ Apologies for chiming in late in the conversation ] >> >> Anshuman Khandual writes: >> >>> On 9/28/22 05:53, Barry Song wrote: >>>> On Tue, Sep 27, 2022 at 10:15 PM Yicong Yang wrote: >>>>> On 2022/9/27 14:16, Anshuman Khandual wrote: >>>>>> [...] >>>>>> >>>>>> On 9/21/22 14:13, Yicong Yang wrote: >>>>>>> +static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) >>>>>>> +{ >>>>>>> + /* for small systems with small number of CPUs, TLB shootdown is cheap */ >>>>>>> + if (num_online_cpus() <= 4) >>>>>> It would be great to have some more inputs from others, whether 4 (which should >>>>>> to be codified into a macro e.g ARM64_NR_CPU_DEFERRED_TLB, or something similar) >>>>>> is optimal for an wide range of arm64 platforms. >>>>>> >>>> I have tested it on a 4-cpus and 8-cpus machine. but i have no machine >>>> with 5,6,7 >>>> cores. >>>> I saw improvement on 8-cpus machines and I found 4-cpus machines don't need >>>> this patch. >>>> >>>> so it seems safe to have >>>> if (num_online_cpus() < 8) >>>> >>>>> Do you prefer this macro to be static or make it configurable through kconfig then >>>>> different platforms can make choice based on their own situations? It maybe hard to >>>>> test on all the arm64 platforms. >>>> Maybe we can have this default enabled on machines with 8 and more cpus and >>>> provide a tlbflush_batched = on or off to allow users enable or >>>> disable it according >>>> to their hardware and products. Similar example: rodata=on or off. >>> No, sounds bit excessive. Kernel command line options should not be added >>> for every possible run time switch options. >>> >>>> Hi Anshuman, Will, Catalin, Andrew, >>>> what do you think about this approach? >>>> >>>> BTW, haoxin mentioned another important user scenarios for tlb bach on arm64: >>>> https://lore.kernel.org/lkml/393d6318-aa38-01ed-6ad8-f9eac89bf0fc@linux.alibaba.com/ >>>> >>>> I do believe we need it based on the expensive cost of tlb shootdown in arm64 >>>> even by hardware broadcast. >>> Alright, for now could we enable ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH selectively >>> with CONFIG_EXPERT and for num_online_cpus() > 8 ? >> When running the test program in the commit in a VM, I saw benefits from >> the patches at all sizes from 2, 4, 8, 32 vcpus. On the test machine, >> ptep_clear_flush() went from ~1% in the unpatched version to not showing >> up. >> >> Yicong mentioned that he didn't see any benefit for <= 4 CPUs but is >> there any overhead? I am wondering what are the downsides of enabling >> the config by default. > As we are deferring tlb flush, but sometimes while we are modifying the vma > which are deferred, we need to do a sync by flush_tlb_batched_pending() in > mprotect() , madvise() to make sure they can see the flushed result. if nobody > is doing mprotect(), madvise() etc in the deferred period, the overhead is zero. Right, it is difficult to justify this overhead for smaller systems, which for sure would not benefit from this batched TLB framework.