From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFB17C38159 for ; Thu, 19 Jan 2023 03:04:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE70A6B0072; Wed, 18 Jan 2023 22:04:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6FA16B0073; Wed, 18 Jan 2023 22:04:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B10246B0074; Wed, 18 Jan 2023 22:04:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9BB966B0072 for ; Wed, 18 Jan 2023 22:04:54 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 63C0A80714 for ; Thu, 19 Jan 2023 03:04:54 +0000 (UTC) X-FDA: 80370056508.04.A947EEA Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf18.hostedemail.com (Postfix) with ESMTP id AE1D01C0002 for ; Thu, 19 Jan 2023 03:04:52 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bhWSFtHF; spf=pass (imf18.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674097492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c/gHZEBJUqbXfG5JX1H5lnam0IN6TNXNjBZYcdLB8wc=; b=dO/gmasg6OMpnzIiGGzj+UpWX6SEQ46ldARI8LjnFc7p/W0g5FlC7gQfVarFFrrzuNmA0m 6lbRvhDXAEK/gD59XBzMfCaxz6IV3C8lM7nT9Ww0/SO1vofwntuhTRV9SFnDApoBlYlHME yYuoQNGi/3AZM9Rt1UgsC1Jy1hnClbk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=bhWSFtHF; spf=pass (imf18.hostedemail.com: domain of npiggin@gmail.com designates 209.85.216.47 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674097492; a=rsa-sha256; cv=none; b=vKoLq/tZTCSFFDpmXpU9IWZ+1XtVkZzsksuCMl+9SYT98h4Rlo6N6liTAjqynD6nGE+7kO eLUrGTx8KzwlwAwoDchnWknII6Hj0j5u9EnE7LKdpau6BCz1t5BKgNyxO98iD40xdfw/EE JzaZAHkIuNwsZHG2lKK6jZ+HwGaZTV8= Received: by mail-pj1-f47.google.com with SMTP id x2-20020a17090a46c200b002295ca9855aso4532292pjg.2 for ; Wed, 18 Jan 2023 19:04:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:references:to:from:subject:cc:message-id:date :content-transfer-encoding:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=c/gHZEBJUqbXfG5JX1H5lnam0IN6TNXNjBZYcdLB8wc=; b=bhWSFtHFWzg3KaP6hdKmTWRc+V9Fox21VYwUq0wTkIBSKY3ClnhvXUqHrGuuAvBhkz WCDVAI3Rzb79DGG3/FspTjpuS3o0BQV8+iwZDwbwkFhojO8Am0TFeO2AS4AJ/1Vpl/bu QOPOLbW6JtRWSdr3FGbSfDZEMf3D6AB4c9WZEho6CHb9N37dU4DdY5qfoaNQJkQZpR9+ TZAo0/OD8U9HQka/hBfHIubzNBRsd14gwDKc0p+jI9ZVwweOmb7ZbrHA1AclDU2wwKPH 9T1TQWRHh9PyQ0Qm/E2ioM7u0ztci3DrxDuMlwc1oQAKgRjh871aYDRUEITBu2ZxqsDS wMHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:references:to:from:subject:cc:message-id:date :content-transfer-encoding:mime-version:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=c/gHZEBJUqbXfG5JX1H5lnam0IN6TNXNjBZYcdLB8wc=; b=UCV0yOQ1mVNNfqk6W7iMJ+o/zg3j+0c8wWzo8/Wnx+12C13AopEg8hwlKpvD7FK5yS PEW0F1NMsYFOOJsNzFj7upVnyUcNM8VIKRlAdUSTjrZHMnaxpFrJDgtdeUQKbCYOOUhZ oPksrloVgrKjkPPRv9VbC5ojViPFJb6fYLbKDW3n9qJUKo8vTyk06y47VeETiqh8DWqZ WlsHnPu6hs0r2B/KLSMx7juL2HdiLFS4KK81U0La3JCDGo4qRsvkKdEi1TqIx9vT6PL9 uyOiqIxNbTdQ0WCODZcoURvCmTIyLg/B7N6D5admX0UhNOsqhkzJ1XQP4zPD70F7QLdf +5nQ== X-Gm-Message-State: AFqh2kqGrAcoCEynkymy8EPK5XxyAbihv5jZzgAlE0TzbpfP676kw4bz wQxeNLB+NhNq6210qV037no= X-Google-Smtp-Source: AMrXdXvH7nWDObatVJJJ0rFrPx7ePH/E+zXu/f0VqlxwbgvnLGgalRPIiDu/+GlHLmvdgYcf9hqo9Q== X-Received: by 2002:a17:902:8a8a:b0:194:480d:6afc with SMTP id p10-20020a1709028a8a00b00194480d6afcmr8663252plo.48.1674097491373; Wed, 18 Jan 2023 19:04:51 -0800 (PST) Received: from localhost (193-116-102-45.tpgi.com.au. [193.116.102.45]) by smtp.gmail.com with ESMTPSA id y20-20020a170902b49400b00194caf3e975sm227479plr.208.2023.01.18.19.04.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 18 Jan 2023 19:04:50 -0800 (PST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Thu, 19 Jan 2023 13:04:44 +1000 Message-Id: Cc: "Andrew Morton" , "linux-arch" , "linux-mm" , Subject: Re: [PATCH v6 4/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN From: "Nicholas Piggin" To: "Linus Torvalds" , "Dave Hansen" , "Andy Lutomirski" , "Peter Zijlstra" , "Catalin Marinas" , "Will Deacon" X-Mailer: aerc 0.13.0 References: <20230118080011.2258375-1-npiggin@gmail.com> <20230118080011.2258375-5-npiggin@gmail.com> In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AE1D01C0002 X-Rspam-User: X-Stat-Signature: tgq1nzmgn3iwyw5yc6sfm4kzpdci39oh X-HE-Tag: 1674097492-63458 X-HE-Meta: U2FsdGVkX1+fFUUxWxMM1kkcFGdyM/0LamwVirgMiaigS+5Az0CMC3Z/SKpcNRLDXTA0Gk1E+p9ZueXYv/k+Pa5dgx/vBQHK6dpfLCOYy6GvvPsbBVrEK1GfcWoz7c3S5+wxfU+2Ak5ObjiEUBGFxsZ7pysyZiOFrF77g/ThyE7xINEbZqjgT1cNPcEaWXvTlJ/88JWRSOAxGpa23wWNejaFEc2zAlN1KiguC3PE3Nz2z27oitu6o/nQ/dgUksOjM+jTdBCv6kzHbNSyHtHz2C2t+XFW0CrFQIQJ4lMnZdMhc4SDENrMDQQgGGAggS5nX7m9WGmXwfloJN0w3qzWdIvU8bCx8K0Hz6vZQD1r8+QwePOQ5a4vPDfXa450ndwrRTHXIQMIQOvUHVnkq5liYn4qxPrCn0qLg21Uwo7dQhECkcJmu3glcjR8DwLGkPv9RnhX2stqcWvLb8CXjO9DO1J0yeiXnCOARlXZXwRRMRdxKIZ4ECIXApCmsvLuBEY9krBeUCZpt6G3TGECju6EW077lrNZ8BkQBg5/iW4vzgAnaVbDdoTq5hKeJdmMr63J3r1O4s3hc+Uu1P2xRKyiqMIykTyTH0x2t3bK7xWnOY/2dNSh034ICoWKxLklYGs4VNmdgD4R5GZjy81xqShAUC/i/6we8CgoMgc92WSoBXFzqz93p6OIF+A6FXDFQpqsToifQPAUSDqRSCTaiey43MXr6E7KGjzMeiaSwGtnzVRDQDLQy53IDJSB9kUSPaFfBdYMQlTp9XT4yEyofryNO+2xVWyY/5cql8Ap5Gkv6jru9B3+D8VJD31DeWufQWWJS/wYUgafFIzDk2yiLWkPksTTNF/PA7vVGnapVxaR/0PwwzJBoWbgRsrVy5jMaJlpFczyPmy+v5czr44rQ1WQDvd633D2bnC+3NtwfcYmaWExIvh4OKDKMfyjML2XDtT2BT2HY03qrx2nmVnJ0fK DzV/WsT4 ALIXEeRh4Px53g85LPwgIT+4Kg4gHiTCuaLXK6m9m/E0AIC5kSrOqPcyvZetav9k98c8gIKNpQclreyVfrwmoPVdTWoG8Nh7hsJLY57R0XLvTigxmzk9Xx0fmqXGjdDW19MFYm88RewcUCNkGwPydh56EFALjUUEmkTd2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu Jan 19, 2023 at 3:30 AM AEST, Linus Torvalds wrote: > [ Adding a few more x86 and arm64 maintainers - while linux-arch is > the right mailing list, I'm not convinced people actually follow it > all that closely ] > > On Wed, Jan 18, 2023 at 12:00 AM Nicholas Piggin wrot= e: > > > > On a 16-socket 192-core POWER8 system, a context switching benchmark > > with as many software threads as CPUs (so each switch will go in and > > out of idle), upstream can achieve a rate of about 1 million context > > switches per second, due to contention on the mm refcount. > > > > 64s meets the prerequisites for CONFIG_MMU_LAZY_TLB_SHOOTDOWN, so enabl= e > > the option. This increases the above benchmark to 118 million context > > switches per second. > > Well, the 1M -> 118M change does seem like a good reason for this series. It was an artificial corner case, mind you. I don't think it's a reason to panic and likely smaller systems with faster atomics will care far less than our big 2-hop systems. Benchmark is will-it-scale: ./context_switch1_threads -t 768 min:2174 max:2690 total:1827952 33.52% [k] finish_task_switch 27.26% [k] interrupt_return 22.66% [k] __schedule 2.30% [k] _raw_spin_trylock ./context_switch1_threads -t 1536 min:103000 max:120100 total:177201906 The top case has 1/2 the switching pairs to available CPU, which makes them all switch the same mm between real and lazy. Bottom case is just switching between user threads so that doesn't hit the lazy refcount. > The patches certainly don't look offensive to me, so Ack as far as I'm > concerned, but honestly, it's been some time since I've personally > been active on the idle and lazy TLB code, so that ack is probably > largely worthless. > > If anything, my main reaction to this all is to wonder whether the > config option is a good idea - maybe we could do this unconditionally, > and make the source code (and logic) simpler to follow when you don't > have to worry about the CONFIG_MMU_LAZY_TLB_REFCOUNT option. > > I wouldn't be surprised to hear that x86 can have the same issue where > the mm_struct refcount is a bigger issue than the possibility of an > extra TLB shootdown at the final exit time. > > But having the config options as a way to switch people over gradually > (and perhaps then removing it later) doesn't sound wrong to me either. IMO it's trivial enough that we could carry both, but everything's a straw on the camel's back so if we can consolidate it would always be preferebale. Let's see how it plays out for a few releases. > And I personally find the argument in patch 3/5 fairly convincing: > > Shootdown IPIs cost could be an issue, but they have not been observed > to be a serious problem with this scheme, because short-lived processes > tend not to migrate CPUs much, therefore they don't get much chance to > leave lazy tlb mm references on remote CPUs. > > Andy? PeterZ? Catalin? > > Nick - it might be good to link to the actual benchmark, and let > people who have access to big machines perhaps just try it out on > non-powerpc platforms... Yep good point, I'll put it in the changelog. I might submit another round to Andrew in a bit with acks and any minor tweaks and minus the last patch, assuming no major changes or objections. Thanks, Nick