From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC86AC4332F for ; Mon, 5 Dec 2022 20:43:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3548B8E0002; Mon, 5 Dec 2022 15:43:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 305138E0001; Mon, 5 Dec 2022 15:43:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A5708E0002; Mon, 5 Dec 2022 15:43:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0AC978E0001 for ; Mon, 5 Dec 2022 15:43:59 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D8E5D140D57 for ; Mon, 5 Dec 2022 20:43:58 +0000 (UTC) X-FDA: 80209429356.12.545EBB9 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) by imf19.hostedemail.com (Postfix) with ESMTP id 68C0B1A000D for ; Mon, 5 Dec 2022 20:43:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=google header.b=VlITxXm5; spf=pass (imf19.hostedemail.com: domain of torvalds@linuxfoundation.org designates 209.85.219.47 as permitted sender) smtp.mailfrom=torvalds@linuxfoundation.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670273037; a=rsa-sha256; cv=none; b=7pUwm0fikMp7Eko6AQ4uT4RVd1wbLJ3If68NxUIRN5hrdvi49GD6z4hlhCQfBQCManoIL1 SVUiaUE8xPek9HypWQDyy37v4McPD0HLEh4c1CwCholRBwT43GPIdmAkeGr20VJHXZcM+u ZC/VMUBjIHpW+RhdBm7PkE5nKS4TOHU= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=google header.b=VlITxXm5; spf=pass (imf19.hostedemail.com: domain of torvalds@linuxfoundation.org designates 209.85.219.47 as permitted sender) smtp.mailfrom=torvalds@linuxfoundation.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670273037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lBWIRVS47RUmDEcOvPdmfWIVH4IaWi6W1xGhZ6HUIpg=; b=GR+MvaGfMMnKEtek5t1gZXJSAd95+QykTR7lc0R+cs7KvYqxxqnlhJN1JQo+IE9RqAyyBq KWQ8cUpcceiA01aF/Mgp5redRxrHDJT1bKHxwVaTH7tQhqKIe3yFZxRAJqXq6BrKgNAiHz ojXsy0WpZzs9ui/LoD7NH88f5XjdYj0= Received: by mail-qv1-f47.google.com with SMTP id e18so9080029qvs.1 for ; Mon, 05 Dec 2022 12:43:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=google; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=lBWIRVS47RUmDEcOvPdmfWIVH4IaWi6W1xGhZ6HUIpg=; b=VlITxXm5biYw6uI6PPXe3docWzenLdY3/Q/s8bcI84Qeqr5ssGWJZkQFxotM28q3kG jidIiH1cGwtjd9ELQ0YAZHqHlktIkQiMXrrnKeMpoCut3nnXF18SB3FqANyHMxvQJoZ2 pp/50ejXAqt+zzjVTkW2LcaWDW6j/KCOcQPT0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lBWIRVS47RUmDEcOvPdmfWIVH4IaWi6W1xGhZ6HUIpg=; b=RlK+NA5moZZ45GZ7fzVeW/g6x5TkG+oGaa2fbcExYu2pt4dfTadTulkCHpwufET6Uu a26kYmlKgZe0nfY09duJyda1A27fe/UDRBOkFALVNQ+pDEc3DF9/NIoLS/fFaKPDeL1n APPtJ5/RwVKtediXxQcsFt28oJrt+7feTBQJWCogpQEoI+ahz/l7VjB4emqiZKgI1DFc 1fusq7j/CVwIEc4ARGFpN5oLAnl86RpJNF2XhKIVR8Kh0vAecbqLQtNq37uGPBuatGx1 BHOvkpETQ53cxsGAMvmwwbp8R2VEoLeo/kAc1cD5QNKNo8UJ1ckpcJRsp11chByUTqIP hgfQ== X-Gm-Message-State: ANoB5pkO++vHR7iOHiQhWwPsL/MfZmFbdLwuOC1Ydwur+xx5LUq3I6Am qJh8ZMifrupZ3rEZGwWFcwUoioR53TI9w/NV X-Google-Smtp-Source: AA0mqf4owWBlrF5+wUn2Feh2EvO//3dWhD1i03O/RP4olq63RoQI8rnmPaNl7+7bIDNhx6eKb+G3sQ== X-Received: by 2002:ad4:4101:0:b0:4b1:856b:4277 with SMTP id i1-20020ad44101000000b004b1856b4277mr60094124qvp.129.1670273036032; Mon, 05 Dec 2022 12:43:56 -0800 (PST) Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com. [209.85.160.172]) by smtp.gmail.com with ESMTPSA id bs21-20020ac86f15000000b003a5c6ad428asm10241676qtb.92.2022.12.05.12.43.53 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 05 Dec 2022 12:43:54 -0800 (PST) Received: by mail-qt1-f172.google.com with SMTP id cg5so12314267qtb.12 for ; Mon, 05 Dec 2022 12:43:53 -0800 (PST) X-Received: by 2002:ac8:688:0:b0:3a5:122:fb79 with SMTP id f8-20020ac80688000000b003a50122fb79mr67106155qth.452.1670273033472; Mon, 05 Dec 2022 12:43:53 -0800 (PST) MIME-Version: 1.0 References: <202212051534.852804af-yujie.liu@intel.com> In-Reply-To: <202212051534.852804af-yujie.liu@intel.com> From: Linus Torvalds Date: Mon, 5 Dec 2022 12:43:37 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [linux-next:master] [mm] 5df397dec7: will-it-scale.per_thread_ops -53.3% regression To: kernel test robot Cc: oe-lkp@lists.linux.dev, lkp@intel.com, Andrew Morton , Johannes Weiner , Hugh Dickins , Nadav Amit , Linux Memory Management List , linux-arch@vger.kernel.org, ying.huang@intel.com, feng.tang@intel.com, zhengjun.xing@linux.intel.com, fengwei.yin@intel.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Spamd-Result: default: False [7.17 / 9.00]; SORBS_IRL_BL(6.00)[209.85.219.47:from,209.85.160.172:received]; BAYES_HAM(-1.73)[85.59%]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; FORGED_SENDER(0.30)[torvalds@linux-foundation.org,torvalds@linuxfoundation.org]; BAD_REP_POLICIES(0.10)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; TAGGED_RCPT(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWELVE(0.00)[13]; DKIM_TRACE(0.00)[linux-foundation.org:+]; R_DKIM_ALLOW(0.00)[linux-foundation.org:s=google]; FROM_NEQ_ENVFROM(0.00)[torvalds@linux-foundation.org,torvalds@linuxfoundation.org]; MIME_TRACE(0.00)[0:+]; R_SPF_ALLOW(0.00)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[linux-foundation.org]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 68C0B1A000D X-Rspamd-Server: rspam01 X-Stat-Signature: pisffcx9dkgb64xp4e4jcyapd593rdfr X-HE-Tag: 1670273037-256903 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Dec 5, 2022 at 1:02 AM kernel test robot wrot= e: > > FYI, we noticed a -53.3% regression of will-it-scale.per_thread_ops due t= o commit: > 5df397dec7c4 ("mm: delay page_remove_rmap() until after the TLB has been = flushed") Sadly, I think this may be at least partially expected. The code fundamentally moves one "loop over pages" and splits it up (with the TLB flush in between). Which can't be great for locality, but it's kind of fundamental for the fix - but some of it might be due to the batch limit logic. I wouldn't have expected it to actually show up in any real loads, but: > in testcase: will-it-scale > test: page_fault3 I assume that this test is doing a lot of mmap/munmap on dirty shared memory regions (both because of the regression, and because of the name of that test ;) So this is hopefully an extreme case. Now, it's likely that this particular case probably also triggers that /* No more batching if we have delayed rmaps pending */ which means that the loops in between the TLB flushes will be smaller, since we don't batch up as many pages as we used to before we force a TLB (and rmap) flush and free them. If it's due to that batching issue it may be fixable - I'll think about this some more, but > Details are as below: The bug it fixes ends up meaning that we run that rmap removal code _after_ the TLB flush, and it looks like this (probably combined with the batching limit) then causes some nasty iTLB load issues: > 2291312 =C4=85 2% +1452.8% 35580378 =C4=85 4% perf-stat.i.iTLB-= loads although it also does look like it's at least partly due to some irq timing issue (and/or bad NUMA/CPU migration luck): > 388169 +267.4% 1426305 =C4=85 6% vmstat.system.in > 161.37 +84.9% 298.43 =C4=85 6% perf-stat.ps.cpu-migra= tions > 172442 =C4=85 4% +26.9% 218745 =C4=85 8% perf-stat.ps.node-= load-misses so it might be that some of the regression comes down to "bad luck" - it happened to run really nicely on that particular machine, and then the timing changes caused some random "phase change" to the load. The profile doesn't actually seem to show all that much more IPI overhead, so maybe these incidental issues are what then causes that big regression. It would be lovely to hear if you see this on other machines and/or loads. Because if it's a one-off, it's probably best ignored. If it shows up elsewhere, I think that batching logic might need looking at. Linus