From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EBC0C48291 for ; Fri, 2 Feb 2024 16:35:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C69166B0088; Fri, 2 Feb 2024 11:35:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF2636B008C; Fri, 2 Feb 2024 11:35:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A6BDA6B0092; Fri, 2 Feb 2024 11:35:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8F5EF6B0088 for ; Fri, 2 Feb 2024 11:35:19 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4B6C0C0926 for ; Fri, 2 Feb 2024 16:35:19 +0000 (UTC) X-FDA: 81747413958.30.0D518CC Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) by imf15.hostedemail.com (Postfix) with ESMTP id 8046EA001F for ; Fri, 2 Feb 2024 16:35:17 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ifplmmxm; spf=pass (imf15.hostedemail.com: domain of mikhail.v.gavrilov@gmail.com designates 209.85.161.46 as permitted sender) smtp.mailfrom=mikhail.v.gavrilov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706891717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q1b7WFP4fLnEs1xL56WMoxp/TFOgr4QRcoCxeMPjVlo=; b=yTdhkFvuoRLY8KGyB4DOHJd/IbIgs3fODwoOWhBaOTsOhY5Rf48AUfMn3bu5JIMjdJEbAC klsl0DHTGTaKmCvuPqTd7xMAn3B2G41JJMkpa24wt31fXWPCyDyFU6rjXZUitYMd+IGKYU OLRnk5uPmSx0LNOlOflqy1dOTrWUoDE= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ifplmmxm; spf=pass (imf15.hostedemail.com: domain of mikhail.v.gavrilov@gmail.com designates 209.85.161.46 as permitted sender) smtp.mailfrom=mikhail.v.gavrilov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706891717; a=rsa-sha256; cv=none; b=oqTQBfTrXkhrd+b2JLgpvk6tV1KlNE4BfBLnVCDf6Nt+ILwOSpAykkCnc+VoA8U0HpwB1e NywMesmJ5cLsXZ4WQRp5tTHx1bDDxB4O49Op98wd2BJATG9XmOH9lG5Pqw25cx3gVMqHSP KZqH4VvMF9UvdHnGLkPyMn8vPHHWO3w= Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-59584f41f1eso683571eaf.1 for ; Fri, 02 Feb 2024 08:35:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706891716; x=1707496516; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Q1b7WFP4fLnEs1xL56WMoxp/TFOgr4QRcoCxeMPjVlo=; b=ifplmmxm6yCQcY9GzwafZ/4abDXEOqVz8MVQUfeQxJno3LsMXtmL0Ang/7fhtq5i/w C4nb4bxWpyWczXcLS3tWuHp/o+1359tSif7Oeqs5FVAjaibnOH5uakwsi6ot3h8hkWrh OTzH4E0E/gPpGsyiWExCf3Spliw5X84yycrnA50QijXZEi5B2+mqjGAC0Qnsw920NHET jsf9zC+tpfTf0SDgIKJ2NyEC1xn9eYOT2BAgDWajIZxrSGvSSZkL8J30+zYsg0zLzTf0 rDdhtq/LSFXIfyJ64NoXftM11pE39MMluPfNvakCL+pnPUBqdQWz3czV1LhQIUh+n6sP GcKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706891716; x=1707496516; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q1b7WFP4fLnEs1xL56WMoxp/TFOgr4QRcoCxeMPjVlo=; b=iOMTWupk1KncZhOOcu1OG81MNMKQLzaTTYnJDIdVrdmh9gL0shf5/OH7M9rucAKmsa obbvEhfkE2YmhrfWVBEEgS6xNnyZnVsbHtxKeJ7D/L70eyLGcvRfAV8pQS5Fmj9p63b4 jKEJMkC1vl2Tgen6OJTgImbvLe2ynpPNRURpyt5YeV7t4WKJkiYfwUxLQvecFAfW09j8 /zsJ3WEL6/TTYdWBG03hl39jX46zajA29DNyb0FdvDMzQltp6RE2PS6po2oge6zzLOK2 PWe1XCI7tle3w2/ufU9v3ia6Pr0MFBcgZAhntd+IlqGYxU6/CVcHHr8ttM9ax50OdePS g97A== X-Forwarded-Encrypted: i=0; AJvYcCVvrYksDSJ9Q+v93fK8y5+3rgyTrUTip61Dtyex4CWmuoM86dAoKzewA0XQ1lJCdEpG3KwJm493cWPiMWO5YLqV/U0= X-Gm-Message-State: AOJu0YyF/i6bnmF9h7UY7aJtxuMSPBbaepaRMepS/gN/7XoTK5naFLbe vo2CuoT2N+vPZ3lnnOKXitJc00xoKnACg+v0NJf+yhzMtBILXei3qcTo8VGEzg6wZNyLV9+Dz+3 YkuxXCIi/qN3iKHQllXVhu+GDmkM= X-Google-Smtp-Source: AGHT+IErnTQmqrpMqoIYTqabnL481xBFUNOvS9QOYdaktWUqC2PEVKx1ai1Cf+/xhKYohW6eUcfmFUF8BtDo2ZNL9hg= X-Received: by 2002:a05:6820:2224:b0:59a:127c:bdfe with SMTP id cj36-20020a056820222400b0059a127cbdfemr7159032oob.1.1706891716497; Fri, 02 Feb 2024 08:35:16 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Mikhail Gavrilov Date: Fri, 2 Feb 2024 21:35:05 +0500 Message-ID: Subject: Re: regression/bisected commit 773688a6cb24b0b3c2ba40354d883348a2befa38 make my system completely unusable under high load To: Marco Elver Cc: Andrey Konovalov , glider@google.com, dvyukov@google.com, eugenis@google.com, Oscar Salvador , Vlastimil Babka , Andrew Morton , Linux List Kernel Mailing , Linux Memory Management List Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 8046EA001F X-Rspam-User: X-Stat-Signature: ozc7zjm9fdx1nu4g5hrixajx1oqsmxb4 X-Rspamd-Server: rspam01 X-HE-Tag: 1706891717-386974 X-HE-Meta: U2FsdGVkX1/8RSe2F//1gog0aFMXmbpe3P1NQscRU7H+xwALtVx61RdePa5WKC0FUlGDRFP/erQCEW+/N+yNT3bsYL1FWRqGf+7CKg18HFSTvB7nZMJ1VEbkFDsmihl1qXty9CrOYhNTw+AxwoPA+KuVX4WauYn+Xl5RRglcYYdPUWDT/Dj+3H4tjnJddmfAJPym3LEOWh3g+358LuFO+ghoN/UdZEiiqnIb1rPPd0/172sVMhUrdstXoOrNK1/PVcC8eXPZzj7Na70Y0rahfCeZjLaBKN8NR3Gn7hYXbeKJ9rND7BRlP/UWB6F11ID9uIPbrL65dVyGbTBfV1t/hlZ33OdqeYFk/e0RdUDIzIrXD1eGCLPB+Ar6Yjk//xiYiTBEI7q91Kfjt7/ORAnOFFHE+YfRgqDI16dt5LpCkrSyK83lsxc48ZzBuEF7DmLMHcAse3UW56DoecIc2sHDodhKZkVrKHGj2YygQITJVr0WDlb7nsLZ+cTwl0YScKiSl20pn8QtoeUBWpqJdDMlNL0CjCuHbJFkwCTbwD+EdtzGPQu5q1exS39hbQUCt4OwjhjJkBnQ0byRimnFYgf+oI7DN34hPfQncQ2mkkjpkeq6PCZyXdht3UX27zKtChm0Q65tI8kjmOJgBS4G5+DPKSts4dIH67VAhkuTWWZZ+aPeSIoHRWJrRzNfdSlyl3Dopk63NG27vAmc/P4HBqt+WYg1Jyr2g0N5jy5bQYh0JLCSDxX6TFDGadZ+xzH2jptGwPuv6Fke+JK25vE51qh8IpJiPjk59EUz8FdUIu5kWR4OrSfdLFhISqo+CguKE4MhA3p8osJUhc1+L1zgFbQRabjDVtOFsHtFNlLmE6ayfo14g2mdfZLO6hF8+cw7WdUszWYTKzOzz5/x0pTw5bW7kbMz3Wtuq0Wt2X8ek6k3Ts2Naq1y16VprmsQxgvASvS4IJ/GEaQT9r2ge4l6K3f bXwN7omk p5R0nLx5V5WtYfQ/ewkqtzoHcG8cAwXaA0knEH9r7/UtopzJ9sw9yw/6e2U/4xGcpWmCepE+Qys5F+9V4twyJFlRJZddQ3hwd1F0bfLzpvhkOZXLhiGafqMg9M1NsODikOpqPVdwBOInSK9+XRJrA19v+LfhL+Fq+dFfrkiF9QDyJwy00lZpk8sUaX7ahYBTVEtU2HDk1kjyZtSQixR4sE+pkRRaVR2Dtw3QDJ884sXR7CprmoAtPUfRzQ6hjTdNHPq8do+rWW0ezucQh+wvWrw6tPVc2rBmcGxWfEuYiym5ERi2KwK83hz/415uk9LinFm5mZ10BJWWUxvs+7qahNmMnVP4yiBF6jT0zNu/E3UkOvU+BskLiYH/c9Niny/vAagumaTGyRC2gWoeJKh9h+CPIh964qR7q/GGfmxu2t1+p92BcLKOG6cuEGAsCC/LYEl6MhUzf0QAA8LsBD8LpEi919l87khi9In8LveYGXGLDdRQTIxBsUKiNnZsBVcH3Ua9bTz2vd+kCi16FgcZDqBs5raR1bEdtt0G0ddjXlP48t8LnVkAUdKjIFEU1r+MCCH7lW4GA6Yha6t/OBpUSL8AuIso2aOxIRAFlwwaQWX1iU8G7p6HlAavGD5KmIHSLaxzLsIs6Ru7XWoky9/4hOdKUrPcvTj/rZJXWt1SiRMwnCq6sq0hirEI4AZk72iFG8otLv+dMdSQ2+L0wxS0YD4VVaAhm0R6pADLIE2RKQxqNKLELFV6j4TP37lXM7M1PoHtx/lF2CpQ6IsCABTckR1/cy8bBiB4u8IKch7COvG4AwbVOFtO8ArjW9kI6hq3/46aIY2wYavnZ6hF+pRbu8iKdDFag0O0NYONqYVFIxOdg88bQm2GcVo/lEMazhGhMqAjxE0DG0GGr44wmYQ/rQenz9vGSPSJnjWxK3+ORp6sGBm99BX8+bwLUZE+Bnv3S/CMll08PteFjRt9o00+37eeC0GbD +xoEAp4G a4jZln3HUSFAHKVnZNkVu1ntNgvAmUfiRW7QecX9/5mCABWzOkJLVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.006920, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 2, 2024 at 2:00=E2=80=AFPM Marco Elver wrote= : > > > Maybe we can try something else? > > That's strange - the patches at [1] definitely revert the change you > bisected to. It's possible there is some other strange side-effect. (I > assume that you are still running all this with a KASAN kernel.) Yes. build .config not changed between kernel builds. > Just so I understand it right: > You say before commit cc478e0b6bdffd20561e1a07941a65f6c8962cab the > game's FPS were good. But that is strange, because at that point we're > already doing stackdepot refcounting, i.e. after commit > 773688a6cb24b0b3c2ba40354d883348a2befa38 which you reported as the > initial performance regression. The patches at [2] fixed that problem. > > So now it's unclear to me how the simple change in > cc478e0b6bdffd20561e1a07941a65f6c8962cab causes the performance > problem, when in fact this is already with KASAN stackdepot > refcounting enabled but without the performance fixes from [1] and > [2]. > > [2] https://lore.kernel.org/all/20240118110216.2539519-2-elver@google.com= / > > My questions now would be: > - What was the game's FPS in the last stable kernel (v6.7)? [6.7] - 83 FPS - 13060 frames during benchmark. > - Can you collect another set of performance profiles between good and > bad? Maybe it would show where the time in the kernel is spent. Yes, please look at [aaa2c9a97c22 perf] and [cc478e0b6bdf perf] > perf diff perf-git-aaa2c9a97c22af5bf011f6dd8e0538219b45af88.data perf-git= -cc478e0b6bdffd20561e1a07941a65f6c8962cab.data No kallsyms or vmlinux with build-id de2a040f828394c5ce34802389239c2a0668fcc7 was found No kallsyms or vmlinux with build-id 33ab1cd545f96f5ffc2a402a4c4cfa647fd727a0 was found # Event 'cycles:P' # # Baseline Delta Abs Shared Object Symbol # ........ ......... .............................................. ...........................................................................= ...........................................................................= ............................... # 48.48% +21.75% [kernel.kallsyms] [k] 0xffffffff860065c0 36.13% -16.49% ShadowOfTheTombRaider [.] 0x00000000001d7f5e 4.43% -2.10% libvulkan_radeon.so [.] 0x000000000006b870 3.28% -0.63% libcef.so [.] 0x00000000021720e0 1.11% -0.53% libc.so.6 [.] syscall 0.65% -0.24% libc.so.6 [.] __memmove_avx512_unaligned_erms 0.31% -0.14% libc.so.6 [.] __memset_avx512_unaligned_erms 0.26% -0.13% libm.so.6 [.] __powf_fma 0.20% -0.10% [amdgpu] [k] amdgpu_bo_placement_from_domain 0.22% -0.09% [amdgpu] [k] amdgpu_vram_mgr_compatible 0.67% -0.09% armada-drm_dri.so [.] 0x00000000000192b4 0.15% -0.08% libc.so.6 [.] sem_post@GLIBC_2.2.5 0.16% -0.07% [amdgpu] [k] amdgpu_vm_bo_update 0.14% -0.07% [amdgpu] [k] amdgpu_bo_list_entry_cmp 0.13% -0.06% libm.so.6 [.] powf@GLIBC_2.2.5 0.14% -0.06% libMangoHud.so [.] 0x000000000001c4c0 0.10% -0.06% libc.so.6 [.] __futex_abstimed_wait_common 0.19% -0.05% libGLESv2.so [.] 0x0000000000160a11 0.07% -0.04% libc.so.6 [.] __new_sem_wait_slow64.constprop.0 0.10% -0.04% radeonsi_dri.so [.] 0x0000000000019454 0.05% -0.03% [amdgpu] [k] optc1_get_position 0.05% -0.03% libc.so.6 [.] sem_wait@@GLIBC_2.34 0.22% -0.02% [vdso] [.] 0x00000000000005a0 0.10% -0.02% libc.so.6 [.] __memcmp_evex_movbe +0.02% [JIT] tid 8383 [.] 0x00007f2de0052823 > - Could it be an inconclusive bisection? I checked twice: [6.7] - 83 FPS [aaa2c9a97c22] - 111 FPS [cc478e0b6bdf] - 64 FPS [6.8-rc2 with patches] - 82 FPS [6.7] https://i.postimg.cc/15yyzZBr/v6-7.png [6.7 perf] https://mega.nz/file/QwJ3hbob#RslLFVYgz1SWMcPR3eF9uEpFuqxdgkwXSa= tWts-1wVA [aaa2c9a97c22] https://i.postimg.cc/Sxv4VYhg/git-aaa2c9a97c22af5bf011f6dd8e= 0538219b45af88.png [aaa2c9a97c22 perf] https://mega.nz/file/dwQxha4J#2_nBF6uNzY11VX-T-Lr_-60WIMrbl1YEvPgY4CuXqEc [cc478e0b6bdf] https://i.postimg.cc/W3cQfMfw/git-cc478e0b6bdffd20561e1a0794= 1a65f6c8962cab.png [cc478e0b6bdf perf] https://mega.nz/file/hl5kwLTC#_4Fg1KBXCnQ-8OElY7EYmPOoDG6ZeZYnKFjamWpklWw [6.8-rc2 with patches] https://i.postimg.cc/26dPpVsR/v6-8-rc2-with-patches.= png [6.8-rc2 with patches perf] https://mega.nz/file/NxgTAb4L#0KO_WU-svpDw60Y3148RZhELPcUtFg3_VCDzJqSyz34 --=20 Best Regards, Mike Gavrilov.