From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E36BC10F1A for ; Tue, 7 May 2024 05:15:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 097416B0088; Tue, 7 May 2024 01:15:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 046B96B0089; Tue, 7 May 2024 01:15:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E295B6B008A; Tue, 7 May 2024 01:15:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C15EB6B0088 for ; Tue, 7 May 2024 01:15:26 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3B87940C9B for ; Tue, 7 May 2024 05:15:26 +0000 (UTC) X-FDA: 82090436652.12.4956926 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by imf21.hostedemail.com (Postfix) with ESMTP id 7F4A31C0010 for ; Tue, 7 May 2024 05:15:23 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WmVZfObb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of andrii.nakryiko@gmail.com designates 209.85.210.41 as permitted sender) smtp.mailfrom=andrii.nakryiko@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715058923; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D8vbZJ5UPiifYuspMMKBbXeF72Brhg4ICn1NMDsG3XQ=; b=14wk1WmITNro2OjcKnEaM393i1oEgJ8xuX0rghGzUo0Wf4Q7ODp7I2LoBu4KEYqqo/esbF ZTiyXX0UgaTwvUPpLiYRVU+fb18dGZk1PAlNeDoh0Teho/gGFh9vzzWYJlbW4rGHH3bi5V VVc/kiFChtmjtEcJm5ltD1jm/4HSTMU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WmVZfObb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of andrii.nakryiko@gmail.com designates 209.85.210.41 as permitted sender) smtp.mailfrom=andrii.nakryiko@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715058923; a=rsa-sha256; cv=none; b=RXakX0aRXjQ7nzY0gPs+VomUn8qX+WL2oOnopHItpaNajRAnOYweVpNEmYgiAo/7U7K1Yz +B0AYtrQo0bejvVx/yJz+qEMQm3amZMMp5j0qSdnTPumRbiUL00+bxjzv58kWbmspOf9Az ND1LjXSXbwBov4f+K/6hCQs1lmjO9sM= Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-6f065bc237aso814623a34.0 for ; Mon, 06 May 2024 22:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715058922; x=1715663722; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=D8vbZJ5UPiifYuspMMKBbXeF72Brhg4ICn1NMDsG3XQ=; b=WmVZfObbH581wyz4JBcAxx/A2qYfC98bsGY+hZA0UIs47x4wJDRZPSn8xc9g3YDBoN cRBZEXGB6finTd4zcSaoxte2GP3IqjV9rG1Z/6JmmhoG3eKFI5Fhv6kfXRUs3wHvdZLp aoivXk0SpQoLMPM9Ttc5ssZofA6IN0kXdE5Rj/Ks2zvFz7IIxYvySCuTUh4EakBBDsuI KkhDNyRE9eJxLudB+oBzr57r6XfaEYF6Cr8kDI76XmrBOcYMuDzNbVxylZ1dIWZeJRzU ucqsYkXOxMkZnbrmwHgCPia3mCpTjQLe2cmHwU8SmW3rlVz3p+Y2MMWs7pYeGeFSobKT fWRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715058922; x=1715663722; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D8vbZJ5UPiifYuspMMKBbXeF72Brhg4ICn1NMDsG3XQ=; b=XL2rOaTw0JXFkNTNFjYhcjG90MKsXVll3mO7vuG+Xn7CPTcDwVL58DMDopvWRKR30p GxWnaRs06uPZNdgahZxC2YTAdFPZRFBlPJYQ+wvlE8adcCM1PFeVTPLGNBzPzYW17VSd OL7YJZ8l1XD0pNFV4Tb0uP3LaFgYxZwkB+GkMgYgcosomgVw9SFkgQnEdgA/QLUCXfgg w0ispQgvxRcRdsg0VOYtkub/8Kw9qJGPb7Mkf2SdJ7UW/2HmnZperI3C4hKCmJk+MkB8 UrlJP4kxE5uofgsU/uM4RiS9S+kRofdUm8wsCK3QeOJUtRtEy/YX5eQ44W8S45xYPBqk Vjhw== X-Forwarded-Encrypted: i=1; AJvYcCXroXJcTRJPMoe8H8SDDNWKVGZlOOVlQA8sDaj69zOFhNxLNwVprGVc4DGdinlIoV0rTXgnS86A2Sdq2a5uY6DFyLE= X-Gm-Message-State: AOJu0Ywg5xzlNmoRpiORDCnuOemJCIDij+7irX/tUk2y2a3mqranrXmy GviSAhGfyfhlCx+stJ9oCrHHB2YTZNnEI4ZfotUBH5mPEhx9drRPHYdpx/CLmk9+XRElyQXx6KL VW/8uc1Q5GcZJvPp5MMpnyxiUg/xTcVKf X-Google-Smtp-Source: AGHT+IEB8vQbtVs+3K7IpXxE9aS+S0ebnjARauIth7zOTQsKYLVCTGe39Yqo8hcl7fuF+7QLXoq2DOEb7xuVPYELUqE= X-Received: by 2002:a17:90b:38c4:b0:2b4:e4d2:e6f0 with SMTP id nn4-20020a17090b38c400b002b4e4d2e6f0mr3870255pjb.45.1715058428309; Mon, 06 May 2024 22:07:08 -0700 (PDT) MIME-Version: 1.0 References: <20240504003006.3303334-1-andrii@kernel.org> <20240504003006.3303334-6-andrii@kernel.org> <2024050404-rectify-romp-4fdb@gregkh> In-Reply-To: From: Andrii Nakryiko Date: Mon, 6 May 2024 22:06:56 -0700 Message-ID: Subject: Re: [PATCH 5/5] selftests/bpf: a simple benchmark tool for /proc//maps APIs To: Ian Rogers Cc: Greg KH , Andrii Nakryiko , linux-fsdevel@vger.kernel.org, brauner@kernel.org, viro@zeniv.linux.org.uk, akpm@linux-foundation.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-mm@kvack.org, Arnaldo Carvalho de Melo , "linux-perf-use." Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7F4A31C0010 X-Stat-Signature: 8751yqkyqi7ok8ca38eqrhcz8acg6frm X-HE-Tag: 1715058923-604101 X-HE-Meta: U2FsdGVkX18bz6mGIJaWZ5rkXCHItbgnpBVQJZwNJXPu+oM4lo+7ynRpaAEKbkUuLjAJCGAuz0AFnNxuYP9tyY0nUEW8nuoVOWlklSt83Sj+343twzO145OC+KtiaSQoquoAHOFJ4BuUhfwveBD+yDlunQ5ghOumoXkj/5G8/DJTh2Lm8Sy0ViNzi/RYTBTwhqpfzzs3AFgowzcVMreG1cwg2lI1zKnPlwnaLuWobpdKGnto5hQggzw9N+i6VLB8QQ4md6bLMbE5Xiy7RmF1WERVOkqcAyRVbSrG4pReLMYjScVJw0S0mXEPLP4DMJCpMHp9dn3qscf0ZBfqM7/ugC+xxCViOgwg6AHa3RTrjQ8S+BzfxSeTkogtC47iA3cX/QyvRQM/qLbC92tlv7fR0bMJjt0s/7rpK0cN5Moafe99THGGmnyaLw6pgiKC2MsHxt0Zm5F59sOFvJmxj/wQYc2brDJiP1WEqaLnKB2dy3L7grVy97IAWzebbhvpUGDpGsnK0bMpK18Fs4XhmHbtl511DsEoPq2RzhdlDSemd9TGrHqerfhOBtkw0UiRg5REuHRhPlovvyQnCVCWwcBd1UvKSirAGFszuCKWwG0/SN1iFjaEyhdRLv0ZtG5sRabOaxx9j3EwQNu5IbedNgmRCxGqu+Xg4+XDFg/bp6hrRIQON3QdilN1jyAiYwL5ftui49uU7gwEwWQblc8z1yLTvUGQ7lI/QP7R0EXJftwHtEkwectxFZbZe4jyZ2zQJhUrWuZjIUM2yYp5KLl2btbdx7Ti5LlK/jz8jS/SjoI4p09B7zGAAbUEE5yeMW0FnCwrbHL5ipS5OE8OD9kkuznKNZeE6rxh/4SfqEmM1OY/5TdNjME8PkehbBw6zG2ORMgMlGaEHYHnQhXBp6pUOWy4miNV4RMI/jlvdIv5KGQl78Yrjef3S4KUv+AiqEJq/1KaFmykqN1HkOy3P3sTCK7 rTBruNpC dSutGhzTBQJBVyySRD39HCyoJZTBuxaUcn0WL0T7kfZtT/YDrzvQKMQ1jnAzrVJ81FSsrG6/Y1ItLd0hbDPOyi5hUlsTE8h4pHPW1yOmeIFls8Foq77r8pK+qrgRT7xxAEkBxGS12czdYmvT28suYeFI9/uW7Uv6a3OpIeK88zYcRnGwdotc2iZGmhGiqPNopcs4KuprEvy+9/3B+wEJRu4Zhzwqp2py1ayS+D7RgLpXi3INvqeeV1KARfbGb+O/kzC2mPDdJGcodFda5vejFgPhKk5ucMxRO2p+8ckznZdbcr0Kyrjq1y9KAzxRYu7GFO7A6yhuV8E/ZiVBmNNaSawgK5LY5PoGDwJtkKY13gjEzdK8hlwyjaErFq6pmfhlJXGtDJ+n3pAxlKBPXX2DVgIIiRy/EJ4+WVNBcM26Re4caKDY4vnJbia7WKAzWG43cOHfo242zMHLox0M9lrS3WbA07XKQuJ9aPf/3oIAS1ZCXxYgS1Xc5g/LlYI+RLJt/1ZNMYHmXKdVixh0SwulVeVnaJuEIz0kFDegBEasxYsCa0KaacxsLlAgV8TjESSRUu54w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 6, 2024 at 11:43=E2=80=AFAM Ian Rogers wro= te: > > On Mon, May 6, 2024 at 11:32=E2=80=AFAM Andrii Nakryiko > wrote: > > > > On Sat, May 4, 2024 at 10:09=E2=80=AFPM Ian Rogers = wrote: > > > > > > On Sat, May 4, 2024 at 2:57=E2=80=AFPM Andrii Nakryiko > > > wrote: > > > > > > > > On Sat, May 4, 2024 at 8:29=E2=80=AFAM Greg KH wrote: > > > > > > > > > > On Fri, May 03, 2024 at 05:30:06PM -0700, Andrii Nakryiko wrote: > > > > > > Implement a simple tool/benchmark for comparing address "resolu= tion" > > > > > > logic based on textual /proc//maps interface and new binar= y > > > > > > ioctl-based PROCFS_PROCMAP_QUERY command. > > > > > > > > > > Of course an artificial benchmark of "read a whole file" vs. "a t= iny > > > > > ioctl" is going to be different, but step back and show how this = is > > > > > going to be used in the real world overall. Pounding on this fil= e is > > > > > not a normal operation, right? > > > > > > > > > > > > > It's not artificial at all. It's *exactly* what, say, blazesym libr= ary > > > > is doing (see [0], it's Rust and part of the overall library API, I > > > > think C code in this patch is way easier to follow for someone not > > > > familiar with implementation of blazesym, but both implementations = are > > > > doing exactly the same sequence of steps). You can do it even less > > > > efficiently by parsing the whole file, building an in-memory lookup > > > > table, then looking up addresses one by one. But that's even slower > > > > and more memory-hungry. So I didn't even bother implementing that, = it > > > > would put /proc//maps at even more disadvantage. > > > > > > > > Other applications that deal with stack traces (including perf) wou= ld > > > > be doing one of those two approaches, depending on circumstances an= d > > > > level of sophistication of code (and sensitivity to performance). > > > > > > The code in perf doing this is here: > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tr= ee/tools/perf/util/synthetic-events.c#n440 > > > The code is using the api/io.h code: > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tr= ee/tools/lib/api/io.h > > > Using perf to profile perf it was observed time was spent allocating > > > buffers and locale related activities when using stdio, so io is a > > > lighter weight alternative, albeit with more verbose code than fscanf= . > > > You could add this as an alternate /proc//maps reader, we have a > > > similar benchmark in `perf bench internals synthesize`. > > > > > > > If I add a new implementation using this ioctl() into > > perf_event__synthesize_mmap_events(), will it be tested from this > > `perf bench internals synthesize`? I'm not too familiar with perf code > > organization, sorry if it's a stupid question. If not, where exactly > > is the code that would be triggered from benchmark? > > Yes it would be triggered :-) Ok, I don't exactly know how to interpret the results (and what the benchmark is doing), but numbers don't seem to be worse. They actually seem to be a bit better. I pushed my code that adds perf integration to [0]. That commit has results, but I'll post them here (with invocation parameters). perf-ioctl is the version with ioctl()-based implementation, perf-parse is, logically, text-parsing version. Here are the results (and see my notes below the results as well): TEXT-BASED =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # ./perf-parse bench internals synthesize # Running 'internals/synthesize' benchmark: Computing performance of single threaded perf event synthesis by synthesizing events on the perf process itself: Average synthesis took: 80.311 usec (+- 0.077 usec) Average num. events: 32.000 (+- 0.000) Average time per event 2.510 usec Average data synthesis took: 84.429 usec (+- 0.066 usec) Average num. events: 179.000 (+- 0.000) Average time per event 0.472 usec # ./perf-parse bench internals synthesize # Running 'internals/synthesize' benchmark: Computing performance of single threaded perf event synthesis by synthesizing events on the perf process itself: Average synthesis took: 79.900 usec (+- 0.077 usec) Average num. events: 32.000 (+- 0.000) Average time per event 2.497 usec Average data synthesis took: 84.832 usec (+- 0.074 usec) Average num. events: 180.000 (+- 0.000) Average time per event 0.471 usec # ./perf-parse bench internals synthesize --mt -M 8 # Running 'internals/synthesize' benchmark: Computing performance of multi threaded perf event synthesis by synthesizing events on CPU 0: Number of synthesis threads: 1 Average synthesis took: 36338.100 usec (+- 406.091 usec) Average num. events: 14091.300 (+- 7.433) Average time per event 2.579 usec Number of synthesis threads: 2 Average synthesis took: 37071.200 usec (+- 746.498 usec) Average num. events: 14085.900 (+- 1.900) Average time per event 2.632 usec Number of synthesis threads: 3 Average synthesis took: 33932.300 usec (+- 626.861 usec) Average num. events: 14085.900 (+- 1.900) Average time per event 2.409 usec Number of synthesis threads: 4 Average synthesis took: 33822.700 usec (+- 506.290 usec) Average num. events: 14099.200 (+- 8.761) Average time per event 2.399 usec Number of synthesis threads: 5 Average synthesis took: 33348.200 usec (+- 389.771 usec) Average num. events: 14085.900 (+- 1.900) Average time per event 2.367 usec Number of synthesis threads: 6 Average synthesis took: 33269.600 usec (+- 350.341 usec) Average num. events: 14084.000 (+- 0.000) Average time per event 2.362 usec Number of synthesis threads: 7 Average synthesis took: 32663.900 usec (+- 338.870 usec) Average num. events: 14085.900 (+- 1.900) Average time per event 2.319 usec Number of synthesis threads: 8 Average synthesis took: 32748.400 usec (+- 285.450 usec) Average num. events: 14085.900 (+- 1.900) Average time per event 2.325 usec IOCTL-BASED =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D # ./perf-ioctl bench internals synthesize # Running 'internals/synthesize' benchmark: Computing performance of single threaded perf event synthesis by synthesizing events on the perf process itself: Average synthesis took: 72.996 usec (+- 0.076 usec) Average num. events: 31.000 (+- 0.000) Average time per event 2.355 usec Average data synthesis took: 79.067 usec (+- 0.074 usec) Average num. events: 178.000 (+- 0.000) Average time per event 0.444 usec # ./perf-ioctl bench internals synthesize # Running 'internals/synthesize' benchmark: Computing performance of single threaded perf event synthesis by synthesizing events on the perf process itself: Average synthesis took: 73.921 usec (+- 0.073 usec) Average num. events: 31.000 (+- 0.000) Average time per event 2.385 usec Average data synthesis took: 80.545 usec (+- 0.070 usec) Average num. events: 178.000 (+- 0.000) Average time per event 0.453 usec # ./perf-ioctl bench internals synthesize --mt -M 8 # Running 'internals/synthesize' benchmark: Computing performance of multi threaded perf event synthesis by synthesizing events on CPU 0: Number of synthesis threads: 1 Average synthesis took: 35609.500 usec (+- 428.576 usec) Average num. events: 14040.700 (+- 1.700) Average time per event 2.536 usec Number of synthesis threads: 2 Average synthesis took: 34293.800 usec (+- 453.811 usec) Average num. events: 14040.700 (+- 1.700) Average time per event 2.442 usec Number of synthesis threads: 3 Average synthesis took: 32385.200 usec (+- 363.106 usec) Average num. events: 14040.700 (+- 1.700) Average time per event 2.307 usec Number of synthesis threads: 4 Average synthesis took: 33113.100 usec (+- 553.931 usec) Average num. events: 14054.500 (+- 11.469) Average time per event 2.356 usec Number of synthesis threads: 5 Average synthesis took: 31600.600 usec (+- 297.349 usec) Average num. events: 14012.500 (+- 4.590) Average time per event 2.255 usec Number of synthesis threads: 6 Average synthesis took: 32309.900 usec (+- 472.225 usec) Average num. events: 14004.000 (+- 0.000) Average time per event 2.307 usec Number of synthesis threads: 7 Average synthesis took: 31400.100 usec (+- 206.261 usec) Average num. events: 14004.800 (+- 0.800) Average time per event 2.242 usec Number of synthesis threads: 8 Average synthesis took: 31601.400 usec (+- 303.350 usec) Average num. events: 14005.700 (+- 1.700) Average time per event 2.256 usec I also double-checked (using strace) that it does what it is supposed to do, and it seems like everything checks out. Here's text-based strace log: openat(AT_FDCWD, "/proc/35876/task/35876/maps", O_RDONLY) =3D 3 read(3, "00400000-0040c000 r--p 00000000 "..., 8192) =3D 3997 read(3, "7f519d4d3000-7f519d516000 r--p 0"..., 8192) =3D 4025 read(3, "7f519dc3d000-7f519dc44000 r-xp 0"..., 8192) =3D 4048 read(3, "7f519dd2d000-7f519dd2f000 r--p 0"..., 8192) =3D 4017 read(3, "7f519dff6000-7f519dff8000 r--p 0"..., 8192) =3D 2744 read(3, "", 8192) =3D 0 close(3) =3D 0 BTW, note how the kernel doesn't serve more than 4KB of data, even though perf provides 8KB buffer (that's to Greg's question about optimizing using bigger buffers, I suspect without seq_file changes, it won't work). And here's an abbreviated log for ioctl version, it has lots more (but much faster) ioctl() syscalls, given it dumps everything: openat(AT_FDCWD, "/proc/36380/task/36380/maps", O_RDONLY) =3D 3 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D 0 ... 195 ioctl() calls in total ... ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D 0 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x9f, 0x1, 0x60), 0x7fff6b603d50) =3D -1 ENOENT (No such file or directory) close(3) =3D 0 So, it's not the optimal usage of this API, and yet it's still better (or at least not worse) than text-based API. [0] https://github.com/anakryiko/linux/commit/0841fe675ed30f5605c5b228e18= f5612ea253b35 > > Thanks, > Ian > > > > Thanks, > > > Ian > > > > > > > [0] https://github.com/libbpf/blazesym/blob/ee9b48a80c0b4499118a1= e8e5d901cddb2b33ab1/src/normalize/user.rs#L193 > > > > > > > > > thanks, > > > > > > > > > > greg k-h > > > >