From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0888C54798 for ; Thu, 7 Mar 2024 18:18:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 545E06B0256; Thu, 7 Mar 2024 13:18:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 51D006B0257; Thu, 7 Mar 2024 13:18:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36F336B0258; Thu, 7 Mar 2024 13:18:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 23B246B0256 for ; Thu, 7 Mar 2024 13:18:10 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F1C81811F6 for ; Thu, 7 Mar 2024 18:18:09 +0000 (UTC) X-FDA: 81871052298.03.15ED877 Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by imf23.hostedemail.com (Postfix) with ESMTP id 2E4B5140006 for ; Thu, 7 Mar 2024 18:18:06 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=H5aOzJ+M; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf23.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709835488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iHaIxY/+BiJn46m1UxNdZN63kb5NNOFAVQARDvBki5g=; b=LmD026YShUM0t36xR5troRY3CvgW8xEYDD4qn6XXRhktNq4iko5BlRc7n8Nq4oBtI9hHp0 18n2CZW6DYU3NYe5055MhdUmn9btjfUYN3hmLffPZrQuIV6vqaLrvVritC99gVvAHuYx9i RhvnTVSD7XkLNRqufpne/wgt4MUHT10= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=H5aOzJ+M; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf23.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.182 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709835488; a=rsa-sha256; cv=none; b=EFyZ09FwYEDqRkUfaW705QQ/J3SznazjNva3pMLp8PocwEZPcIDXKii//hcpGA+ycZ1xkl UE0Wq0hTwBG/1jTe2L2oo9r6RklonaAx3ut6yd2mE2EDKlN05KPRXcVtvHfIdufNBVkKZY EnSqDeXxmnDLU62aLTqMBZkNLeWiTN4= Date: Thu, 7 Mar 2024 13:17:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1709835485; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=iHaIxY/+BiJn46m1UxNdZN63kb5NNOFAVQARDvBki5g=; b=H5aOzJ+MeszkUjW7Gdq8GIb9xZvvYFafE54Qajm1X53jP+9kSZaHykmws15lGZbxbVC4sP xB+aj3bB+XZoL9vVS6ke4qTd3NSX0xuXSFczxjIN4xYv/lKG+5cGIuGnJ1kVWq3Q8k+XsG i3YlKbhrl9zIpdvnTlhFtaDJQqe9fLg= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Randy Dunlap Cc: Suren Baghdasaryan , akpm@linux-foundation.org, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org Subject: Re: [PATCH v5 37/37] memprofiling: Documentation Message-ID: References: <20240306182440.2003814-1-surenb@google.com> <20240306182440.2003814-38-surenb@google.com> <10a95079-86e4-41bf-8e82-e387936c437d@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <10a95079-86e4-41bf-8e82-e387936c437d@infradead.org> X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 2E4B5140006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: a3pz8s6jfitym8sqokt7gbzfzdqe5sz3 X-HE-Tag: 1709835486-847178 X-HE-Meta: U2FsdGVkX19vkhBXzVdtXDa4kCVMM09gOq836vY5c68Q1uPhUSdLCcGihWhl6/zf6Fxl8Tf8x+eMVAc5+CbC7rWa8HO/aHtXBYKLADBx/0SkzpCns+68JmQ0+I0/I4PjgTsmkPRjjgjBL7vxTe3A4dwagLMWAQQFIzNSeOn4d65O2bYXl1uZc8cr8AanaPIlbihTKy/m3OXd37+yzIh6JHiv6n7U4vkizbQrtoz54SxYG5AymJhT2vYCvMcDa119MWuLp4f9YvBc9RPoA9wmCK0KPY7tE0qkk4fZdL0JTnSEMst4/s2v6yvsWm+CVwV0gRW7jCOWgFxPYHJ/pSqghBHelPHMn418A4eOzP++Fe3P/XqBX+E6B6k81XFYPoXwGEC8XVjxbZWTYYTUA3Onrw2MHFX9P4gcdEjzy/Hw9vjqVgAsUsUuvZhfJ8HUDnspXbAj5EigBZ6BmdcPk2RMGgAJtgYKPgmRAdjqACthBFTjTq+bMe7nAdTFFxMg+YbhiwDdBNNsSllX72gmEBYLIz5u+OYIHG2htOlj5aWi0so5hSqkOncWESmGdCsuulMhp+vdRtwbmZRhac4WsyflpNPYCPsj11W5yILUVcVCSmfHhI/kmV8y4fcwKidoSFF0fj3V2DrCNvWqBA9BzRgvO/n36LMOXJORc4PGAt3vso/6cLbqMf6cQ61wbLRJTeCB/RBriLEsUlzfYhL/NoAkrDUKY2BjIpuagjUiBGuQFWvshb18tqXCA9TX7XDdQqqWexCLVlWr16uo9qxtTqAO7lLDL89divUOnsabO1CZM179AT3ELzs2+K01IFbGrTu88QJ2iO5tcluUN/BUvWGD67kDc5JwCeqhJu46ux5/yZZ8896IQPo6rg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 06, 2024 at 07:18:57PM -0800, Randy Dunlap wrote: > Hi, > This includes some editing suggestions and some doc build fixes. > > > On 3/6/24 10:24, Suren Baghdasaryan wrote: > > From: Kent Overstreet > > > > Provide documentation for memory allocation profiling. > > > > Signed-off-by: Kent Overstreet > > Signed-off-by: Suren Baghdasaryan > > --- > > Documentation/mm/allocation-profiling.rst | 91 +++++++++++++++++++++++ > > 1 file changed, 91 insertions(+) > > create mode 100644 Documentation/mm/allocation-profiling.rst > > > > diff --git a/Documentation/mm/allocation-profiling.rst b/Documentation/mm/allocation-profiling.rst > > new file mode 100644 > > index 000000000000..8a862c7d3aab > > --- /dev/null > > +++ b/Documentation/mm/allocation-profiling.rst > > @@ -0,0 +1,91 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > + > > +=========================== > > +MEMORY ALLOCATION PROFILING > > +=========================== > > + > > +Low overhead (suitable for production) accounting of all memory allocations, > > +tracked by file and line number. > > + > > +Usage: > > +kconfig options: > > + - CONFIG_MEM_ALLOC_PROFILING > > + - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT > > + - CONFIG_MEM_ALLOC_PROFILING_DEBUG > > + adds warnings for allocations that weren't accounted because of a > > + missing annotation > > + > > +Boot parameter: > > + sysctl.vm.mem_profiling=0|1|never > > + > > + When set to "never", memory allocation profiling overheads is minimized and it > > overhead is > > > + cannot be enabled at runtime (sysctl becomes read-only). > > + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=y, default value is "1". > > + When CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT=n, default value is "never". > > + > > +sysctl: > > + /proc/sys/vm/mem_profiling > > + > > +Runtime info: > > + /proc/allocinfo > > + > > +Example output: > > + root@moria-kvm:~# sort -g /proc/allocinfo|tail|numfmt --to=iec > > + 2.8M 22648 fs/kernfs/dir.c:615 func:__kernfs_new_node > > + 3.8M 953 mm/memory.c:4214 func:alloc_anon_folio > > + 4.0M 1010 drivers/staging/ctagmod/ctagmod.c:20 [ctagmod] func:ctagmod_start > > + 4.1M 4 net/netfilter/nf_conntrack_core.c:2567 func:nf_ct_alloc_hashtable > > + 6.0M 1532 mm/filemap.c:1919 func:__filemap_get_folio > > + 8.8M 2785 kernel/fork.c:307 func:alloc_thread_stack_node > > + 13M 234 block/blk-mq.c:3421 func:blk_mq_alloc_rqs > > + 14M 3520 mm/mm_init.c:2530 func:alloc_large_system_hash > > + 15M 3656 mm/readahead.c:247 func:page_cache_ra_unbounded > > + 55M 4887 mm/slub.c:2259 func:alloc_slab_page > > + 122M 31168 mm/page_ext.c:270 func:alloc_page_ext > > +=================== > > +Theory of operation > > +=================== > > + > > +Memory allocation profiling builds off of code tagging, which is a library for > > +declaring static structs (that typcially describe a file and line number in > > typically > > > +some way, hence code tagging) and then finding and operating on them at runtime > > at runtime, > > > +- i.e. iterating over them to print them in debugfs/procfs. > > i.e., iterating i.e. latin id est, that is: grammatically my version is fine > > > + > > +To add accounting for an allocation call, we replace it with a macro > > +invocation, alloc_hooks(), that > > + - declares a code tag > > + - stashes a pointer to it in task_struct > > + - calls the real allocation function > > + - and finally, restores the task_struct alloc tag pointer to its previous value. > > + > > +This allows for alloc_hooks() calls to be nested, with the most recent one > > +taking effect. This is important for allocations internal to the mm/ code that > > +do not properly belong to the outer allocation context and should be counted > > +separately: for example, slab object extension vectors, or when the slab > > +allocates pages from the page allocator. > > + > > +Thus, proper usage requires determining which function in an allocation call > > +stack should be tagged. There are many helper functions that essentially wrap > > +e.g. kmalloc() and do a little more work, then are called in multiple places; > > +we'll generally want the accounting to happen in the callers of these helpers, > > +not in the helpers themselves. > > + > > +To fix up a given helper, for example foo(), do the following: > > + - switch its allocation call to the _noprof() version, e.g. kmalloc_noprof() > > + - rename it to foo_noprof() > > + - define a macro version of foo() like so: > > + #define foo(...) alloc_hooks(foo_noprof(__VA_ARGS__)) > > + > > +It's also possible to stash a pointer to an alloc tag in your own data structures. > > + > > +Do this when you're implementing a generic data structure that does allocations > > +"on behalf of" some other code - for example, the rhashtable code. This way, > > +instead of seeing a large line in /proc/allocinfo for rhashtable.c, we can > > +break it out by rhashtable type. > > + > > +To do so: > > + - Hook your data structure's init function, like any other allocation function > > maybe end the line above with a '.' like the following line. > > > + - Within your init function, use the convenience macro alloc_tag_record() to > > + record alloc tag in your data structure. > > + - Then, use the following form for your allocations: > > + alloc_hooks_tag(ht->your_saved_tag, kmalloc_noprof(...)) > > > Finally, there are a number of documentation build warnings in this patch. > I'm no ReST expert, but the attached patch fixes them for me. > > -- > #Randy