From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF250C67861 for ; Fri, 5 Apr 2024 15:21:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 705F26B0087; Fri, 5 Apr 2024 11:21:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 68F096B0089; Fri, 5 Apr 2024 11:21:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 530036B008A; Fri, 5 Apr 2024 11:21:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3838E6B0087 for ; Fri, 5 Apr 2024 11:21:02 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C4770161187 for ; Fri, 5 Apr 2024 15:21:01 +0000 (UTC) X-FDA: 81975841122.20.E95CA14 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf28.hostedemail.com (Postfix) with ESMTP id E585DC0014 for ; Fri, 5 Apr 2024 15:20:59 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eWASSIjv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of surenb@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712330459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xfDVWiPNFB4hCGgsx7SCawcdvp63A70UKMWvRcHoJQg=; b=CW3YtB4A5z/zTXuo7RmHRtLrPzoop2CMfxfm8RdhSOZhi9sbDHMtxJecBxU3l3eeITURdo /N/PtvMEB+tl09NfLVjQ1RTnUeb5ay55WdYc7P612bQprh9r/m+Ztv3HtHPyuZ1YTtAp+U WVuDphTYc1gznyEqJglFwA9BC9Xq9QE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eWASSIjv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of surenb@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712330459; a=rsa-sha256; cv=none; b=zZvZjvbh3BiC2kriVXgCzmwmJ97+43cYfJR7w7J9Hn4QJE9m+3Hp3MM+xZXWm2mTTwxcWh 0frPrQvwov//38L6vyZEDdmZgbCUqlyt4d2/7lCShHQ07fKNN4+YrltK965PK17xq4PnOq 90WB98zAH11aJ4xRZN2darasMDbBAi4= Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-dc6cbe1ac75so1740712276.1 for ; Fri, 05 Apr 2024 08:20:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1712330459; x=1712935259; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=xfDVWiPNFB4hCGgsx7SCawcdvp63A70UKMWvRcHoJQg=; b=eWASSIjvQ16BFBtqxHGaAn+NizMbETpmzfQt7e2jtf4C6em0cAGnjlAinF8GuXGPdY gDiDHBLleUkucWCbhd4v34Tkm32y9EInZE0djhEXRomp0gdgmhB6tvaFyH6Toe+eqaTM v43hl5hO6pHxaJjLciL956w3zUp9KU6PzHgUjvV2oklTTTdIVWWUuSCGfa1BtsIfFBiN Rl0HfTj+kmG+g5GvVyRIKwx8uNGcR5UY+FISGQtJflWsF41diWLrckNhVcMwK9pz//4w RRffx3861M34qdymYZJ80KULzS1KvVCQMp4Nu0JtKc3bBkjSWHykx0hh9JZ1QM8UNNaL 0WRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712330459; x=1712935259; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xfDVWiPNFB4hCGgsx7SCawcdvp63A70UKMWvRcHoJQg=; b=kd3aUS2JPhDKPyWyAF/3JF+wGUskaypg0DoCLYW+oFsPxXsQ3oOo4C97vH6Kk80ihR KWkU9ANTufM6wNMD7J5slN17vvAGcUUoSW5W4dfiaIAwPhovE1VVUtJ4TT5/w9t4+e/E AXoqFakBOkEPGn7Mmut0rnEg0RlYJjRD+cPJfIrp2oLY4QvZ5mU570OBTZf9PuplvuCO 21CFVF/bb2lp1s1MAv2dujozhXnq4V0nnC7PY2+41B8BQxH4P5TFGKkG7dmOj/XMmrKt jAybdOrTNn186aFd8vhXqbEwB5fwHs5R+ZNo8krdgkK7AMzO3qpyRJKWjFCW7TM0HW7O AWkg== X-Forwarded-Encrypted: i=1; AJvYcCUI+THnxiB/Wga3SVCv1nZ4HBvf02EUMzYkRwopbIdAsghzJlZz8b8Z1exonMMUBn3Cc3zxxAt6yhlU2utZbE8P91w= X-Gm-Message-State: AOJu0Yzc38ZF3s1fCR7fAUG9mez7NCJfCgfLwtgfOeoB81ZMRQYkOBJm SHPw2LvoB+kFdEJgnOM+UDl37DxXzsGUiF52Mgy63pOj3u1iGUz1vfLBE9K3UbsXutQYKnnuX/H BGMvZB5cQe3Z8hi6FJEL7VXiJhUrwgbgywAr3 X-Google-Smtp-Source: AGHT+IFEI5zu5AiOteuhDQ2sEIS5wB7/bvXsQ5XrghDRi0ac0xrycb3LZS7XKg1pBjRTjmrFbQ1PTicmGSgIyMm6syU= X-Received: by 2002:a25:6ec3:0:b0:dc6:d258:c694 with SMTP id j186-20020a256ec3000000b00dc6d258c694mr2014282ybc.19.1712330458412; Fri, 05 Apr 2024 08:20:58 -0700 (PDT) MIME-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> <41328d5a-3e41-4936-bcb7-c0a85e6ce332@gmail.com> In-Reply-To: <41328d5a-3e41-4936-bcb7-c0a85e6ce332@gmail.com> From: Suren Baghdasaryan Date: Fri, 5 Apr 2024 08:20:44 -0700 Message-ID: Subject: Re: [PATCH v6 00/37] Memory allocation profiling To: Klara Modin Cc: akpm@linux-foundation.org, kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: E585DC0014 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: sh9prm4bsg4uoruyf5fhjj8g38xfu8b4 X-HE-Tag: 1712330459-608622 X-HE-Meta: U2FsdGVkX1+lliqZHlkmzB/6sMTWMIhwrxfR5dw4aMWVdFu46i3TNB58XSzSPZ7ej9f48RCbAorpmx/BWjJUFh9gP+CPpTVd6q7y0utGF3m0q+5vpfyfX5p0Hw3EkQfBrC63f7G65bdGjvRC+cv692MC3uoE4fAkusDAL2/40yJpOTpTuwPAmlcUuTtcWeeXgNZZJQQYUGLAxpuGPgwouB+UTA5ATbFL6qBzaalECfd1tk1Gmkp9BNxZnq+W14ec3WzXSfHScmDzQsWxU19MD+1MuheFgX+T7PWlOcjGAbJbYj0j3SiSvp8eWevVIAiAFmzbi80ICHwPmbgmC9Yew5HBduBdUFlIN/UAQkGU363x11PhVt+F9Ffue/HDSNj9B4/Loxf4Eh3zzeHIYHXowRbB7ELw+S1TbGrPC8Jqh31TkedRrCzkT5j3Nqif1Sq4RoKgfS2pPO+FBFSsjVCqJoM6AtLq8GafGqoZPMivtontSOQFLwrTZtLEqs1qRHGkagZy7Bf4A1VOjJJVwWEgbQomdMx1G+giE2KpSWRoTAw/92QJ8Of6r5NupkiqSN1i/zGHqSn2gYfVsAOPuNfl7mLmhc8sLVdsEWSCijM/bXUUymmgrou2kqJLnFXWi2PfuXuttMP0BNlgHLxbNeJ5R9LC+7QnPf9ZwatR4xBb5S14wEHlbwu1QoR3YRMe2ff9s0LJz/VWG9uD8e5SeQwfGdL54ywHCbsxPkSI6npniaLljhifJljhRUlmT1lImxzz1H8Tz0ZSIkbfxSwsrs6sVyI8ELq0oCCGV59QiiosebhC1Jw+yGHIg6w5WAj0gzKYNKStwLVf5qOAi863ZYUUHZPb7mMnWrVGI/B8URTRdn/P18JdjPMlIeRhFySCAt0UDlgsWOZKBmfWaQRZe0vDPgeeE3BvcQXZ8k7zHsQn+hcuk8+BqzXlXR9sG/2ZJDhEvfONNrcevi9gX+NQX8D QHJoq4Rm Mt2YXOqB5+YHp1VRJoQsSMcpvwNU6w+3rBS0X9jd/NHHvXIMWeQKmcd/HOsevN+aYukSOx1dk2f9IMttachcXX0u5XsWtrgtzMwHKrcHipSqld5nXNHeHoeu1e+BdzK0sHjCGco3D/vX+zptRoh5jEao9jSClUhihmf3EfFJpRGqWhtpaWfmUB3/wurH07Svn3sVhfYwsTmSafm78CMbXzKy34iCIs9XBZRYUK+x755ucMwUt0s0mwEKr4GgS7uzxJ8DDKcA4MgZT/T3QZjKohIoyxBwRK06CTkbXqbgB9L+1cbEVtf36FAe28fNOudVRuI7W X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 5, 2024 at 7:30=E2=80=AFAM Klara Modin = wrote: > > On 2024-04-05 16:14, Suren Baghdasaryan wrote: > > On Fri, Apr 5, 2024 at 6:37=E2=80=AFAM Klara Modin wrote: > >> If I enable this, I consistently get percpu allocation failures. I can > >> occasionally reproduce it in qemu. I've attached the logs and my confi= g, > >> please let me know if there's anything else that could be relevant. > > > > Thanks for the report! > > In debug_alloc_profiling.log I see: > > > > [ 7.445127] percpu: limit reached, disable warning > > > > That's probably the reason. I'll take a closer look at the cause of > > that and how we can fix it. > > Thanks! In the build that produced debug_alloc_profiling.log I think we are consuming all the per-cpu memory reserved for the modules. Could you please try this change and see if that fixes the issue: include/linux/percpu.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/percpu.h b/include/linux/percpu.h index a790afba9386..03053de557cf 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -17,7 +17,7 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES #ifdef CONFIG_MEM_ALLOC_PROFILING -#define PERCPU_MODULE_RESERVE (8 << 12) +#define PERCPU_MODULE_RESERVE (8 << 13) #else #define PERCPU_MODULE_RESERVE (8 << 10) #endif > > > > > In qemu-alloc3.log I see couple of warnings: > > > > [ 1.111620] alloc_tag was not set > > [ 1.111880] WARNING: CPU: 0 PID: 164 at > > include/linux/alloc_tag.h:118 kfree (./include/linux/alloc_tag.h:118 > > (discriminator 1) ./include/linux/alloc_tag.h:161 (discriminator 1) > > mm/slub.c:2043 ... > > > > [ 1.161710] alloc_tag was not cleared (got tag for fs/squashfs/cache= .c:413) > > [ 1.162289] WARNING: CPU: 0 PID: 195 at > > include/linux/alloc_tag.h:109 kmalloc_trace_noprof > > (./include/linux/alloc_tag.h:109 (discriminator 1) > > ./include/linux/alloc_tag.h:149 (discriminator 1) ... > > > > Which means we missed to instrument some allocation. Can you please > > check if disabling CONFIG_MEM_ALLOC_PROFILING_DEBUG fixes QEMU case? > > In the meantime I'll try to reproduce and fix this. > > Thanks, > > Suren. > > That does seem to be the case from what I can tell. I didn't get the > warning in qemu consistently, but it hasn't reappeared for a number of > times at least with the debugging option off. > > Regards, > Klara Modin