From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2616FC02183 for ; Mon, 13 Jan 2025 21:56:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AAC216B008C; Mon, 13 Jan 2025 16:56:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A5CDF6B0092; Mon, 13 Jan 2025 16:56:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D6BD6B0093; Mon, 13 Jan 2025 16:56:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6A3946B008C for ; Mon, 13 Jan 2025 16:56:37 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id EA8A1AEA82 for ; Mon, 13 Jan 2025 21:56:36 +0000 (UTC) X-FDA: 83003788392.06.9799D98 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) by imf12.hostedemail.com (Postfix) with ESMTP id 0B7D64000C for ; Mon, 13 Jan 2025 21:56:34 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="eW/pUGjc"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of surenb@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736805395; a=rsa-sha256; cv=none; b=nOu8+bq2AUw/863ty2F3YjuOTl8POF9mwHxdk7WfAiC6SK+f7e7D/MD+2ZTH+ZxAaFrC3y 1nhIeaXqD+eJwxGWXBxSsz3DR47lzb3mRwll4flF7s5x0LOiLPor2oSJMJs7fILdLklLcc 21h3dxDi9SkLfyJGoKBECdq2kzdZ2vE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="eW/pUGjc"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf12.hostedemail.com: domain of surenb@google.com designates 209.85.160.181 as permitted sender) smtp.mailfrom=surenb@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736805395; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G84V4eoUdbKVxwZugEKXDJQdU66KCiiWaYxfkEnXs1A=; b=z292zH7Sj4rRoahYgOIz2TavYVYuadcvkNniBzpcfRGyDOlm2+FdNwMj/u365pnwAdxilz 0CxLILlF7TaIlNpZVq50mg2Q4vWkoKt9xL1aHYXdU/BDKi+4zM8Bq9e8XY1itPWXK9yqvH LzqOuiwz6QfvQ6Kw5NfDCBSUMGjRePg= Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-4679b5c66d0so16171cf.1 for ; Mon, 13 Jan 2025 13:56:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736805394; x=1737410194; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=G84V4eoUdbKVxwZugEKXDJQdU66KCiiWaYxfkEnXs1A=; b=eW/pUGjcpjmiNEhkmF/NHcffKayaeiEjIDLLh3QPtOWd/FuACwvmzU3fCNsYi14UDy L35mvcojZUH0OE7iSBgYpvWGSH557nK8JvOztCsbZuyyvmV57V4+gz9JrQpPz5kwOg2o /xBCYoQVdtlvoFtVaTgL+mYEHRIkiAmjso5vn/7VUSgEhrw7cCLXtYvNi5nYYESc7ao6 +IxxqncTeZRa96VOKZO5LX9xz0YVwUHlADqanqPbyr/tMDdqcn1fzg7Kofwn3HWbSh22 L1t46+pSpYalCn8W6VL1W0XSfysdVfIRBXDsLGLCTyr+whOc7a9/txUq7cQKlXXPlXBq H8eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736805394; x=1737410194; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G84V4eoUdbKVxwZugEKXDJQdU66KCiiWaYxfkEnXs1A=; b=tGPB4X1msa7amc5xM+AZxlgIEqBolE2e7vx1z8+lcpKezEnI3347qnwoXmGjzPSU6i wCLKBmMBv9Pr10Ptt+FundhiiRmZlxmpsHwZXuAlw7ii+NhmUyt7kKLdh3shngp2fFam LuqCZ3yG0tRLG2FT2YYqTjd499iXu+YypbWplyxDOWuoIR/qX0Dhg+RmC2bPD6JJoYs3 d8DMCZnJk5mUc33RBxLXueCochoKXlrjF0O8KP4p/GceIevxwhm3+8Bw9LP6zr2HF+Ty Auhi8LvJ5AIP0uJJ28fK6+mjP/iYuAA5wm3MSeW9wFt4gtYaGudBo9lhCxdu5JEmve4x vt9Q== X-Forwarded-Encrypted: i=1; AJvYcCXSZy3Ckf/Gu8Czuo2OPPVBnk/e8DLQHAH0FGwYg24oSkvXxTpTG0vZxZEb6MlY8wEHAIqMod3Tww==@kvack.org X-Gm-Message-State: AOJu0YxTyBqW1FFaLepNjB+jChGgnqaIKgAaOMgVD0t5RCTop02nJtaL aRkVry2PRnffPZmW2jqHL/JndtTYwLHxAMGBBF/QE3iu8WtX4CwyMcX0VV6Jh73QgYFIdpQm40P 33VYcx8qXgQKMQqbPileDHgDOZqNbJ82tap7k X-Gm-Gg: ASbGncuEJs1H+2k4zqLJojzIFTSYyCnVAJPTn2ntaOOaLNjMiCpt/3TvOKUmHBLaRV5 4lNhQimNW/gRbP6wmGhoAV0ES6LKnt7SYRUA5nA== X-Google-Smtp-Source: AGHT+IEtqXLLNqaVi1sBp2p7m2+mJ/cfVCXTGyAEk7OPWTXwdc9+LGh400ZjRHsRgiCCcWu/EHvF9kRFFE9MR6kQzPQ= X-Received: by 2002:a05:622a:1aa0:b0:466:8f39:fc93 with SMTP id d75a77b69052e-46dea6b5116mr94581cf.3.1736805393862; Mon, 13 Jan 2025 13:56:33 -0800 (PST) MIME-Version: 1.0 References: <20250106112103.25401-1-hao.ge@linux.dev> <48f208b6.32ab.19455c70dbe.Coremail.00107082@163.com> <254a4857.b2b.19458d0dbc2.Coremail.00107082@163.com> <213ff7d2.7c6c.1945eb0c2ff.Coremail.00107082@163.com> In-Reply-To: <213ff7d2.7c6c.1945eb0c2ff.Coremail.00107082@163.com> From: Suren Baghdasaryan Date: Mon, 13 Jan 2025 13:56:23 -0800 X-Gm-Features: AbW1kvZKYFSclgiPBhkZQt5fUujqTQvOIJvcNaSMMhqTiWK0rv4STJ2y0cfHlws Message-ID: Subject: Re: memory alloc profiling seems not work properly during bootup? To: David Wang <00107082@163.com> Cc: kent.overstreet@linux.dev, Hao Ge , akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Hao Ge , Alessio Balsini , Pasha Tatashin , Sourav Panda Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: s3twp7r8hx6gpwswh7tue97askduo3nn X-Rspam-User: X-Rspamd-Queue-Id: 0B7D64000C X-Rspamd-Server: rspam08 X-HE-Tag: 1736805394-827267 X-HE-Meta: U2FsdGVkX19PqD9/tCwwoTXvIPLnHURTt+06nCD/kxuO/nBeqcs+PmbcMx8PyXAxGFuWJveHL9Vv1egs3pLV+44XGUc8dnmwTMEOeJKjP8cHiVJad9UTMz5znj795BscTcRv6P3nYrhg6E2PntYuOqiMWh+D/JSn/ZtlVrqnS7tlmJuy/AixvxbP3p1Ek66GbUcz2fsZP2ZYUpbC9QW/H9OUAA9Rd79pkwPqDKg+ZgQLHYLq6V/tofAL6MpslGBFcAem1LTs77Tp5RH1PxXf4rDjkes+FovF+6oBmMFmgjDjuEPdGM2ona7FdmFrlPaLdGAOj99iDaAaq+XqK+dhCi29K686uHuALpHBUMdL5TKZ7IUNRsnFtVv/r/tKNpGtNFvfT26S6jEIPTGeU6FqdfdA/o2/loJN1Sy1j5EWksoRlcmw2Vfbs0RFjfEg1cd7xDkuNFDzCAR7Hy8IDGQJYoAePu2kws9vfJO1MSz0URSWNSGje/oFduOGu5I9BVGydwPT/MED6xRLhQP/iW631ZAPNK/eJgzaM95xoAgshy99qO3eqVOrBgxe2U4J1H3iPqiqRuWhirtvuboSyOh0yjlDDizC19sESSfhPXnl35DgfxHLe2VI1a11i4eJ770YtxC9bczdq7N/8qpyakEpVnsptNePJUb36n78gYkk6FPdDFhGOZdLlOZwRIUF84VTxKzVyqh7h++emLYRpaYMHKKEsUPf1+TwTzcnvyBfrxPWXbBeZc9iiMmH1rrBd1OTW36eQpVcn6DRePMXJ6ye9kDvFRZWwsGohrtnUTB3SPYKOpUVIDyNdqJl9O669ej4BpsvwOhrDLyrC34+6g1H+Ch/Ulqke7XfbWv+yAIcXkzdFaj68jod0bGoEcozQ6A5ukYOFopVtxVTBzNxNUDpvyY0hMMwFBcRh3xHAYMCqO5oSBglBfriSxlTsmDrIl2+BmV/ZJ5fn4PilbXUr77 yvPxoBiU X8kwJCazuFrAVtPeij46uYYy4rxZRW/Ntwc3vve+r9HGZHkvDDFZvUwdtzvJu2a+PJPAGdNSI0t2d9vgAsiJhFQAlKf23+3tspcUtgkxJtUz445BnpHtSZkxIXXj4HDYPRlQj3kuIIU4YG2Jl2reRZ2I+GjrdO31iQHGzjRjtptXSSqDWT/F5+BMvCiqXDfGe7j0fro1Ah29m5pBqSPWjSY7lu2TUU4JpuvPLejyNl1rlmthIJyXBSYRwYisZov7E4GWFFbRNRVCyg+cKikIIq2Ua1ahi+j6xDk+HB0/L/N0w45Ea4Gr7bbNQFWDtmsWKWF0dDuRsgFUqfiWxEcCG5xkj1BbEf0Kp2WOd X-Bogosity: Ham, tests=bogofilter, spamicity=0.009541, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jan 13, 2025 at 12:04=E2=80=AFAM David Wang <00107082@163.com> wrot= e: > > Hi, > > More update, > > When I boot up my system, no alloc_percpu was accounted in kernel/sched/= topology.c > > 996 14 kernel/sched/topology.c:2275 func:__sdt_alloc 80 > 996 14 kernel/sched/topology.c:2266 func:__sdt_alloc 80 > 96 6 kernel/sched/topology.c:2259 func:__sdt_alloc 80 > 12388 24 kernel/sched/topology.c:2252 func:__sdt_alloc 80 > 612 1 kernel/sched/topology.c:1961 func:sched_init_numa 1 > > And then after suspend/resume, those alloc_percpu shows up. > > 996 14 kernel/sched/topology.c:2275 func:__sdt_alloc 395 > 996 14 kernel/sched/topology.c:2266 func:__sdt_alloc 395 > 96 6 kernel/sched/topology.c:2259 func:__sdt_alloc 395 > 12388 24 kernel/sched/topology.c:2252 func:__sdt_alloc 395 > 0 0 kernel/sched/topology.c:2242 func:__sdt_alloc 70 = <--- > 0 0 kernel/sched/topology.c:2238 func:__sdt_alloc 70 = <--- > 0 0 kernel/sched/topology.c:2234 func:__sdt_alloc 70 = <--- > 0 0 kernel/sched/topology.c:2230 func:__sdt_alloc 70 = <--- > 612 1 kernel/sched/topology.c:1961 func:sched_init_numa 1 > > I have my accumulative counter patch and filter out items with 0 accumula= tive counter, > I am almost sure the patch would not cause this accounting issue, but not= 100%..... Have you tested this without your accumulative counter patch? IIUC, that patch filters out any allocation which has never been hit. So, if suspend/resume path contains allocations which were never hit before then those allocations would become suddenly visible, like in your case. That's why I'm against filtering allocinfo data in the kernel. Please try this without your patch and see if the data becomes more consistent. Thanks, Suren. > > > It seems to me, during boot up, some alloc_percpu is not registered. > > > FYI > David > > > > At 2025-01-12 12:41:10, "David Wang" <00107082@163.com> wrote: > > > > > >At 2025-01-11 22:31:36, "David Wang" <00107082@163.com> wrote: > >>Hi, > >> > >>I have using this feature for a long while, and I believe this memory a= lloc profiling feature > >>is quite powerful. > >> > >>But, I have been wondering how to use this data, specifically: > >>how anomaly could be detected, what pattern should be defined as anomal= y? > >> > >>So far, I have tools collecting those data (via prometheus), make basic= analysis, i.e. top-k, group-by or rate. > >>Those analysis help me understand my system, but I cannot tell whether = it is abnormal or not. > >> > >>And sometimes I would just read through /proc/allocinfo, trying to pick= up something. > >>(Sometimes get lucky, actually only once, find the underflow problem we= eks ago.) > >> > >>A tool would be more helpful if it can identify anomalies, and we can a= dd more pattern as develop along. > >> > >>A pattern may be hard to define, especially when it involves context. F= or example, > >>I happened to notice following strange things recently: > >> > >> 896 14 kernel/sched/topology.c:2275 func:__sdt_alloc 102= 5 > >> 896 14 kernel/sched/topology.c:2266 func:__sdt_alloc 102= 5 > >> 96 6 kernel/sched/topology.c:2259 func:__sdt_alloc 102= 5 > >> 12288 24 kernel/sched/topology.c:2252 func:__sdt_alloc 102= 5 <----- B > >> 0 0 kernel/sched/topology.c:2242 func:__sdt_alloc 210 > >> 0 0 kernel/sched/topology.c:2238 func:__sdt_alloc 210 > >> 0 0 kernel/sched/topology.c:2234 func:__sdt_alloc 210 > >> 0 0 kernel/sched/topology.c:2230 func:__sdt_alloc 210= <----- A > >>Code A > >>2230 sdd->sd =3D alloc_percpu(struct sched_domain *); > >>2231 if (!sdd->sd) > >>2232 return -ENOMEM; > >>2233 > >> > >>Code B > >>2246 for_each_cpu(j, cpu_map) { > >> ... > >> > >>2251 > >>2252 sd =3D kzalloc_node(sizeof(struct sched_do= main) + cpumask_size(), > >>2253 GFP_KERNEL, cpu_to_node(j)= ); > >>2254 if (!sd) > >>2255 return -ENOMEM; > >>2256 > >>2257 *per_cpu_ptr(sdd->sd, j) =3D sd; > >> > >> > >>The address of memory alloced by 'Code B', is stored in memory "Code A'= , the allocation counter for 'Code A' > >>is *0*, while 'Code B' is not *0*. Something odd happens here, either = it is expected and some ownership changes happened somewhere > >>, or it is a leak, or it is an accounting problem. > >> > >>If a tool can help identify this kind of pattern, that would be great!~ > >> > >> > >>Any suggestions about how to proceed with the memory problem of kernel/= sched/topology.c mentioneded > >> above?, or is it a problem at all? > >> > > > >Update: > > > >It seems the memory alloced by 'Code B' could be handovered via claim_al= locations: > >1530 /* > >1531 * NULL the sd_data elements we've used to build the sched_domain a= nd > >1532 * sched_group structure so that the subsequent __free_domain_alloc= s() > >1533 * will not free the data we're using. > >1534 */ > >1535 static void claim_allocations(int cpu, struct sched_domain *sd) > > > >So most likely, this is neither a leak nor a accounting issue. False ala= rm, sorry.... > > > >The reason I brought this up is that the profiling data is rich, but a u= ser who is not familiar > >with code detail could not make much out of it. If a tool can tell wheth= er the system is drifting away somewhere, > >like a healthcheck based on profiling data, it would be quite helpful. > > > >Thanks > >David > > > >