From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2133DC2D0CD for ; Mon, 19 May 2025 13:33:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 899666B00D5; Mon, 19 May 2025 09:33:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8480D6B00D6; Mon, 19 May 2025 09:33:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E8506B00D7; Mon, 19 May 2025 09:33:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4BE7D6B00D5 for ; Mon, 19 May 2025 09:33:32 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D5C5216081E for ; Mon, 19 May 2025 13:33:33 +0000 (UTC) X-FDA: 83459749506.23.D51A3C4 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf17.hostedemail.com (Postfix) with ESMTP id B5AF540009 for ; Mon, 19 May 2025 13:33:31 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="YtQp6U/S"; spf=pass (imf17.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747661611; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dq+HWWyDqsg/ZEBS6xelB5MUoV4ZPRa2fZFwi8bgY1U=; b=oseYh3tBxMpYOEyAxh3IZsJWxGE+VMppQPa3Gv+50JkMiaOePobOfNtpkjgCR9MSozFvnZ c34n3Oifr27xui+1VZ98/UiUUDuiC5M3JKMdVydl9Gz4/IOVnm917yEPoGcoEWG0whM+4A 8GG+kLRtmf5M/s/52NB6uP3TzKlNxj8= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="YtQp6U/S"; spf=pass (imf17.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747661611; a=rsa-sha256; cv=none; b=3vzqsf11LJHIKvu2aezlVS41a5yQqy17rJKMsi01vWm7lSzQvw0ZdzAxo9isxc6qa9svaf odMVPugdFzA/fuj5ZvxKxp698O3Px5gcMbrDSS+7C8TPSnjJmrr89dACmljDXNf2t4at0Z YIAlTHNtaOywcyvoad1cjU20zaerTD0= Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-ad53a96baf9so426509866b.3 for ; Mon, 19 May 2025 06:33:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747661610; x=1748266410; darn=kvack.org; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id:from:to :cc:subject:date:message-id:reply-to; bh=Dq+HWWyDqsg/ZEBS6xelB5MUoV4ZPRa2fZFwi8bgY1U=; b=YtQp6U/SKZjf9tj91MH+gUTEcccTIRmx8XWaaKQnYeU3VtSmY8a/88W9rcHoYQMheK t97DF4aN++1F6k1U2SUyjyeL/e04koyjcL0eG1BgWnP46ohRySOcionwzS4V8my+UiNU mqOqc7LkdCFN0CV/M4U2hDDJ0hrBueJImRbyZWRclvN8gu8/TcX7P5baSsPjZS6WfGnx xnuEPF5luJZ0eHtKxpwtyuzEBNy3KJq+QP04+of0Flk+mpDeNCRB3VA5nH203t7un6RY dFmKKeIoTk31nC8slUluaZa7ehNHyUtfMJ+EE538BQY13YFbtU5TTUeaU3DcH6FNv9hA MgHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747661610; x=1748266410; h=content-transfer-encoding:in-reply-to:content-language:references :cc:to:from:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dq+HWWyDqsg/ZEBS6xelB5MUoV4ZPRa2fZFwi8bgY1U=; b=o6W6zASwNm4GZ+tBccFqNtBk2Z+85tHJX25fDtzYURPF953EE5GC+u8oSFmoKCmiOb DOcjb1lLmHe77wKPJ/ZvUgFtG79/42Hs8LesBxaauBwbJEN4dvBqFzEDr2V+GpGPFCPB Qq6nBWmORSZ7lxPVAe4M98adusSy5VzkkKw9tqsJqJ+EP8Q2EmzKW9/7+1zj0XHkrN7m IeMDC5yFVdtjz0PGW/r/p8OiBolD+bBa4RDRcDCw7g4ma8R5PJUdX8erR8GQ6boCfrl8 C+x14oU8PRcDyqn9Hc/jHXC6NEDn+L/4E+4OsSWMU38TAQZq2rpQODfl20KSayn0t8gX yTYg== X-Forwarded-Encrypted: i=1; AJvYcCUas2Z/fEbrryOFkAXXrGAWAqEH5EhcVUW+3R44qa0W0Wt0UMcH3JahsFRExyQQRYnZrKqHvnppgA==@kvack.org X-Gm-Message-State: AOJu0Yy+p23hVKAsNWAFs/250lZlw0Ax+tb8GAKLpn4g4bor5w9A8Pz2 mQV7x7ptXeTlEYCKy7MdmYl0Qk4eAlkp0yqUsaUTL7O7KHK7eH1PMWKX X-Gm-Gg: ASbGncuyO4XGIBxD2hS25z91xlnzvjrFo6nKtP/oLBn2vgZgqHBOSrmUpIx1tPKcJcp EYrRnuo2Heo1tROiTVGRj7UEUp+cwpMAigq06MwwXO2MtmP06yW37u5R9BzXuodVG5KoftvMaJH jhDn2UQ39gsQLz2ODBqIgP76AmR7u2DppqfPtIU9sLlPmoYTasKtfRBiJiJ36B1PiyNTf2kS6wI WjLPGg1rlO+/hh2gyIi/9+oS2P4GY3VNEQA5huwLz4YPL2N/p9Hm11+ZTUlSppwUPQ5HOXlBpEL fRX4UgHcm+OydhTSnD6c2VcPtjYf09raNOG09bl4MSyVHnWpBe2SHMvJRgAogTy45q/KLJ036N9 fWwvXuv3Wd0atspGxQZwKhQaxQB63phtfCbqSthKp X-Google-Smtp-Source: AGHT+IGEUQVQieMZN8AVV7+2m1G3PtYiQy4+LUGMtdD8LhNzxcL5WvHctGy4zTZhNDB8ybllxvb/VQ== X-Received: by 2002:a17:907:1c24:b0:ad4:d0ba:ef93 with SMTP id a640c23a62f3a-ad52d42bdbbmr1093197266b.11.1747661609821; Mon, 19 May 2025 06:33:29 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:1c0a:f3ac:4087:51c8? ([2620:10d:c092:500::6:19cb]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ad52d04b068sm587456766b.22.2025.05.19.06.33.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 19 May 2025 06:33:28 -0700 (PDT) Message-ID: Date: Mon, 19 May 2025 14:33:28 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Memory allocation profiling warnings in memory bound systems From: Usama Arif To: Suren Baghdasaryan Cc: Johannes Weiner , Shakeel Butt , Linux Memory Management List , kent.overstreet@linux.dev, vlad.wing@gmail.com References: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> Content-Language: en-US In-Reply-To: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam10 X-Stat-Signature: u5ht4h5i3h8zffhmqqkn4paz9ian8knf X-Rspamd-Queue-Id: B5AF540009 X-Rspam-User: X-HE-Tag: 1747661611-285587 X-HE-Meta: U2FsdGVkX18gUdKseJb7b7vZb//6tohqBkv2a95kkrCG3X95C+HB1r0aNBuHbxw3E80HMZmGaTYUWCwm4KebjvOv7Ig2CeO4H1i9rNLC94tPUe5jc9dzivaINRzzDvxPcxjppUDBa02hKhAhkmnYfGr+wZES2po7NRni6puD6T8Ffl5akXqcwnGSs9tA3t/PxJMRDTD9dqJSZlS5CHGtVehVR0WKufIjD9vC8ydhztzXJchbGySwTCNuCKr/94vM9nRTTM8ebEzijq88md0nt5ckS3JYgZ7qzjZXmBFa4pIQf0L6blPIHOO9/KInOBW5mxxuRbqWTzn/X1Y0tQwHAoZL19fNFe6Rutbyg2ewlIg22pDQASYWLi7DMvJ7o9Yny5vSkpMQ0luX/rB2vH1jz44ieFgROU3yoOfiQm6CF78kVviSTPcfgMS6dbEtKin0uwZycuzTJqbMMiQaycUBkfksw0vXxPhD+9o07VK5owxpi+aS5G8T8uTXynXU3O3q0ZA4lL0vnCDwItpmqpULBCEzidJqMD+kwr2juWODf+GdxAAvyDRcwvp+fMe7u9Ok+yNywDZ54J+buv7d6vCmCkRhxoBiOxDzesGaKZfkrHNBi/tYCRvTTB81UElZ9DAEvez+QrDJZ4I5aZ3iCzeuNlDr5BimiCBdBDBHz0fAd2QdlpLmWdLOyV0cHAWvkTvJC5zq19MGYAERIa9b9a+nG1iUAzN2qcNy65hOM8KHYUvZAOVsd2tNFmlG7Li8/m+VhaYcqhNhHqxkt4eELXX1qAd91Gr5HejZ52MkbtwyZZhfwnjtKkjZo1XGHV1mDwspdJ5kMVIxOPpKY6jCGOAWfQKoTgCbG4Zx7I12q0W1LqsVyx7ZxwX6Xc/oV/VCSawdOHKKDIlLTWFWl2B3VF4nADC+o1jMZYPpArt1gHWwQ9Ng6AZC2RDuv6ViJ1KoyxepEPju3pq5+REBsRaP8pD +QCGaqC+ fOKKymIt36qBETezhcSHjo8uVu8MY/gWnLMiX6JWhPHc//jy2IwF/8a37rMdFjnZC/jv2UgiWujq1S97PEjCh7YBsF3YdxgVmJgfb5hxek8IRiNGdq+dWYCdhO8vrI6bQshKccoMZXtI/SoYNbtigYXGh8EdX9tyXMSABsU3yqRfog6P4n/+atvYbwL6y+iv5qTKwE9N1jlKFZ0mp5VZutme9mulbn5HBdCebBzbm7tHMxWxZDVtCfgr6QoAlf9yzB6JBCORuaav8EU4Up4TYxcWQMM8VX0eXVXGPF3CH1LB7VNMBwxwD0Q8sUUKhmzuljSj0nWquhjjCTQURH4fpuc3Bmq3ger/Mo8hrskP7OTQ9rZOjIXrr1PjDbJP5FBJ57MELTQAmftIiBT9sGR8iXxORrt+8OD413e/fPPZCtHDX2XRHq7mQYkO45Irwpp5uQ6gD1Klyh88O2X7q4aBIVahfQ5y8EpVibeWEuiQC9zyGOfho16iCjhPD0wf3Zj/4m+Do1Lu5UzkmgIH3Dxv9pmfCds6NoKnEem0p X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: +cc Vlad On 19/05/2025 14:31, Usama Arif wrote: > Hi, > > We have started enabling memory allocation profiling (with kernel 6.13) in our fleet > and are seeing a large number of warnings (reported by Vlad Poenaru) due to failure > in allocation of slab object extensions on services that are memory bound. I have attached > one of the logs at the end. > > Does it make sense to change the slabobj_ext to be allocated via kvcalloc and also change > the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A large number of > prints for this in a short time may mask any real issues in the system during memory > pressure being reported in dmesg. I tried to see if there were any changes after 6.13 > to this code but didn't find any, but thought will check before sending below as a patch. > > diff --git a/mm/slub.c b/mm/slub.c > index c2151c9fee22..4595ca190cd9 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, > gfp &= ~OBJCGS_CLEAR_MASK; > /* Prevent recursive extension vector allocation */ > gfp |= __GFP_NO_OBJ_EXT; > - vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, > + vec = kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp, > slab_nid(slab)); > if (!vec) { > /* Mark vectors which failed to allocate */ > @@ -2069,7 +2069,7 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) > > slab = virt_to_slab(p); > if (!slab_obj_exts(slab) && > - WARN(alloc_slab_obj_exts(slab, s, flags, false), > + WARN_ONCE(alloc_slab_obj_exts(slab, s, flags, false), > "%s, %s: Failed to create slab extension vector!\n", > __func__, s->name)) > return NULL; > > > > > [ 1824.754108] prepare_slab_obj_exts_hook, zs_handle-zswap1: Failed to create slab extension vector! > [ 1824.771857] WARNING: CPU: 17 PID: 118473 at mm/slub.c:2074 kmem_cache_alloc_noprof+0x780/0x1620 > ... > [ 1824.967011] RIP: 0010:kmem_cache_alloc_noprof+0x780/0x1620 > [ 1824.978004] Code: 48 8b 14 24 44 8b 4c 24 08 4c 8b 44 24 18 e9 fd f8 ff ff 49 8b 56 60 48 c7 c7 cc 76 5c 82 48 c7 c6 46 f6 6e 82 e8 10 14 63 ff <0f> 0b 44 8b 4c 24 08 e9 50 f9 ff ff f0 49 0f ba 2c 24 00 0f 82 cd > [ 1825.015516] RSP: 0000:ffffc9004a6cb228 EFLAGS: 00010286 > [ 1825.025967] RAX: 0000000000000055 RBX: 0000000000000000 RCX: 0000000000000000 > [ 1825.040248] RDX: ffff889fff6b0158 RSI: ffff889fff6a1c60 RDI: ffff889fff6a1c60 > [ 1825.054534] RBP: ffffea009a9e6dc0 R08: ffffffff83268ec0 R09: 000000000002fffd > [ 1825.068817] R10: 0000000000000000 R11: ffffc9003de2f9d0 R12: ffffffff7fff0000 > [ 1825.083086] R13: 0000777f80000000 R14: ffff8881caa20700 R15: ffff88a6a79b7000 > [ 1825.097371] FS: 00007f545a9447c0(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000 > [ 1825.113572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 1825.125063] CR2: 00007f4e5c3d0020 CR3: 0000001a2d9ad001 CR4: 00000000007726f0 > [ 1825.139328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 1825.153596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 1825.167858] PKRU: 55555554 > [ 1825.173281] Call Trace: > [ 1825.178193] > [ 1825.182402] ? __warn+0xa4/0x140 > [ 1825.188882] ? kmem_cache_alloc_noprof+0x780/0x1620 > [ 1825.198647] ? report_bug+0xe1/0x140 > [ 1825.205821] ? kmem_cache_alloc_noprof+0x780/0x1620 > [ 1825.215594] ? handle_bug+0x5e/0x90 > [ 1825.222579] ? exc_invalid_op+0x16/0x40 > [ 1825.230257] ? asm_exc_invalid_op+0x16/0x20 > [ 1825.238633] ? kmem_cache_alloc_noprof+0x780/0x1620 > [ 1825.248410] ? kmem_cache_alloc_noprof+0x780/0x1620 > [ 1825.258212] ? zs_malloc+0x958/0x9d0 > [ 1825.265393] ? ZSTD_compress2+0x73/0xb0 > [ 1825.273099] zs_malloc+0x958/0x9d0 > [ 1825.279930] ? zstd_scompress+0x4c/0x70 > [ 1825.287625] ? scomp_acomp_comp_decomp.llvm.7924360292990082857+0x128/0x1d0 > [ 1825.301568] zs_zpool_malloc+0xe/0x30 > [ 1825.308915] zswap_store+0x4b7/0x8e0 > [ 1825.316094] ? __lruvec_stat_mod_folio+0x11a/0x240 > [ 1825.325724] swap_writepage+0x12f/0x370 > [ 1825.333426] shrink_folio_list+0x809/0x13a0 > [ 1825.341837] ? 0xffffffff81000000 > [ 1825.348511] ? isolate_lru_folios+0x242/0x4d0 > [ 1825.357263] ? sysvec_call_function+0xa/0x80 > [ 1825.365840] ? asm_sysvec_call_function+0x16/0x20 > [ 1825.375285] shrink_lruvec+0x50c/0xb50 > [ 1825.382823] shrink_node+0x38e/0x8e0 > [ 1825.390000] do_try_to_free_pages+0xc4/0x530 > [ 1825.398560] try_to_free_pages+0x191/0x460 > [ 1825.406805] __alloc_pages_noprof+0x2997/0x4f60 > [ 1825.415906] vma_alloc_folio_noprof+0x132/0x530 > [ 1825.425017] folio_prealloc+0xc8/0xf0 > [ 1825.432383] do_pte_missing+0x605/0x1020 > [ 1825.440268] handle_mm_fault+0x3f9/0x1190 > [ 1825.448333] ? task_tick_fair.llvm.18033716799305157738+0x43/0x190 > [ 1825.460729] do_user_addr_fault+0x196/0x5c0 > [ 1825.469150] exc_page_fault+0x69/0x130 > [ 1825.476687] asm_exc_page_fault+0x22/0x30 > [ 1825.484737] RIP: 0033:0x346333 > [ 1825.490879] Code: 5e 41 5f 5d c3 48 8b 41 08 48 8b 10 48 89 51 08 eb e4 49 8b 4e 08 4c 89 f6 48 2b 35 cf 8a 29 00 48 c1 ee 04 69 f6 ab aa aa aa <89> 71 20 c7 41 24 ff ff 00 00 48 8d b1 00 40 00 00 49 89 76 08 e9 > [ 1825.528413] RSP: 002b:00007ffcabaf4b70 EFLAGS: 00010a07 > [ 1825.538891] RAX: 000000000000000c RBX: 0000000000000018 RCX: 00007f4e5c3d0000 > [ 1825.553175] RDX: 000052f00026ea40 RSI: 0000000000000221 RDI: 0000000000000018 > [ 1825.567459] RBP: 00007ffcabaf4bc0 R08: 0000512000000040 R09: 0000000000000000 > [ 1825.581746] R10: 00007f5428b0c6a0 R11: 0000000000000002 R12: 0000534000000800 > [ 1825.596030] R13: 0000000000000190 R14: 000052f00026ea30 R15: 0000512000000040 > [ 1825.610328] > > > Thanks, > Usama