From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4719EC2D0CD for ; Mon, 19 May 2025 13:31:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4180A6B0095; Mon, 19 May 2025 09:31:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3CB616B00C6; Mon, 19 May 2025 09:31:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2689D6B00D3; Mon, 19 May 2025 09:31:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 02F7A6B0095 for ; Mon, 19 May 2025 09:31:04 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1CBFFE2564 for ; Mon, 19 May 2025 13:31:06 +0000 (UTC) X-FDA: 83459743332.25.6A8FDD2 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf01.hostedemail.com (Postfix) with ESMTP id 27C634000A for ; Mon, 19 May 2025 13:31:03 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nZKN0Yjn; spf=pass (imf01.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747661464; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Bo1pg6w58jtBqrFob5z57I9akpg3uCxPpoHY6OtFhHI=; b=TPdYCB/tT1uOFbHs7V1rywwH3CrhlygGkFl3FHAZLui6+yUhPCVtBi9kLcnbOG5VJk38XM DtYXnCQhMzZL7NKbWQO/abbpapTqj3IFW7zBAu6aznGO5Fe2PEeyRSdFx1PXbkriBb1oNi KboaUMw3S5hKXrR1SHmn8sCmRIqOg+M= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=nZKN0Yjn; spf=pass (imf01.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747661464; a=rsa-sha256; cv=none; b=5LE6hkQiB8TimVAYDeu8EYhNOJyYjJQtiffTGju1dFtEw1ACblFQItyC95PofgMiFyyP3I ighHrTdVoK7v9YP5prOiDk6o4D70h5P6giyouy2UlLuzc9X7NJNx8vJDzJB9MOfGpprOY+ buMk8d88kAwmsOpR/aExXRy8oHy4BjM= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-601afe51106so2240975a12.1 for ; Mon, 19 May 2025 06:31:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747661462; x=1748266262; darn=kvack.org; h=content-transfer-encoding:subject:from:cc:to:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=Bo1pg6w58jtBqrFob5z57I9akpg3uCxPpoHY6OtFhHI=; b=nZKN0Yjn6blbM8AVQDAIplaes3WvqzsBQFaV718gTBVQ5K1wvbxpO+5RGmhkHIZIS5 8448WlnnYq7Rg42myl5AqA4n+CBNtu41Ch8+38pJ2P1YNmLc92x6yzOU+38la0XeqdRB b4Wmz6E05/QXpAwEIGkY9YCUbPVS2PIpt6Lkb4K1+cdWI1hyAFeI6q9YLyA6LmZrO6vw Ogi37U6+y3Xjc58sYtYs2sLLweaSOivM3eSDWjhWXuc/gfHV29WvW410rUhAd5rAYTsM CyScFTfWmDartWVb7EHj5XEAHUtWrhuPDrUmtajPZ3WKzjisIc2Sv2v/BtnEl8kNaPpq mFuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747661462; x=1748266262; h=content-transfer-encoding:subject:from:cc:to:content-language :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Bo1pg6w58jtBqrFob5z57I9akpg3uCxPpoHY6OtFhHI=; b=WYwgMX0cMLyrXuIavPr4Li4op1kCsBNeA8bGx/I0TP2GT7gihECMNrSdgI63kMjELV duOqHQk5gCGn3KGIqDn6WW/ecG3KNzIMuovtXia1H5cVsAk9nUhxa3C+eJLFxjnCVIAM OOh5Mw2M4Jl7uWU+lVyO/dLUNw2ufyoqsnY+zhyufTGcxy7ZDH2ZqDpWNuK3Y3xx3o4b M7n4tgDu1L6dt+LdWgxr52RSFooNX4abynXuKj/KxeeRbFECcfj5NAati8/dkCUFmhcb L0jcJG97H8l8S872MhNpd+5Cf6s8tCGWOSKnZAuj+BJlJcnFIc/FTFJssygBt7QzIjBY SUoA== X-Forwarded-Encrypted: i=1; AJvYcCW002MhKMID5C1fauj2UOY3M2Td9A7nBxp1oo2QrCCGr3a8nHgddvU9DZljMzjuQe3BdiGf9qI0Aw==@kvack.org X-Gm-Message-State: AOJu0YyDvOdDnwdvWqnj2R+uzrMQGiFg+27+3fzjII4461Jim7iT+6e5 jEIx49Wr9gyfwvwngZgwMm9wkhkx5KiCM4YOmSsxVgvw1WjD7GwSjNhv X-Gm-Gg: ASbGncujD6yCY/5yYNmmjOsynh9B2nRGWIpuI2l7ETsOF1lZn+YlZErX1Fy/8j8ohq7 Hmlzi1QyBxv30/d0eWWKTbUrlBZ0B6qIAK0+6dwwGHqfWSrBzycH24RMAXKOXoEsTdMznODmpsZ tlOW4JffI6YN2+ZHZSXOBUlFQszYUlwc4Q8H/4n15Z3RDe/VuSmZwtiQ0lfSqW+orRJa67oS+yH O/OgnxbZpqGPisqn9m5eS3v4PWlWDsmHKqpltcx1BO2g19XTxlqa5Yd9ZHC5bQzZ0FHN5zLN2ZK SQ5hPZ0bpwrjh2cb3E0o5JEGw2lDxQoXH48vvL+q7KbryMLENgjSM5aBzqgYw13RD/HRRXBWZ59 fXZcN24LP5UkoNPkQTcuLKRvkNDiFwA== X-Google-Smtp-Source: AGHT+IHWS3o3NoQ01ocwrnQloeUNl0gY1qAiKoTFh+6MDLGi8PalyzimeR1LK8j+0c2XylihenanmQ== X-Received: by 2002:a05:6402:1e8f:b0:5f7:29e0:5cf9 with SMTP id 4fb4d7f45d1cf-6008a392cc1mr10595136a12.5.1747661462240; Mon, 19 May 2025 06:31:02 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:1c0a:f3ac:4087:51c8? ([2620:10d:c092:500::6:19cb]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-6004d4f2ce3sm5747185a12.4.2025.05.19.06.31.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 19 May 2025 06:31:01 -0700 (PDT) Message-ID: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> Date: Mon, 19 May 2025 14:31:01 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: Suren Baghdasaryan Cc: Johannes Weiner , Shakeel Butt , Linux Memory Management List , kent.overstreet@linux.dev From: Usama Arif Subject: Memory allocation profiling warnings in memory bound systems Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 27C634000A X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ok94x61j6na5ic47y4m8gckqp74r936d X-HE-Tag: 1747661463-991251 X-HE-Meta: U2FsdGVkX1/YRL31Hr+vBtf9XchPtO0HOQBaZ/PFTu2HlMuw6grYeYTaUkEmu4ZMnpXYTZ4hNYnjGilUxWZfhnHv5/lxmwYYgr9aTk95bORk0I3um3QvoRErbpuE9Id9AVAnyajfZX+aNJninFLdl6u3ApDpsfVhmkj01tVRALzwQrmh0wVFHzJw9AOE1W9ggRupRwbZlf7hvvpVaRuXU2VfXzS2Bb8XKGCx3UP+SxYDCiBUIAVBl7pwQ9lQCVA7umx+LNQdIBUaX1cc/W16NVRwyRSmHuia1caMwq5b8bhYuRPPyrKYnDmKbWN/h2ze93H50dCnptxNC0QGgem7g6GmWHlUm4iUWy/YVgffhC4VnLTwQZAA+GSwwyTmvUqi7eOG/uhynIc268BRtVb+r+dxxiQ2m+1DK4Seg0ZggatDAy7urpPiE11HAh2IM3u/hzuPOwGbVL2Jvy+2kI9xN5p5+IyAvaBeAHtsw9pgYLpSD0u5LDg/a+iEihzawi5A+mAg2C866kY0r1f5jB9w1kV6KnbptL0/JGUuoVPU11z0hFEUllf3mZT8wYgNJ7T7h/P/WDCXUjhGi/TgJx3KI93QN+p+u4BXjjhdxiB5IlQJBzlgAlomRc6khy4bOnH5U7RYjGO540qZrJMJRDC0ZIsayRvQgGN+tMS2lwWdrUEwb2BaYgIFwJzD6iiY1Uuc+X1vr5rP9aP453lhJ6lf8YMovD0+ffiN0Y4pd7aqDNuZ3Q+AY4tFnXrP8Rl+cz+PsTvjSxJaWabgO+tgQ79bVsrlHowewWE3z9K+F69KPFu/Uuf3rhZPRcGclL1+oFylvhmBx4so8ZRU1MIBlYwIzxZhliBG8ESMA72vwt14t36kga27bO2qDQyttgFBajlgJHxZvU4WJuTct6aXHAn0XRAyBjF0WkP5w611uTr71psz9kHi9Jp1+OXNhLl2/+kyMa7nAOeKaCa3rnKFHX1 TJ5qKIm0 Xgnc9+i/Jvrim9/Eea2OIATe8Oc7P/6HayCiRekaywoViY93djx989EQp54r2WQtWCy5WMo3FTjjDAxwf3BHkMIYXsDIVK9g3ezsNBVBb9pBXQGnwSEshJvzbVv7Rr/Wgwo1qV24nIe8Q312g2r2pKi4By6xZBBQokPHw8BpewmLSYjoJrxfXh82wJKOZzeZZvWXy/gBbAnwzRJV58Gf2+QNKrGexuhE1dsOEvVVJ82qEYhyPMUxbgMTuQlSeir0wKBT+hUp6tPxOn8vlfY+Qhp1pnPRdfFbm/xGE5FiLTK90ZqK0c2CV1i09PASz0gJdyD1e3ifHWBVZy2HpSy/nxWfJ1oqEdEbMrCmQ/AUHEEBTaU0SxINWcqQWPEnr9xy5FImAXneeyeRSvYYWkN4TvkBE5FLimMTcZBUKk3xvCpYSKE79Q7V5ak9zzNyWgaU384oQWtxpvD1VMvlr7AvnPFrD4Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi, We have started enabling memory allocation profiling (with kernel 6.13) in our fleet and are seeing a large number of warnings (reported by Vlad Poenaru) due to failure in allocation of slab object extensions on services that are memory bound. I have attached one of the logs at the end. Does it make sense to change the slabobj_ext to be allocated via kvcalloc and also change the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A large number of prints for this in a short time may mask any real issues in the system during memory pressure being reported in dmesg. I tried to see if there were any changes after 6.13 to this code but didn't find any, but thought will check before sending below as a patch. diff --git a/mm/slub.c b/mm/slub.c index c2151c9fee22..4595ca190cd9 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, gfp &= ~OBJCGS_CLEAR_MASK; /* Prevent recursive extension vector allocation */ gfp |= __GFP_NO_OBJ_EXT; - vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, + vec = kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp, slab_nid(slab)); if (!vec) { /* Mark vectors which failed to allocate */ @@ -2069,7 +2069,7 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, gfp_t flags, void *p) slab = virt_to_slab(p); if (!slab_obj_exts(slab) && - WARN(alloc_slab_obj_exts(slab, s, flags, false), + WARN_ONCE(alloc_slab_obj_exts(slab, s, flags, false), "%s, %s: Failed to create slab extension vector!\n", __func__, s->name)) return NULL; [ 1824.754108] prepare_slab_obj_exts_hook, zs_handle-zswap1: Failed to create slab extension vector! [ 1824.771857] WARNING: CPU: 17 PID: 118473 at mm/slub.c:2074 kmem_cache_alloc_noprof+0x780/0x1620 ... [ 1824.967011] RIP: 0010:kmem_cache_alloc_noprof+0x780/0x1620 [ 1824.978004] Code: 48 8b 14 24 44 8b 4c 24 08 4c 8b 44 24 18 e9 fd f8 ff ff 49 8b 56 60 48 c7 c7 cc 76 5c 82 48 c7 c6 46 f6 6e 82 e8 10 14 63 ff <0f> 0b 44 8b 4c 24 08 e9 50 f9 ff ff f0 49 0f ba 2c 24 00 0f 82 cd [ 1825.015516] RSP: 0000:ffffc9004a6cb228 EFLAGS: 00010286 [ 1825.025967] RAX: 0000000000000055 RBX: 0000000000000000 RCX: 0000000000000000 [ 1825.040248] RDX: ffff889fff6b0158 RSI: ffff889fff6a1c60 RDI: ffff889fff6a1c60 [ 1825.054534] RBP: ffffea009a9e6dc0 R08: ffffffff83268ec0 R09: 000000000002fffd [ 1825.068817] R10: 0000000000000000 R11: ffffc9003de2f9d0 R12: ffffffff7fff0000 [ 1825.083086] R13: 0000777f80000000 R14: ffff8881caa20700 R15: ffff88a6a79b7000 [ 1825.097371] FS: 00007f545a9447c0(0000) GS:ffff889fff680000(0000) knlGS:0000000000000000 [ 1825.113572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1825.125063] CR2: 00007f4e5c3d0020 CR3: 0000001a2d9ad001 CR4: 00000000007726f0 [ 1825.139328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1825.153596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1825.167858] PKRU: 55555554 [ 1825.173281] Call Trace: [ 1825.178193] [ 1825.182402] ? __warn+0xa4/0x140 [ 1825.188882] ? kmem_cache_alloc_noprof+0x780/0x1620 [ 1825.198647] ? report_bug+0xe1/0x140 [ 1825.205821] ? kmem_cache_alloc_noprof+0x780/0x1620 [ 1825.215594] ? handle_bug+0x5e/0x90 [ 1825.222579] ? exc_invalid_op+0x16/0x40 [ 1825.230257] ? asm_exc_invalid_op+0x16/0x20 [ 1825.238633] ? kmem_cache_alloc_noprof+0x780/0x1620 [ 1825.248410] ? kmem_cache_alloc_noprof+0x780/0x1620 [ 1825.258212] ? zs_malloc+0x958/0x9d0 [ 1825.265393] ? ZSTD_compress2+0x73/0xb0 [ 1825.273099] zs_malloc+0x958/0x9d0 [ 1825.279930] ? zstd_scompress+0x4c/0x70 [ 1825.287625] ? scomp_acomp_comp_decomp.llvm.7924360292990082857+0x128/0x1d0 [ 1825.301568] zs_zpool_malloc+0xe/0x30 [ 1825.308915] zswap_store+0x4b7/0x8e0 [ 1825.316094] ? __lruvec_stat_mod_folio+0x11a/0x240 [ 1825.325724] swap_writepage+0x12f/0x370 [ 1825.333426] shrink_folio_list+0x809/0x13a0 [ 1825.341837] ? 0xffffffff81000000 [ 1825.348511] ? isolate_lru_folios+0x242/0x4d0 [ 1825.357263] ? sysvec_call_function+0xa/0x80 [ 1825.365840] ? asm_sysvec_call_function+0x16/0x20 [ 1825.375285] shrink_lruvec+0x50c/0xb50 [ 1825.382823] shrink_node+0x38e/0x8e0 [ 1825.390000] do_try_to_free_pages+0xc4/0x530 [ 1825.398560] try_to_free_pages+0x191/0x460 [ 1825.406805] __alloc_pages_noprof+0x2997/0x4f60 [ 1825.415906] vma_alloc_folio_noprof+0x132/0x530 [ 1825.425017] folio_prealloc+0xc8/0xf0 [ 1825.432383] do_pte_missing+0x605/0x1020 [ 1825.440268] handle_mm_fault+0x3f9/0x1190 [ 1825.448333] ? task_tick_fair.llvm.18033716799305157738+0x43/0x190 [ 1825.460729] do_user_addr_fault+0x196/0x5c0 [ 1825.469150] exc_page_fault+0x69/0x130 [ 1825.476687] asm_exc_page_fault+0x22/0x30 [ 1825.484737] RIP: 0033:0x346333 [ 1825.490879] Code: 5e 41 5f 5d c3 48 8b 41 08 48 8b 10 48 89 51 08 eb e4 49 8b 4e 08 4c 89 f6 48 2b 35 cf 8a 29 00 48 c1 ee 04 69 f6 ab aa aa aa <89> 71 20 c7 41 24 ff ff 00 00 48 8d b1 00 40 00 00 49 89 76 08 e9 [ 1825.528413] RSP: 002b:00007ffcabaf4b70 EFLAGS: 00010a07 [ 1825.538891] RAX: 000000000000000c RBX: 0000000000000018 RCX: 00007f4e5c3d0000 [ 1825.553175] RDX: 000052f00026ea40 RSI: 0000000000000221 RDI: 0000000000000018 [ 1825.567459] RBP: 00007ffcabaf4bc0 R08: 0000512000000040 R09: 0000000000000000 [ 1825.581746] R10: 00007f5428b0c6a0 R11: 0000000000000002 R12: 0000534000000800 [ 1825.596030] R13: 0000000000000190 R14: 000052f00026ea30 R15: 0000512000000040 [ 1825.610328] Thanks, Usama