From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EF4BC3ABDD for ; Mon, 19 May 2025 15:50:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C9CA56B00D3; Mon, 19 May 2025 11:50:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4A8C6B00D4; Mon, 19 May 2025 11:50:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AECF96B00D5; Mon, 19 May 2025 11:50:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8DBD86B00D3 for ; Mon, 19 May 2025 11:50:42 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BB13712047B for ; Mon, 19 May 2025 15:50:43 +0000 (UTC) X-FDA: 83460095166.07.BA177B6 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf30.hostedemail.com (Postfix) with ESMTP id DA98C80006 for ; Mon, 19 May 2025 15:50:41 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bVJ45Fk+; spf=pass (imf30.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747669841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OhEDe/Q7m4mjolkEBnThDIdcvVSA1XBoos0o4+WU0d4=; b=ozNtwzr6gR17jisdm145vYWbQ+jKKJqaUjMVUlue3cdpfKt0JISe6VGKcZlH8n/lSsh9WX E4Lib+SN/TdjKC8tuhC9LdNj1Pj8Ur7Qol9pnQys/aPZtbN2TGksxaxqOro0Oc7CgcenER ZoXTOOTb+KrygbEjz4RDUaWj1xHqtzc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=bVJ45Fk+; spf=pass (imf30.hostedemail.com: domain of surenb@google.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747669841; a=rsa-sha256; cv=none; b=7I0sbQLCPVyKprIlKsFNWfzrCm/271nne+68WS9VXbCu8RSD0xq8ksVLCi9VK3X72yeIV+ Wi9SJjuMm2Ab22xgeYwRVqiiakg1yT8Szsw0gUNT6ai1Rm5eOIYi0sMLUcPz+UEdeTtdU/ F6avDm6NUbYjtnrKnov6mgiWSFD6Ijo= Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-47e9fea29easo776151cf.1 for ; Mon, 19 May 2025 08:50:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1747669841; x=1748274641; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=OhEDe/Q7m4mjolkEBnThDIdcvVSA1XBoos0o4+WU0d4=; b=bVJ45Fk+qD97VG/OTg5i/v7P7yUqxQVtrvWVAzYYDpJ+8HSA4R1MSP0QTWNLipQEvq sJv21JRq3atPVgNB4XxV7R4QqhJuVKlBXGPuL+EMv7LIUvMXSy+Vn+AiZ5dk0asi2koj 1VapBUytvSbpLijvZ8KCacE9VVtC3VbMK3BBZrSFR3ZsL7tdf5jiz5xn0GNRlI56ey7e 15xy11DJFrv299k4gO07T/JC+NZw8dYM9Es/DAtVbvUMi+6Aq/p4hnNHDC6IUp2EfSgc NNZ+5gfh9hEaIAuyIPEJVf6sVIde4zkLNyN9dEAlJTeuRzLrZ4dTd5qoVGD1CPIs0yxo wPEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747669841; x=1748274641; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OhEDe/Q7m4mjolkEBnThDIdcvVSA1XBoos0o4+WU0d4=; b=I1EnXTMfpCjeOW8ryyD4xR8CPjaj4LtZKid0sFrIsk3a+ZKYyd0NBFSOV30h+xOvVU 40ZXbiC17G8WoPbMTWdfVJfVf85Rwzq+qc4OBGheTBEKxuP0Emtl2Tl3YDSjwxLi3YUu Gg737EVrJzzKN1wqD+h/sBJcZo76nYOWq2mVeAAN70Yh0qVpZieo0WJ/5H1TzdMPkWs3 8Ji21rOuNSwBfCQFfSkJog38dB89Ipn7CXAo9myGUlQ2mPofXw0vAETm7kfXK26Qu6GS 14HCOvux7cl30ne7geEvFvMKl7CCoLwqYM3zb3VmF9qgVDwcwMkj9WoPGPZeCHwRt7IN r5pQ== X-Forwarded-Encrypted: i=1; AJvYcCWTJiAmX2X7GwPi5pivBuDtZ3iNgdB9EK5rjmGzGIeI2tFCgoz0KzXIX43DElSghRWJsrhSFNUVuw==@kvack.org X-Gm-Message-State: AOJu0Yw77wpUCDOWJ6AT2XlvXon5AHp+Uabab/cbQf7qizz+tZ2H50Xr WLeTsFnZKLP5iBZGYUxzPhYQmXcNyPSVGLuhAcbBu0BplqGlhv7SGYG5q7ja1W1qQSYKLoK6gJk NpE0OZpcb8c1qDe9KPfMPrHAyqOHKeiiCvR4mX+1e X-Gm-Gg: ASbGncti3uuL7P+TL1b9fwPUCcFqIE6uJ48yl+HwaJwFvvLVqmO262WVS1EbszDVBcz l4IXg/KhGXATMAnpOV6Kyo6ixz7/rOlbmgKaYRSHsmuIFs3BI/SBrd0HpDKasJWWNcqCYGLDyJS qVfRATTXhrhOzjaSITvXsxCOK/2yxX6lg= X-Google-Smtp-Source: AGHT+IHZq5Jiu7Ql3rrDUw/lfwVHNONwE60NPEGzceDIy4DJjL2P4ZnoVgPsBxkQwkNxeTko3k/ltEdD6SQE6Rx5QZk= X-Received: by 2002:a05:622a:1826:b0:494:af4b:59fd with SMTP id d75a77b69052e-49595d52c6emr7535291cf.18.1747669840664; Mon, 19 May 2025 08:50:40 -0700 (PDT) MIME-Version: 1.0 References: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> In-Reply-To: From: Suren Baghdasaryan Date: Mon, 19 May 2025 08:50:28 -0700 X-Gm-Features: AX0GCFsJD1NdIwgo-Xq1YzmgrXZBh03Mn0Y5I3DFV9ejlGPBBjaojgBdMGx-jtc Message-ID: Subject: Re: Memory allocation profiling warnings in memory bound systems To: Usama Arif Cc: Johannes Weiner , Shakeel Butt , Linux Memory Management List , kent.overstreet@linux.dev, vlad.wing@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: DA98C80006 X-Rspamd-Server: rspam09 X-Stat-Signature: ic8neuhqs1ckaeaoe6hurcc85sdgjbfb X-HE-Tag: 1747669841-166159 X-HE-Meta: U2FsdGVkX18Csqo+Y0L8rKyimPNleWzng2JLZ5PegcWjtiH+VjIlnmmqe7sN9gcfbo4a24G8AQ/hFqdMKPKrDmteabvoPKs1gOV+Jzh1usI9zvafKoku2M8vabiKPiA9EtdrqCd3XKHBBIUFUVozINIgjMy40jyzK3DZ4dyEPdJAukDheZkfwRUVb/ZwPhmCdzlLHC3OShQCAAgIUyc2rMKrcn3X0PIZi9j3nH6EGAu8PaN5Ijt046bSy89sHd0gecClpbZJXXjnmMerusUAwE6P6KnO3xz9qVboW3TkTcmq56a8EkaM0DXP+KB9LUKvpQZMnhh4Hn7cqBDQNQ22jdRw01eXUg9GsuulT8i3VZ+EkrGEnBkbfQcERLOVKLdzYWpRSs4KYbcaTbvhalWmRZsH0Dq2Q1IRTl6QQLEeP5Tz3LVJgjd7CVq2GCxPScqadqOclnY/I00fkf0DRLRcKPDZd921ZZAxaw7yeLQ9sAHmrphk3AcRgF2ulEPH4u4bWE8TBfSDvgHNSuEpRWjSv9Yd8gPVuS1cnaSPjEy5yhCSXHFFiXdtakNUhiNEKExmNsJVwYS2ZjXLSzFuRd2cEPdy3OVq3Sr7su185fGETTEpPZL97jS/ipmZRL5T3Mj6jmEefwIM12H1+qc5MkLAWTatQUU54EvtE/Zey+xHSY3llz6hTSdNnFQZ9fChs5CZawUBZPvO4Aye6nyv7Cz8dNtdarEVjn+Ncxw4/jgN6Frh2xHoqxkTpBWrr5w4hfbqDRH3IOcDDuzHaKhZyTtGvDQ8f3MMSmVDfAd7B6C8RaI+HMG5I9YC2HA4vXahNxjkyJvpIknjpNWKBfcl4Lqt1BJCAtjtWi7X8G74/IWjDkro19pCozP1MDpdQu9TqwEstOmeEwvP0WnaSIOGsUwo9vLNoknak2n7+aJZ0wYWhHLz4xX5XXmBzSu6EvdBD4AE2Pw4XzHQKbqbRVCg7DN EprZVUz9 yOrUbpCxEF3xvGXh9O0/fAr8uy+m5LByVrd0ersA03FgLBCAJU6hNMDsxJLGZbgG1x0TgxHr5dwXrxAEcZhvKoCJbJqtx7KHdx4Zd2ga7324ZOENtnmiTuzhUjic+N6fwHwdY6RjHeKIg1ea6hm/DBAXZf4Y4ifPd72GeObk5ygtdBtoNrTErR6JTztqxxExs8bR135y8xEEzZlsekfXdRjciSPs1oQ7xkF09q1A2JjE2/si7syPk9cuxUI5F5xCa+GXCv0JPwQSToh/lmrYBvZUHg3rtZ/vI7O8ODpfSuzbd8MytFmtXmv8n7GTtzqP97Z5fxhUoIZER3mgAfntBzMcQpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, May 19, 2025 at 6:33=E2=80=AFAM Usama Arif = wrote: > > > +cc Vlad > > On 19/05/2025 14:31, Usama Arif wrote: > > Hi, > > > > We have started enabling memory allocation profiling (with kernel 6.13)= in our fleet > > and are seeing a large number of warnings (reported by Vlad Poenaru) du= e to failure > > in allocation of slab object extensions on services that are memory bou= nd. I have attached > > one of the logs at the end. > > > > Does it make sense to change the slabobj_ext to be allocated via kvcall= oc and also change > > the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A = large number of > > prints for this in a short time may mask any real issues in the system = during memory > > pressure being reported in dmesg. I tried to see if there were any chan= ges after 6.13 > > to this code but didn't find any, but thought will check before sending= below as a patch. > > > > diff --git a/mm/slub.c b/mm/slub.c > > index c2151c9fee22..4595ca190cd9 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct= kmem_cache *s, > > gfp &=3D ~OBJCGS_CLEAR_MASK; > > /* Prevent recursive extension vector allocation */ > > gfp |=3D __GFP_NO_OBJ_EXT; > > - vec =3D kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, > > + vec =3D kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp, Hi Usama, Is the allocation larger than page size? IIUC, unless allocation size is over PAGE_SIZE, kvcalloc_node() will not fall back to vmalloc (see: https://elixir.bootlin.com/linux/v6.14.7/source/mm/util.c#L668). How big is the allocation when it fails in your case? > > slab_nid(slab)); > > if (!vec) { > > /* Mark vectors which failed to allocate */ > > @@ -2069,7 +2069,7 @@ prepare_slab_obj_exts_hook(struct kmem_cache *s, = gfp_t flags, void *p) > > > > slab =3D virt_to_slab(p); > > if (!slab_obj_exts(slab) && > > - WARN(alloc_slab_obj_exts(slab, s, flags, false), > > + WARN_ONCE(alloc_slab_obj_exts(slab, s, flags, false), Makes sense if you see lots of these but I'm still wondering how big these failing allocations are. > > "%s, %s: Failed to create slab extension vector!\n", > > __func__, s->name)) > > return NULL; > > > > > > > > > > [ 1824.754108] prepare_slab_obj_exts_hook, zs_handle-zswap1: Failed to = create slab extension vector! > > [ 1824.771857] WARNING: CPU: 17 PID: 118473 at mm/slub.c:2074 kmem_cach= e_alloc_noprof+0x780/0x1620 > > ... > > [ 1824.967011] RIP: 0010:kmem_cache_alloc_noprof+0x780/0x1620 > > [ 1824.978004] Code: 48 8b 14 24 44 8b 4c 24 08 4c 8b 44 24 18 e9 fd f8= ff ff 49 8b 56 60 48 c7 c7 cc 76 5c 82 48 c7 c6 46 f6 6e 82 e8 10 14 63 ff= <0f> 0b 44 8b 4c 24 08 e9 50 f9 ff ff f0 49 0f ba 2c 24 00 0f 82 cd > > [ 1825.015516] RSP: 0000:ffffc9004a6cb228 EFLAGS: 00010286 > > [ 1825.025967] RAX: 0000000000000055 RBX: 0000000000000000 RCX: 0000000= 000000000 > > [ 1825.040248] RDX: ffff889fff6b0158 RSI: ffff889fff6a1c60 RDI: ffff889= fff6a1c60 > > [ 1825.054534] RBP: ffffea009a9e6dc0 R08: ffffffff83268ec0 R09: 0000000= 00002fffd > > [ 1825.068817] R10: 0000000000000000 R11: ffffc9003de2f9d0 R12: fffffff= f7fff0000 > > [ 1825.083086] R13: 0000777f80000000 R14: ffff8881caa20700 R15: ffff88a= 6a79b7000 > > [ 1825.097371] FS: 00007f545a9447c0(0000) GS:ffff889fff680000(0000) kn= lGS:0000000000000000 > > [ 1825.113572] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 1825.125063] CR2: 00007f4e5c3d0020 CR3: 0000001a2d9ad001 CR4: 0000000= 0007726f0 > > [ 1825.139328] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000= 000000000 > > [ 1825.153596] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000= 000000400 > > [ 1825.167858] PKRU: 55555554 > > [ 1825.173281] Call Trace: > > [ 1825.178193] > > [ 1825.182402] ? __warn+0xa4/0x140 > > [ 1825.188882] ? kmem_cache_alloc_noprof+0x780/0x1620 > > [ 1825.198647] ? report_bug+0xe1/0x140 > > [ 1825.205821] ? kmem_cache_alloc_noprof+0x780/0x1620 > > [ 1825.215594] ? handle_bug+0x5e/0x90 > > [ 1825.222579] ? exc_invalid_op+0x16/0x40 > > [ 1825.230257] ? asm_exc_invalid_op+0x16/0x20 > > [ 1825.238633] ? kmem_cache_alloc_noprof+0x780/0x1620 > > [ 1825.248410] ? kmem_cache_alloc_noprof+0x780/0x1620 > > [ 1825.258212] ? zs_malloc+0x958/0x9d0 > > [ 1825.265393] ? ZSTD_compress2+0x73/0xb0 > > [ 1825.273099] zs_malloc+0x958/0x9d0 > > [ 1825.279930] ? zstd_scompress+0x4c/0x70 > > [ 1825.287625] ? scomp_acomp_comp_decomp.llvm.7924360292990082857+0x12= 8/0x1d0 > > [ 1825.301568] zs_zpool_malloc+0xe/0x30 > > [ 1825.308915] zswap_store+0x4b7/0x8e0 > > [ 1825.316094] ? __lruvec_stat_mod_folio+0x11a/0x240 > > [ 1825.325724] swap_writepage+0x12f/0x370 > > [ 1825.333426] shrink_folio_list+0x809/0x13a0 > > [ 1825.341837] ? 0xffffffff81000000 > > [ 1825.348511] ? isolate_lru_folios+0x242/0x4d0 > > [ 1825.357263] ? sysvec_call_function+0xa/0x80 > > [ 1825.365840] ? asm_sysvec_call_function+0x16/0x20 > > [ 1825.375285] shrink_lruvec+0x50c/0xb50 > > [ 1825.382823] shrink_node+0x38e/0x8e0 > > [ 1825.390000] do_try_to_free_pages+0xc4/0x530 > > [ 1825.398560] try_to_free_pages+0x191/0x460 > > [ 1825.406805] __alloc_pages_noprof+0x2997/0x4f60 > > [ 1825.415906] vma_alloc_folio_noprof+0x132/0x530 > > [ 1825.425017] folio_prealloc+0xc8/0xf0 > > [ 1825.432383] do_pte_missing+0x605/0x1020 > > [ 1825.440268] handle_mm_fault+0x3f9/0x1190 > > [ 1825.448333] ? task_tick_fair.llvm.18033716799305157738+0x43/0x190 > > [ 1825.460729] do_user_addr_fault+0x196/0x5c0 > > [ 1825.469150] exc_page_fault+0x69/0x130 > > [ 1825.476687] asm_exc_page_fault+0x22/0x30 > > [ 1825.484737] RIP: 0033:0x346333 > > [ 1825.490879] Code: 5e 41 5f 5d c3 48 8b 41 08 48 8b 10 48 89 51 08 eb= e4 49 8b 4e 08 4c 89 f6 48 2b 35 cf 8a 29 00 48 c1 ee 04 69 f6 ab aa aa aa= <89> 71 20 c7 41 24 ff ff 00 00 48 8d b1 00 40 00 00 49 89 76 08 e9 > > [ 1825.528413] RSP: 002b:00007ffcabaf4b70 EFLAGS: 00010a07 > > [ 1825.538891] RAX: 000000000000000c RBX: 0000000000000018 RCX: 00007f4= e5c3d0000 > > [ 1825.553175] RDX: 000052f00026ea40 RSI: 0000000000000221 RDI: 0000000= 000000018 > > [ 1825.567459] RBP: 00007ffcabaf4bc0 R08: 0000512000000040 R09: 0000000= 000000000 > > [ 1825.581746] R10: 00007f5428b0c6a0 R11: 0000000000000002 R12: 0000534= 000000800 > > [ 1825.596030] R13: 0000000000000190 R14: 000052f00026ea30 R15: 0000512= 000000040 > > [ 1825.610328] > > > > > > Thanks, > > Usama >