From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E727DC2D0CD for ; Mon, 19 May 2025 17:24:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 413656B00D1; Mon, 19 May 2025 13:24:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C39D6B00D2; Mon, 19 May 2025 13:24:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23E186B00D3; Mon, 19 May 2025 13:24:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EEF2B6B00D1 for ; Mon, 19 May 2025 13:24:01 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C68DF80B98 for ; Mon, 19 May 2025 17:24:04 +0000 (UTC) X-FDA: 83460330408.01.3D831C9 Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by imf26.hostedemail.com (Postfix) with ESMTP id C480C140009 for ; Mon, 19 May 2025 17:24:02 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RvaMm0Bb; spf=pass (imf26.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.208.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747675442; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3rsv9jkm5eLmngTLYRbXBLDmS8lLeNb225jEXQuDNAk=; b=2mX4b5Z3kL2Bq75OGcs+yhYWkIYkoQ2vKO5+/fTbt1hSY4ir7GYLJdzFycM+OiVFdm3xCk Ajese3OseWqCp50z4Wu7F3AXjtxV5CQBPGjG0T70DBC1rbdD+ghc1xXTCK8311Ei7T7BE3 Ww21sNVVz/cpJKyblVnGIp3gBZbq3fU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747675442; a=rsa-sha256; cv=none; b=IpA6HkOERVXvOxi1k/wzsVmk+8GD5fb211wEJxAfh6GdaNWXpAL8/YulRlEciqTuxr88Pe DmXVIiE2rqdXvMXNZCCflZTHIZ1tIAX9lOx/xE6X2dhcgxMYolJXJd4epqgUccLshpSdKv t4H7YzObDKaC2n/V4BAdICjcKLrzSBI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=RvaMm0Bb; spf=pass (imf26.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.208.42 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-f42.google.com with SMTP id 4fb4d7f45d1cf-6019b564d0bso4657610a12.2 for ; Mon, 19 May 2025 10:24:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1747675441; x=1748280241; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=3rsv9jkm5eLmngTLYRbXBLDmS8lLeNb225jEXQuDNAk=; b=RvaMm0BbEJ9HRei+XryVWLYhOybaZ/5mxWVKqKxd6qqii8esjiQSLK0s8QgmIQIyYo reGo3awGOCnRZNAWN+S2vGW6ZPffT4FJaQIpwRsOxR6+jINMfGHQVjCtRuHoFiHASQCJ GRyLYmACQpZwmsrlE001+M8Z1AWbKYm45/yePabi8rJGblbFddjZx4IF686UHosQGSM9 BmmcEERySmG+8ukD8sQa8wV4yBy1iHbtTxoUP5myQwlUVXA1wiPmWr3RCTHM+HWJBs4/ VNjDaG/q5JCw4IAxmkhqzG79iZ/IRy9UdA3zyZXrExg3QndQ+ufdguZNK61aQWAGe0sV 33yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747675441; x=1748280241; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3rsv9jkm5eLmngTLYRbXBLDmS8lLeNb225jEXQuDNAk=; b=wWUbvbiSE4OL+jjF2vLmwOU19leuI60Lr4l1Lno/J5altu51BfgoCCsxqTe5Dj2Mca iTJnfUhZiOK9t42snKnB80GOCrOQ6YmSS0FPGl40e2PVn7rGcjYUBEqPB2unH6OM7TBG eArklG9CerVD0zvScxnf0dGQqavNjwob3HxnQ2OaP/kprrQCji93xlnNpZy083lO5D/I fEgHbZB3YXjJHQ94jVYHTwatYCGxXyEDist3oGOoSFUH110xFm1QpLETed+D0uNc99Tc CQeVd8r8b8heSh9MwTSrDNWfuc3RojWPh4jwuyCJxpMzUK1+T37Ff8B/xAgUfAOv75Cy zHBQ== X-Forwarded-Encrypted: i=1; AJvYcCUib/LWhtCHP/hPlJCOWMyEeAWrAuiQ+nVXR1bVDyIgZfCYEOpLmvXY5Xcb7wjg864GGDk/jlDF9g==@kvack.org X-Gm-Message-State: AOJu0YxyiuV7A9TKg8Z9QRJFVorLGnwk4bntnD2crzkUhpIrwGJEmdGa vOets46+zzDC0YN4oltQfpI7FdX9OWRRPBoAtNmOE6OJ4zpObUkgl6Bp X-Gm-Gg: ASbGncs2uNqPVssvORNouATUA5x8iLowgrlKXii1epMJzhuSyeJ9MffglVoF4nIxwPt K435qUaFVcQKf7ro/cdeTMIi8Yz753gFsV0firczCLLodF+IPQLA8avmlPadm/bNbcjEyTPWwIW tE+P1FYdRhy8to5YhEZL3okuLzUUXuGeRyZwoGiM/xLvnkrfmvSM11+EkzH7BBb1AEBM6Wed/Hl NT+fE7lZfcuquXf13qQhlTgghWS1MrTh2/EY+dq8sUXQTM87yXpV7uLn59tROHyDH/9DdoeMpKc 0bpuL7To8m+5rAd+z3Azvy6bAEYv9lS6Iwuc7phyzYwjmUKH3ApH3nHkEJFXZMoykdDTPb/oW9D ll1ieEaYhrCIl1ID8egKyxnRlFRTRqzOtq75ITJw+ X-Google-Smtp-Source: AGHT+IGYv4CgsNOfNMsxPoRRqHFLg71gpMe3Apiu5yQmdBLnuZ57VJdKHnsCbDpvPGkunS4dI2Rw0w== X-Received: by 2002:a05:6402:348b:b0:601:834a:e678 with SMTP id 4fb4d7f45d1cf-601834ae764mr10857737a12.17.1747675440875; Mon, 19 May 2025 10:24:00 -0700 (PDT) Received: from ?IPV6:2a03:83e0:1126:4:1c0a:f3ac:4087:51c8? ([2620:10d:c092:500::6:19cb]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-6004d4f5fe5sm6025084a12.7.2025.05.19.10.24.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 19 May 2025 10:24:00 -0700 (PDT) Message-ID: Date: Mon, 19 May 2025 18:23:59 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Memory allocation profiling warnings in memory bound systems To: Suren Baghdasaryan , Johannes Weiner Cc: Shakeel Butt , Linux Memory Management List , kent.overstreet@linux.dev, vlad.wing@gmail.com References: <17fab2d6-5a74-4573-bcc3-b75951508f0a@gmail.com> <20250519160846.GA773385@cmpxchg.org> Content-Language: en-US From: Usama Arif In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Stat-Signature: 6pdgg714rkconz1h1gsz3x6pn4c7y94a X-Rspamd-Queue-Id: C480C140009 X-Rspam-User: X-Rspamd-Server: rspam02 X-HE-Tag: 1747675442-53386 X-HE-Meta: U2FsdGVkX1/M5ov1YLdyydd6fXBk89/0TEoPZIbnVTKQkbT9JcSIAlamBXbQo43CTxOZ8TrguAyxlhXzNirFhEEkwbxW2h4xx64vqTtQn/FLhRNwbhp4173Y1R9jdnsBgdVb9FXaXQ+w2Og0uStZxQFZs6rQE+Pk2pd3LHHEaTgBKfW7uc72VY90QcRMdnD1zj8LiIf/Iwc3hJbhM07SPuSRWqI/v7FpAso7v2QpdEm2l4QG8j6PnGFzAXg5DJwvMGgdnJNG0B80zyamLba7nXL3UJeM79Z4VPj+AcZOgKEav8L1AfrEQhKctY5uCrO6b7rtqLnd8CmwPGYLt/1h/GhNY1tBnXDxJ/Zsy3HZqiv31J5le7bKpScSakFJtJ1ShBbe7I8XYYa7/SlwNu+6SiPsoQO1zrf7WG6F5mVKXf4e4WbIHU2VtBnc50xnfQS/vaoETL3zQXSfdGpKTFZAhW3VE59Iccebx7xJNRq0/zQ4AgAUnP3Vqu53nwrdQHuhs0XRl1ZvKf4f8nyTKbhbnuhn4QJUmCXLhKF5jJt5oAXY1/o9ZLmudFghmQ8gQh5UcBaW2pkGy2NyhbA/4YA6W4wKRl1pZToCGYRiZW5HMxIhSMMF+rX24FtUPrAqyfu3qhdBypZHbW7vJ/F0K+rV5BhUdC+DH8njfMJBm+0MszuS5gBzWwBKwxAsdAcwUBCy/MVz/GePQJOamUHX3J3TVO6FuA0qPfLYPqLL+WW/W9XDx7EZpCeGopBpKPfwuBfuuP8siv1+owyYzqLyRNJ4aoZ0LssY9YAUz6YK/t+mIkeiIziPRUPzV+GFAnse3qe/Ldxrp6VDud+8kfAely4/LBe6ww0uY1faqK5TkZX9K5giOCzOqXy2wLu+CnKfDolKBUbVVu+qltc4n9YuK0ykawveQ9iFRgIwKMyayZkaxDcezr4T0VEuWvndEvfzMZ6F3FUwVjgCrC3Y5CF3qKZ AASR/WHn B2mPEzqi3Rmaort12ZV+GTwofPcc6bFH51UfvalqItsFYYfrOdO+5+FCH5BgPVH2HlneRuvvvC1qacxMhmW/413EevXRne4zrGNaYKqfmnk2O9bAyI+A8Y8d+lZbU0XsI9cBaHXstjJQg0JaUhDyIS8wisdz2qzUWQluLSNyPmJ6PfA6Pszvmj3jThvVJbxXLKxefXpnJlWakLHAasr3+8v1cc8aGB5dWN7eHJR80VuYoWuhnEN6wzmzy0Z9+wyHXx/gNaUXtjVn6Y8Qubj5fmhOma5lSn+cgVBowwWmftYokVGmb76np2q5dd2u0FvxsiXVM48hkwtNuKf4zHKaGaxwU/QMc9UMTDmr9t0/6CqV32GG7MccppdbolGtmm6jWwLFvTSh9XlzW3L+Glgb8NPiItN8IbFFPBHEBVG1eciHw44X9st8EG9CcXfRB+7HAbSnnCAQdU9GsxDSua70QkAmdzUF66mZ+dENF2iXf/lkpUqiR90DL/UnmBhF/2GT2IE/vETOi8Q7ctCc7IfIoJg1UA+1NI8YUiZZDjHzVc+WtEX/1GzAC5EPMFGTkbKIeDJoR/6HZqn2dPAl5oJIaMsoeTvXbzYVGpGftAMJpmjmVH3ZxSaarClessV8SJkZnpGeN0vLmCyVhWJDA+qr/2oHCVQqTpesM8nH1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 19/05/2025 17:42, Suren Baghdasaryan wrote: > On Mon, May 19, 2025 at 9:08 AM Johannes Weiner wrote: >> >> On Mon, May 19, 2025 at 08:50:28AM -0700, Suren Baghdasaryan wrote: >>> On Mon, May 19, 2025 at 6:33 AM Usama Arif wrote: >>>> >>>> >>>> +cc Vlad >>>> >>>> On 19/05/2025 14:31, Usama Arif wrote: >>>>> Hi, >>>>> >>>>> We have started enabling memory allocation profiling (with kernel 6.13) in our fleet >>>>> and are seeing a large number of warnings (reported by Vlad Poenaru) due to failure >>>>> in allocation of slab object extensions on services that are memory bound. I have attached >>>>> one of the logs at the end. >>>>> >>>>> Does it make sense to change the slabobj_ext to be allocated via kvcalloc and also change >>>>> the WARN to WARN_ONCE (or maybe even pr_debug?) like the diff below? A large number of >>>>> prints for this in a short time may mask any real issues in the system during memory >>>>> pressure being reported in dmesg. I tried to see if there were any changes after 6.13 >>>>> to this code but didn't find any, but thought will check before sending below as a patch. >>>>> >>>>> diff --git a/mm/slub.c b/mm/slub.c >>>>> index c2151c9fee22..4595ca190cd9 100644 >>>>> --- a/mm/slub.c >>>>> +++ b/mm/slub.c >>>>> @@ -1961,7 +1961,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s, >>>>> gfp &= ~OBJCGS_CLEAR_MASK; >>>>> /* Prevent recursive extension vector allocation */ >>>>> gfp |= __GFP_NO_OBJ_EXT; >>>>> - vec = kcalloc_node(objects, sizeof(struct slabobj_ext), gfp, >>>>> + vec = kvcalloc_node(objects, sizeof(struct slabobj_ext), gfp, >>> >>> Hi Usama, >>> Is the allocation larger than page size? IIUC, unless allocation size >>> is over PAGE_SIZE, kvcalloc_node() will not fall back to vmalloc (see: >>> https://elixir.bootlin.com/linux/v6.14.7/source/mm/util.c#L668). How >>> big is the allocation when it fails in your case? >> >> Digging through the reports, it appears we're encountering both. We've >> seen a zswap slab where the slab is order-0 and slabext is >> higher-order (8 byte objects, 512 objsperslab, 1 pageperslab), but >> also biovec-max where it's the other way round (4k byte objects, 8 >> objsperslab, 8 pagesperslab). >> >> In the first case, vmalloc would help. In the second it wouldn't. > > Ok, then I don't see any downside to changing to kvcalloc_node() here. > Let's do it. > >> >> The second case is interesting. The higher-order slab succeeds because >> bios use a mempool; but the system is so depleted that the order-0 for >> the slabext fails. > > I see. > >> >> I'm not sure there is much we can do about this tbh. It would seem >> overkill to add a mempool or grant the tracking access to system-wide >> emergency reserves. > > Yeah, with the system under so much memory pressure we probably have > bigger issues than extension vector allocation failures. > >> >> A warn-once would probably make sense nonetheless. > > Agree. > >> >> It might also make sense to flag the line item for that callsite in >> the reporting file, to make it obvious that the counter is compromised >> and is missing allocations? > > Good idea. We could output something like 'X' instead of the number if > the value is known to be invalid. I can look into it. Will also have > to raise the file version so that parsers can handle this change. > Thanks, I will send the above diff as patches. For when the value is inaccurate, it might be better to have the number and [X] next to it to reflect its inaccurate? Maybe an inaccurate number is better than no number?