From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 014A0C2BA15 for ; Tue, 18 Jun 2024 21:32:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D6D68D0063; Tue, 18 Jun 2024 17:32:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 660438D005C; Tue, 18 Jun 2024 17:32:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B35E8D0063; Tue, 18 Jun 2024 17:32:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 21EF98D005C for ; Tue, 18 Jun 2024 17:32:33 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A6E53140AFB for ; Tue, 18 Jun 2024 21:32:32 +0000 (UTC) X-FDA: 82245308544.03.D1AA4F8 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) by imf11.hostedemail.com (Postfix) with ESMTP id 4727F40017 for ; Tue, 18 Jun 2024 21:32:30 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ebiederm@xmission.com designates 166.70.13.233 as permitted sender) smtp.mailfrom=ebiederm@xmission.com; dmarc=pass (policy=none) header.from=xmission.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718746347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EOmFwF9fcU0mr0sU6zJL/K7F1fVgIHm5kCuv8djGXqc=; b=X5pM5U8IoQvmms5T3/KXFp/xmR5AUi+fAQtz5W9Sn8SSvfnAvw6pTOCpqUT6sAHjqRtSe0 3K0xJ1srlyPH21CPhnmhlWBGBxszT74a2K29Q1gQ8/XD4wmWq7G7Tk8aKxvovMsTEd7gkx hZb49KVVaQ6RUtq1jDGUhuknLAvZ1Kc= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ebiederm@xmission.com designates 166.70.13.233 as permitted sender) smtp.mailfrom=ebiederm@xmission.com; dmarc=pass (policy=none) header.from=xmission.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718746347; a=rsa-sha256; cv=none; b=XiPPgFHDYH7N8ME0Dsqij9VqLOk/aHZ9Ur9Y8eOmMrsyRmKui9H0BFZrP+qOJ7V1QkMSJ+ q7pxNjVT1KGFHz/GQF4k/8JMPiyoE0fOBPD+Xzvf/WL0unMVuAFrDZMpXkOLZw8DYM0j5A 6SSBODkc7TWSBMvL/KBNI55NgPUiQCE= Received: from in02.mta.xmission.com ([166.70.13.52]:47062) by out03.mta.xmission.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1sJgRE-005R6Y-Kl; Tue, 18 Jun 2024 15:32:24 -0600 Received: from ip68-227-168-167.om.om.cox.net ([68.227.168.167]:34896 helo=email.froward.int.ebiederm.org.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.93) (envelope-from ) id 1sJgRD-0023Sy-FS; Tue, 18 Jun 2024 15:32:24 -0600 From: "Eric W. Biederman" To: Roman Kisel Cc: Sebastian Andrzej Siewior , akpm@linux-foundation.org, apais@linux.microsoft.com, ardb@kernel.org, brauner@kernel.org, jack@suse.cz, keescook@chromium.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, nagvijay@microsoft.com, oleg@redhat.com, tandersen@netflix.com, vincent.whitchurch@axis.com, viro@zeniv.linux.org.uk, apais@microsoft.com, ssengar@microsoft.com, sunilmut@microsoft.com, vdso@hexbites.dev References: <20240617234133.1167523-1-romank@linux.microsoft.com> <20240617234133.1167523-2-romank@linux.microsoft.com> <20240618061849.Vh9N3ds2@linutronix.de> Date: Tue, 18 Jun 2024 16:21:09 -0500 In-Reply-To: (Roman Kisel's message of "Tue, 18 Jun 2024 09:30:22 -0700") Message-ID: <87sexakkvu.fsf@email.froward.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1sJgRD-0023Sy-FS;;;mid=<87sexakkvu.fsf@email.froward.int.ebiederm.org>;;;hst=in02.mta.xmission.com;;;ip=68.227.168.167;;;frm=ebiederm@xmission.com;;;spf=pass X-XM-AID: U2FsdGVkX1/gB2O6TobRXQAk0tnw1xCtXBe434z0Hw0= X-SA-Exim-Connect-IP: 68.227.168.167 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: [PATCH 1/1] binfmt_elf, coredump: Log the reason of the failed core dumps X-SA-Exim-Version: 4.2.1 (built Sat, 08 Feb 2020 21:53:50 +0000) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: 4727F40017 X-Stat-Signature: deg66bjnk9gb6dyd6ieapg1zugkoh1jj X-HE-Tag: 1718746350-954031 X-HE-Meta: U2FsdGVkX1/20acxF6Zi//ngzFgJvXVBEGlJhpp0zNYkupYm8SPQrX8/3NKY713LLKHghp8Vx6y6/uP7Li56qTuuDbiPpPGDMSMr9WO4ko1iS038yYCJZJNd1kkmESaSZKNIdxJ8dN8hNpfR52axZWLhDJjydXTWzFEJu5o0+bquyYtjinAA+BF+DhXWfIYKzvd9l49FOd7+H9HCR9fZcE7kwNUvbVIr7Ee6IMWT54JHmSimkZwFsT8p2nZnrTeuJ08LDsiOWqFpocqSJtGJnAeOdPSaWua/g61aANsXRs9SeMCxTr93ZCIYvQ58GcDzqC2nYvWvqsIkf0kE96Vez5zQTFjQjpoVkMjzsl3dNPsw11avMh7NCOZKV1sgoh2//rPBm9IiL/18f0IYSt59HqSw782s68YH4kcVaWIaetaU2eycfSpXB/Owtc2Vs97rfPTsU1mV4FXIHoXGSN/985aBqADte86K7DY2JvnSKC2WahI4xQUSZIlM3xfX7ZAUGXTDiLyyVEGD/P4ZSlKFKUg1TGI+K/tUJfXgYGTEz1EHMqFKGIyr5L7ZlR4k7FAZosTND/5IFkBpY4G1ziAyJr6Wz+CrquvF6PrKYMMTETF8FCLSC2cIGLj90eRr0h3UW0RZdxamYAk+hzc5odTcJVnqtxw1lEfvdcYrjhMbykNMfeM874L4f+BYj7fz/MC2jDkvo6IHQn3/euV5eXEVIHbRo6KrXO2pi8Rr5LiG4vrgSurIVmNaK6azrxhDE8le5L4s7vbjxCAD210bGWsoCWXtuXkgmZy9Krqh1jJjxbxgaPmXYBX3taBOVc4WmBxUuNDT8tCGxgTWGbM1sFRG22K8PShy65eHdjzRSoXJjcxVtXzR2+WNMyBcWRQeUe2e8eUGh2wYTfygeonVkaqULxr/QcRve3blYvX3Lrczgb2jj5E7RKUBlv/pRnZ7ytoXEMR/I/vr8zXLg9DPtwU w6UXrROv P0YfVfsJMPLnzfScM/QU6kQ1YVm11PjyB0P1mmQX1BCbyiaQ3ASo+ssnWHynrhiOwueNXZgLO3WzTqNyDoNSYpmMNNBj54cDaxsLM12z8zx5Lt40eay1GwKQUUplhN97L7cW8 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Roman Kisel writes: > On 6/17/2024 11:18 PM, Sebastian Andrzej Siewior wrote: >> On 2024-06-17 16:41:30 [-0700], Roman Kisel wrote: >>> Missing, failed, or corrupted core dumps might impede crash >>> investigations. To improve reliability of that process and consequently >>> the programs themselves, one needs to trace the path from producing >>> a core dumpfile to analyzing it. That path starts from the core dump file >>> written to the disk by the kernel or to the standard input of a user >>> mode helper program to which the kernel streams the coredump contents. >>> There are cases where the kernel will interrupt writing the core out or >>> produce a truncated/not-well-formed core dump. >> How much of this happened and how much of this is just "let me handle >> everything that could go wrong". > Some of that must be happening as there are truncated dump files. Haven't run > the logging code at large scale yet with the systems being stressed a lot by the > customer workloads to hit all edge cases. Sent the changes to the kernel mail > list out of abundance of caution first, and being ecstatic about that: on the > other thread Kees noticed I didn't use the ratelimited logging. That has > absolutely made me day and whole week, just glowing :) Might've been a close > call due to something in a crash loop. Another reason you could have truncated coredumps is the coredumping process being killed. I suspect if you want reasons why the coredump is truncated you are going to want to instrument dump_interrupted, dump_skip and dump_emit rather than their callers. As they don't actually report why the failed. Are you using systemd-coredump? Or another pipe based coredump collector? It might be the dump collector is truncating things. Do you know if your application uses io_uring? There were some weird issues with io_uring and coredumps that were causing things to get truncation at one point. As I recall a hack was put in the coredump code so that it worked but maybe there is another odd case that still needs to be handled. > > I think it'd be fair to say that I am asking to please "let me handle (log) > everything that could go wrong", ratelimited, as these error cases are present > in the code, and logging can give a clue why the core dump collection didn't > succeed and what one would need to explore to increase reliability of the > system. If you are looking for reasons you definitely want to instrument fs/coredump.c much more than fs/binfmt_elf.c. As fs/coredump.c is the code that actually performs the writes. One of these days if someone is ambitious we should probably merge the coredump code from fs/binfmt_elf.c and fs/binfmt_elf_fdpic.c and just hardcode the coredump code to always produce an elf format coredump. Just for the simplicity of it all. Eric