From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0A8BC02183 for ; Fri, 17 Jan 2025 13:40:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B2D1280002; Fri, 17 Jan 2025 08:40:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66279280001; Fri, 17 Jan 2025 08:40:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 551B8280002; Fri, 17 Jan 2025 08:40:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 326F0280001 for ; Fri, 17 Jan 2025 08:40:35 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D78301A1BC9 for ; Fri, 17 Jan 2025 13:40:34 +0000 (UTC) X-FDA: 83017053588.06.1C811C5 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 0DAFDC0016 for ; Fri, 17 Jan 2025 13:40:32 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf28.hostedemail.com: domain of "SRS0=X/cO=UJ=goodmis.org=rostedt@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=X/cO=UJ=goodmis.org=rostedt@kernel.org" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737121233; a=rsa-sha256; cv=none; b=l4i9SkRIZy6UJPD02EC8YYyd/03PpZdt+QTEeHjqOTHbgD0iJGKJguVzDKpwjI3O5QaFPV YOUryYuakdIKvpoentpOBI6E/Ewme1RE/zc4ZdmPZdpsds/+3f9Zl2h7JU2zOX90vPkhpc 6wg+by/HHDX/0XhEoz1FQS51U9z7j48= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf28.hostedemail.com: domain of "SRS0=X/cO=UJ=goodmis.org=rostedt@kernel.org" designates 139.178.84.217 as permitted sender) smtp.mailfrom="SRS0=X/cO=UJ=goodmis.org=rostedt@kernel.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737121233; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4MSlfMWTenTA16vG2VXIQkLZt+h3+UvMMHrxS1reDNY=; b=2CYKPAOr/zWIp50ximjWocTisk7XtZdxXe6vmfJT0YnoCVSX1MxYTLojs3lLXF47gCswnF F0EDIXzBUT7Oo2BkrzY7Ggq5iZlHKarcX1R5nWyZalDZcPOa6AUz/RVzkzfVH9R1rOXzWD Z7qyhzK9PoILmzUBJGiurfQg+5mncYc= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 5B6D85C61F0; Fri, 17 Jan 2025 13:39:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2266AC4CEE1; Fri, 17 Jan 2025 13:40:31 +0000 (UTC) Date: Fri, 17 Jan 2025 08:40:38 -0500 From: Steven Rostedt To: Alexandre Ferrieux Cc: linux-trace-users@vger.kernel.org, Lorenzo Stoakes , LKML , linux-mm@kvack.org Subject: Re: Bug: broken /proc/kcore in 6.13 Message-ID: <20250117084038.79f40307@gandalf.local.home> In-Reply-To: <05ea473e-d7e9-4ca5-ad91-ba8c00618fb4@orange.com> References: <05ea473e-d7e9-4ca5-ad91-ba8c00618fb4@orange.com> X-Mailer: Claws Mail 3.20.0git84 (GTK+ 2.24.33; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 0DAFDC0016 X-Rspamd-Server: rspam10 X-Stat-Signature: gtcfkbz8jhakysztipos6u46opzkkgca X-HE-Tag: 1737121232-987510 X-HE-Meta: U2FsdGVkX19NzehhIoFTr1liVepOVGDfevr3KwlKJeuRNsdQIdhDutlWUqUhxwaGBLt9YUk77+UiNpE9GoJBzVXwW9i8vbtJHWkIjk9eyH+datpgKH0s5ZNu3xE6MEAL92GdH1DUyHt2zlJDix9ahhVxrR/+ktchFHhDObS5Llhb62vqBsqFbyVfQ65MPGm4THjy5BlBuQjxez/lutz9wC27ko3yhrJojo2xR4srO0nvWoCiFstvesUDujiAXUFkS5A0HlG1qcOuqbxAqDz5+/CgHajfxykqw+7Etgq/VVJRw9djNt+6VcnthsmokP3e6fyl7Qpub/PFsQpAYa5LS+J1sEKiTFJXwTFDfm5Tgw2AByP/BsDrX7CHGTfMJnp+TOwrcvj4ffhsHZYK7k8OXW1UmDa8FUPFkKEd/4JEFOuVdNqr2ycRiKPxcH5g0G2HR/vWMkd7OAFX241tXgfI2KWDNS6luhWgzJXp6tohd6n1wJvhoOkPcU/xX9dPeLh4VppIjtcmIZGg2qUwprQdefEbqBMmOTyyR0QL3LIxgMZjPHN0MuwWvk9J1MKxiHrF8G+cg/O1LGgFMcl9qsFK6ysjNCK1uCox8CLsTDl1OmFgk+ZTKD2xSLy+YBWm5z7LU7kt4wBRJqmaF8VY6Z9rqFJWCraPvBpTtXZKgcnx02eIR57OPBO/CzUjetx67+0RAqXB18E5+w02ZG/QkBfaMXx+C9b46RZmjSwCSRN4z3YnkBS5maXXcZSf9f9jcEAZtjMYdwO3HIeUewkOY6GW1E9DxdLip+g+jJzc1lQyyPRCLlaQk0YNNX++JX/fcRpxoayBavIuV42AXNe3ppTq4vjseIN4jb+FqHclR3kf+zS1IA03wQAb1RirLk+mJADZrhs4spD+38zQMcqP+iyksYuKz773D/dAP7/ZpV/fY8MFiydBHl1H34Nj+tItPAqiWb6J3ms51YxVWcn8IyJ NaigUnyp jfaagQ4z/SnHbT2TLh7L7+oTF+TNXPpl5B5PeH0A4fVBxI7XRGb+jWmos+kHd+C7TCEeHhk/3fdOi3eperPbyF4KFSUCGmSWJJtQZC09pzsL2LJ8HcPta41Df5KSxdNm6TLG3Ql3YbQ1hUYAkZiDV0PCxpHEgpmGCuMVM9oPhdY6rRH5C6p8IRvEOR5g68yKN3WfkF5lhF0as+bIdw87ML/6T8nXYgGMpDTqvvSETP1g6KUYY8WKNi9KCwOUuCh+L35dMJA9wE2aLGogNBfhzSZXrpA1f7XyP9+hIxLolHDfZb3mF0ojhznwEdK8e3zUW8bk9uiVvpG8zfiT/4/36xw/XD1FDfK/VgrJRSyi1Bxus082snsfCn8qHQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: [ Cc'ing the proper folks ] -- Steve On Fri, 17 Jan 2025 11:36:05 +0100 Alexandre Ferrieux wrote: > Hi, > > Somewhere in the 6.13 branch (not bisected yet, sorry), it stopped being > possible to disassemble the running kernel from gdb through /proc/kcore. > > More precisely: > > - look up a function in /proc/kallsyms => 0xADDRESS > - tell gdb to "core /proc/kcore" > - tell gdb to "disass 0xADDRESS,+LENGTH" (no need for a symbol table) > > * if the function is within the main kernel text, it is okay > * if the function is within a module's text, an infinite loop happens: > > > Example: > > # egrep -w ice_process_skb_fields\|ksys_write /proc/kallsyms > ffffffffaf296c80 T ksys_write > ffffffffc0b67180 t ice_process_skb_fields [ice] > > # gdb -ex "core /proc/kcore" -ex "disass 0xffffffffaf296c80,+256" -ex quit > ... > Dump of assembler code from 0xffffffffaf296c80 to 0xffffffffaf296d80: > ... > End of assembler dump. > > # gdb -ex "core /proc/kcore" -ex "disass 0xffffffffc0b67180,+256" -ex quit > ... > Dump of assembler code from 0xffffffffc0b67180 to 0xffffffffc0b67280: > (***NOTHING***) > ^C <= inefficient, need kill -9 > > > Ftrace (see below) shows in this case read_kcore_iter() calls vread_iter() in an > infinite loop: > > while (true) { > read += vread_iter(iter, src, left); > if (read == tsz) > break; > > src += read; > left -= read; > > if (fault_in_iov_iter_writeable(iter, left)) { > ret = -EFAULT; > goto out; > } > } > > As it turns out, in the offending situation, vread_iter() keeps returning 0, > with "read" staying at its initial value of 0, and "tsz" nonzero. As a > consequence, "src" stays stuck in a place where vread_iter() fails. > > A cursory "git blame" shows that this interplay (vread_iter() legitimately > returning zero, and read_kcore_iter() *not* testing it) has been there from > quite some time. So, while this is arguably fragile, possibly the new situation > lies in the actual memory layout that triggers the failing path. > > Thanks for any insight, as this completely breaks debugging the running kernel > in 6.13. > > -Alex > > > ------------ > # tracer: nop > # > # entries-in-buffer/entries-written: 0/0 #P:48 > # > # TASK-PID CPU# TIMESTAMP FUNCTION > # | | | | | > <...>-3304 [045] 487.295283: kprobe_read_kcore_iter: > (read_kcore_iter+0x4/0xae0) pos=0x7fffc0b6b000 > <...>-3304 [045] 487.295298: kprobe_vread_iter: > (vread_iter+0x4/0x4e0) addr=0xffffffffc0b67000 len=384 > <...>-3304 [045] 487.295326: kretprobe_vread_iter: > (read_kcore_iter+0x3e6/0xae0 <- vread_iter) arg1=0 > <...>-3304 [045] 487.295329: kprobe_vread_iter: > (vread_iter+0x4/0x4e0) addr=0xffffffffc0b67000 len=384 > <...>-3304 [045] 487.295338: kretprobe_vread_iter: > (read_kcore_iter+0x3e6/0xae0 <- vread_iter) arg1=0 > <...>-3304 [045] 487.295339: kprobe_vread_iter: > (vread_iter+0x4/0x4e0) addr=0xffffffffc0b67000 len=384 > <...>-3304 [045] 487.295345: kretprobe_vread_iter: > (read_kcore_iter+0x3e6/0xae0 <- vread_iter) arg1=0 > <...>-3304 [045] 487.295347: kprobe_vread_iter: > (vread_iter+0x4/0x4e0) addr=0xffffffffc0b67000 len=384 > <...>-3304 [045] 487.295352: kretprobe_vread_iter: > (read_kcore_iter+0x3e6/0xae0 <- vread_iter) arg1=0 > <...>-3304 [045] 487.295353: kprobe_vread_iter: > (vread_iter+0x4/0x4e0) addr=0xffffffffc0b67000 len=384 > ... >