From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A084FC27C77 for ; Wed, 12 Jun 2024 15:37:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 37E4B6B009D; Wed, 12 Jun 2024 11:37:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 32E466B009E; Wed, 12 Jun 2024 11:37:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A7B06B00A0; Wed, 12 Jun 2024 11:37:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F217F6B009D for ; Wed, 12 Jun 2024 11:37:52 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A1ECC1C2A9D for ; Wed, 12 Jun 2024 15:37:52 +0000 (UTC) X-FDA: 82222641984.27.18F9FB6 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf08.hostedemail.com (Postfix) with ESMTP id 21AFE160019 for ; Wed, 12 Jun 2024 15:37:48 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=daXsUq4X; spf=pass (imf08.hostedemail.com: domain of iii@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=iii@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718206669; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sUlVOpLsB7ZgvZM2Yku0ftpL1kxmfJTOLslTgiO7Wc0=; b=7OYEhMbbBrrvsmlkJBHlaAr+W1rgeDYXGdQw+j8PpVtfx2UFpSVQ+I1D2JjUktVHLigHvW RtlyX1MNFxQXH3Nw1vXxu4KHC6T2+14w+r+/F+0M6Zq8A+M4iWDbd1+B1Wss2CQfliAlkf TLhL4mjUGTIKcq4saj/ZFAad0cMeXTU= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=daXsUq4X; spf=pass (imf08.hostedemail.com: domain of iii@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=iii@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718206669; a=rsa-sha256; cv=none; b=F1xtD7fYdbenn8IOalUhLkKCJzWIncuRrEuGl6zmNWsQqkmjyaphK1TA2F6eHpjLLd/nyR +fxnMep13nJwuas+1Endjk3gadhmXP3WqCa5XZM7Oq2AkoIoR+WZJVfVASvKND5AqFrPsC we73NaHqTvFktT7DDM33KWElLqmxPP0= Received: from pps.filterd (m0353723.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45CFTKxU001593; Wed, 12 Jun 2024 15:37:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h= message-id:subject:from:to:cc:date:in-reply-to:references :content-type:content-transfer-encoding:mime-version; s=pp1; bh= sUlVOpLsB7ZgvZM2Yku0ftpL1kxmfJTOLslTgiO7Wc0=; b=daXsUq4XnojjxyDh rPr9puRhWpsPHbxnHsDA45+Ak7X/gc6/eWJlq6VdxuuBhEH2cVk8eSh7J3u1QF2r 2iz25E/yYhBBavM9hj7cdamlNfKbIpe5TRTbWTPTPtOZrlKfDz80FREgwAB1N/DF 39EGVItTyh5uFdosCCg+oduzzdN/nIs3unsVyF9+nEFGpXi4KsxPH20qUqW/6H4J h0rKBW/mTZGiAldxPaAojr7RqIwxh800XCERGPKVz5VhZN8i0eFcZOXIH+jt0ZYr HflGxrUP/wP4pLCTgE2r490Ih3S0EoY2sqIJLRKzcJdjw/YPj5w8LUWPnl0fi+2N AoA2qg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3yqebx80pn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 12 Jun 2024 15:37:40 +0000 (GMT) Received: from m0353723.ppops.net (m0353723.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 45CFbdDb014086; Wed, 12 Jun 2024 15:37:40 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3yqebx80pj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 12 Jun 2024 15:37:39 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 45CFLocC003881; Wed, 12 Jun 2024 15:37:39 GMT Received: from smtprelay04.fra02v.mail.ibm.com ([9.218.2.228]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 3yn2mpy4ha-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 12 Jun 2024 15:37:39 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay04.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 45CFbXI411403582 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 12 Jun 2024 15:37:35 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 187342004E; Wed, 12 Jun 2024 15:37:33 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B02D720040; Wed, 12 Jun 2024 15:37:32 +0000 (GMT) Received: from [9.155.200.166] (unknown [9.155.200.166]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 12 Jun 2024 15:37:32 +0000 (GMT) Message-ID: <6403223315eda4e8023a828d6f40353c694d474e.camel@linux.ibm.com> Subject: Re: [PATCH v3 01/34] ftrace: Unpoison ftrace_regs in ftrace_ops_list_func() From: Ilya Leoshkevich To: Steven Rostedt Cc: Alexander Gordeev , Alexander Potapenko , Andrew Morton , Christoph Lameter , David Rientjes , Heiko Carstens , Joonsoo Kim , Marco Elver , Masami Hiramatsu , Pekka Enberg , Vasily Gorbik , Vlastimil Babka , Christian Borntraeger , Dmitry Vyukov , Hyeonggon Yoo <42.hyeyoo@gmail.com>, kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-s390@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Mark Rutland , Roman Gushchin , Sven Schnelle Date: Wed, 12 Jun 2024 17:37:32 +0200 In-Reply-To: <20240102101712.515e0fe3@gandalf.local.home> References: <20231213233605.661251-1-iii@linux.ibm.com> <20231213233605.661251-2-iii@linux.ibm.com> <20240102101712.515e0fe3@gandalf.local.home> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.50.4 (3.50.4-1.fc39) X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: rmQltXYdpFw9fTbDS6VlEHPVsYz_YHE3 X-Proofpoint-GUID: Ivcj_IlvBtvvOQzLTRy84LG3MKhmjE9t Content-Transfer-Encoding: quoted-printable X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-12_08,2024-06-12_02,2024-05-17_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 phishscore=0 lowpriorityscore=0 adultscore=0 clxscore=1011 impostorscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2405170001 definitions=main-2406120110 X-Rspam-User: X-Rspamd-Queue-Id: 21AFE160019 X-Rspamd-Server: rspam01 X-Stat-Signature: w8njksfe158xw54xsetzu69k6794pyck X-HE-Tag: 1718206668-558761 X-HE-Meta: U2FsdGVkX1/sKkoATOXTponFcjdYvnT4t4ECojwUszYyG9W++EjUfyRoyVTA0cCY0qSn6TDO6WQwsSyIwz1BeuUfsrepCqwJUYb/Qste2po7/IBtH004Ohe8PkgLLcJkYkjUr36MoG/hkjEmiOQcXUAPEQuq6I6TsN+vjXmoQEjbr4S17buhL8BmDbhlOgkGXFFuzE8GEBOPna4EwDso1F00L0aHHUhmZCpWSIwbAr9mvnuJHcdZs7l/zg7cvWgfVUlxKt8Bhw4khB8Wl3kFXMEeuFi7kENVY5S5sqsHzSFKKMpG8Wld9k2So1fu4KmS0fjTEMw4xlFVp16cqWmtgyxTegJ6fbD9uwNUNw49leLOTRA7S53qlA9BYeALPDpzEmv6rj21EDubsllxH+QLPRj23LOh2h4VmVuaTJxyneVMVtxZq/RZJxzPlJVSK0vx28/F3487+/cl7pbG/Cu7Df7OoeejxzDwPESymPMIk2053DvLTODAfbC+7lPB6xMBdA7oSLawvPNl0M23+qOkq3AytrVZi1nE9nAxd7ZD+T1sL7ur4Hs6W7PAG6TuwOrSJIYz/URbX2fnj1Pwh+TsF55W9EI5h/7yAyC3HhHtYVMNsN7coyEu16DbtUpVmdy2UlOYt1uSkinK7SliCzsH5h6b1chmL8N8n+Yq9gETud18RzHbzTRof2FqnMpiswaJDhAfFhDWr+OY/v8pWZOZMjnwU1ZkyEWsKrSwqnftk433PjJCy9hs5uNJ9KgsCXAyHTDR1oivRnCS79bmXN7kk9mlH9AuSqHcHmkJWlBH3ICDqLBWWw7FKdrtet179UkJs4vcCPbo8wh9yZ9/jS5enSl3dcwZq0WiNgFtMd8xFPkARfd+1/sn12lbsd2xsNNsGYx2CB5ALn2f2xLnDxSH7vJ6XEucr+pRliqFMwPtwmDyC715tJzR4QVrN02ZuyjNme6QTXK6Zw+B6c493HP PDL04xlp URkZx2q6YOccqAllawMteHcG75MVW9zTpRVWc9MQz1d/6cLaEi/oCbZbJgYtRq/Nyio5rqyCIVCz3qupEZp2hwH0qwMp2v7lyUao6u3NXNBcyn4ZnNumZ15fC2zdGRFWRouDNiAYfjlXW3MoMmqklzjFTSFsdHavb/flkdiOySAjm7zeZwQXmituO2cNrAehvonhwXJ1s+fgoLfeLeKdYghFij8lERf/J4ylfRHQlGZCXJ1vgzUiD5lCRm1BZxOeYrpLVLfDRLIMpmT1ZGuTXd4zS+GBqa40BYtLhWfAwR8sOdqP60uvnK9XkMfJOm6XIp61iNB6xt3wvOtiqGTsiygUSnIRjH8nID7kFrVcHnPaEUlic1bxr5sFuyY4+AZpZw89c+UFfTjjOCOF1/4cjN1+bawISSiV55LSxOJrpK099Q6a7j5weiLTjEzZyXzs/qK5rufWNcw8oFu4U8wPYe62HYxUoeM+gmYE+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, 2024-01-02 at 10:17 -0500, Steven Rostedt wrote: > On Thu, 14 Dec 2023 00:24:21 +0100 > Ilya Leoshkevich wrote: >=20 > > Architectures use assembly code to initialize ftrace_regs and call > > ftrace_ops_list_func(). Therefore, from the KMSAN's point of view, > > ftrace_regs is poisoned on ftrace_ops_list_func entry(). This > > causes > > KMSAN warnings when running the ftrace testsuite. >=20 > BTW, why is this only a problem for s390 and no other architectures? >=20 > If it is only a s390 thing, then we should do this instead: >=20 > in include/linux/ftrace.h: >=20 > /* Add a comment here to why this is needed */ > #ifndef ftrace_list_func_unpoison > # define ftrace_list_func_unpoison(fregs) do { } while(0) > #endif >=20 > In arch/s390/include/asm/ftrace.h: >=20 > /* Add a comment to why s390 is special */ > # define ftrace_list_func_unpoison(fregs) > kmsan_unpoison_memory(fregs, sizeof(*fregs)) >=20 > >=20 > > Fix by trusting the architecture-specific assembly code and always > > unpoisoning ftrace_regs in ftrace_ops_list_func. > >=20 > > Acked-by: Steven Rostedt (Google) >=20 > I'm taking my ack away for this change in favor of what I'm > suggesting now. >=20 > > Reviewed-by: Alexander Potapenko > > Signed-off-by: Ilya Leoshkevich > > --- > > =C2=A0kernel/trace/ftrace.c | 1 + > > =C2=A01 file changed, 1 insertion(+) > >=20 > > diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c > > index 8de8bec5f366..dfb8b26966aa 100644 > > --- a/kernel/trace/ftrace.c > > +++ b/kernel/trace/ftrace.c > > @@ -7399,6 +7399,7 @@ __ftrace_ops_list_func(unsigned long ip, > > unsigned long parent_ip, > > =C2=A0void arch_ftrace_ops_list_func(unsigned long ip, unsigned long > > parent_ip, > > =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct ftrace_ops *op, st= ruct > > ftrace_regs *fregs) > > =C2=A0{ > > + kmsan_unpoison_memory(fregs, sizeof(*fregs)); >=20 > And here have: >=20 > ftrace_list_func_unpoison(fregs); >=20 > That way we only do it for archs that really need it, and do not > affect > archs that do not. >=20 >=20 > I want to know why this only affects s390, because if we are just > doing > this because "it works", it could be just covering up a symptom of > something else and not actually doing the "right thing". >=20 >=20 > -- Steve >=20 >=20 > > =C2=A0 __ftrace_ops_list_func(ip, parent_ip, NULL, fregs); > > =C2=A0} > > =C2=A0#else >=20 Ok, it has been a while, but I believe I have a good answer now. KMSAN shadow for memory above $rsp is essentially random. Here is an example (you'll need a GDB hack from [1] if you want to try this at home): (gdb) x/5i do_nanosleep 0xffffffff843607c0 : call 0xffffffffc0201000 Thread 3 hit Breakpoint 1, 0xffffffffc0201000 in ?? () (gdb) x/64bx kmsan_get_metadata($rsp - 64, 0) 0xffffd1000087bd38: 0x00 0x00 0x00 0x00 0x00 0x00 =20 0x00 0x00 0xffffd1000087bd40: 0x00 0x00 0x00 0x00 0x00 0x00 =20 0x00 0x00 0xffffd1000087bd48: 0x00 0x00 0x00 0x00 0x00 0x00 =20 0x00 0x00 0xffffd1000087bd50: 0x00 0x00 0x00 0x00 0xff 0xff =20 0xff 0xff 0xffffd1000087bd58: 0x00 0x00 0x00 0x00 0x00 0x00 =20 0x00 0x00 0xffffd1000087bd60: 0xff 0xff 0xff 0xff 0xff 0xff =20 0xff 0xff 0xffffd1000087bd68: 0xff 0xff 0xff 0xff 0xff 0xff =20 0xff 0xff 0xffffd1000087bd70: 0xff 0xff 0xff 0xff 0xff 0xff =20 0xff 0xff So if assembly (in this case ftrace_regs_caller) allocates struct pt_regs on stack, it may or may not be poisoned depending on what was called before. So, by accident, on s390x it's poisoned and trips KMSAN, and on x86_64 it's not. Based on this observation, I'd say we need an unpoison call in all ftrace handlers (e.g., kprobe_ftrace_handler), and not just this one. But why is this the case? Kernel stacks are created by alloc_thread_stack_node() using __vmalloc_node_range(__GFP_ZERO), so they are fully unpoisoned. Then functions are called and return, their locals are poisoned and unpoisoned. Interestingly enough, on return, they are not poisoned back, even though commit 37ad4ee8364255c73026a3c343403b5977fa7e79 Author: Alexander Potapenko Date: Thu Sep 15 17:04:13 2022 +0200 x86: kmsan: don't instrument stack walking functions says they do. So what if we introduce that [2]? # echo "p:nanosleep do_nanosleep %di" >/sys/kernel/tracing/kprobe_events # echo 1 >/sys/kernel/debug/tracing/events/kprobes/nanosleep/enable # sleep 1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D BUG: KMSAN: uninit-value in kprobe_ftrace_handler+0x5b9/0x790 kprobe_ftrace_handler+0x5b9/0x790 0xffffffffc02010de do_nanosleep+0x5/0x670 hrtimer_nanosleep+0x169/0x3b0 common_nsleep+0xc7/0x100 __x64_sys_clock_nanosleep+0x4e2/0x650 do_syscall_64+0x6e/0x120 entry_SYSCALL_64_after_hwframe+0x76/0x7e Local variable nd created at: do_filp_open+0x3b2/0x5e0 Quite similar to s390. Local variable nd is a random leftover from a different call stack, which the modified instrumentation poisoned on return from do_filp_open(). Alexander, what do you think about adding [2] upstream as an option that can be enabled from the command line? Also, what do you think about poisoning kernel stacks? Formally they are zeroed out, but I think valid code has no business reading these zeroes. [1] https://sourceware.org/bugzilla/show_bug.cgi?id=3D31878 [2] https://github.com/iii-i/llvm-project/commits/msan-poison-allocas-before-re= turning-2024-06-12/