From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF811EB64D8 for ; Wed, 14 Jun 2023 13:23:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 413206B0074; Wed, 14 Jun 2023 09:23:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C3F16B0075; Wed, 14 Jun 2023 09:23:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28C5B6B0078; Wed, 14 Jun 2023 09:23:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1A0416B0074 for ; Wed, 14 Jun 2023 09:23:49 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CF33BA038B for ; Wed, 14 Jun 2023 13:23:48 +0000 (UTC) X-FDA: 80901420936.23.31A3696 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf28.hostedemail.com (Postfix) with ESMTP id 2F20BC0011 for ; Wed, 14 Jun 2023 13:23:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=nuEcGBGh; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf28.hostedemail.com: domain of lee@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=lee@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686749027; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=xcKDF1WQTxN1LEhxz8EbgGwOf4+r0hOZdUqq2Gv20XE=; b=Q/N4LyNcAUkm369V8e9WB+rFHtYqVDr8+e8Gb5/AHfttg2mmRKPAmmjbJB0wF2GbVOpx8k WuFJem0hdxSSYhX07K0QFTQ+yIiFgtlCasURQmgUT6Hsgul9S0J0lzgDrXcnfPE3mIMn4u E085j44FscqLqb1lcnXUouLfeSI/5a4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=nuEcGBGh; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf28.hostedemail.com: domain of lee@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=lee@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686749027; a=rsa-sha256; cv=none; b=Y6pN7gHeIbKD6RKzpsaZR2FDtXdDYHi0dhoiCvstIftTznydmu6PT69z8q4yY723U77EYS A9iPG/hMMSZdf/Wed7G840VPgZOM65BfREFCt4k9ex3IsDckowzyomvMvVMUgIUbhLcOEb OpWTm0QpIXuiExctphY9GdQKTOyi198= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 51D8C6423D; Wed, 14 Jun 2023 13:23:46 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 397FFC433C8; Wed, 14 Jun 2023 13:23:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1686749025; bh=qAp1EyPAJ/3UHgu6vuw4vJIYncffs63ClelMAcaMES0=; h=Date:From:To:Cc:Subject:From; b=nuEcGBGhCCIa7Yu11c5lQemBvsdW38aqdyZgdfqotQj5+acb+Uwy89w3+LsFxL7rB /BxtO1UZEDaB3C8ZUZ7OtyBnAZQLpX7qEZgDkwLlFFinv17rzfygp9ia90XC6NS478 lZLYmERk92GMlz4oGHgqIhb3hYta9MOC8Ilv0lIukbzCFe0O5PCxmEHV2wxuylr4B/ 8oH3thNg0MAIEEMfcU4o4l0DzaCBUtYmgJ4QEqEodk7aXQAIqToPsnyFXqu0i9Hpos aR1r03TE2/mtSmioXP2cUZolTPXNcH2xdvGfYob2ZZIPW58mcEfBCPtadQWw0iGrbj PFijLfuTAoegA== Date: Wed, 14 Jun 2023 14:23:39 +0100 From: Lee Jones To: Dave Hansen , Andy Lutomirski , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Lee Jones Subject: x86: pgtable / kaslr initialisation (OOB) help Message-ID: <20230614132339.GS3635807@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 2F20BC0011 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: moh1x87waw7rmi99har7cj5rh5om1y57 X-HE-Tag: 1686749026-176058 X-HE-Meta: U2FsdGVkX1+StrN6IP5XecEbZeAoB2L5hq8phVn5ML8KJn6zqgjshRfpvB9Pb6ykj1/DDJIQLx0fDQl4O3ZEp/9JYLvQhItqevwPMbUggCdiscEXOPQHTaPaikUMmP0hAjIF2LYOK5f9UVJZkezOMqeStKOO1akTvyZkv1lv7Mtys/2dLSj0SexpImErqFQ8g7uhPcoKfzGhhR7COLBwiCPEnMSvLzkUVGfGzXc745jRLKU6TkVPjCWz0kYOH38Y2L/IOfZGKiu5UFC+9H+dTGBiUycg+c8akXFajqF6XacX48S45jsX115ZBdIAvmxR43lgAtzh/26yDEoK+hvpyqLNP6SV3HPYlCyMSOuMvGWT70azb35DN5iy3HyH1PIXpF0aCP/f212HFLX4nCsY9CcrPEhQFtCNGrKs2qOeIFvIWxCAeVKUl8/6UDNdtnSEG3mmu8wUP2AVRvf4D3jqftOBEFxNz3BtJyniwrLQVgi4fVgickGFIdfqKSMC5iWYc8RBeCAkuyZpfr+Hp8jLNtV6L0jdyNxA5wWISFW4SjM5Xnoo89zoIfCwtMwwcxQRhhP2mhbLJTvjJC8BR3VkZStJJiDkGDcIZkgKonXl3A9gzBIETOVS1xFM6vKGPqU74S2B/GtTMJpNL/DIkrAL3kc8n82XDNRkK1vMMMaxF9s6v3G3fyPjsvbvYa5QMISQhclK52feEmaYY6zROYMVQ9q/E6extyUwTM9ASKNGelysaF/tMFp16UgtLOwih6rf/jzKtJZ6T9AqNffXxGKKuFPALQ01QWM5nbS9hf5YZZCs1CIZ8b3AGQnUvfrosBM5IN6wRdLIijaeMFTAIuSuThG7TCDH+LxcRgvf2W+wELqMyZHX+VoLNHY963Ku59pgoI4k6q5Nt2pPzC27K23dOoAyP7VcemkU+Zr5tgjWI6ZhbFp0Ka0WGkiHG5hMP89zAxqgUV5IOjgticS2V0Z ZRpTtIXW 70p21ETK7A3yAY5zhQ50JqGEk5Rp0LEVcKMhQ2p9the2L4OMGlxdymkcvf1Ie/9ORFGftNYbOixqtW7M8KiClZJ1G2xOXTi9wLkTT9vlS9Bi/hxytgHrwmc+78byTj6DI0585sjU1mQkE4eJMTnEdi6RDbdkglX2x3iScOPN2uJZP6P8kEkbjcguv8ThCtk6UySq1r8vyGmayO0NKpnGxDBl5NuvrDCCDS2+tYx9N1OVvPjGvMALYCPe95xxtu+ph8Zp58yqn2Et4zgOqwz61SO3GaD7yQc0DmXkjcIUqd6nlX7tDHhbs4FnhRw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Good afternoon, I'm looking for a little expert help with a out-of-bounds issue I've been working on. Please let me know if my present understanding does not quite match reality. - Report The following highly indeterministic kernel panic was reported to occur every few hundred boots: Kernel panic - not syncing: Fatal exception RIP: 0010:alloc_ucounts+0x68/0x280 RSP: 0018:ffffb53ac13dfe98 EFLAGS: 00010006 RAX: 0000000000000000 RBX: 000000015b804063 RCX: ffff9bb60d5aa500 RDX: 0000000000000001 RSI: 00000000000003fb RDI: 0000000000000001 RBP: ffffb53ac13dfec0 R08: 0000000000000000 R09: ffffd53abf600000 R10: ffff9bb60b4bdd80 R11: ffffffffbcde7bb0 R12: ffffffffbf04e710 R13: ffffffffbf405160 R14: 00000000000003fb R15: ffffffffbf1afc60 FS: 00007f5be44d5000(0000) GS:ffff9bb6b9cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000015b80407b CR3: 00000001038ea000 CR4: 00000000000406a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: __sys_setuid+0x1a0/0x290 __x64_sys_setuid+0xc/0x10 do_syscall_64+0x43/0x90 ? asm_exc_page_fault+0x8/0x30 entry_SYSCALL_64_after_hwframe+0x44/0xae - Problem The issue was eventually tracked down to the attempted dereference of a value located in a corrupted hash table. ucounts_hashtable is an array of 1024 struct hlists. Each element is the head of its own linked list where previous ucount allocations are stored. The [20]th element of ucounts_hashtable was being consistently trashed on each and every boot. However the indeterminism comes from it being accessed only every few hundred boots. The issue disappeared, or was at least unidentifiable when !(LTO=full) or when memory base randomisation (a.k.a. KASLR) was disabled, rending GDB all but impossible to use effectively. The cause of the corruption was uncovered using a verity of different debugging techniques and was eventually tracked down to page table manipulation in early architecture setup. The following line in arch/x86/realmode/init.c [0] allocates a variable, just 8 Bytes in size, to "hold the pgd entry used on booting additional CPUs": pgd_t trampoline_pgd_entry; The address of that variable is then passed from init_trampoline_kaslr() via a call to set_pgd() [1] to have a value (not too important here) assigned to it. Numerous abstractions take place, eventually leading to native_set_p4d(), an inline function [2] contained in arch/x86/include/asm/pgtable_64.h. >From here, intentionally or otherwise, a call to pti_set_user_pgtbl() is made. This is where the out-of-bounds write eventually occurs. It is not known (by me) why this function is called. The returned result is subsequently used as a value to write using the WRITE_ONCE macro. Perhaps the premature write is not intended. This is what I hope to find out. A little way down in pti_set_user_pgtbl() [3] the following line occurs: kernel_to_user_pgdp(pgdp)->pgd = pgd.pgd The kernel_to_user_pgdp() part takes the address of pgdp (a.k.a. trampoline_pgd_entry) and ends up flipping the 12th bit, essentially adding 4k (0x1000) to the address. Then the part at the end assigns our value (still not important here) to it. However, if we remember that only 8 Bytes was allocated (globally) for trampoline_pgd_entry, then means we just stored the value into the outlands (what we now know to be allocated to another global storage user ucounts_hashtable). - Ask Hopefully I haven't messed anything up in the explanation here. Please let me know if this is the case. I'm keen to understand what the intention was and what we might do to fix it, if of course this hasn't already been remedied somewhere. As ever, any help / guidance would be gratefully received. Kind regards, Lee [0] https://elixir.bootlin.com/linux/latest/source/arch/x86/realmode/init.c#L18 [1] https://elixir.bootlin.com/linux/latest/source/arch/x86/mm/kaslr.c#L178 [2] https://elixir.bootlin.com/linux/latest/source/arch/x86/include/asm/pgtable_64.h#L142 [3] https://elixir.bootlin.com/linux/latest/source/arch/x86/mm/pti.c#L142 -- Lee Jones [李琼斯]