MCEs on MIPS: multiple matching TLB entries

[Yu Zhao <yuzhao@google.com>]

Hi,

OpenWrt folks ran into MCEs caused by multiple matching TLB entries
[1], after they updated their kernel from v6.1 to v6.6.

I reported similar crashes previously [2], on v6.4. So they asked me
whether I'm aware of a fix from the mainline, which I am not.

I took a quick look from the MM's POV and found nothing obviously
wrong. I'm hoping they have better luck with the MIPS experts.

Thanks!

[1] https://github.com/openwrt/openwrt/pull/15635
[2] https://lore.kernel.org/linux-mm/CAOUHufbAjZd4Mxkio9OGct-TZ=L0QRG+_6Xa7atQVFN_4ez86w@mail.gmail.com/

Copying and pasting one of the crashes from OpenWrt:

CFE for WNR3500L version: v1.0.36
Build Date: Tue Aug 11 15:09:14 CST 2009
Init Arena
Init Devs.
Boot partition size = 262144(0x40000)
Found a 8MB ST compatible serial flash
et0: Broadcom BCM47XX 10/100/1000 Mbps Ethernet Controller 5.10.56.28
CPU type 0x19740: 453MHz
Tot mem: 65536 KBytes

Device eth0:  hwaddr 00-FF-FF-FF-FF-FF, ipaddr 192.168.1.1, mask 255.255.255.0
        gateway not set, nameserver not set
too long file.
LZMA boot failed
Loader:raw Filesys:raw Dev:flash0.os File: Options:(null)
Loading: .. 3808 bytes read
Entry at 0x80001000
Closing network.
Starting program at 0x80001000
[    0.000000] Linux version 6.6.35 (user@connors)
(mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 13.3.0
r25518+987-f7a68458b4) 13.3.0, GNU ld (GNU Binutils) 2.42) #0 Sun Jun
23 09:14:12 2024
[    0.000000] CPU0 revision is: 00019740 (MIPS 74Kc)
[    0.000000] bcm47xx: Using bcma bus
[    0.000000] (NULL device *): bus0: Found chip with id 0x4716, rev
0x01 and package 0x0A
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases,
linesize 32 bytes
[    0.000000] This processor doesn't support highmem. -65536k highmem ignored
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000]   HighMem  empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000003ffffff]
[    0.000000] Kernel command line: noinitrd console=ttyS0,115200
[    0.000000] Dentry cache hash table entries: 8192 (order: 3, 32768
bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 2, 16384
bytes, linear)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 16240
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 56672K/65536K available (5819K kernel code,
596K rwdata, 1244K rodata, 204K init, 297K bss, 8864K reserved, 0K
cma-reserved, 0K highmem)
[    0.000000] NR_IRQS: 256
[    0.000000]  bcm47xx_soc: bus0: Core 0 found: ChipCommon (manuf
0x4BF, id 0x800, rev 0x1F, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 1 found: IEEE 802.11 (manuf
0x4BF, id 0x812, rev 0x11, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 2 found: GBit MAC (manuf
0x4BF, id 0x82D, rev 0x00, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 3 found: MIPS 74K (manuf
0x4A7, id 0x82C, rev 0x01, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 4 found: USB 2.0 Host (manuf
0x4BF, id 0x819, rev 0x04, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 5 found: PCIe (manuf 0x4BF, id
0x820, rev 0x0E, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 6 found: DDR1/DDR2 Memory
Controller (manuf 0x4BF, id 0x82E, rev 0x01, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 7 found: Internal Memory
(manuf 0x4BF, id 0x80E, rev 0x07, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Core 8 found: I2S (manuf 0x4BF, id
0x834, rev 0x00, class 0x0)
[    0.000000]  bcm47xx_soc: bus0: Found M25P64 serial flash (size:
8192KiB, blocksize: 0x10000, blocks: 128)
[    0.000000]  bcm47xx_soc: bus0: Early bus registered
[    0.000000] MIPS: machine is Netgear WNR3500L
[    0.000000] bcm47xx: Setting up vectored interrupts
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 8438235966 ns
[    0.000003] sched_clock: 32 bits at 227MHz, resolution 4ns, wraps
every 9481163773ns
[    0.009630] Calibrating delay loop... 226.09 BogoMIPS (lpj=1130496)
[    0.080067] pid_max: default: 32768 minimum: 301
[    0.098070] Mount-cache hash table entries: 1024 (order: 0, 4096
bytes, linear)
[    0.098182] Mountpoint-cache hash table entries: 1024 (order: 0,
4096 bytes, linear)
[    0.115630] RCU Tasks Trace: Setting shift to 0 and lim to 1
rcu_task_cb_adjust=1.
[    0.119327] clocksource: jiffies: mask: 0xffffffff max_cycles:
0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.119449] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.127277] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.147294] clocksource: Switched to clocksource MIPS
[    0.162062] NET: Registered PF_INET protocol family
[    0.162678] IP idents hash table entries: 2048 (order: 2, 16384
bytes, linear)
[    0.164718] tcp_listen_portaddr_hash hash table entries: 1024
(order: 0, 4096 bytes, linear)
[    0.167040] Table-perturb hash table entries: 65536 (order: 6,
262144 bytes, linear)
[    0.167138] TCP established hash table entries: 1024 (order: 0,
4096 bytes, linear)
[    0.167258] TCP bind hash table entries: 1024 (order: 1, 8192 bytes, linear)
[    0.167386] TCP: Hash tables configured (established 1024 bind 1024)
[    0.168124] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.168379] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.169663] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.169984] PCI: CLS 0 bytes, default 32
[    0.201695]  bcm47xx_soc: bus0: PCIEcore in host mode found
[    0.201712]  bcm47xx_soc: bus0: This PCIE core is disabled and not working
[    0.203394] gpio gpiochip0: Static allocation of GPIO base is
deprecated, use dynamic allocation.
[    0.204527]  bcm47xx_soc: bus0: Bus registered
[    0.230753] workingset: timestamp_bits=14 max_order=14 bucket_order=0
[    0.233286] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.233331] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME)
(CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.244147] Serial: 8250/16550 driver, 2 ports, IRQ sharing enabled
[    0.246592] printk: console [ttyS0] disabled
[    0.267632] serial8250.0: ttyS0 at MMIO 0xb8000300 (irq = 2,
base_baud = 1250000) is a U6_16550A
[    0.267762] printk: console [ttyS0] enabled
[    0.793749] 3 bcm47xxpart partitions found on MTD device bcm47xxsflash
[    0.800497] Creating 3 MTD partitions on "bcm47xxsflash":
[    0.806143] 0x000000000000-0x000000040000 : "boot"
[    0.844873] 0x000000040000-0x0000007f0000 : "firmware"
[    0.853949] failed to parse "brcm,trx-magic" DT attribute, using default: -89
[    0.861402] 3 trx partitions found on MTD device firmware
[    0.866918] Creating 3 MTD partitions on "firmware":
[    0.872045] 0x00000000001c-0x000000000928 : "loader"
[    0.877127] mtd: partition "loader" doesn't start on an erase/write
block boundary -- force read-only
[    0.892292] 0x000000000928-0x00000024f800 : "linux"
[    0.897409] mtd: partition "linux" doesn't start on an erase/write
block boundary -- force read-only
[    0.911927] 0x00000024f800-0x0000007b0000 : "rootfs"
[    0.917042] mtd: partition "rootfs" doesn't start on an erase/write
block boundary -- force read-only
[    0.930529] mtd: setting mtd4 (rootfs) as root device
[    0.935780] 1 squashfs-split partitions found on MTD device rootfs
[    0.942210] 0x000000560000-0x0000007b0000 : "rootfs_data"
[    0.953347] 0x0000007f0000-0x000000800000 : "nvram"
[    0.976291] bgmac_bcma bcma0:2: Found PHY addr: 30 (NOREGS)
[    1.089641] b53_common: found switch: BCM53115, rev 8
[    1.095225] bgmac_bcma bcma0:2: Support for Roboswitch not implemented
[    1.104192] bgmac_bcma: Broadcom 47xx GBit MAC driver loaded
[    1.110909] bcm47xx-wdt bcm47xx-wdt.0: BCM47xx Watchdog Timer
enabled (30 seconds)
[    1.121569] NET: Registered PF_INET6 protocol family
[    1.146114] Segment Routing with IPv6
[    1.150302] In-situ OAM (IOAM) with IPv6
[    1.154877] NET: Registered PF_PACKET protocol family
[    1.160253] 8021q: 802.1Q VLAN Support v1.8
[    1.273304] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.282239] Freeing unused kernel image (initmem) memory: 204K
[    1.288306] This architecture does not have kernel memory protection.
[    1.294893] Run /sbin/init as init process
[    2.348011] init: Console is alive
[    2.352416] init: - watchdog -
[    3.465026] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.645823] usbcore: registered new interface driver usbfs
[    3.651824] usbcore: registered new interface driver hub
[    3.657587] usbcore: registered new device driver usb
[    3.678605] gpio_button_hotplug: loading out-of-tree module taints kernel.
[    3.722626] ehci-platform ehci-platform.0: EHCI Host Controller
[    3.728888] ehci-platform ehci-platform.0: new USB bus registered,
assigned bus number 1
[    3.737524] ehci-platform ehci-platform.0: irq 5, io mem 0x18004000
[    3.767369] ehci-platform ehci-platform.0: USB 2.0 started, EHCI 1.00
[    3.776631] hub 1-0:1.0: USB hub found
[    3.782776] hub 1-0:1.0: 2 ports detected
[    3.813642] ohci-platform ohci-platform.0: Generic Platform OHCI controller
[    3.820986] ohci-platform ohci-platform.0: new USB bus registered,
assigned bus number 2
[    3.829703] ohci-platform ohci-platform.0: irq 5, io mem 0x18009000
[    3.903777] hub 2-0:1.0: USB hub found
[    3.909946] hub 2-0:1.0: 2 ports detected
[    3.940493] kmodloader: done loading kernel modules from
/etc/modules-boot.d/*
[    3.959102] init: - preinit -
[    3.965715] Got mcheck at 800104ec
[    3.969240] CPU: 0 PID: 245 Comm: init Tainted: G           O       6.6.35 #0
[    3.976522] $ 0   : 00000000 00000001 fffd5000 00000001
[    3.981900] $ 4   : 00000004 00000003 00026edf 7fc8c008
[    3.987273] $ 8   : 00000000 00000001 0000001b 00000068
[    3.992635] $12   : 7fc8d600 81c2376c 81c2370c ffffff00
[    3.998007] $16   : 7fc8d000 8192dad0 00000000 806ed280
[    4.003379] $20   : 8192dad0 00000001 807b1000 7fc8d7c0
[    4.008751] $24   : 77d75000 ffffffff
[    4.014122] $28   : 81ad6000 81ad7dc8 00000000 80017678
[    4.019486] Hi    : 00000071
[    4.022428] Lo    : 0ceb0000
[    4.025369] epc   : 800104ec __kmap_pgprot+0xdc/0x108
[    4.030547] ra    : 80017678 r4k_flush_cache_page+0x24c/0x29c
[    4.036430] Status: 1120a402 KERNEL EXL
[    4.040455] Cause : 00800060 (ExcCode 18)
[    4.044541] PrId  : 00019740 (MIPS 74Kc)
[    4.048543] CPU: 0 PID: 245 Comm: init Tainted: G           O       6.6.35 #0
[    4.055816] Stack : 00000000 00000001 807b1000 800625ac 00000000
00000004 00000000 00000000
[    4.064399]         81ad7c8c 807b0000 80780000 8064d57c 81ac1528
00000001 81ad7c30 860f9307
[    4.072985]         00000000 00000000 8064d57c 81ad7b70 ffffefff
00000000 00000000 ffffffea
[    4.081562]         00000000 81ad7b7c 000000a9 806f5a88 00000000
8064d57c 00000000 806ed280
[    4.090140]         8192dad0 00000001 807b1000 7fc8d7c0 00000018
80324b6c 89052010 00000060
[    4.098718]         ...
[    4.101229] Call Trace:
[    4.103730] [<80006fd0>] show_stack+0x28/0xf0
[    4.108219] [<8057ec8c>] dump_stack_lvl+0x38/0x60
[    4.113077] [<80008108>] do_mcheck+0x2c/0xa0
[    4.117462] [<80003d34>] handle_mcheck_int+0x3c/0x48
[    4.122547]
[    4.124072] Index    : 3
[    4.126661] PageMask : 0
[    4.129241] EntryHi  : fffd4000
[    4.132447] EntryLo0 : 00026edf
[    4.135652] EntryLo1 : 00026edf
[    4.138848] Wired    : 4
[    4.141430]
[    4.142954] Index:  0 pgmask=4kb va=fffd4000 asid=00
[    4.142954] [pa=007af000 c=3 d=1 v=1 g=1] [pa=007af000 c=3 d=1 v=1 g=1]
[    4.154868] Index:  1 pgmask=4kb va=fffd2000 asid=00
[    4.154868] [pa=0105b000 c=3 d=1 v=1 g=1] [pa=0105b000 c=3 d=1 v=1 g=1]
[    4.166775] Index:  2 pgmask=4kb va=fffd0000 asid=00
[    4.166775] [pa=01018000 c=3 d=1 v=1 g=1] [pa=01018000 c=3 d=1 v=1 g=1]
[    4.178682] Index:  3 pgmask=4kb va=fffd4000 asid=00
[    4.178682] [pa=009bb000 c=3 d=1 v=1 g=1] [pa=009bb000 c=3 d=1 v=1 g=1]
[    4.190590] Index: 19 pgmask=4kb va=80026000 asid=00
[    4.190590] [pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
[    4.202496] Index: 28 pgmask=4kb va=80038000 asid=00
[    4.202496] [pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
[    4.214400] Index: 29 pgmask=4kb va=8003a000 asid=00
[    4.214400] [pa=00000000 c=0 d=0 v=0 g=0] [pa=00000000 c=0 d=0 v=0 g=0]
[    4.226312]
[    4.227840] Code: 40843000  40850000  000000c0 <42000002> 000000c0
40875000  10600002  41606000  41606020
[    4.237866] Kernel panic - not syncing: Caught Machine Check
exception - caused by multiple matching entries in the TLB.
[    4.248920] Rebooting in 1 seconds..
[    5.250906] bcm47xx: Please stand by while rebooting the system...
Decompressing..........done