From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EF366C282CD for ; Mon, 3 Mar 2025 14:27:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D3446B007B; Mon, 3 Mar 2025 09:27:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 65BE96B0083; Mon, 3 Mar 2025 09:27:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 523456B0085; Mon, 3 Mar 2025 09:27:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 33FEA6B007B for ; Mon, 3 Mar 2025 09:27:11 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C785680A1B for ; Mon, 3 Mar 2025 14:27:10 +0000 (UTC) X-FDA: 83180467020.21.59CED8A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id D6D5214000C for ; Mon, 3 Mar 2025 14:27:08 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jpeDY1sd; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741012029; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PTzadO2DnckvRJ6jl7j+mB7ira4oDWZGsSy/iOCEG/E=; b=AHLeDRwIJEKy6TE4YHpfk6dxBVKLKnWjiZGR67ncoay1RaRbiFwTu+beMpErSDPtd2ota+ jd+nS3Mf7K/VqL6STtNuqCoF4OT25MsCvLrlnkXUYpkvR4Rr2BiQzehoxIHl8bQzHHn/xn 6TLX3qcf44Nah4AQlylR59+mD/so5U0= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jpeDY1sd; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741012029; a=rsa-sha256; cv=none; b=7wYLrQFCNG1Rov5EMtLc97/gNO27uXjR4bsXDjs59zV7t5txxwriAfvlyaNUxW3YZdAhYa sQdIffrjj46OUMsN2qiUppM7Go6sERFyjWtauvEyIkmJKdSEKwJe4tsnJJzZkUE/zPnz/q 0c0PPg+vWgmVTyMkFi08IiHAXqLpELQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Transfer-Encoding: Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Sender:Reply-To:Content-ID:Content-Description; bh=PTzadO2DnckvRJ6jl7j+mB7ira4oDWZGsSy/iOCEG/E=; b=jpeDY1sdsl7kdGEJjFfCB8tVfI rpzWNIEMkPCYt9vCLtKaZ7RzjBk7/QIkx7YExiVJi3aR1tXLj15uRKWtufG8fVXJQOIvTrH+S77Ra 27MoDAoLk51t9BBE+VNq7e8zZ3rzh1Zd6cUNrgbO8wQo+ef/Ovx4eW2DoHNPgfGh4x/xN5NEuMqKG O5Mnvnfo2wEj7aj1i7PufG3faSwYi0WrIkYaq59bUAT1ESdRwxYbgAx3AYYQ39Sr5NljXJ0ZiArD/ FaRPqJCKRQEPFj4WPyAkD3CMDrxcypdnQ75u6B4lA5aEwgCBOBkSNPcKWzmPG6hXvfm/REohuVoxj r8O8RoOA==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tp6l8-0000000Bkxo-1gBn; Mon, 03 Mar 2025 14:27:06 +0000 Date: Mon, 3 Mar 2025 14:27:06 +0000 From: Matthew Wilcox To: Hannes Reinecke Cc: Sagi Grimberg , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , linux-mm@kvack.org Subject: Re: Kernel oops with 6.14 when enabling TLS Message-ID: References: <08c29e4b-2f71-4b6d-8046-27e407214d8c@suse.com> <509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com> X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: D6D5214000C X-Stat-Signature: 6i3osrx1dzt6rp5a7sm7twbmyz7facib X-HE-Tag: 1741012028-479771 X-HE-Meta: U2FsdGVkX1/1qeHuL+xnTgeOnUpnZuAjKKU6yE4Jn/dlpGITYvwwdtJp+e9NE/ww0wzxJuGVGMBD8y4m++cJIQCOenuHHd4Dv8IGP9fa305eiRENZUi8dfzrC9E4eWuxrzwU4K15cXa8XNK9kUzMP9UGhjt+Qo6Oldzi+/2UwSNv00N/Zl2npVkJTVk7wuRA+EEaqDMGOlSCi5sjRDyTzmqfXZrD4iiEuvF203rYfkMA7TVNpTNRnTs+xnw/Ig3EqmfV6D7zi2PEyghBHOYZRnspkcSQBHTslYpq+LK1LXt5WbkW82IMlu7ffPvdUJ93UMU4ra8nZgGvJmy+uD9HJtcbLBFh1M8qLDyxrXcvYma9Qo7xSWZ2jYwipxzsenzrhXqeYvZVxB7MCEBaIof7wgysA4yxbNclV9pKy4vkYOxEiDZ6Kd2RfYtvLRzSV8W9HAGqZXN5D3EoPWpdtsbP0nQqyoPs6UR8vIWA2mpY5wTzbxO691E+FMcqne8B/Y9J91HiqfcaUC1m1f0YDtwOHDGG4CWU6F7f6mZgOJ23Pv4YOZ5meHjr5uTQWSSJUTPqkeSXWXlW05pXJQgA/KmTFLlBFSFNb7sdVnYlIczHWpGPoGW4RhY/h6+EugUFq49/SEvIEZDtRpHKCMrGRpi7GoR9VFYoBP991L9H5mo1BOgm39Jml5OJwYJBNUoVAInpe2U8NwJZUSs+Ne5spLhZjSEr5WIe/kqQJOGfxv3Ovr3RPL7VMzVt62pMZl+dbmj5/DZ4pvaEET62OCfbDFHX2oTfvGMUc1kkjjDt8VSTPLcrvjjAiWpIY5/mC9keTrejwPFD0bGK//AuTKDMdeE/C5yHWpqfUgmAJ9iogiv6mu80wLdIJJM3PsiSe28eiWa5aqRK8oWvqCZyaw0DVpZUNejp9m6cuEtj7q8vgfq5xYlVJkh1z8mua34w6scUnCZQ5vp0j7jxgUVt82i+XF8 U6MBIz4F 2+rvdGAKw0pDRoIxeBF5GQ1VdkvzcnsLsrSRTZsLpzxAuFftsEKGozFilZAPki62C3vmualOTd+fsCt1wNS00rURzugPWOpgm4hVFBG6ga26TLWeGM1V24EZnqaaqvsGZXM1Z0iTARtOHCtpCfOLPthLOfNV7SO37nKnKNGsJPApfoaYXKARYeViqk5mNCyZGsJ44UF9zBG5J6ZFtVB86rnRB94cUZxewp2DYq82S34JrHyMwy1Sgc2qlg1Z78CwSxzNUJODfAyL8zB6diFhSBBrVygbRYLhDgeJ6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Mar 03, 2025 at 08:48:09AM +0100, Hannes Reinecke wrote: > On 2/28/25 11:47, Hannes Reinecke wrote: > > Hi Sagi, > > > > enabling TLS on latest linus tree reliably crashes my system: > > > > [  487.018058] ------------[ cut here ]------------ > > [  487.024046] WARNING: CPU: 9 PID: 6159 at mm/slub.c:4719 > > free_large_kmalloc+0x15/0xa0 That's: if (WARN_ON_ONCE(order == 0)) pr_warn_once("object pointer: 0x%p\n", object); And while the object pointer is obfuscated (hashed pointers), this wouldn't be helpful in trying to track down the problem. Perhaps we could make this a VM_WARN_ON_ONCE_FOLIO() so we get the dump_page()? I'm tempted to believe this is a double-free, but then I'm not sure why it'd be triggered by this patch. > > [  487.296801]  kfree+0x234/0x320 > > [  487.332084]  nvmf_connect_admin_queue+0x105/0x1a0 [nvme_fabrics > > 34d997d53c805aa2fae8e8baee6a736e8da38358] > > [  487.332093]  nvme_tcp_start_queue+0x18f/0x310 [nvme_tcp > > 68f6be106f52ac467179f8a0922f02aeb6fa1f1c] > > [  487.332102]  nvme_tcp_setup_ctrl+0xf8/0x700 [nvme_tcp > > 68f6be106f52ac467179f8a0922f02aeb6fa1f1c] > > [  487.394495]  nvme_tcp_create_ctrl+0x2e3/0x4d0 [nvme_tcp > > 68f6be106f52ac467179f8a0922f02aeb6fa1f1c] > > [  487.394503]  nvmf_dev_write+0x323/0x3d0 [nvme_fabrics > > 34d997d53c805aa2fae8e8baee6a736e8da38358] > > [  487.394514]  vfs_write+0xd9/0x430 > > [  487.551642] object pointer: 0x00000000346cb6fc Oh, wait, that's not the crash! We continue to free the folio. Even though we hit the "can't happen" case. That's dangerous. > > [  489.405197] Oops: general protection fault, probably for non- > > canonical address 0xdead000000000100: 0000 [#1] PREEMPT SMP NOPTI I think we all recognise that as list poison. I bet this is a double-free. Or it could be a wild-free. I mean, look at kfree(): folio = virt_to_folio(object); if (unlikely(!folio_test_slab(folio))) { free_large_kmalloc(folio, (void *)object); return; } So if you call kfree() on a random pointer, chances are it's not part of slab, and we jump into the free_large_kmalloc() path. We have a _lot_ of page types available. We should mark large kmallocs as such. I'll send a patch to do that.