From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5D82C282D1 for ; Mon, 3 Mar 2025 15:48:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D9A876B0083; Mon, 3 Mar 2025 10:48:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D499F6B0085; Mon, 3 Mar 2025 10:48:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C108E280001; Mon, 3 Mar 2025 10:48:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9E8EB6B0083 for ; Mon, 3 Mar 2025 10:48:56 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B700E140B94 for ; Mon, 3 Mar 2025 15:48:55 +0000 (UTC) X-FDA: 83180673030.05.22F41C0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 79E7D8001D for ; Mon, 3 Mar 2025 15:48:53 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KF4dtobH; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741016934; a=rsa-sha256; cv=none; b=Iak5QTwBbULiYkYHHjIQ/w8KYBkFJ6+Y0YbhUzQTDos6BXWOICx38UTy3wgTS4UrqAthDW xsnaxpjB6fhBrtfQLt6uprTf60jBNW1FCdFKZ5SPobQoBz4wG5zhHf2g78b5+g34kLovES JxqtHtutRZxuqXCLIV1eEPJzsBmshTk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=KF4dtobH; dmarc=none; spf=none (imf30.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741016934; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oBr3Q35PYOX8KLcyP5TMvc9JdpscxzNbBR95ep2biiE=; b=58RLgSD1KaEZtQLRmzWAV96VV/W1UpkLQ4DgFBmFqYkhu8Of7C87DEh3F7BTNiswm8ICBQ CjVWK0xdPETbYtF8Qf3HBDNxPb7YQ6l6CImbc/sLPKwq+nzuWkWIidjn10tm5KkK/NJUcU uKUkHGJJBSaTMln9cAv8tNnxl49lCAA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=oBr3Q35PYOX8KLcyP5TMvc9JdpscxzNbBR95ep2biiE=; b=KF4dtobHd1vxFyKHtfGmYLTnJu 3geMPg49K4GbOBm2pLeLDxmdEyM8wxBYxMf0A6ZCUckskxQC8SObjg7eTENqRnD+xezhPXhoVYHt7 BIa6qhmEz9fV+4iAUt8AtSB0mCb9bo/tn0/azePHbgZeoTb9n1epLw+QhwDJCzecaA/PCHoFKXcTX YzLAysAqb1fhfLjGJtAwfR99COhkmANTeVDCN0GV88kJod+UULBOqZjVdywdyIZq8Xtd7uUnlRtjW bHDeAxxeCqrp+xFeD2K166PKbV/B9nTwf9tEdMJHfR7IP0C7C7IqpVZ1OCAMaIrfuxbxcT6EvnVve YABkLQIw==; Received: from willy by casper.infradead.org with local (Exim 4.98 #2 (Red Hat Linux)) id 1tp82C-0000000BqnG-2N01; Mon, 03 Mar 2025 15:48:48 +0000 Date: Mon, 3 Mar 2025 15:48:48 +0000 From: Matthew Wilcox To: Hannes Reinecke Cc: Vlastimil Babka , Sagi Grimberg , "linux-nvme@lists.infradead.org" , "linux-block@vger.kernel.org" , linux-mm@kvack.org Subject: Re: Kernel oops with 6.14 when enabling TLS Message-ID: References: <08c29e4b-2f71-4b6d-8046-27e407214d8c@suse.com> <509dd4d3-85e9-40b2-a967-8c937909a1bf@suse.com> <15be2446-f096-45b9-aaf3-b371a694049d@suse.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <15be2446-f096-45b9-aaf3-b371a694049d@suse.com> X-Rspamd-Queue-Id: 79E7D8001D X-Stat-Signature: 4jm3hnm8y7bqysqe3darnqf5uf5qqaj3 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1741016933-556676 X-HE-Meta: U2FsdGVkX18pIttgsEg41SH7UIlQ8BTp/Dc4dhBaA3a1usj/M/ax0XooN7EHwiNlWJogw7Qnx0MVXlPEisweAlRrGPME5MTVtLekvYBKqgiRoj5IfS2bbVVyxZnv3h3uu7SOdU1YCId+Ihw641Y4oO5KVNNU/8BuVgdpA/6sQbUcWbDxx33Io4h/h/C8xI+9wzU50lOpmFvujKoD1I4ytfWegtFsuU/0LryKHTIkXuTS5OnD+lCo5Lk/SWyp5tg1BZQTjZTBZK9ehhxn7oyojGEJl+VcTwjeA7yenWPYQRFQW20Yh6WtiQCXANLI59TcdYtwfJET89+6JzqrIDQYk4KfUTpfhPpwQVyuA0bGQm2Ax7rprg94Gwvw7FOIb5+RLXAx7jfMm0+QB9F7XaZEs+DAAtdZXIZ0SgfSzNQNS11DgJzuKjwx9NvfzhmYVWzOaHoeTNpEsQ6UgXWuTqnu8gRYxplON6X+1IfYFwc6kbMoQ0hEUJFO3U/8dF9S23Dgh0xYH5x6U81ssSzJ2DMpq5NWHlMjS5bYebNHBV2qrAgCjY90S9vESoSm0WQGAKwtomH50R1/OoG/lad5hZWvYcwF9dEAHFgq9RsK4HhMTATLuP6LgFLkWhwyFgmr4GoUPyIm71gUZtxlpdnroq08nieE9Wr1+N3q8dzlfXhKlBrrdxJjpRbhv1anYbbudcJobIONFGc1uKcLLkRI8ktqqtsffBJjq4Aga9l0EsFs6lNUjJ5mOLmdvRxm3rU0m5OT2PZIBVpagVnW+4+QGQ7zAmoZfg+MN3NQipvtMTZWLwcw9ecoCTtY1AaOA3iEK9pRcrv+Wx3fBBVWGKh1sLSvH5t1gSYHcxXP83Xy+imgIfLY4D12MiYU4HEvbBjzZNv7I+KT4eK1us4ScHAOqTHtoCDkyJ8aHYIPVU6ceEgplgF94+BWV20dllrcnnzEEdDstMDAgxgkEnQaZrwukUy r2ON6/nA fdPdMSu5QiM/MCkzlYymf7p7tSqkmtbJtTF0lMtqZz8aZE2Cenz6O0EAwaAfhbDvkaBMCJa2aqJ5aLjfOXzlypOoivCi9jIImVda0pn/h5xYOzD4di+1nlq+OAgaRidc/ND5PECPvDrkOswwbI730yexnFYCdNtnjzOtVtMAtPR9rJS9iFQICm5TUXCqfkRLM7ngSv6t5Zp6rNuogU5Rc+UxssRYrYYEcqI3Sdnezn7dnWNI4gAiXZaR/b5RTm2wVe2sM80kRizJDHhE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Mar 03, 2025 at 04:39:47PM +0100, Hannes Reinecke wrote: > On 3/3/25 15:42, Matthew Wilcox wrote: > > On Mon, Mar 03, 2025 at 02:27:06PM +0000, Matthew Wilcox wrote: > > > We have a _lot_ of page types available. We should mark large kmallocs > > > as such. I'll send a patch to do that. > > > > Can you try this? It should fix the crash, at least. Not sure why the > > frozen patch triggered it. > > Still crashes: It warns, but doesn't crash! This is an improvement. > [ 63.658068] WARNING: CPU: 6 PID: 5216 at mm/slub.c:4720 > free_large_kmalloc+0x89/0xa0 > [ 63.667728] RIP: 0010:free_large_kmalloc+0x89/0xa0 > [ 63.842773] Call Trace: > [ 63.934398] kfree+0x2a5/0x340 > [ 63.987632] nvmf_connect_admin_queue+0x105/0x1a0 [nvme_fabrics > 18bfa9223bf0bd1ec571f5f45774adcc919a867e] > [ 63.987641] nvme_tcp_start_queue+0x192/0x310 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 63.987649] nvme_tcp_setup_ctrl+0xf8/0x700 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 64.043323] nvme_tcp_create_ctrl+0x2e3/0x4d0 [nvme_tcp > a0629454ac5200d03b72a09e4d2b1e27dfa113e9] > [ 64.043332] nvmf_dev_write+0x323/0x3d0 [nvme_fabrics > 18bfa9223bf0bd1ec571f5f45774adcc919a867e] > [ 64.043344] vfs_write+0xd9/0x430 > [ 64.108458] ---[ end trace 0000000000000000 ]--- > [ 64.108461] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x2 pfn:0x5e3a > [ 64.108465] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 64.108469] raw: 000fffffc0000000 0000000000000000 fffb0b48c0178e90 > 0000000000000000 > [ 64.108472] raw: 0000000000000002 0000000000000000 00000000ffffffff > 0000000000000000 > [ 64.108473] page dumped because: Not a kmalloc allocation Right. So you called kfree() on something that isn't currently kmalloced memory. Either it used to be kmalloced memory and we freed the slab that it used to be in, or it's a wild pointer. Whichever it is, that's a bug in the caller, not in slab. Why it bisected to that commit, I can't say. Maybe it changed the timing, or maybe it was just luck (whether the allocation which is now being freed is the last allocation in the slab or not). > [ 66.084156] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x0 pfn:0x5de5 > [ 66.093770] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.101810] raw: 000fffffc0000000 0000000000000000 dead000000000122 > 0000000000000000 > [ 66.111311] raw: 0000000000000000 0000000000000000 00000000ffffffff > 0000000000000000 > [ 66.111314] page dumped because: Not a kmalloc allocation > [ 66.112001] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0xdc pfn:0x5de3 > [ 66.137452] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.137460] raw: 000fffffc0000000 ff45d9a24d93f420 ff45d9a24d93f420 > 0000000000000000 > [ 66.137464] raw: 00000000000000dc 0000000000000000 00000000ffffffff > 0000000000000000 It happened again ;-) > [ 66.137466] page dumped because: Not a kmalloc allocation > [ 66.138095] page: refcount:0 mapcount:0 mapping:0000000000000000 > index:0x0 pfn:0x5de5 > [ 66.180944] flags: 0xfffffc0000000(node=0|zone=1|lastcpupid=0x1fffff) > [ 66.180950] raw: 000fffffc0000000 ff45d9a24da3f420 ff45d9a24da3f420 > 0000000000000000 > [ 66.180953] raw: 0000000000000000 0000000000000000 00000000ffffffff > 0000000000000000 > [ 66.180954] page dumped because: Not a kmalloc allocation And again ... > [ 66.181672] BUG: unable to handle page fault for address: > ff40e4ea8fa50250 Oh, now it crashed. But we have so much evidence of a bug in the caller at this point that I don't think we can blame slab for falling over. If you're double-freeing something that's _not_ in a freed slab, this is the kind of thing we might expect? You need to turn on the debugging options Vlastimil mentioned and try to figure out what nvme is doing wrong.