From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C63C1CAC5BB for ; Sun, 28 Sep 2025 15:09:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D26438E0015; Sun, 28 Sep 2025 11:09:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD6E28E0001; Sun, 28 Sep 2025 11:09:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9E9D8E0015; Sun, 28 Sep 2025 11:09:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A47DE8E0001 for ; Sun, 28 Sep 2025 11:09:35 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 57CDD1D9262 for ; Sun, 28 Sep 2025 15:09:35 +0000 (UTC) X-FDA: 83938993110.09.C04531E Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf14.hostedemail.com (Postfix) with ESMTP id D1B3A10000D for ; Sun, 28 Sep 2025 15:09:32 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Gg4EgO6D; spf=pass (imf14.hostedemail.com: domain of nilay@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=nilay@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1759072173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JNpeYR+IfQUFp0FlIyW2XW1Up7YQWvi6w8S3hBXfZ0Q=; b=WC3slEy38YD/4bN/3aED/bbWy2eDmqBTYpvGbWYdq5bU2WnDjptrjbotGvMYC0u5MPRNnZ mG1FeWo5ZxxeWw/Qh8vkV8pdMpiQ+eIT86sOcjHTfHVV98ufK2WqIpoqOZG4SAiJXkt+pS 8hOW2LBp1bJ2jvXugT32jpjuzvaQXag= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1759072173; a=rsa-sha256; cv=none; b=O/GrX02SthavqtBLwRMLzEyXOdoETvk21Kajen3i3tnTN2rMSfzQelTIYDBryUyur29j+j ZYR/xhdELI3fe/1re6tfHxAr7qk1/Q8ptiEYnZY1SBUmPy6h13/gdhOmIfnBXJhjCq4+WV B9rPg6xVkoxw/1ACKP1+NN1vNuNHIjQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=Gg4EgO6D; spf=pass (imf14.hostedemail.com: domain of nilay@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=nilay@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 58S7U4t2026791; Sun, 28 Sep 2025 15:09:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=JNpeYR +IfQUFp0FlIyW2XW1Up7YQWvi6w8S3hBXfZ0Q=; b=Gg4EgO6DgbVe6ofWZakAQF z4nxx0jfYGogJddxFPpBL2fm/ntBC8aWHuye7HTJunxC5SxziH2k9lrBxxwywh6S vS4vLitQNTFxjK/tpvvfYhFfzw05r580deVaVoJ6argfAi1fcqb/Ld7DwbTU2jjh r9rsX6fGlWS1Yv/BlSKpcA/XxI5AC2+Kb2XxOrlLiABr+s66GJHaTtZYkDSJKATP R8bQU1JhIHt3GUMXXV90kC99uu1EdcKn2WQqBORmiUlSlzDAuga3sGAOQlTece/f 49XWqgDxK+DsHNkkIqFrfT4xArJCwak99qKpbHiHOwX3vNqvhwvkqcPfN79P+7SQ == Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 49e7ktx0d5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 28 Sep 2025 15:09:30 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 58SCCvRv003314; Sun, 28 Sep 2025 15:09:29 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 49etmxj80x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 28 Sep 2025 15:09:29 +0000 Received: from smtpav04.wdc07v.mail.ibm.com (smtpav04.wdc07v.mail.ibm.com [10.39.53.231]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 58SF9T6N27656782 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 28 Sep 2025 15:09:29 GMT Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1CE0358045; Sun, 28 Sep 2025 15:09:29 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C56C158052; Sun, 28 Sep 2025 15:09:23 +0000 (GMT) Received: from [9.43.71.234] (unknown [9.43.71.234]) by smtpav04.wdc07v.mail.ibm.com (Postfix) with ESMTP; Sun, 28 Sep 2025 15:09:23 +0000 (GMT) Message-ID: Date: Sun, 28 Sep 2025 20:39:20 +0530 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [BUG] Double-free in blk_mq_free_sched_tags() after commit f5a6604f7a44 To: Ming Lei , Niklas Fischer Cc: linux-mm@kvack.org, linux-block@vger.kernel.org, vbabka@suse.cz, akpm@linux-foundation.org, axboe@kernel.dk References: <37087b24-24f7-46a9-95c4-2a2f3dced09b@niklasfi.de> Content-Language: en-US From: Nilay Shroff In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=T7WBjvKQ c=1 sm=1 tr=0 ts=68d94faa cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=IkcTkHD0fZMA:10 a=yJojWOMRYYMA:10 a=VwQbUJbxAAAA:8 a=BNlTfnLp_OdMHRcVVn4A:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=cPQSjfK2_nFv0Q5t_7PE:22 a=HhbK4dLum7pmb74im6QT:22 a=pHzHmUro8NiASowvMSCR:22 a=Ew2E2A-JSTLzCXPT_086:22 X-Proofpoint-GUID: OUPLr9phSgYtTV7jg1Aq2OZUhzGes3_m X-Proofpoint-ORIG-GUID: OUPLr9phSgYtTV7jg1Aq2OZUhzGes3_m X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwOTI3MDAyNSBTYWx0ZWRfXx2ileBIFQfyt MTweuzUniWlioSf9ptg9rzOr3DPcLp5rnPZ1DxGjR4YOHVg8n8okNlzreGYIjTB/vueABS0ay1u I914xBZFgQvKlpan6zDV03iPgcUozEIHx+D/SS2SI7EgJ/nIMggaa3Dhmdiw92fv49L9rqURoWT Xqj/+9ZkmZhhqBq+6fyz/Ug6K6qvEEdsuLnqJcJjmLKDCqZNsNNA5MS6svlFZa1nun2W4PrKL03 v5n5hx3gR4r6SKoDtd8b/cGm8E0/a4XtOOOsnFTVbN7XGq5o6Wl2cd6Utv6XMbY1720wwaEdne+ gz79d0+jczXD2EP6syUfv1HMFsOagLfnntVggC0lqh+RoxfmiRNcforXgl12mzLfEbMV6GGsbmF 8e1oTYDknkPigXYoovbz3kOJn2jGCw== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1117,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-09-28_05,2025-09-26_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1011 spamscore=0 suspectscore=0 priorityscore=1501 bulkscore=0 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2509150000 definitions=main-2509270025 X-Stat-Signature: dqiymujkjetew5fxrbh9dcmk9b63pcxt X-Rspamd-Queue-Id: D1B3A10000D X-Rspam-User: X-Rspamd-Server: rspam03 X-HE-Tag: 1759072172-753510 X-HE-Meta: U2FsdGVkX18JIQNpwOF1mFovJII2HCnO9TdzCG8m1unQvTZZFBisgtdQUId6aXWO5vWrzZBe+Loo0ogXWpIDMqL+4q1IgXbhEljkiVnwol8TYYPMQMsWpoUPe9TcE5/7YAcZihtFR3EKhjd9269aihjpPT5aRW5QvkvG8YxG5SOBPm318G9iKGZikwpY8YL4CS6Z6ZlFf4/lnS6pt+bcX+8N2OPgkZObzkKyhZ93xXgApo2zScTaSyxnQID7gnH27iQ+G2sHX+Bm5o14uOVnNq6cnGBxjq+BdjkOJUAoHrUGnkfxxwuiY6RhM7zcnaXyFG+tNIRp2Lr6jzpOo+26L1hm3fygccqFkmPNyy6qKwCrMLuCr+HAH75MN7XnxsWPL8fKYxbk2iZNH4ZRelCl34q724AGgCWjrx3og01xRRaW/sjGcBm7124gDGxiNp9AzV1znv+AMD8pE/3+++bn9PPmJ9iqqu5p4AIzMi8RIj6dsTByVy0r/WXK/pMJY4fwXLMLQISJ4GjjSmBDc+cfXQcN1yLshgOcJX9L2PUignN9lZGtpAnXebYwByYI5jHAPn9azlEuBuCZk/BnolXh+wXf85ZbYXVs1lGHh37B0QnwOaI652CiR8/YFQ4eY+v4G4edgSUfnEIvsO20fE/rI3+rV8u8r5IyQnysBzSIUni/Grsn4tO72FMXFxBFu6/R2qg+8mnjRn8F6zlExOce7T/+zpVTn4Gw3g0JSpnb9XYBhzYi1NBE0lNddaKDVYPqxSBD1C//6s4kma09IaiHRzETl2LqUlXdneKm+y3XjwYXOIsQ+6X0ZN+1mL7HFEjXuh1rtDpCsOX9KV7MQ4/8SBumBNNYppoIieFz4g3ASWC32h0c4hpuSvTFo2kzYx3DlbtcAxlYnmbV0e7Y7AQGIZ5pYCLILAsQ1lkM88AVuv0175Y7Gj1Ow4P2Cq0yiG/Iqg2ggIeuZgxYze05/LE mU4/WOjG TwgsRngETWOzgv7Dojc499aolqchakrAkLcp2IG8nuj6AALc1KrBzoz0LNQmZ3dMEOhOzAV0mm/u0DiGLBlgGghGKj3wzvOdcbLejRHHnMU2xlinSkf7CQsbPDhit+47tfK8Ek9rQEVKT8eGOOL6+m/DI9MYrMtblvxKTMltT/PgRnx93X3Hgubp0YcL1QuNhkrkV4Ddz/z2+/CuhO6UyVFZAZnhZA3KNrGHGvljUqyohKzNiE3w3WWJgF672ZQosA+o2QnGdew7dPuVtsKLJxX5Fgvm+Nm5Fc+3oiatgNhaIL7q/OeooRTShJE7vNtxLxnB7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 9/28/25 6:48 PM, Ming Lei wrote: > On Sun, Sep 28, 2025 at 8:18 PM Niklas Fischer wrote: >> >> Hello, >> >> I'm reporting a kernel crash that occurs during boot on systems with >> multiple storage devices. The issue manifests as a double-free bug in >> the SLUB allocator, triggered by block layer elevator switching code. >> >> === Problem Summary === >> >> The system crashes during early boot when udev configures I/O schedulers >> on multiple storage devices. The crash occurs in mm/slub.c with a >> double-free detection, traced back to blk_mq_free_sched_tags(). >> >> === Crash Details === >> >> Multiple crashes occur during boot, showing a severe race condition. >> Seven separate kernel oops/panics are observed: >> >> * Oops #1 (CPU 13, PID 928): General protection fault in >> kfree+0x69/0x3b0 - corrupted address 0x14b9d856a995288 >> * Oops #2-4, #6-7 (multiple CPUs/PIDs): kernel BUG at mm/slub.c:546 in >> __slab_free+0x111/0x2a0 - SLUB double-free detection >> * Oops #5 (CPU 1, PID 952): General protection fault in kfree+0x69/0x3b0 >> - corrupted address 0x2480af562995288 >> >> All crashes share the same call stack pattern: >> >> elv_iosched_store+0x149/0x180 >> elevator_change+0xdb/0x180 >> elevator_change_done+0x4a/0x1f0 >> blk_mq_free_sched_tags+0x34/0x70 >> blk_mq_free_tags+0x4b/0x60 >> kfree+0x334/0x3b0 <-- crash here >> >> === Bisection Results === >> >> I bisected the issue to this commit: >> >> commit f5a6604f7a4405450e4a1f54e5430f47290c500f > > It should be solved by the following commit: > > https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-6.18/block&id=ba28afbd9eff2a6370f23ef4e6a036ab0cfda409 > > Thanks, > Oh, I hadn’t noticed this message before sending my previous email. It’s quite possible that this could address the observed symptom, though I didn’t see any traces of nr_request being modified in the provided dmesg.txt. Nonetheless, this change could be a potential fix and may be worth trying out in a custom kernel. Thanks, --Nilay