From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE25AC001DC for ; Mon, 17 Jul 2023 14:35:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 772316B007D; Mon, 17 Jul 2023 10:35:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 722226B007E; Mon, 17 Jul 2023 10:35:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EA568D0001; Mon, 17 Jul 2023 10:35:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4CE916B007D for ; Mon, 17 Jul 2023 10:35:05 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 03B71C039E for ; Mon, 17 Jul 2023 14:35:04 +0000 (UTC) X-FDA: 81021350970.20.BABB9A2 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf08.hostedemail.com (Postfix) with ESMTP id C3B5C16000F for ; Mon, 17 Jul 2023 14:35:01 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jPSv5Sk5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of ritesh.list@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ritesh.list@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689604501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:references:dkim-signature; bh=4BYLcNQI8huu9QFMy9I1uV9Z2M9k8SXeF4zF9gh3CZU=; b=uUM7cnp1Vcz6mDb8+XyJNnqojCowAoGbsF7+s/Nb71yjGkt2nWOua3lPSkiTKft2Kq2FB/ 8tFRYV8KI8YVSYXU+r4218/Jcb4onTioL8pa+wy16ff4kdk79kDFcjMyDMOI3EYWkcYE+R /b7ElwhcskYr6eX89FOI8NVYMPrcfQI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=jPSv5Sk5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf08.hostedemail.com: domain of ritesh.list@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=ritesh.list@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689604501; a=rsa-sha256; cv=none; b=pMxdA7LOQTITfyfjJo1GpW9rZBsXqPBV42gJD2NBoxLmB6LrzyvzhlKzCFZcaW1rlruJyA MF8/tX31X1uVwXvijwyImP8pqV+c9r7s8qy6/gACyIvNHrxHoyZHFmFKbfqAKFb1bM0JWj JU4ZDw+FSi7BRUa5P09CbRGPxuaaGuE= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6686708c986so4692975b3a.0 for ; Mon, 17 Jul 2023 07:35:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689604500; x=1692196500; h=in-reply-to:subject:cc:to:from:message-id:date:from:to:cc:subject :date:message-id:reply-to; bh=4BYLcNQI8huu9QFMy9I1uV9Z2M9k8SXeF4zF9gh3CZU=; b=jPSv5Sk5pxmVc8p5jjDApn+MZrG0aK3Tzn7t9YcSUHTKnNZ1ADUAoGJXASUubASmEn WIQZelQhm2559jwQKDsREioZrkpoQRPaJZTIsiwPhnOAai2fgyPccFV7xlAIRT1XGHWd 9xUVxB/XdBj1+BOnFX4vWbjn6uhNRMR+FOrmt/sEz1Za8Xez5r9L+xzGGrluP33eLJG6 UqX43s+AvqaiaYuIG8eV9E8aeSpqf0rRIOdsh5MhPbopjsxBbmAnYQUkpDLYhBzyj5Du oJe8KI1E0YafvghyjQitXy9cya8WlgGbGSDoma0wTrwweyYP353kU6wOINFVHJj/cx3Y Rl+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689604500; x=1692196500; h=in-reply-to:subject:cc:to:from:message-id:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=4BYLcNQI8huu9QFMy9I1uV9Z2M9k8SXeF4zF9gh3CZU=; b=d/zT8faxKnOBz1znfi/uNTx9eZj9D7+AytWlOYFwJdO4nrIaWs0ReDFaII4lL6yyIm hx3t2QiYBLPa6DEtWyS1Z94Z2Lc31emjvdUd/EeF90cBWV2CqjkUoq3ligT6aTsNioDW 3Ycnfs9FMAv3KiUljcbKg01m0alNfU3n/fG2VD/8yk1vGkq7TaZR8x78ERIdFX0vX8ew rFXz+AajySCRgA++nzEcxOTRVqT/zpqBMG6jQBSCtHWahSJdbgyb4k/0giEkTsMJRTG9 QvSDFtdGqaAj7157Kl/ROYwyJoag3FM5VOLbPSb0UMa+Ugu5pqKvTqTfF4XRsUAMIBKu EC/g== X-Gm-Message-State: ABy/qLad2T4xHmIdaof6PZpFkSVvV5ajncHRcRsGqhIO4c5GxBr5vTZK BlMFnNjZRlSu4KkXlj9GI8k= X-Google-Smtp-Source: APBJJlHAPdcJ6KFA2sv9z3Ia/B+Z9Y4x09UnZq7fBkGBsVJ2BJn5QQ5Fk3EK5lO+9+2Glx7Z2FZW0Q== X-Received: by 2002:a05:6a00:2e05:b0:681:415d:ba2c with SMTP id fc5-20020a056a002e0500b00681415dba2cmr16138558pfb.31.1689604499783; Mon, 17 Jul 2023 07:34:59 -0700 (PDT) Received: from dw-tp ([49.207.232.207]) by smtp.gmail.com with ESMTPSA id x18-20020aa784d2000000b0068288aaf23esm11992747pfn.100.2023.07.17.07.34.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Jul 2023 07:34:59 -0700 (PDT) Date: Mon, 17 Jul 2023 20:04:54 +0530 Message-Id: <87jzuyobch.fsf@doe.com> From: Ritesh Harjani (IBM) To: Naresh Kamboju , open list , linux-mm , lkft-triage@lists.linaro.org, linux-ext4 , LTP List Cc: Andrew Morton , Arnd Bergmann , Dan Carpenter , Theodore Ts'o , Andreas Dilger , Ojaswin Mujoo Subject: Re: next: kernel BUG at fs/ext4/mballoc.c:4369! In-Reply-To: <87o7kbnle9.fsf@doe.com> X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C3B5C16000F X-Stat-Signature: 7xcyqsr5j774k8zj7mmjpgtwttx5rf9p X-Rspam-User: X-HE-Tag: 1689604501-839816 X-HE-Meta: U2FsdGVkX1+/kZGXBG4Rgfx19BwzwSM3zBXg1RjaJ0Xy4azva9+MSoHrYpt80Y1SdkhhyYV5uw7MpkwP87rC/MJJBbe5X39i8Nrbbjmnz6H0upMZWiA4w1quZrrKYhWsBIlQphJZALWElWmx5nxYYpABCbOV8rnKbAm9tGQcQL55ElZrzAICUqvLxZw1c5Vyj/AxReNAsN0BMyvbQvZoBfj9kOD/SgDmjmPKg9LG3isMEVFAvTsPK5d3tf8Ugpa5qxaSUpyNI+4eqC28aE2eUXnnpqXX8ouPyfS/9RxNhCUsZf/0UKbf0vVWjTlP4EQDhuhXkKVGmZCNnm74/PLALhVDrauf/CaWOVWYoYy+GfJa8KLyMAsPHDEUgWFnKDSrgWoEVjh1VPlg34P4EkzcZEKXsq2KSCVWSEk5qqBjP7+bZNI9S1MyX74cJCUGC7xAToLOsnwQNJaBWZLbD2QnqtotW4OLy1tp4oBWhmPjXv071MPxHnVKl0tEkEfZrjVJo7djJUYViZqPV6fhkhuE3vno8EJmzsuK9iL3FRbzzvwe8JI6JSt+1HhRCUM9cf06ARJWCC7KymFeERx2nb8cvYP+0W2QF9z2wcieGkbd3izclEnuO6HgWxLLb6FYiyq2+R986mrCoI7rn4/xIUb1vMWf/7CYmlf67JS6U5y7lyh8Gf3Fr8ecic54DLaxhR8HIQbSo02rli010cNAjKtv7iwWaScZ+LYx6Hppt79x+ZKV43jbzSG/p8dLVAZZiJpkA6auftfp/QkmeDryW+rdhM66+uTHdKA9cNj+jgw5dy4/aBN5sdcSuwr6Av49+3JDCBHndsGsotjNt9kLobrMDm3sSBOka0xUuKPAf07eG6djL3cwiuG4v+jKNFLHuGL6ZgyZvDbqkwocya+RqLTr7aKAWcInQaMa24deVoWbf7s++wkLoM6eS6rdidWyaZ9uvSe60fByttKkbQ4U63e mFnKDk3t YQQv+wRAzE9jqOv8dQRLpfpHw53buyhvT3+Sy+CHGHZfOyadLVuj9wz77JDkbfMLfyXBIQ/oMjh3S+mUuV6RD36HnHA6bI31PlydjHnaVaSfjQzxiDJtE4uVWMby7wrcfvQTR0qoHiaAVjxN1FqZwliL0jViO0idwYcG10h8rhkkJYdFpYBC6totRx/kC+ZQzZHG3mubPtalNxgyXouv/iVDmemGJBzimThC4J4P4LiZhWEMHqXqoLxVhstASNrDjRAXsO2z1hUKCOnDg96AJmAWbJOfCK47E9IqkTocWIlwyVwuAfGGfTO9LgXsjtQ9iKiKEvf27uSzzjt86d3To7y3nhXr4OFqWIu7ukCiOk9vtL4NR8Eb7bvCNTERqVpYZdxuTMQHnwV1og4MmcK6ZIFGfQUwhfbApDjOdtGzHfmTQmcd6PRuDcti6GvW4fa//amBh1I4etLmZvUhsaBmtWKVVw7XpCaIRDg8IhAMvRzG3uhd8ZY7+81XaurbG1+rq0bV9SDAvkoriLarJfsxB7napGw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Ritesh Harjani (IBM) writes: > Naresh Kamboju writes: > >> Following kernel BUG noticed while testing LTP fs testing on x86_64 >> arch x86_64 on the Linux next-20230716 built with clang toolchain. >> >> I see a similar crash log on arm64 Juno-r2. The logs are shared below. >> >> Reported-by: Linux Kernel Functional Testing >> >> x86 log: >> ------- >> tst_test.c:1634: TINFO: === Testing on ext2 === >> tst_test.c:1093: TINFO: Formatting /dev/loop0 with ext2 opts='' extra opts='' >> mke2fs 1.46.5 (30-Dec-2021) >> [ 1393.346989] EXT4-fs (loop0): mounting ext2 file system using the >> ext4 subsystem > > ext4 driver is used for ext2 filesystem here. It will be using indirect > block mapping path. > >> [ 1393.396754] EXT4-fs (loop0): mounted filesystem >> 7ca8e239-bc8f-488c-af12-5e0ef12d17a5 r/w without journal. Quota mode: >> none. >> fs_fill.c:115: TINFO: Running 6 writer threads >> tst_fill_fs.c:109: TINFO: writev(\"mntpoint/subdir/thread6/AOF\", iov, >> 512): ENOSPC >> tst_fill_fs.c:109: TINFO: writev(\"mntpoint/subdir/thread5/AOF\", iov, >> 512): ENOSPC >> ... >> tst_fill_fs.c:109: TINFO: writev(\"mntpoint/subdir/thread6/AOF\", iov, >> 512): ENOSPC >> tst_fill_fs.c:109: TINFO: writev(\"mntpoint/subdir/thread3/AOF\", iov, >> 512): ENOSPC >> tst_fill_fs.c:109: TINF[ 1393.817197] ------------[ cut here ]------------ >> [ 1393.823305] kernel BUG at fs/ext4/mballoc.c:4369! > > It's hard to trigger the race I guess. But here are some debugging > information. > > [ 955.508751] EXT4-fs (loop1): mounting ext2 file system using the ext4 subsystem > [ 955.515527] EXT4-fs (loop1): mounted filesystem 57096378-d173-4bc5-ac06-9cd53c1dfa1c r/w without journal. Quota mode: none. > [ 959.289672] EXT4-fs (loop1): unmounting filesystem 57096378-d173-4bc5-ac06-9cd53c1dfa1c. > [ 959.490548] EXT4-fs (loop1): mounting ext3 file system using the ext4 subsystem > [ 959.503719] EXT4-fs (loop1): mounted filesystem 841c90bd-4d83-4bc5-be10-39452034e84b r/w with ordered data mode. Quota mode: none. > [ 960.553669] ext4_mb_pa_adjust_overlap: ==== This should not happend ==== left_pa=ffff8881471c7f50 deleted=0 lstart=6144 len=656 right_pa=0000000000000000 > [ 960.557437] ext4_mb_pa_adjust_overlap: pa = ffff8881471c7540, deleted=1 lstart=5872 len=272 pstart=34560 > [ 960.560659] ext4_mb_pa_adjust_overlap: pa = ffff8881471c7f50, deleted=0 lstart=6144 len=656 pstart=26848 > [ 960.563855] ext4_mb_pa_adjust_overlap: pa = ffff8881471c7ee0, deleted=1 lstart=6623 len=2 pstart=45503 > > (This is the rbtree printed ^^^ ) > > (gdb) p ac->ac_o_ex > $8 = { > fe_logical = 6625, > fe_start = 27328, > fe_group = 0, > fe_len = 1 > } > (gdb) p new_start > $9 = 6144 > (gdb) p new_end > $10 = 8192 > (gdb) p left_pa_end > $11 = 6800 > > > The bug one happens here - > > if (left_pa) { > left_pa_end = > left_pa->pa_lstart + EXT4_C2B(sbi, left_pa->pa_len); > BUG_ON(left_pa_end > ac->ac_o_ex.fe_logical); > } > > i.e. left_pa_end(6144 + 656 = 6800) > ac->ac_o_ex.fe_logical(6625) > > Thought of sharing this info which can save time for others. > Ok, so looks like we have some idea of what could be going wrong here. ext4_mb_pa_adjust_overlap() account and adjust for PAs that are marked deleted as well. However ext4_mb_use_preallocated() doesn't. It will simply skip the PAs which are marked deleted and move forward with searching in the rbtree. This could create problems while searching when we had PAs marked as deleted which were fixed in ext4_mb_adjust_overlap(). For e.g. when we have below tree... [ 5473.519335] ext4_mb_pa_adjust_overlap: pa = ffff88814a2ed1c0, deleted=1 lstart=1040 len=16 [ 5473.515741] ext4_mb_pa_adjust_overlap: pa = ffff88814a2ed4d0, deleted=0 lstart=1024 len=46 (Note the entries have overlapping ranges). (gdb) p ac->ac_o_ex $26 = { fe_logical = 1042, fe_start = 21967, fe_group = 0, fe_len = 1 } ... and we are allocating for ac_o_ex (1042) and root is pa = 0xffff88814a2ed1c0 (lstart=1040). The root pa covers the requested range but since it is marked as deleted, we ignore and search further. Since 1042 > 1040, we go to right. But we won't find any PA in the right subtree of pa (1040). This could cause PAs to be skipped for e.g. pa with lstart = 1024 will not be considered which ideally should have been used. This then causes a bug_on in ext4_mb_adjust_overlap() function (normalization path) when it finds an already allocated overlapping PA. @Ojaswin mentioned the same problem was solved in ext4_mb_pa_adjust_overlap(), however the logic was never added to ext4_mb_use_preallocated(). These can basically trigger in extremely low memory space and only when such ranges exist in the PA rbtree. Hence, I guess it is a little hard to tigger race. -ritesh