From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EC30C27C79 for ; Thu, 20 Jun 2024 18:02:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AE3D6B0480; Thu, 20 Jun 2024 14:02:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75C366B0481; Thu, 20 Jun 2024 14:02:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FDDD6B0482; Thu, 20 Jun 2024 14:02:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 404786B0480 for ; Thu, 20 Jun 2024 14:02:57 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DA5FAA1F29 for ; Thu, 20 Jun 2024 18:02:56 +0000 (UTC) X-FDA: 82252037952.23.4BD3B12 Received: from mail-il1-f181.google.com (mail-il1-f181.google.com [209.85.166.181]) by imf29.hostedemail.com (Postfix) with ESMTP id CAE9812002D for ; Thu, 20 Jun 2024 18:02:54 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=draconx-ca.20230601.gappssmtp.com header.s=20230601 header.b=c9AXEHvg; spf=none (imf29.hostedemail.com: domain of nbowler@draconx.ca has no SPF policy when checking 209.85.166.181) smtp.mailfrom=nbowler@draconx.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718906570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+Qz9jlWmDkZmtbdUlVdk29k4QOcWOt4wWsvvxN5FWug=; b=EPdNlVdIA9ADsxyKGRohWCZDgv2aJNVXsgqm8Ul0hJYX8jD7YpCAKxB2nTjMWEJ0T6GFW8 w5oj3e55V7pA/aimbZLraFi0So0Ii2x0PtTKr/5rfgcbk9Pk1gBleVDMLm04J+ebZag3Vo g/MKiUTHPG6kkbevJ3zQLUylRJNQc24= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=draconx-ca.20230601.gappssmtp.com header.s=20230601 header.b=c9AXEHvg; spf=none (imf29.hostedemail.com: domain of nbowler@draconx.ca has no SPF policy when checking 209.85.166.181) smtp.mailfrom=nbowler@draconx.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718906570; a=rsa-sha256; cv=none; b=PuCJqnB+aVJ3GBBQUXpm2quKcaYja4jjGoSIoHkJfQKW+zTqh1v2aZehaz13Smo3VHU+1q SD37cTqOYGUvIbehoYFIpmA9x3JKVhJ/Gu2qIjw7vyqq2fOQWQuTjLBwKq3FNDq3kgnRlN T2j/fC5Vo7+TB0X9Ad9I04S6BG5+ZFo= Received: by mail-il1-f181.google.com with SMTP id e9e14a558f8ab-3748ebe7e53so4522905ab.1 for ; Thu, 20 Jun 2024 11:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=draconx-ca.20230601.gappssmtp.com; s=20230601; t=1718906574; x=1719511374; darn=kvack.org; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=+Qz9jlWmDkZmtbdUlVdk29k4QOcWOt4wWsvvxN5FWug=; b=c9AXEHvg3y3jfoOo4CzhX4Msy6+JIYIDo9nc7Al1emp5VlBlTvp1e3u/pIa+ANOyh/ 8/3lzrFQsMdhIleVG4yiFtu9KK+eU2jAkmDMeKKptJwP4dqkcxTVPEuRukc+D2xX+pH7 UfSx/VUupaJMGMevUuiIGBFoJT/w2jHtObCEcbQUSrnbooRJnWP8sAI5GtpbCW6Rkqra yWsVhE4xZXdY9rzRB48mRTmERtYQk9U0BoJn6zHG0xTlUe3qbGVLHnIfypACd7/cV1CV uQSE8ktlhbvv/3ohB3G/BN3OFYq9dJTHi4fwxQsXCwZwhBgISP5LnjaCNms076eCFEeh saKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718906574; x=1719511374; h=content-transfer-encoding:in-reply-to:references:to:from :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+Qz9jlWmDkZmtbdUlVdk29k4QOcWOt4wWsvvxN5FWug=; b=qSOwLkS3al6+EnghyFsDg+GV7BzyZR3MOoF3qXS2LO0cvEbuDCEfWirsfq+UwYvRGt AFDfkQA59yAsCSYvPUvNXvVD64ugvavJQV3BPiGWbvsE44XNcg9+a7KcmjFx9TUuPCSg XIrbnzu2ZIsUJUuMENgIjmgAe4WkAZv/Lwfe0dbBKTXcSnyMsOkSLIEfw0JMkXhw2lIb lhNRYU1Zp6mYSxIYR1j5fpf2yFJ3RCzRJTpFmBVoW85VHwAW53/eLWDrlVy7pG9q7/fO xC2oLKGdnJ84/0u4g8A2+2OnmfxF29xh50d9pUWXTiyh/80pHdM7oi5J5txtAQaB22hf LISg== X-Forwarded-Encrypted: i=1; AJvYcCV9E/IrPvS7RYb1TXx7+Sr6o6bX9pjqJYjetmev+600vMk+9L1+dBmklRee3yRHw+nPhjPgeE0IZnJCd0DkzDk3QSM= X-Gm-Message-State: AOJu0Yx/1iKCDND5hhSRbyP0dudsWR1XhM0h5zG9/0rzJrg7HCVo73K9 hAyljFmG53eFFsUoQbU3X9gxu7STEChxSE7OxI80XMRqW7niuhuvm6jwoQ4CJDg= X-Google-Smtp-Source: AGHT+IFfEEgDs4n2+MF/0tHKPsSrZUMKCSuyV8ocM4F5arwnroC2vy4H2Ws/7SUtA9bHKoG4bq/m/g== X-Received: by 2002:a05:6e02:148c:b0:375:b19d:deb4 with SMTP id e9e14a558f8ab-3761d657299mr74849545ab.7.1718906573851; Thu, 20 Jun 2024 11:02:53 -0700 (PDT) Received: from [192.168.0.50] (dhcp-24-53-241-2.cable.user.start.ca. [24.53.241.2]) by smtp.gmail.com with ESMTPSA id 8926c6da1cb9f-4b956934522sm4442432173.58.2024.06.20.11.02.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 20 Jun 2024 11:02:53 -0700 (PDT) Message-ID: <00d74f24-c49c-460e-871c-d5af64701306@draconx.ca> Date: Thu, 20 Jun 2024 14:02:51 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: PROBLEM: kernel crashes when running xfsdump since ~6.4 Content-Language: en-US From: Nick Bowler To: linux-kernel@vger.kernel.org, Linux regressions mailing list , linux-mm@kvack.org, sparclinux@vger.kernel.org, "Uladzislau Rezki (Sony)" , Andrew Morton References: <75e17b57-1178-4288-b792-4ae68b19915e@draconx.ca> In-Reply-To: <75e17b57-1178-4288-b792-4ae68b19915e@draconx.ca> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: CAE9812002D X-Stat-Signature: zwjde371iqb6zyq8o1rpz3umz7ps43je X-HE-Tag: 1718906574-775874 X-HE-Meta: U2FsdGVkX1/voEnW01vV0FYVg91WHALHXCo1fc085s/0IRePM8m+f/r2+jb3lb3jr4+2j+tvsE7MF7R6s+wtLmAIN1auK2gW6Elx85kwTP2cqy5ujUfa0NK29vxN08vLgF/xLVfOpp7xYu0Xs15I/P8ujQ7B8TsZhM/zL+6RN8UA86iRHFzfpmc+1qdy5/8Y3/QsTgzpw/DpdWO7+sRgjD268gwjLoyRbawUidhSPBUJu3iw6Vb+opwToaAZ7bZmNrJ5dO8JU4jvwAoQbc+/90rsT7k1GBkgTWfHPnYoG/npspFI3lERBAFlhajFirBrM6wHOhNwnvkBTwGFYoDec2LI72tKzpmP2cQKonbSb8omhp2IF4qHhw8AI6QY3cJxEXmnoz6BzBCVZlfB0VwkSbhMfuHOdNmxL7yoNyoXnDx00RTXA34jR+kEvpfM2WfoyZkuVJXC7p+KnacB60qX9Aw8SPSrPVkOrZmE2oq0ugOGKU+UEJ0VN0R737ZsPIm+QOT4Q52CzsWttNzcWh48IfH04kP0C9AD3G1EuM7LDqZoLzvskClWCbSTNh3hmzITScPdxlwIBL4rmWkdhyFyv1pZRfSUVwF/M3TwcgOruQmOeQuxNTAAU0WapKFVqMUo87DdfdWxJFW4T0sWtksmJ/3GVYTcaBSnRZVMvguS67tnbgoG0A184xaaS6jKYHLqqioprouLLqLkITWxD5wPOQ+1kwPA5S3qH4jEeFPUNjzulZD0UVbPvClF8XIdFzVqmleeOBV1ugtCTeod3CU/2CqI6z1lRD5uzRXCB+ullWu6xhENLCzFo8Ww5Np13/vgyMdXPjLPSv3V81m1TQzUwb4OX58o/AToRDdkB1ZCnFcmJuYLBYCdlKDHi5/sH/83WDp92ZxCMhI+4T5NFxPiMgKJDQekIIm4aJ5202MWwNzAtC00X1WUNmGL5bguMWKMhn+fPETeXAMnUMefgUO CqKKxEPk nvw/XgNXx+MEemkxztEBxPY2iGGMxcRxf5ZEL8siIuVr//NN6jRz6y79hFBlhiBkphBV6TP1EodxONhNsougCLtmqNtxSEyBC3JFpXDwIDVnpcYC4qDEbV2v9NK3CW7N83sxT4lLPS4xAkB/v3+CeT/e/v8OMaWPCBnA5xI51eJOLdVJXqXBg+5CINheLK8hAqwZd33DfsEHeZyGuS2vlTy4VfNjSg4My5zGX+clbWTH7e/Qr6ZGvuriQgeLWqzC1y/j9nFDQ/RcrLMS79G2SadJHL5oAxTxwfm+FK0F8StZveN7oth4B6aPFCKTVgiXbZpyk2TGvn345BPIJMgKA7VlEfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2024-06-20 02:19, Nick Bowler wrote: > After upgrading my sparc to 6.9.5 I noticed that attempting to run > xfsdump instantly (within a couple seconds) and reliably crashes the > kernel. The same problem is also observed on 6.10-rc4. [...] > 062eacf57ad91b5c272f89dc964fd6dd9715ea7d is the first bad commit > commit 062eacf57ad91b5c272f89dc964fd6dd9715ea7d > Author: Uladzislau Rezki (Sony) > Date: Thu Mar 30 21:06:38 2023 +0200 > > mm: vmalloc: remove a global vmap_blocks xarray I think I might see what is happening here. On this machine, there are two CPUs numbered 0 and 2 (there is no CPU1). The per-cpu variables in mm/vmalloc.c are initialized like this, in vmalloc_init for_each_possible_cpu(i) { /* ... */ vbq = &per_cpu(vmap_block_queue, i); /* initialize stuff in vbq */ } This loops over the set bits of cpu_possible_mask, bits 0 and 2 are set, so it initializes stuff with i=0 and i=2, skipping i=1 (I added prints to confirm this). Then, in vm_map_ram, with the problematic change it calls the new function addr_to_vb_xa, which does this: int index = (addr / VMAP_BLOCK_SIZE) % num_possible_cpus(); return &per_cpu(vmap_block_queue, index).vmap_blocks; The num_possible_cpus() function counts the number of set bits in cpu_possible_mask, so it returns 2. Thus, index is either 0 or 1, which does not correspond to what was initialized (0 or 2). The crash occurs when the computed index is 1 in this function. In this case, the returned value appears to be garbage (I added prints to confirm this). If I change addr_to_vb_xa function to this: int index = ((addr / VMAP_BLOCK_SIZE) & 1) << 1; /* 0 or 2 */ return &per_cpu(vmap_block_queue, index).vmap_blocks; xfsdump is working again. Cheers, Nick