From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF79FC2BBCA for ; Tue, 25 Jun 2024 12:40:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 281316B009B; Tue, 25 Jun 2024 08:40:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2311E6B02FE; Tue, 25 Jun 2024 08:40:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D1586B00A9; Tue, 25 Jun 2024 08:40:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E43A96B02FE for ; Tue, 25 Jun 2024 08:40:26 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A63941A17A6 for ; Tue, 25 Jun 2024 12:40:26 +0000 (UTC) X-FDA: 82269369252.16.16E3410 Received: from mail-ed1-f45.google.com (mail-ed1-f45.google.com [209.85.208.45]) by imf10.hostedemail.com (Postfix) with ESMTP id B9197C0004 for ; Tue, 25 Jun 2024 12:40:24 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kHSekaFK; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719319210; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WoVkX+/SqvLNF+tl9I//nQP7hSOM7jd83RwihftTHdk=; b=CdK9vrLxplgBcmg37XOM1T47bmagM5LETrPfnDZsJKOfDVVsC0xDNTXbXxUtkYudgUKSn5 pwIps89w7hAt2kPLCnsPTlvAA7elpv41WqgAaI+1kYttnSZAJ2x2OpGOJeI+NHTDZF+MP/ snesWwjFTshB5+McGjt0Anyro8s1F+A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719319210; a=rsa-sha256; cv=none; b=Z1IKpWP7LEFZr3o2CrxlzQlLEvkfRBF/sm8Jj9iGGM7/nkcP1pZqVFwQcJHEFfmSn9fxyS K9E5nsUVumO/6Pb+GNSFSpa51szf04wwLOtE+Tptv9tS/hYZZ4G1vaEvANMRsCEHs+2n/h 6ZCvTJOkLrX1hCAhEgQDI4qDhgY1KPQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=kHSekaFK; spf=pass (imf10.hostedemail.com: domain of urezki@gmail.com designates 209.85.208.45 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-f45.google.com with SMTP id 4fb4d7f45d1cf-57cbc2a2496so6315619a12.0 for ; Tue, 25 Jun 2024 05:40:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1719319223; x=1719924023; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=WoVkX+/SqvLNF+tl9I//nQP7hSOM7jd83RwihftTHdk=; b=kHSekaFK0CRHgtU7Gd/WSKnQYUHw8fNWiY+Iu7ZjUrPzNcr7nArmEnHl4vVSgWuTmW Sx8tap+/0dGr2NuaDw97M0a7dJPNm52JqxooiDzHarDqpxuO7BXABPZNW0jAoQv3DHPA uMQpbc1XYWk8AJezIxLWsi7tjc0tVztRAHaNKqtCvv37RE3UOYrHjOHX3CW+4EVN1lED YpkLIH8V38TET9qKLRcGNBqR5vut58SVaeOFhZSyzjPt0v8n6fGMx0hJXWWLr8DPuERo w+xYXJ+Xh6MPQjOCv3hLAC6wtp7qTeubV1TIQsqdZ2ENuIjGRN65GknKe+zFho2iKng2 NaOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719319223; x=1719924023; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WoVkX+/SqvLNF+tl9I//nQP7hSOM7jd83RwihftTHdk=; b=r2rj2dZnLHgmU9eeT1xJSGeOr9H/mNOGdee9jBDr4zNKjbIQWPrI0qbKuhBdei3FEi JW4kP6KOjeU+EBVEEm0X2nEYn0EtipHlR3egjaaJqkPclrApQ8OWB/uY0AO7846/guhK nx5Uf8ISVyUDel6Uwm/0z7AJ7XNNiu38O0411bMbd48+RGep/MfKhfgQ7Y6h7dFT0uIl njjCTraPx5Hx1GIZgwmD91UT4m2v42Jb30j7+rfuumqhxJRiFtpejIdopWDLQQ9SuN5H XWRWx+l7pbzxbc5NZjhVEae9AB7I/gQ2Wnx0e8YJVTWIC8C+utGQ7ywEbHiIcTlc+eq6 rgCw== X-Forwarded-Encrypted: i=1; AJvYcCXB4pmaH0xXdeJURt3TFPJy0vWIejUw1vHg6JxD4DlxkSRfiDafnbkcCiSRIyQhdQWiqZ1StQ7OwLwaGaYdoC7vaYc= X-Gm-Message-State: AOJu0YzbrJbqD48jhSMunQBV9SahbtX2HGa9bDmEhSepQjjlZq1M0c2R Y1lOa0DzhvHJEd/HcbneIBvyLrxomjSf7LL6fMv3lRpqgBNtKAY8 X-Google-Smtp-Source: AGHT+IFxTLa8c3S5ivxUnXxBLRu48xXYi/65NyP5o9yS4QN739SX8vgF3IZu8OCLLzy1iVUqusy0fA== X-Received: by 2002:aa7:cb59:0:b0:57d:4fd8:db59 with SMTP id 4fb4d7f45d1cf-57d4fd8dc94mr6295848a12.0.1719319222848; Tue, 25 Jun 2024 05:40:22 -0700 (PDT) Received: from pc638.lan (host-185-121-47-193.sydskane.nu. [185.121.47.193]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-57d30563440sm5853522a12.84.2024.06.25.05.40.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Jun 2024 05:40:22 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Tue, 25 Jun 2024 14:40:20 +0200 To: Baoquan He Cc: Uladzislau Rezki , Nick Bowler , Hailong Liu , linux-kernel@vger.kernel.org, Linux regressions mailing list , linux-mm@kvack.org, sparclinux@vger.kernel.org, Andrew Morton Subject: Re: PROBLEM: kernel crashes when running xfsdump since ~6.4 Message-ID: References: <75e17b57-1178-4288-b792-4ae68b19915e@draconx.ca> <00d74f24-c49c-460e-871c-d5af64701306@draconx.ca> <20240621033005.6mccm7waduelb4m5@oppo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B9197C0004 X-Stat-Signature: h4gkcxo5iuag3scnsw1mucgid51dzcat X-HE-Tag: 1719319224-506863 X-HE-Meta: U2FsdGVkX19Xm4FACwsH3bfWgv+FYt/41BBthtYgzxVnaX7PxMZQto93Xn5rHuPf5GKliJV3ztJS//cCGkerCTV3Vw1yZgO+UdxJlmgqhxng1kgLiaZ4STDAMCGQn2ftG5b/x4AP7W2gwNchnCZxE4vPc17jBcNpAD5+zqbH5VZcLnA/33hK0slQqNnulHp1UBvCVDQVcMQNVZTqFDP4r4EQu8IiwTsktN5pOTznzSES0N7FIlR2bn0FWYzyKulQYW7PapUIYwYsOjbYoA61/q0HdlOXPmLCkYbYrKkMv+/YQSZy3jlYECkZ6XUHwvwY+P9tjT02jirUn3T8a5vh8T/lruZw8aBx7WRlwkjWxJjkqJXP/ZG4ES4diVph2mRsz276mXo7UILziYql+3Q1vaUoySnoeJTw4zEa1JqPCTyehdxmrZNNVWkNOHUYm2EZXBQR6EfsbMFHfCf03Q0hTJpc1JSFlTA22LdIpWWNOYVSNIwRmCTgdDqIYp7g0RqjOrDyxNRxqW5SWyYE1VM9lMbHUpWRuT36zDHWCFUAPi739/NIjMrqhGPjbcf08W/iSd1xlWWmCxIi9RBfZb9FaBoF04O1ln682k5JJ8yXX1VDeV5FKvaRNwqkEM9I9dpRItxhUGo7QB5YEyZtZKUYJ1PIgt6NZgdnDvq4I56qSCjJYWqIPBoafHAh+BDd3P6N7lXgrPmqlNtrLi/K37T1JybXknctEizquNrIrdZdcN6AJfMFPiMwYqURv9pFVQUuiA0lIZd+2lFDmYRlPOdAkIQbRvrJU8ZCU0idNz+/yinDiCHuFlx3MaOodR3+g3P7knL038VeoiiuLj1Z+UAQMwLfnqOSIO6hR1aH8uIzX2d/LtBuantGg52exDpTkaitT8zPHwERxGlZ+/9NfXJL5Pn8Eym6vPWdoHWqYUlpZssXdrwGT0fUe3GqmXz3ihlVpTZEVbAhvblpqWOpN7g FwcD2T2y ClFigwJet/W9VHVCnajNvk45x1zupqLnvLgXCLIs8IUrs0f41lYb0HX6OX9IRS8kdfEN4xbXz5kdygm+Ee6L6N88H/1KMp+m7PFDH2NWveGrBICkv5v+SDlg+wNUyEuAVygKox/L92o4DwwAkHLUvh5Y/hR5zkhNmXilb0SpJf6GW/V+sL4bUXTZRvy7E/ZIbho3KmifrE4b950joEr+Brow/A+yTbK+R/j5qC4rKVBbE+l0pkGC/uGwolYNYta1n1r4mkcDf5sN1Ppmq3HrbSlcSGFpq7OZcLkXYPk3WHcsOf3z88y7xPLF4c9QLPCYlHjqDZQIycz8c11w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 25, 2024 at 07:40:21PM +0800, Baoquan He wrote: > On 06/25/24 at 12:32pm, Uladzislau Rezki wrote: > > On Tue, Jun 25, 2024 at 11:30:33AM +0800, Baoquan He wrote: > > > On 06/24/24 at 02:16pm, Uladzislau Rezki wrote: > > > > On Fri, Jun 21, 2024 at 10:02:50PM +0800, Baoquan He wrote: > > > > > On 06/21/24 at 11:44am, Uladzislau Rezki wrote: > > > > > > On Fri, Jun 21, 2024 at 03:07:16PM +0800, Baoquan He wrote: > > > > > > > On 06/21/24 at 11:30am, Hailong Liu wrote: > > > > > > > > On Thu, 20. Jun 14:02, Nick Bowler wrote: > > > > > > > > > On 2024-06-20 02:19, Nick Bowler wrote: > > > > > ...... > > > > > > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > > > > > > > index be2dd281ea76..18e87cafbaf2 100644 > > > > > > > --- a/mm/vmalloc.c > > > > > > > +++ b/mm/vmalloc.c > > > > > > > @@ -2542,7 +2542,7 @@ static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); > > > > > > > static struct xarray * > > > > > > > addr_to_vb_xa(unsigned long addr) > > > > > > > { > > > > > > > - int index = (addr / VMAP_BLOCK_SIZE) % num_possible_cpus(); > > > > > > > + int index = (addr / VMAP_BLOCK_SIZE) % nr_cpu_ids; > > > > > > > > > > > > > > return &per_cpu(vmap_block_queue, index).vmap_blocks; > > > > > > > } > > > > > > > > > > > > > The problem i see is about not-initializing of the: > > > > > > > > > > > > for_each_possible_cpu(i) { > > > > > > struct vmap_block_queue *vbq; > > > > > > struct vfree_deferred *p; > > > > > > > > > > > > vbq = &per_cpu(vmap_block_queue, i); > > > > > > spin_lock_init(&vbq->lock); > > > > > > INIT_LIST_HEAD(&vbq->free); > > > > > > p = &per_cpu(vfree_deferred, i); > > > > > > init_llist_head(&p->list); > > > > > > INIT_WORK(&p->wq, delayed_vfree_work); > > > > > > xa_init(&vbq->vmap_blocks); > > > > > > } > > > > > > > > > > > > > > > > > > correctly or fully. It is my bad i did not think that CPUs in a possible mask > > > > > > can be non sequential :-/ > > > > > > > > > > > > nr_cpu_ids - is not the max possible CPU. For example, in Nick case, > > > > > > when he has two CPUs, num_possible_cpus() and nr_cpu_ids are the same. > > > > > > > > > > I checked the generic version of setup_nr_cpu_ids(), from codes, they > > > > > are different with my understanding. > > > > > > > > > > kernel/smp.c > > > > > void __init setup_nr_cpu_ids(void) > > > > > { > > > > > set_nr_cpu_ids(find_last_bit(cpumask_bits(cpu_possible_mask), NR_CPUS) + 1); > > > > > } > > > > > > > > > I see that it is not a weak function, so it is generic, thus the > > > > behavior can not be overwritten, which is great. This does what we > > > > need. > > > > > > > > Thank you for checking this you are right! > > > > > > Thanks for confirming this. > > > > > > > > > > > Then it is just a matter of proper initialization of the hash: > > > > > > > > > > > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > > > > index 5d3aa2dc88a8..1733946f7a12 100644 > > > > --- a/mm/vmalloc.c > > > > +++ b/mm/vmalloc.c > > > > @@ -5087,7 +5087,13 @@ void __init vmalloc_init(void) > > > > */ > > > > vmap_area_cachep = KMEM_CACHE(vmap_area, SLAB_PANIC); > > > > > > > > - for_each_possible_cpu(i) { > > > > + /* > > > > + * We use "nr_cpu_ids" here because some architectures > > > > + * may have "gaps" in cpu-possible-mask. It is OK for > > > > + * per-cpu approaches but is not OK for cases where it > > > > + * can be used as hashes also. > > > > + */ > > > > + for (i = 0; i < nr_cpu_ids; i++) { > > > > > > I was wrong about earlier comments. Percpu variables are only available > > > on possible CPUs. For those nonexistent possible CPUs of static percpu > > > variable vmap_block_queue, there isn't memory allocated and mapped for > > > them. So accessing into them will cause problem. > > > > > > In Nick's case, there are only CPU0, CPU2. If you access > > > &per_cpu(vmap_block_queue, 1), problem occurs. So I think we may need to > > > change to take other way for vbq. E.g: > > > 1) Storing the vb in the nearest neighbouring vbq on possible CPU as > > > below draft patch; > > > 2) create an normal array to store vbq of size nr_cpu_ids, then we can > > > store/fetch each vbq on non-possible CPU? > > > > > A correct way, i think, is to create a normal array. A quick fix can be > > to stick to a next possible CPU. > > > > > The way 1) is simpler, the existing code can be adapted a little just as > > > below. > > > > > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > > > index 633363997dec..59a8951cc6c0 100644 > > > --- a/mm/vmalloc.c > > > +++ b/mm/vmalloc.c > > > @@ -2542,7 +2542,10 @@ static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); > > > static struct xarray * > > > addr_to_vb_xa(unsigned long addr) > > > { > > > - int index = (addr / VMAP_BLOCK_SIZE) % num_possible_cpus(); > > > + int index = (addr / VMAP_BLOCK_SIZE) % nr_cpu_ids; > > > + > > > + if (!cpu_possible(idex)) > > > + index = cpumask_next(index, cpu_possible_mask); > > > > > cpumask_next() can return nr_cpu_ids if no next bits set. > > It won't. nr_cpu_ids is the largest index + 1, the hashed index will > be: 0 =< index <= (nr_cpu_ids - 1) e.g cpu_possible_mask is > b10001111, the nr_cpu_ids is 8, the largest bit is cpu7. > cpu_possible(index) will check that. So the largest bit of cpumask_next() > returns is (nr_cpu_ids - 1). > /** * cpumask_next - get the next cpu in a cpumask * @n: the cpu prior to the place to search (i.e. return will be > @n) * @srcp: the cpumask pointer * * Return: >= nr_cpu_ids if no further cpus set. */ static inline unsigned int cpumask_next(int n, const struct cpumask *srcp) { /* -1 is a legal arg here. */ if (n != -1) cpumask_check(n); return find_next_bit(cpumask_bits(srcp), small_cpumask_bits, n + 1); } -- Uladzislau Rezki