From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,UNPARSEABLE_RELAY,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCBA6C433B4 for ; Wed, 7 Apr 2021 13:09:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C579661353 for ; Wed, 7 Apr 2021 13:09:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C579661353 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=e16-tech.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F0C786B007D; Wed, 7 Apr 2021 09:09:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBC6F6B007E; Wed, 7 Apr 2021 09:09:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D84EA6B0080; Wed, 7 Apr 2021 09:09:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id BDDEC6B007D for ; Wed, 7 Apr 2021 09:09:10 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7DDBF1E02 for ; Wed, 7 Apr 2021 13:09:10 +0000 (UTC) X-FDA: 78005601660.20.9907004 Received: from out20-1.mail.aliyun.com (out20-1.mail.aliyun.com [115.124.20.1]) by imf07.hostedemail.com (Postfix) with ESMTP id 4AB3CA0003B3 for ; Wed, 7 Apr 2021 13:09:07 +0000 (UTC) X-Alimail-AntiSpam:AC=CONTINUE;BC=0.04462136|-1;CH=green;DM=|CONTINUE|false|;DS=CONTINUE|ham_system_inform|0.0133605-0.000248439-0.986391;FP=0|0|0|0|0|-1|-1|-1;HT=ay29a033018047188;MF=wangyugui@e16-tech.com;NM=1;PH=DS;RN=4;RT=4;SR=0;TI=SMTPD_---.JwOA6aC_1617800942; Received: from 192.168.2.112(mailfrom:wangyugui@e16-tech.com fp:SMTPD_---.JwOA6aC_1617800942) by smtp.aliyun-inc.com(10.147.40.200); Wed, 07 Apr 2021 21:09:02 +0800 Date: Wed, 07 Apr 2021 21:09:07 +0800 From: Wang Yugui To: Vlastimil Babka Subject: Re: unexpected -ENOMEM from percpu_counter_init() Cc: linux-mm@kvack.org, linux-btrfs@vger.kernel.org, wangyugui@e16-tech.com In-Reply-To: <60e9b994-e37c-d059-4af5-0cb7860ca4f3@suse.cz> References: <20210401185158.3275.409509F4@e16-tech.com> <60e9b994-e37c-d059-4af5-0cb7860ca4f3@suse.cz> Message-Id: <20210407210905.F790.409509F4@e16-tech.com> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Becky! ver. 2.75.03 [en] X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 4AB3CA0003B3 X-Stat-Signature: yrt88b5ibzcuwrhiuhmjqm7jciti8tje Received-SPF: none (e16-tech.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=out20-1.mail.aliyun.com; client-ip=115.124.20.1 X-HE-DKIM-Result: none/none X-HE-Tag: 1617800947-396072 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, > +CC btrfs > > On 4/1/21 12:51 PM, Wang Yugui wrote: > > Hi, > > > > an unexpected -ENOMEM from percpu_counter_init() happened when xfstest > > with kernel 5.11.10 and 5.10.27 > > Is there a dmesg log showing allocation failure or something? When unexpected -ENOMEM of percpu_counter_init(), btrfs as upper caller finally output something to dmesg. And we add one trace to btrfs source to make sure that. > if (ret == -ENOMEM) printk("ENOMEM btrfs_drew_lock_init\n"); Now the reproduce frequency become from >50% to not happen or very slow with the flowing change. diff --git a/mm/percpu.c b/mm/percpu.c index 6596a0a..0127be1 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -104,8 +104,8 @@ /* chunks in slots below this are subject to being sidelined on failed alloc */ #define PCPU_SLOT_FAIL_THRESHOLD 3 -#define PCPU_EMPTY_POP_PAGES_LOW 2 -#define PCPU_EMPTY_POP_PAGES_HIGH 4 +#define PCPU_EMPTY_POP_PAGES_LOW 8 +#define PCPU_EMPTY_POP_PAGES_HIGH 16 #ifdef CONFIG_SMP /* default addr <-> pcpu_ptr mapping, override in asm/percpu.h if necessary */ diff --git a/include/linux/percpu.h b/include/linux/percpu.h index 5e76af7..8cc091b 100644 --- a/include/linux/percpu.h +++ b/include/linux/percpu.h @@ -14,7 +14,7 @@ /* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES -#define PERCPU_MODULE_RESERVE (8 << 10) +#define PERCPU_MODULE_RESERVE (32 << 10) #else #define PERCPU_MODULE_RESERVE 0 #endif Just some guess, 1) maybe some releationship to the trigger of 'vm.dirty_bytes=10737418240'. this problem happen in server/T7610 with E5-2660v2 *2 and SSD/SAS(6Gb/s) and 192G memory but not happen in server/T620 with E5-2680v2 *2 and SSD/NVMe and 192G memory. 2) maybe some releationship to numa. 128G memory in node1(CPU1), and 64G in node2(CPU2) Best Regards Wang Yugui (wangyugui@e16-tech.com) 2021/04/07 > > direct caller: > > int btrfs_drew_lock_init(struct btrfs_drew_lock *lock) > > { > > int ret; > > > > ret = percpu_counter_init(&lock->writers, 0, GFP_KERNEL); > > if (ret) > > return ret; > > > > atomic_set(&lock->readers, 0); > > init_waitqueue_head(&lock->pending_readers); > > init_waitqueue_head(&lock->pending_writers); > > > > return 0; > > } > > > > upper caller: > > nofs_flag = memalloc_nofs_save(); > > ret = btrfs_drew_lock_init(&root->snapshot_lock); > > memalloc_nofs_restore(nofs_flag); > > if (ret == -ENOMEM) printk("ENOMEM btrfs_drew_lock_init\n"); > > if (ret) > > goto fail; > > > > The hardware of this server: > > CPU: Xeon(R) CPU E5-2660 v2(10 core) *2 > > memory: 192G, no swap > > > > Only one xfstests job is running in this server, and about 7% of memory > > is used. > > > > Any advice please. > > > > Best Regards > > Wang Yugui (wangyugui@e16-tech.com) > > 2021/04/01 > > > >