From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52A66C4828D for ; Wed, 7 Feb 2024 03:29:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D60A06B0083; Tue, 6 Feb 2024 22:29:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CE8978D0002; Tue, 6 Feb 2024 22:29:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB1128D0001; Tue, 6 Feb 2024 22:29:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AB9746B0083 for ; Tue, 6 Feb 2024 22:29:02 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5C1A41C12A3 for ; Wed, 7 Feb 2024 03:29:02 +0000 (UTC) X-FDA: 81763576524.22.ACB71F3 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by imf16.hostedemail.com (Postfix) with ESMTP id 7049A18000B for ; Wed, 7 Feb 2024 03:29:00 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IjBYdx31; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rulin.huang@intel.com designates 192.198.163.7 as permitted sender) smtp.mailfrom=rulin.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707276540; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=jsG0XDttIX7FxcZAixSH2765TJo5G5tcEXA/2l1SILg=; b=vWM+v7WY67TgEUGOyEh9U4X7vFGzjW1i0n1w3cFH3TzJJElUAOSNDkTUywVdCWQvmo8Ioq bmMO7qaC4Rqo2ontigWEQ6MK5iDvmOII5PUAbr/vc6kCxUvFIHfUFxwFwfqEjLCxLQ8BKY N6TasqCAk6ysuCbyT3BsrlLIfIrXBl8= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IjBYdx31; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf16.hostedemail.com: domain of rulin.huang@intel.com designates 192.198.163.7 as permitted sender) smtp.mailfrom=rulin.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707276540; a=rsa-sha256; cv=none; b=zdlDKO2xF+K88W4gZE8GhBPbtLDL0BpuHLBHN2UfUi7s6jUZGODDJ4s3I+sPv5nxIYcj21 bw0I+AwKoLa48EEC19I1YL5mJDLimTzG/LHY1q9/dj3m03auxckRA8GiA+9aGi6DAcdCq2 rap67WX3+sE1zjngLnZYicONO13A7Gk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1707276540; x=1738812540; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=5z8kY6K3nvIR3A14IdHiEafFgYEBuUDMqEiZjPP8O+Y=; b=IjBYdx31S828hlmx7Iv+rn38BUoJPFSrirsaMhWRpoRCOsFUqx+E+eOw cx26lSQfZdyQ+yBJYg9xSdjQIGaBej1/nlZ9PO9l0gIJ5BeZnLHVMn8al NPXs/CF879j3PjsKTF2M3QR2uwPFAzDimJAs+PGQcI1/nVzaT+frQD3uM mAzDTAVkFyN82VvpbZceiFE1stUsBkGMqSleOn/P3s7PWFN+sQGh/edbB 2vj1OdTcPi6wt8IjYgH/cd3KQmY5acsZrFpXooc46ER1/v0t1qR4+6PM7 8q+ML9wwENJ4Cy6sTVctcx5FyRLvKxEqF5CN3vp8v5Ev7zHLRzZR405os Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10976"; a="26344745" X-IronPort-AV: E=Sophos;i="6.05,250,1701158400"; d="scan'208";a="26344745" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2024 19:28:55 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10976"; a="824384659" X-IronPort-AV: E=Sophos;i="6.05,250,1701158400"; d="scan'208";a="824384659" Received: from linux-pnp-server-09.sh.intel.com ([10.239.176.190]) by orsmga001.jf.intel.com with ESMTP; 06 Feb 2024 19:28:52 -0800 From: rulinhuang To: akpm@linux-foundation.org Cc: urezki@gmail.com, hch@infradead.org, lstoakes@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, tim.c.chen@intel.com, colin.king@intel.com, zhiguo.zhou@intel.com, rulinhuang Subject: [PATCH] mm/vmalloc: lock contention optimization under multi-threading Date: Tue, 6 Feb 2024 22:30:59 -0500 Message-ID: <20240207033059.1565623-1-rulin.huang@intel.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7049A18000B X-Stat-Signature: mjxmix3mw1qke6wwne5315yz8e7xnquj X-Rspam-User: X-HE-Tag: 1707276540-825209 X-HE-Meta: U2FsdGVkX18iF7bxMARSRg5a1oAAvNm049dp2Z40A3WF2o7D7QhIlTUJvceH2RObQffS9d6SM3BvJaxpdRSlLZsnplAuvx0kdPlBsBFSan4vixaeFlg5B8DndQCiD+wbktI7oSkxg+5KiE7hx2OYRbzFVbpj2wzBIGq23ECDqd3TgMWJRH/w6sDM6D9pV1eBUjkdqXNuL0veavFC1IlO83fF/jflA9Gqz7iDodCvSxTtj1DSliqPxzQ5pkkMedUnm5nfEhInpxYnt0xVmqp9yhUCxRUTd3emKUSPRhWeO1e1hI/2V2Kd18Q4Ba13OaXw+zXp/SI1C4ZFSP0Ams59Uijgz56EZgekj/5nV3rIhcJvM/ODPGMr4/YuoOm9Aw7NY4leniKl/wDz8CjzLrdPQwulzx4Qfnb0QeEOmcN07ufFJYdnIDoqXKFqXW5p1b+qjhlwoNtgcQI8eTn7R/iJ4VFQcKFyAY279pNruT8EBpc/UQwh701FGhERPswtA4+SCl++7FNhJswcm91rcZAUHHU01fmrMKuoibGR4HdDbEaBJSJUfN2oC+By/JD6FKXpvTncbBZJ9I00hUUslObW66IamFJXT0kzV+7t5m/1qb46sPLzjPJtdRM4HFcp5kLLhKhjGYkXheLdRfA5FBbWSPs0HrRucIuGNXfekeqHjmi1Y6zN3nJS7fOyMJYyWPAxNUzpfVCvuKwgyj2ZIadKi2decObXVdh9SImB1Ndib/oMRCVxEzjmpOh+lemit31xA/OXQ76KTxB+Kj+yK3fDfxGNdriZZcbzN3g5BkN53Cg6xeiWS9hClwlRkiuLjl9wDDq06wG1sosvKUk3v7meVH+Z5jWLKpoep5rhsR2LQ9O3VpexlR/QWSMAE7D1hd3nJcKF0Qjal7x8/KVzYGyOeRLI73lzQiK8ihWKSYKaegu2jneYzOOP8NvyEDqfX3V2ZmeLKQjbYSE89LLPhmn 4AG3RpEn YTMo5lgrURy6v+OxS2NpFeZH90x8wC9tRF8T2bu5vIZWuu5xRaC0D1lC1MCl9N6Ul/exGnzw65EcjhO+TgO+cmRqsFX1VZ0WDdL589TcojlM+rQrXwnvNRhMTdCBnj0G+PGWNlw34GnpIK8GWFIRzdswGFM4NZvYG3Bzw9Al4C4DggT7uJEfqMm9NVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When allocating a new memory area where the mapping address range is known, it is observed that the vmap_area lock is acquired twice. The first acquisition occurs in the alloc_vmap_area() function when inserting the vm area into the vm mapping red-black tree. The second acquisition occurs in the setup_vmalloc_vm() function when updating the properties of the vm, such as flags and address, etc. Combine these two operations together in alloc_vmap_area(), which improves scalability when the vmap_area lock is contended. By doing so, the need to acquire the lock twice can also be eliminated. With the above change, tested on intel icelake platform(160 vcpu, kernel v6.7), a 6% performance improvement and a 7% reduction in overall spinlock hotspot are gained on stress-ng/pthread(https://github.com/ColinIanKing/stress-ng), which is the stress test of thread creations. Reviewed-by: "Chen, Tim C" Reviewed-by: "King, Colin" Signed-off-by: rulinhuang --- mm/vmalloc.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c171..3b1f616e8ecf0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1577,13 +1577,13 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) /* * Allocate a region of KVA of the specified size and alignment, within the - * vstart and vend. + * vstart and vend. If vm is passed in, the two will also be bound. */ static struct vmap_area *alloc_vmap_area(unsigned long size, unsigned long align, unsigned long vstart, unsigned long vend, int node, gfp_t gfp_mask, - unsigned long va_flags) + unsigned long va_flags, struct vm_struct *vm) { struct vmap_area *va; unsigned long freed; @@ -1627,9 +1627,12 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->va_start = addr; va->va_end = addr + size; - va->vm = NULL; + va->vm = vm; va->flags = va_flags; + if (vm != NULL) + vm->addr = (void *)addr; + spin_lock(&vmap_area_lock); insert_vmap_area(va, &vmap_area_root, &vmap_area_list); spin_unlock(&vmap_area_lock); @@ -2039,7 +2042,8 @@ static void *new_vmap_block(unsigned int order, gfp_t gfp_mask) va = alloc_vmap_area(VMAP_BLOCK_SIZE, VMAP_BLOCK_SIZE, VMALLOC_START, VMALLOC_END, node, gfp_mask, - VMAP_RAM|VMAP_BLOCK); + VMAP_RAM|VMAP_BLOCK, + NULL); if (IS_ERR(va)) { kfree(vb); return ERR_CAST(va); @@ -2394,7 +2398,8 @@ void *vm_map_ram(struct page **pages, unsigned int count, int node) struct vmap_area *va; va = alloc_vmap_area(size, PAGE_SIZE, VMALLOC_START, VMALLOC_END, - node, GFP_KERNEL, VMAP_RAM); + node, GFP_KERNEL, VMAP_RAM, + NULL); if (IS_ERR(va)) return NULL; @@ -2548,14 +2553,6 @@ static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, va->vm = vm; } -static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, - unsigned long flags, const void *caller) -{ - spin_lock(&vmap_area_lock); - setup_vmalloc_vm_locked(vm, va, flags, caller); - spin_unlock(&vmap_area_lock); -} - static void clear_vm_uninitialized_flag(struct vm_struct *vm) { /* @@ -2592,14 +2589,16 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, if (!(flags & VM_NO_GUARD)) size += PAGE_SIZE; - va = alloc_vmap_area(size, align, start, end, node, gfp_mask, 0); + area->flags = flags; + area->caller = caller; + area->size = size; + + va = alloc_vmap_area(size, align, start, end, node, gfp_mask, 0, area); if (IS_ERR(va)) { kfree(area); return NULL; } - setup_vmalloc_vm(area, va, flags, caller); - /* * Mark pages for non-VM_ALLOC mappings as accessible. Do it now as a * best-effort approach, as they can be mapped outside of vmalloc code. base-commit: de927f6c0b07d9e698416c5b287c521b07694cac -- 2.43.0