From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2510FC4332F for ; Thu, 22 Dec 2022 14:22:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55F1E900003; Thu, 22 Dec 2022 09:22:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E84A900002; Thu, 22 Dec 2022 09:22:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33ACC900003; Thu, 22 Dec 2022 09:22:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1C5EE900002 for ; Thu, 22 Dec 2022 09:22:08 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D96D4C0742 for ; Thu, 22 Dec 2022 14:22:07 +0000 (UTC) X-FDA: 80270156694.10.43AE2BF Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf15.hostedemail.com (Postfix) with ESMTP id F00B8A000E for ; Thu, 22 Dec 2022 14:22:05 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Oqxt1IEW; spf=pass (imf15.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671718926; a=rsa-sha256; cv=none; b=CqXnco/ddA6RxVdobJWTmhRF1EyR9W70lY3GjV04DfJ+XQwA7o77LQTFKSv+KoHQ8piN4q vs5Ynl2SdSOgQHy2TAuWLM3HTtycDA9K1RWtTHAkWz9murXJPjmsfuOIZUrrFL65H+hLTf 7INL7exbpMT+woXk1yZlFyjMVwuU4/k= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Oqxt1IEW; spf=pass (imf15.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671718926; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WDd7wGgsS2+0SAKYEIGw8VztMH3gr95vwLLvFognSgs=; b=kw66/lj2Pz16W3fGCxLcVzCxUh3hqYL/hRDmDci8vDPcDWOTGJ8wvgjwNxaG+Jw7phmDB8 SsDRCFe4z1x9wbvUit2vZVtfvWgoBEaO5ChCO8SCuMxCENfJvogPqjXSdmPeAvxaQ7vw0Y 3z4Gz7zUYXs6m7K4eMKK3Y7jM0g5uSY= Received: by mail-lf1-f51.google.com with SMTP id o6so2947374lfi.5 for ; Thu, 22 Dec 2022 06:22:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:from:to:cc:subject:date:message-id:reply-to; bh=WDd7wGgsS2+0SAKYEIGw8VztMH3gr95vwLLvFognSgs=; b=Oqxt1IEWoAUAhvn6kaRaX+EwSjlEDb2kViD+Jq7Pzb80ZQhbTeRASJVyHAdlqk4DDO KDFBY3WZ0E8davmCLOOw7wNLE8EDHSBkUPmH5+nAWfBQlOLRLFeCUko1Vt4nfHG+NZDD YFQVhPAK6xI3o03fiwfQUu8yfTMOwqWkgDp7axfKHYQBILZZYbJJu0RtFus+xfLLCVLo 9/SGogEQCweufi6nLirldk0qkKjFDSfV/sr3ZTBMBC4Nl7nkSs/UQKbNZjWNrpSK+qX6 2QlyaPKRMG34uMOoc2VvOd5yZDAO1XYzdIp9ln7/4166/bubOjChH0fKHwAVfV9G15JT H+nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:date:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=WDd7wGgsS2+0SAKYEIGw8VztMH3gr95vwLLvFognSgs=; b=yowf9AW6HCoJRHbAvefsqYDXmPKcLsd+6Ohsn17xmQZtgBf16uM0lzRxxDopYJQ4Iy eZONopesfG/2+EcPtUdvMMKQu2FsWdLs/jWoeYaMJsOSWKpaIzmaLqDYU8SBnrq8+aZZ nW7XJ+V1nZVO/odLoRPB9Sr5U2e3Q+tKC8s3sSo8hLol1KaXggZ2+2yA/Sw1z2drbWmZ lXQvTbxJPvGP9XMj3ADQE/X464CizhUFOuED27xDDcl7BxWMJokNRlVnbQ2RhYVbZ9cq iiVu88/T2AI/rnvhimodR0m3IHsbDcWv1hgWydNr4UmozRjhZ70213+vNxPtg05d3e/C GIxA== X-Gm-Message-State: AFqh2kqPk2gHY4JVOZu9gY3CMrjF8a/N0vCqYBRw75JQ6e2elgPnutE4 VeYTJag0A8r6+VAuHXFOh1jMDl8baus= X-Google-Smtp-Source: AMrXdXtEL69S3W027zNimuG2jPMNkd1i6DxcGUOJOqJa/Z7nHMEwHbuOlU2URBcplUEghBYoQarZkw== X-Received: by 2002:a05:6512:2393:b0:4a4:68b9:1a00 with SMTP id c19-20020a056512239300b004a468b91a00mr2269494lfv.40.1671718923807; Thu, 22 Dec 2022 06:22:03 -0800 (PST) Received: from pc636 (host-90-233-218-120.mobileonline.telia.com. [90.233.218.120]) by smtp.gmail.com with ESMTPSA id d21-20020ac24c95000000b0049465afdd38sm83755lfl.108.2022.12.22.06.22.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 06:22:03 -0800 (PST) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Thu, 22 Dec 2022 15:22:01 +0100 To: Lorenzo Stoakes Cc: "Uladzislau Rezki (Sony)" , Andrew Morton , linux-mm@kvack.org, LKML , Baoquan He , Christoph Hellwig , Matthew Wilcox , Nicholas Piggin , Oleksiy Avramchenko , Roman Gushchin Subject: Re: [PATCH v2 1/3] mm: vmalloc: Avoid of calling __find_vmap_area() twise in __vunmap() Message-ID: References: <20221221174454.1085130-1-urezki@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: F00B8A000E X-Stat-Signature: kx1hn7dnu31j8ekuecxpzgsto4k7a5eu X-HE-Tag: 1671718925-328159 X-HE-Meta: U2FsdGVkX1+ygGtVXQDTEcHALAuYOrxN+Ws0CNcVhSEsXegzi8+AZYL7Wi/3iUzaIUkHZa86RZ1AODfYdx07S7EVcB0LnGtDN6iqe4mp/eg0M9MvbOApeAP1Iz+kV3BZU/Lhz9JIErZwo3BVUvVq+iWiqyrkXKzhleRJ2YRrM7e9KlXKY1/pzSQXPHPzgZu/dhK33ybLG1Fk0m+A7bgsLUsC0MSMcGV/3SJ3moGbdqS6eewGTsWGd7TJ0rgdY98i7Uje3+5YF0375NBMrDnRZyWBCXNtH/6V+USOwprl85/Jx/jTlcb/cRabA4bDAk5kDqTyyh/NrAsq/rfAXStNlunLc+dDsKzXkBDqZ83LBeTzbEVneTH4OJwNtFVQs7ewdjaGMiuD0nMxA/XhO8jbCujPMDf0/nFWZ0miSVbRAqORQXaDhSN/E6X9nlqacwFbLM9Zk5tuRC1n36efQDw9keJnF7ccDKXImu7RSjznq1DGfTQCb1v5MwpeGn9zUfzCPS2Vrc0tbYp+14ExpIpLknW2XOvbkBGr/hLOn0CfpDE+Qfe3f1MZRi6awQXvt0M2Ss9YHxXUR5qjgYvZkaScfJVixfteegyMTgY4tTilfZ1sWwDiPh4qB4+JfF+NI1ekXA5sGXfpKJeo9KRwUhQ884NPTQ5Gnw88rz+IIgJ8mOEKy6AxXYQyPezu1Jck4lx/ozf6WiSRCgYbp2O1eGbMsJVBe4lEVDjQGKywVGuqnbTKuzea/PcGfkxo3GV8w0RqX7ZOvMIg8DYvYPd8vC1QKD/1UTH5iHJQikT0dJ/6LIrMEHgd2KhUj6mFU9kxZc1WzIlM0ib+509EjnHzUVbACMQzjc4ug+Rg4HBRi+wAlpTJ+PyO5y6bNVi86mdfP6iSiZvpvm7jU1mmTWIigAFXv56f0g2tv4vEYIMZ4T65nLOFvQVplOHo3sy77ResvlnipZChN0zm6rGZ33/pPjr JOp9GzW8 R5wxik59nAuNSD17hZPY1QISCP9lEnOaZcMSL2mijRPIQI2lj9KKPnEumhZUtKyuT/XFD3rv2ai1dKJI59THsg1tiIpgTkx66Ej5qldGgwf+PVqQAlq73vn67rVe9iHPSnws3BHmz7/bCkvw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: > Some pedantic grammar/spelling stuff:- > > (I know it can be a little annoying to get grammatical suggestions so I do hope > that it isn't too irritating!) > It is absolutely OK :) > > For the Subject line:- > 'mm: vmalloc: Avoid of calling __find_vmap_area() twise in __vunmap()' -> > 'mm: vmalloc: Avoid calling __find_vmap_area() twice in __vunmap()' > Will fix in the v3. > On Wed, Dec 21, 2022 at 06:44:52PM +0100, Uladzislau Rezki (Sony) wrote: > > Currently __vunmap() path calls __find_vmap_area() two times. One on > > entry to check that area exists, second time inside remove_vm_area() > > function that also performs a new search of VA. > > Perhaps slightly tweak to:- > > "Currently the __vunmap() path calls __find_vmap_area() twice. Once on entry > to check that the area exists, then inside the remove_vm_area() function > which also performs a new search for the VA." > Will fix in the v3. > > > > In order to improvie it from a performance point of view we split > > remove_vm_area() into two new parts: > > - find_unlink_vmap_area() that does a search and unlink from tree; > > - __remove_vm_area() that does a removing but without searching. > > 'that does a removing but without searching' reads better I think as > 'that removes without searching'. > Will fix in the v3. > > > > In this case there is no any functional change for remove_vm_area() > > whereas vm_remove_mappings(), where a second search happens, switches > > to the __remove_vm_area() variant where already detached VA is passed > > as a parameter, so there is no need to find it again. > > > > 'where already detached VA' -> 'where the already detached VA' as a minor nit > here! > Will fix it. > > Performance wise, i use test_vmalloc.sh with 32 threads doing alloc > > free on a 64-CPUs-x86_64-box: > > > > perf without this patch: > > - 31.41% 0.50% vmalloc_test/10 [kernel.vmlinux] [k] __vunmap > > - 30.92% __vunmap > > - 17.67% _raw_spin_lock > > native_queued_spin_lock_slowpath > > - 12.33% remove_vm_area > > - 11.79% free_vmap_area_noflush > > - 11.18% _raw_spin_lock > > native_queued_spin_lock_slowpath > > 0.76% free_unref_page > > > > perf with this patch: > > - 11.35% 0.13% vmalloc_test/14 [kernel.vmlinux] [k] __vunmap > > - 11.23% __vunmap > > - 8.28% find_unlink_vmap_area > > - 7.95% _raw_spin_lock > > 7.44% native_queued_spin_lock_slowpath > > - 1.93% free_vmap_area_noflush > > - 0.56% _raw_spin_lock > > 0.53% native_queued_spin_lock_slowpath > > 0.60% __vunmap_range_noflush > > > > __vunmap() consumes around ~20% less CPU cycles on this test. > > Very nice, amazing work! > Thanks! > > > > Reported-by: Roman Gushchin > > Signed-off-by: Uladzislau Rezki (Sony) > > --- > > mm/vmalloc.c | 66 +++++++++++++++++++++++++++++++++------------------- > > 1 file changed, 42 insertions(+), 24 deletions(-) > > > > diff --git a/mm/vmalloc.c b/mm/vmalloc.c > > index 9e30f0b39203..28030d2441f1 100644 > > --- a/mm/vmalloc.c > > +++ b/mm/vmalloc.c > > @@ -1825,9 +1825,11 @@ static void free_vmap_area_noflush(struct vmap_area *va) > > unsigned long va_start = va->va_start; > > unsigned long nr_lazy; > > > > - spin_lock(&vmap_area_lock); > > - unlink_va(va, &vmap_area_root); > > - spin_unlock(&vmap_area_lock); > > + if (!list_empty(&va->list)) { > > + spin_lock(&vmap_area_lock); > > + unlink_va(va, &vmap_area_root); > > + spin_unlock(&vmap_area_lock); > > + } > > Do we want to do the same in free_vmap_area()? > The free_vmap_area() is a bit special. It only pairs with alloc_vmap_area(). There are two users and both invoke free_vmap_area() in a error path. So probably it would be good to remove it fully. But it requires some refactoring. > > > > nr_lazy = atomic_long_add_return((va->va_end - va->va_start) >> > > PAGE_SHIFT, &vmap_lazy_nr); > > @@ -1871,6 +1873,19 @@ struct vmap_area *find_vmap_area(unsigned long addr) > > return va; > > } > > > > +static struct vmap_area *find_unlink_vmap_area(unsigned long addr) > > +{ > > + struct vmap_area *va; > > + > > + spin_lock(&vmap_area_lock); > > + va = __find_vmap_area(addr, &vmap_area_root); > > + if (va) > > + unlink_va(va, &vmap_area_root); > > + spin_unlock(&vmap_area_lock); > > + > > + return va; > > +} > > + > > /*** Per cpu kva allocator ***/ > > > > /* > > @@ -2591,6 +2606,20 @@ struct vm_struct *find_vm_area(const void *addr) > > return va->vm; > > } > > > > +static struct vm_struct *__remove_vm_area(struct vmap_area *va) > > +{ > > + struct vm_struct *vm; > > + > > + if (!va || !va->vm) > > + return NULL; > > + > > + vm = va->vm; > > + kasan_free_module_shadow(vm); > > + free_unmap_vmap_area(va); > > + > > + return vm; > > +} > > + > > /** > > * remove_vm_area - find and remove a continuous kernel virtual area > > * @addr: base address > > @@ -2607,22 +2636,8 @@ struct vm_struct *remove_vm_area(const void *addr) > > > > might_sleep(); > > > > - spin_lock(&vmap_area_lock); > > - va = __find_vmap_area((unsigned long)addr, &vmap_area_root); > > - if (va && va->vm) { > > - struct vm_struct *vm = va->vm; > > - > > - va->vm = NULL; > > - spin_unlock(&vmap_area_lock); > > - > > - kasan_free_module_shadow(vm); > > - free_unmap_vmap_area(va); > > - > > - return vm; > > - } > > - > > - spin_unlock(&vmap_area_lock); > > - return NULL; > > + va = find_unlink_vmap_area((unsigned long) addr); > > + return __remove_vm_area(va); > > } > > Really nice separation of concerns and cleanup. > Thanks! > > > > static inline void set_area_direct_map(const struct vm_struct *area, > > @@ -2637,15 +2652,16 @@ static inline void set_area_direct_map(const struct vm_struct *area, > > } > > > > /* Handle removing and resetting vm mappings related to the vm_struct. */ > > -static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) > > +static void vm_remove_mappings(struct vmap_area *va, int deallocate_pages) > > Perhaps rename this to va_remove_mappings() or vmap_area_remove_mappings() since > it is now explicitly accepting a vmap_area rather than vm_struct? > I agree. There is a discrepancy. I can rename it to the va_remove_mappings() if there are no any complains from others. > > { > > + struct vm_struct *area = va->vm; > > unsigned long start = ULONG_MAX, end = 0; > > unsigned int page_order = vm_area_page_order(area); > > int flush_reset = area->flags & VM_FLUSH_RESET_PERMS; > > int flush_dmap = 0; > > int i; > > > > - remove_vm_area(area->addr); > > + __remove_vm_area(va); > > > > /* If this is not VM_FLUSH_RESET_PERMS memory, no need for the below. */ > > if (!flush_reset) > > @@ -2690,6 +2706,7 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) > > static void __vunmap(const void *addr, int deallocate_pages) > > { > > struct vm_struct *area; > > Feels like it's getting a bit confusing with 'va' representing vmap_area and > 'area' which represents... vm_struct (this file has a bunch of naming > inconsistencies like this actually), perhaps rename this to 'vm'? > We can. I think it should be a separate patch-set for refactoring of things like: va, vm, area, vmap, etc :) > > + struct vmap_area *va; > > > > if (!addr) > > return; > > @@ -2698,19 +2715,20 @@ static void __vunmap(const void *addr, int deallocate_pages) > > addr)) > > return; > > > > - area = find_vm_area(addr); > > - if (unlikely(!area)) { > > + va = find_unlink_vmap_area((unsigned long)addr); > > + if (unlikely(!va)) { > > WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n", > > addr); > > return; > > } > > > > + area = va->vm; > > debug_check_no_locks_freed(area->addr, get_vm_area_size(area)); > > debug_check_no_obj_freed(area->addr, get_vm_area_size(area)); > > > > kasan_poison_vmalloc(area->addr, get_vm_area_size(area)); > > > > - vm_remove_mappings(area, deallocate_pages); > > + vm_remove_mappings(va, deallocate_pages); > > > > if (deallocate_pages) { > > int i; > > -- > > 2.30.2 > > > > Other than some pendatic points about grammar/naming this looks really good! > Thank you for the review! -- Uladzislau Rezki