From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5454C77B75 for ; Mon, 22 May 2023 11:09:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C45E280008; Mon, 22 May 2023 07:09:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 973CA280002; Mon, 22 May 2023 07:09:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72908280008; Mon, 22 May 2023 07:09:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5E79A280002 for ; Mon, 22 May 2023 07:09:01 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 40A83160DF3 for ; Mon, 22 May 2023 11:09:01 +0000 (UTC) X-FDA: 80817618882.02.00D70D5 Received: from mail-ej1-f42.google.com (mail-ej1-f42.google.com [209.85.218.42]) by imf02.hostedemail.com (Postfix) with ESMTP id 5F60280014 for ; Mon, 22 May 2023 11:08:58 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (imf02.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684753739; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=O5zHBHTvGEb2Zu4p5/DPXJf8BJ5SNzb/BtA238s3igGpnUINc0PruhHGL1pVSr8XmuRAMR Tm7h2GVhd6ELP+Qcur4aFRRrIsFPpnYtS2Thy2x3Vk4I5JnuVnOxR3tToAtHtDRuEHDfhV EDqZKoMJio54EVd+CMCJ2XV65x/uMrk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (imf02.hostedemail.com: domain of urezki@gmail.com designates 209.85.218.42 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684753739; a=rsa-sha256; cv=none; b=eVFl3ZqVVrecrAP/IrT/6UTj5n/NvOd95tGbwricJbdbnRG7sWPmB5ihv+l4gaz4yKHuQo 82t2KgpSa7lxymfkYpDVTLIT9QQ/nCO+KkHzioZh8PYDcAt6TaNXUWwdPS4NA8xDcHDHFR VrYu3Hobv+yQ9jkrJXDl8ujjmQ8YJQU= Received: by mail-ej1-f42.google.com with SMTP id a640c23a62f3a-95fde138693so705259566b.0 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=W+JtWfgKVMFA959Uc5C8wiOgCsYc9+SYai8nupDO9KlapBo9VlFnbVWgJAQ004f/QK mZQgkKrt4awtWG7HH1AMVsq0BI7HDBAsUxUanMdXJuT1NjGXA8j91P6MNQVZ7/2q/uOl u8b46ysLxNwbLImafPVAYrKfHx5ge512UBdEXib49XuWenSq7+MfDhUI4cQN2Y29QWPG 8ukh8J/sV1fqVvb+o7S1LN2QlrXlkMocODGZyaNkalUBH3toAXu0mQeN6PrvNuypiv2Q zaaZIwTdJUy/uadYITY2t/C6AJEJ4AgkDslYAlw0hmiXnARf/cNIGN7mPPlYm11bAx6x 3eRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=HVqIffRRoQ/R9gChCVVzOCE9CDOwTzg6AkiKoP1DnQDWeBMpODsZsD2wqwDszuGCKR RZnfW9zpBH7IMD5VD4pRbLHqBxOkArLltThcYKW85hLhsXPk0pq8lQkavTC7WyMyeDge NQdVG7gnBWJYZGfHKy01zjxZiztMEXLj6vxbC7zbhJnjtEj9k8pnQw0+z2u9LwGY25Rg CFQikfhXgxGBAKNk1Fh0S/Qwa5WB1kC63wNoAmeMP0/2U/d/CxV6BcbmR7RBOnAPGK0R 7OL3obBeg6Fd7kNH0YBFMcNFrOgJyVRM3PsHxSwhAuk5PcDC8QlL37EVnKOfvV6zIor6 Gi5w== X-Gm-Message-State: AC+VfDz5xgoLpAkVSm3jIFgaDGqSEkk1ca1kOczq+yjLlzo+0YMnzNB1 DQ6DTkJI92ALczRoH+Lvzxur/dZ/cG6qCw== X-Google-Smtp-Source: ACHHUZ4lpJZI6b774mW0GGFUcH9S7bFeeu1TaXyRQOPcceEGL/AxG3iytu1ZZc95ZLsmsCWn+xcxSw== X-Received: by 2002:a17:907:3eaa:b0:96f:8d00:43be with SMTP id hs42-20020a1709073eaa00b0096f8d0043bemr6754703ejc.0.1684753737692; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 7/9] mm: vmalloc: Insert lazy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:47 +0200 Message-Id: <20230522110849.2921-8-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 5F60280014 X-Rspam-User: X-Stat-Signature: wsxdjkmfy5sm949j7xdjmssrugii375n X-Rspamd-Server: rspam01 X-HE-Tag: 1684753738-835957 X-HE-Meta: U2FsdGVkX18xxNeK2sxwAAD+C+IaaNobyZDW6ZpxQVMtA5t3BoId3m4NYVWNwev1r18yxBCg+GoOUYxZGI+nuJbRYAGdXcmfCrTbWPg+SZzXPytMAr2+zxmWBFIZFTe3q7d74b8eBWpSrpLuJ6VnepPYBCp9gwx+ga4A5x0rxQw0vzV9mv79asr9JruF3dBdchaYDa7/NOXeKsj5eeXa/ud4TPeIpZgMAHQ7pZibr3N+sxKeM6ohqwCbDWYnouwwMjjW6/IWiegUqkvJZG5RSL3/XVtIXxMP029T+6sjITD+sP6iR4BFXkgw/7+wmaLQ/JO60i/CsNptDnjJMmeUh1u65bkgY5yo2z9NIjHBN5L1Qds3R2vKMczDxJ84DjGsCYLYrokyrXqAUaF+Sx8KsZlih3qaBS/7JO0YxHx6Ky1c5ZfBXUhGji9dmogoVLHhmaF783R6pe7DjtxkN1+78hE3hjD4iSQkSFlPspnC8SSfIq6oGdSsHRfk/Xe76IaDbhAU6gdLL0L14M7kjIak5bn6nqWeTYqc8aRmEdP0h5eaUklJ9K+GLzQrJZMsmL+Fcj9+muJGC4tZImZNE6ImRTlVEup2toRVWmXc/eQ0C2wp85JLnf8IKd8WBvhtNHAJK2X3ObWJFfJg5hZD77ymvycoWKWVB+yuSPSkvFUWb20WDbKKnqmBuPdK5dtK7ck9QwVH/PCUg3i5SGVbZgn4/nugfOsussXhOG3JO+SXd1FvpWKDThokbLJEl9ZvDIqH8zAkNBYfRgxFZzaDBXwjACsydWkzbk7L2yR5EWMf7CpzzqOW9h8kGrNsiy8CuZjvr5lMIi+nlGnnLoGDkuD8SX75d4FaTBnof4XMeoC9JEkV94qb81lWZWfSDoNQsuAifQfzFfnP3VWXh7XrjL1dRNQACFwBDCAzdYS47BpPi+RjNKPr19DT6p0quuXAzIoibmvT5/wkuBYaeygStps n2sSU7vv kET8RP/S7tl/hLJoJOYJPo1pFrEIrttfrcT2W/A3NJlKnPyp8x9FBxDYm7cvN/OQEg51CUkkRzfyEVQoAWndtn51YKF3NzxP7Fw0aBcvaPv3nH22l/rhVNnd4kgqoWMXI20+YRb24VmcBlqWAG3B0OF16XLYi373fVYWqZMD86v/NjYQ/2/T4wT19hXh1WQ2Ad59pdihaztokwQKojDVwg0rNYESozA3gGCvOgIEgjKxbAqPUkz0tNV+sh0yFW7bX4xHt/NTlH5XwokVzVlDo/G4ttAtpT14wO+N9SKg1hoqfRskD2QTfYEdbx2NDsBcTfBV2/cCrV9pzLmZveC6y6Pk/LmHlScmdJnArmd5hyppDHOsIPxUR8lSBw0i8BfhCeAsxkEhLWItVjoiGt+R4lvsFxkvEjBJWlk8ZMI0NpN9ymHtpIFGEva4ZZGo7U2Z3Vdv/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to busy VAs, lazy ones are stored per a CPU zone also. Freed address is converted into a correct zone it belongs to and resides there for further handling. Such approach does not require to have any global locking primitive, instead an access becomes scalable to number of CPUs. This patch removes a global purge-lock, global purge-tree and list. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 56 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dd83deb5ef4f..fe993c0561dd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -734,10 +734,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock); LIST_HEAD(vmap_area_list); static bool vmap_initialized __read_mostly; -static struct rb_root purge_vmap_area_root = RB_ROOT; -static LIST_HEAD(purge_vmap_area_list); -static DEFINE_SPINLOCK(purge_vmap_area_lock); - /* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to @@ -1792,39 +1788,17 @@ static DEFINE_MUTEX(vmap_purge_lock); /* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus(void); -/* - * Purges all lazily-freed vmap areas. - */ -static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +static unsigned long +purge_cpu_vmap_zone(struct cpu_vmap_zone *z) { - unsigned long resched_threshold; - unsigned int num_purged_areas = 0; - struct list_head local_purge_list; + unsigned long num_purged_areas = 0; struct vmap_area *va, *n_va; - lockdep_assert_held(&vmap_purge_lock); - - spin_lock(&purge_vmap_area_lock); - purge_vmap_area_root = RB_ROOT; - list_replace_init(&purge_vmap_area_list, &local_purge_list); - spin_unlock(&purge_vmap_area_lock); - - if (unlikely(list_empty(&local_purge_list))) + if (list_empty(&z->purge_list)) goto out; - start = min(start, - list_first_entry(&local_purge_list, - struct vmap_area, list)->va_start); - - end = max(end, - list_last_entry(&local_purge_list, - struct vmap_area, list)->va_end); - - flush_tlb_kernel_range(start, end); - resched_threshold = lazy_max_pages() << 1; - spin_lock(&free_vmap_area_lock); - list_for_each_entry_safe(va, n_va, &local_purge_list, list) { + list_for_each_entry_safe(va, n_va, &z->purge_list, list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; unsigned long orig_start = va->va_start; unsigned long orig_end = va->va_end; @@ -1846,13 +1820,57 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) atomic_long_sub(nr, &vmap_lazy_nr); num_purged_areas++; - - if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) - cond_resched_lock(&free_vmap_area_lock); } spin_unlock(&free_vmap_area_lock); out: + return num_purged_areas; +} + +/* + * Purges all lazily-freed vmap areas. + */ +static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +{ + unsigned long num_purged_areas = 0; + struct cpu_vmap_zone *z; + int need_purge = 0; + int i; + + lockdep_assert_held(&vmap_purge_lock); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + INIT_LIST_HEAD(&z->purge_list); + + if (RB_EMPTY_ROOT(&fbl_root(z, LAZY))) + continue; + + fbl_lock(z, LAZY); + WRITE_ONCE(fbl(z, LAZY, root.rb_node), NULL); + list_replace_init(&fbl_head(z, LAZY), &z->purge_list); + fbl_unlock(z, LAZY); + + start = min(start, + list_first_entry(&z->purge_list, + struct vmap_area, list)->va_start); + + end = max(end, + list_last_entry(&z->purge_list, + struct vmap_area, list)->va_end); + + need_purge++; + } + + if (need_purge) { + flush_tlb_kernel_range(start, end); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + num_purged_areas += purge_cpu_vmap_zone(z); + } + } + trace_purge_vmap_area_lazy(start, end, num_purged_areas); return num_purged_areas > 0; } @@ -1870,16 +1888,9 @@ static void purge_vmap_area_lazy(void) static void drain_vmap_area_work(struct work_struct *work) { - unsigned long nr_lazy; - - do { - mutex_lock(&vmap_purge_lock); - __purge_vmap_area_lazy(ULONG_MAX, 0); - mutex_unlock(&vmap_purge_lock); - - /* Recheck if further work is required. */ - nr_lazy = atomic_long_read(&vmap_lazy_nr); - } while (nr_lazy > lazy_max_pages()); + mutex_lock(&vmap_purge_lock); + __purge_vmap_area_lazy(ULONG_MAX, 0); + mutex_unlock(&vmap_purge_lock); } /* @@ -1889,6 +1900,7 @@ static void drain_vmap_area_work(struct work_struct *work) */ static void free_vmap_area_noflush(struct vmap_area *va) { + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); unsigned long nr_lazy_max = lazy_max_pages(); unsigned long va_start = va->va_start; unsigned long nr_lazy; @@ -1902,10 +1914,9 @@ static void free_vmap_area_noflush(struct vmap_area *va) /* * Merge or place it to the purge tree/list. */ - spin_lock(&purge_vmap_area_lock); - merge_or_add_vmap_area(va, - &purge_vmap_area_root, &purge_vmap_area_list); - spin_unlock(&purge_vmap_area_lock); + fbl_lock(z, LAZY); + merge_or_add_vmap_area(va, &fbl_root(z, LAZY), &fbl_head(z, LAZY)); + fbl_unlock(z, LAZY); trace_free_vmap_area_noflush(va_start, nr_lazy, nr_lazy_max); @@ -4199,17 +4210,21 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) static void show_purge_info(struct seq_file *m) { + struct cpu_vmap_zone *z; struct vmap_area *va; + int i; - mutex_lock(&vmap_purge_lock); - spin_lock(&purge_vmap_area_lock); - list_for_each_entry(va, &purge_vmap_area_list, list) { - seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + + fbl_lock(z, LAZY); + list_for_each_entry(va, &fbl_head(z, LAZY), list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } + fbl_unlock(z, LAZY); } - spin_unlock(&purge_vmap_area_lock); - mutex_unlock(&vmap_purge_lock); } static int s_show(struct seq_file *m, void *p) -- 2.30.2