From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24594C433DF for ; Wed, 3 Jun 2020 17:14:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB87920674 for ; Wed, 3 Jun 2020 17:14:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="qtPiRTay" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DB87920674 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 318EF80007; Wed, 3 Jun 2020 13:14:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C8D78E0006; Wed, 3 Jun 2020 13:14:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1DEAE80007; Wed, 3 Jun 2020 13:14:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id 044748E0006 for ; Wed, 3 Jun 2020 13:14:52 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id B5FFB1388DD for ; Wed, 3 Jun 2020 17:14:52 +0000 (UTC) X-FDA: 76888550424.14.thing04_479dab2bc433f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 580CD1868BA63 for ; Wed, 3 Jun 2020 17:14:52 +0000 (UTC) X-HE-Tag: thing04_479dab2bc433f X-Filterd-Recvd-Size: 5005 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Jun 2020 17:14:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type: Content-ID:Content-Description:In-Reply-To:References; bh=4Q/XcaC/L2EioezqtHqpFhDtoWqsVCKp+4jwJq/wzHQ=; b=qtPiRTaymdbkdupQDGkE7FxSNw yTWXX3jbGekamRMIDMCsqN4MJFSI+dJAFYFs1EIyKSME+UuPoSzT9K4sOZMPgc1zjoX9e8x3T9J+9 0MRzXHwY1SHK+8MVkqvmQ+WLIACTgmwTWLbMdb0L8fZUK8z/OSjuAqm4AjU0zMfK/jhj7YYP7UQCn jba+CFLzm8p9PLX6dhY2Wu/8g5oazsenooTQ3tCYtZlgMV722VDmuV0t/JUdpbUJ8dRIWjWrCkHaM PDuLLdueNiQaGtX/hyS5x2Q3g/w/mQAyrAD5apMbvR2w5vrKXMDzlO17Mcqe323kLxQ9dZ9pOl2KO B5TZOkmw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jgWyY-0001fa-Bh; Wed, 03 Jun 2020 17:14:50 +0000 From: Matthew Wilcox To: Andrew Morton , linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" Subject: [PATCH] vmalloc: Convert to XArray Date: Wed, 3 Jun 2020 10:14:48 -0700 Message-Id: <20200603171448.5894-1-willy@infradead.org> X-Mailer: git-send-email 2.21.1 MIME-Version: 1.0 X-Rspamd-Queue-Id: 580CD1868BA63 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The radix tree of vmap blocks is simpler to express as an XArray. Reduces both the text and data sizes of the object file and eliminates a user of the radix tree preload API. Signed-off-by: Matthew Wilcox (Oracle) --- mm/vmalloc.c | 40 +++++++++++----------------------------- 1 file changed, 11 insertions(+), 29 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1e94497b7388..375bbb410a94 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -25,7 +25,7 @@ #include #include #include -#include +#include #include #include #include @@ -1513,12 +1513,11 @@ struct vmap_block { static DEFINE_PER_CPU(struct vmap_block_queue, vmap_block_queue); =20 /* - * Radix tree of vmap blocks, indexed by address, to quickly find a vmap= block + * XArray of vmap blocks, indexed by address, to quickly find a vmap blo= ck * in the free path. Could get rid of this if we change the API to retur= n a * "cookie" from alloc, to be passed to free. But no big deal yet. */ -static DEFINE_SPINLOCK(vmap_block_tree_lock); -static RADIX_TREE(vmap_block_tree, GFP_ATOMIC); +static DEFINE_XARRAY(vmap_blocks); =20 /* * We should probably have a fallback mechanism to allocate virtual memo= ry @@ -1575,13 +1574,6 @@ static void *new_vmap_block(unsigned int order, gf= p_t gfp_mask) return ERR_CAST(va); } =20 - err =3D radix_tree_preload(gfp_mask); - if (unlikely(err)) { - kfree(vb); - free_vmap_area(va); - return ERR_PTR(err); - } - vaddr =3D vmap_block_vaddr(va->va_start, 0); spin_lock_init(&vb->lock); vb->va =3D va; @@ -1594,11 +1586,12 @@ static void *new_vmap_block(unsigned int order, g= fp_t gfp_mask) INIT_LIST_HEAD(&vb->free_list); =20 vb_idx =3D addr_to_vb_idx(va->va_start); - spin_lock(&vmap_block_tree_lock); - err =3D radix_tree_insert(&vmap_block_tree, vb_idx, vb); - spin_unlock(&vmap_block_tree_lock); - BUG_ON(err); - radix_tree_preload_end(); + err =3D xa_insert(&vmap_blocks, vb_idx, vb, gfp_mask); + if (err) { + kfree(vb); + free_vmap_area(va); + return ERR_PTR(err); + } =20 vbq =3D &get_cpu_var(vmap_block_queue); spin_lock(&vbq->lock); @@ -1612,12 +1605,8 @@ static void *new_vmap_block(unsigned int order, gf= p_t gfp_mask) static void free_vmap_block(struct vmap_block *vb) { struct vmap_block *tmp; - unsigned long vb_idx; =20 - vb_idx =3D addr_to_vb_idx(vb->va->va_start); - spin_lock(&vmap_block_tree_lock); - tmp =3D radix_tree_delete(&vmap_block_tree, vb_idx); - spin_unlock(&vmap_block_tree_lock); + tmp =3D xa_erase(&vmap_blocks, addr_to_vb_idx(vb->va->va_start)); BUG_ON(tmp !=3D vb); =20 free_vmap_area_noflush(vb->va); @@ -1723,7 +1712,6 @@ static void *vb_alloc(unsigned long size, gfp_t gfp= _mask) static void vb_free(unsigned long addr, unsigned long size) { unsigned long offset; - unsigned long vb_idx; unsigned int order; struct vmap_block *vb; =20 @@ -1733,14 +1721,8 @@ static void vb_free(unsigned long addr, unsigned l= ong size) flush_cache_vunmap(addr, addr + size); =20 order =3D get_order(size); - offset =3D (addr & (VMAP_BLOCK_SIZE - 1)) >> PAGE_SHIFT; - - vb_idx =3D addr_to_vb_idx(addr); - rcu_read_lock(); - vb =3D radix_tree_lookup(&vmap_block_tree, vb_idx); - rcu_read_unlock(); - BUG_ON(!vb); + vb =3D xa_load(&vmap_blocks, addr_to_vb_idx(addr)); =20 unmap_kernel_range_noflush(addr, size); =20 --=20 2.26.2