From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C885BC77B73 for ; Sat, 27 May 2023 17:38:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5DA0900002; Sat, 27 May 2023 13:38:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0D526B0072; Sat, 27 May 2023 13:38:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CD642900002; Sat, 27 May 2023 13:38:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BCB896B0071 for ; Sat, 27 May 2023 13:38:46 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 78D6DC029B for ; Sat, 27 May 2023 17:38:46 +0000 (UTC) X-FDA: 80836745052.09.C9FF92E Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf04.hostedemail.com (Postfix) with ESMTP id 937DF40002 for ; Sat, 27 May 2023 17:38:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=oTvcJliD; spf=pass (imf04.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685209124; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=T2QEnc04K6XQU731V+g/CSxV+BmIh9ersch9wSssbZ4=; b=dKhw+FLGvIq3oyKFHyYDuwKH2J4MX3JNMb78qaN1WCXTj1wgQ4ZKNHFfK2oa0CpZwRnG2z OzZ1Li+z9Vyv6lCXSlGe0oJHGnTSV9iBRfqk2Pkfn6fQP5jbkAZaQzHowF9KbdU/hop5BT 8xFwADngdeE1pVofKcHK5y53vnv98l8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685209124; a=rsa-sha256; cv=none; b=g5DfNAprMqYSSid1Co2AaAIRXP7AMU9cZsmFPP7gyV1C/cgHWGiLd0NWn6pDOZYTE5Pzni ICDo1Io5uXgwlvIrN+uA+GmrhlhMUjC+YlHPSqEY0yIPn0zq7EtUb5OM5lsnsgJcqb/rAU WcKzHcza+Pm7Ofjq5ybPPp/rv+gOgdc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=oTvcJliD; spf=pass (imf04.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=lstoakes@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-3f6042d605dso12595435e9.2 for ; Sat, 27 May 2023 10:38:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685209123; x=1687801123; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=T2QEnc04K6XQU731V+g/CSxV+BmIh9ersch9wSssbZ4=; b=oTvcJliDtct0H6q1cTQIpLQjma/hXqi5aL2A7en+aELa+o1HZC0H6icMt7/SIn3/31 PI/H8GwoZ7sk5YPrG4kNan/2UMVBfAu9mEFgujTJ3MMm90byNXg3+jDgIuNhgC/EfOPu J5IaCKwKNcvP1UZD+bKDOlMObvsGiWcnwBeKEtuYNFtuRAFq4baj2lt8j+0aPLo3xVD/ tznxlNQ7z0INkN6qcvQ5f7+KGJWbEzPVZRbXLoyBzHhTTRBIlJnxNT2UL/TaRVzR9D9J 8ttD8fYBpc4XPgUXbIcaHe9iVqE/QpDQS1XGTJZmiX9OzP3Ji9FXXlX91ibrojQNroP3 TkDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685209123; x=1687801123; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=T2QEnc04K6XQU731V+g/CSxV+BmIh9ersch9wSssbZ4=; b=Lvb0APKrrcepYcxeUBK5ql9GcWyNxpFVsXy8TvZQAb18Db7O7JGldGPcnud8IkImK/ Fsa3C++DEJBAuHNgsB8LQ4MfdnngcHOjCGEtr8j7Tz2ZV8URJycQClSle05eIeBQgONb IUefWXkVz1yt4f3jFPYYOv0on069txRnmLk8xu1L4Lz/wyucDbIPjt8A1qW/bMXsxbQw ++l20dgNhVyR01kOkA3inZLVc5gacPxJTc43YS/wDb+jeGN3w+smzIoMFogkFAUllGJ4 j8hjd5TzfXYTBRL+uz/rhQEk91OqbFj+jytkOSzv4fxIiqbkRR6m+teMDhR/cXxOM8IN g+wQ== X-Gm-Message-State: AC+VfDwAs6dLYvTsE4cMqIIF/gkjBDp/sHNuUEMmTkzhu6Bwj2jSccRj Ti1kRTuwQSenv3e/HX4zOf0= X-Google-Smtp-Source: ACHHUZ5BlC90RdGhJZNpQkXcfbrwWt3eOnfrrvxltYUxh7mMZerO3XymyWBUTBu5wUeceEg/1BWk7Q== X-Received: by 2002:a05:600c:2284:b0:3f6:42ce:a384 with SMTP id 4-20020a05600c228400b003f642cea384mr4048863wmf.11.1685209122751; Sat, 27 May 2023 10:38:42 -0700 (PDT) Received: from localhost (host81-154-179-160.range81-154.btcentralplus.com. [81.154.179.160]) by smtp.gmail.com with ESMTPSA id v15-20020a5d4a4f000000b00307c0afc030sm8729157wrs.4.2023.05.27.10.38.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 27 May 2023 10:38:41 -0700 (PDT) Date: Sat, 27 May 2023 18:38:40 +0100 From: Lorenzo Stoakes To: Thomas Gleixner Cc: linux-mm@kvack.org, Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Peter Zijlstra , Baoquan He Subject: Re: [V2 patch 2/6] mm/vmalloc: Avoid iterating over per CPU vmap blocks twice Message-ID: <08636589-ed5e-4e22-a19f-e4bc05eb1fec@lucifer.local> References: <20230525122342.109672430@linutronix.de> <20230525124504.633469722@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230525124504.633469722@linutronix.de> X-Rspamd-Queue-Id: 937DF40002 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: t55wnb7gsn3nbwox6kk6t8d655qa4st5 X-HE-Tag: 1685209124-574981 X-HE-Meta: U2FsdGVkX19dPtQHlaMKekQCen7vpqQozdOOY4bNmJu7Ok+l2klDCmySJDLBR+HqO61DKzMBssjWsWK74gmf4PPVI+qLPDtsFTvr+kfNOoqFQ+2msy6Pp2xrcPjDO8G0BMqTcNpKeh2NmHiFabK7GRRVonp79bOSl0D8lLHGiNdSBI2FNZxwcuBRoMgmL/sppePRGR4cli6OsGbB/j1HI+9V3QwjoR5mnJ1ajKx4i2PM0qj/nNI/28hZGWev5PeH3E9gBB5AuC99DRkMkpDNUQi6WnFaht0MSu+1ssvXbzp91ojhflygSObAEv0MZIDsNBsUJViSwX5f+lmcIzxctO0GProGOONcbYTlJPneZevks9xbjvc6O2GDNI3tMCvQO8k6IzpzENVDKL/zP/8KjUqTRM86gyylNL8DXPbBynG+3qaGwuYDz9U6kUScFqU+4oeL5qwZMNPCSfcAw/6KzNeeUWMhMEuJM+Y1qjnUWsy0hhYpOQ+scTcYi2yfgc+NTp4PP9FExziqMdHFKsk3q6yduKQa418l//EuLHgrnUwE2Yg1xIRANurqZuAqQgZmRlRKSC3EVKgOtNNh35w9q0A0TRe7mdguw+dNR4ENcRFnfjeocKAnc4g82TaKi5UbLylzpnX8Jb2fsXQFFQkSjJPXRhU5ddmterJEZgqQ1gdnBA1R93IxfrfQBI1zH70NjHsSspxZdhStNmm8ofAcRfhgKrz2L5EileuFRvAnuyiDvtISlgGV82x6vs1mUPzPyFFl9WuroaLDtzfvhs0uDGGVMnmFb6/oDQ8xL/LSIMGng39gdcVCRJkESuxEKaA8+j0QApBCRaXSCHqKFTQO0/am8/7kIGvTw0hyx+5iaHtoIDGTH6oiLSBsH/QMPUlDJlM/MFei8TcQn01P5mvi+Nsq+6uMjwTGw0k/JrEnMSWayvczJfYjunUhjDMB+PUOftSuq3i5dcWTTgAjE9g dJqvwCYJ G60A6qsI7vqBDuDCz6gWMgR7iGu/yQPA9PzgDr2TEk6y5hNeuBwsgmpGf/KgeDaZcBbuRioLLvVNX4USBDcQYiRjFasmzz5FmaftZ3nVyhH5Sct6Mv775wsrrs2mBI6kFQ7m0C3IfPt8A0teBCX/eCrG/VWePJOjLRvvXd/i6Ki70nq7WhlDX1aGORPXUjcrCPdN1unhvD/3cT6MSTf7d1JPgulT3U2P2z/lAq22E98vKCr4Q4RqXjfX63BotDlcziQ9JvQ9oOtBzcDf1w1vYPGwbF55cZAos0VLbgRFw0EToB55VCJZKu/rMjoAGgN4obbEj9JD7rDVGMz90CiS2zk8ByfICzPRCyJhwKa3ZVqOdJ2QJQHeZi6W/V2t831pB6IslF1Lcq3001y4tAQrTG1CKJo5M6PIQAGpb X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, May 25, 2023 at 02:57:04PM +0200, Thomas Gleixner wrote: > _vunmap_aliases() walks the per CPU xarrays to find partially unmapped > blocks and then walks the per cpu free lists to purge fragmented blocks. > > Arguably that's waste of CPU cycles and cache lines as the full xarray walk > already touches every block. > > Avoid this double iteration: > > - Split out the code to purge one block and the code to free the local > purge list into helper functions. > > - Try to purge the fragmented blocks in the xarray walk before looking at > their dirty space. > > Signed-off-by: Thomas Gleixner > --- > V2: Fix coding style issues - Christoph > --- > mm/vmalloc.c | 70 ++++++++++++++++++++++++++++++++++++++--------------------- > 1 file changed, 46 insertions(+), 24 deletions(-) > > --- a/mm/vmalloc.c > +++ b/mm/vmalloc.c > @@ -2086,39 +2086,54 @@ static void free_vmap_block(struct vmap_ > kfree_rcu(vb, rcu_head); > } > > +static bool purge_fragmented_block(struct vmap_block *vb, > + struct vmap_block_queue *vbq, struct list_head *purge_list) > +{ > + if (vb->free + vb->dirty != VMAP_BBMAP_BITS || > + vb->dirty == VMAP_BBMAP_BITS) > + return false; > + > + /* prevent further allocs after releasing lock */ > + vb->free = 0; > + /* prevent purging it again */ > + vb->dirty = VMAP_BBMAP_BITS; > + vb->dirty_min = 0; > + vb->dirty_max = VMAP_BBMAP_BITS; > + spin_lock(&vbq->lock); > + list_del_rcu(&vb->free_list); > + spin_unlock(&vbq->lock); > + list_add_tail(&vb->purge, purge_list); > + return true; > +} > + > +static void free_purged_blocks(struct list_head *purge_list) > +{ > + struct vmap_block *vb, *n_vb; > + > + list_for_each_entry_safe(vb, n_vb, purge_list, purge) { > + list_del(&vb->purge); > + free_vmap_block(vb); > + } > +} > + > static void purge_fragmented_blocks(int cpu) > { > LIST_HEAD(purge); > struct vmap_block *vb; > - struct vmap_block *n_vb; > struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); > > rcu_read_lock(); > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > - > - if (!(vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS)) > + if (vb->free + vb->dirty != VMAP_BBMAP_BITS || > + vb->dirty == VMAP_BBMAP_BITS) > continue; > > spin_lock(&vb->lock); > - if (vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS) { > - vb->free = 0; /* prevent further allocs after releasing lock */ > - vb->dirty = VMAP_BBMAP_BITS; /* prevent purging it again */ > - vb->dirty_min = 0; > - vb->dirty_max = VMAP_BBMAP_BITS; > - spin_lock(&vbq->lock); > - list_del_rcu(&vb->free_list); > - spin_unlock(&vbq->lock); > - spin_unlock(&vb->lock); > - list_add_tail(&vb->purge, &purge); > - } else > - spin_unlock(&vb->lock); > + purge_fragmented_block(vb, vbq, &purge); > + spin_unlock(&vb->lock); > } > rcu_read_unlock(); > - > - list_for_each_entry_safe(vb, n_vb, &purge, purge) { > - list_del(&vb->purge); > - free_vmap_block(vb); > - } > + free_purged_blocks(&purge); > } > > static void purge_fragmented_blocks_allcpus(void) > @@ -2226,12 +2241,13 @@ static void vb_free(unsigned long addr, > > static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) > { > + LIST_HEAD(purge_list); > int cpu; > > if (unlikely(!vmap_initialized)) > return; > > - might_sleep(); > + mutex_lock(&vmap_purge_lock); > > for_each_possible_cpu(cpu) { > struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); > @@ -2241,7 +2257,14 @@ static void _vm_unmap_aliases(unsigned l > rcu_read_lock(); > xa_for_each(&vbq->vmap_blocks, idx, vb) { > spin_lock(&vb->lock); > - if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > + > + /* > + * Try to purge a fragmented block first. If it's > + * not purgeable, check whether there is dirty > + * space to be flushed. > + */ > + if (!purge_fragmented_block(vb, vbq, &purge_list) && > + vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > unsigned long va_start = vb->va->va_start; > unsigned long s, e; > > @@ -2257,9 +2280,8 @@ static void _vm_unmap_aliases(unsigned l > } > rcu_read_unlock(); > } > + free_purged_blocks(&purge_list); > > - mutex_lock(&vmap_purge_lock); > - purge_fragmented_blocks_allcpus(); Nice catch... and ugh that we were repeating work here. > if (!__purge_vmap_area_lazy(start, end) && flush) > flush_tlb_kernel_range(start, end); > mutex_unlock(&vmap_purge_lock); > > Overall nice cleanup, it feels like vmalloc has a lot of dirty, musty corners. Glad you've broken out the feather duster :) Reviewed-by: Lorenzo Stoakes