From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19084C77B7E for ; Thu, 25 May 2023 12:57:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7425900004; Thu, 25 May 2023 08:57:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A245E900002; Thu, 25 May 2023 08:57:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91462900004; Thu, 25 May 2023 08:57:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 77150900002 for ; Thu, 25 May 2023 08:57:07 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3C9D9AD925 for ; Thu, 25 May 2023 12:57:07 +0000 (UTC) X-FDA: 80828777694.25.D63B752 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf02.hostedemail.com (Postfix) with ESMTP id 4B13D80014 for ; Thu, 25 May 2023 12:57:04 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=KEv1FJcW; dkim=pass header.d=linutronix.de header.s=2020e header.b="G7/CpQCG"; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf02.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685019425; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=TpSIWN1012C/jnF1qSmFVH5HMCSWBYgnygUEcktXhkI=; b=hzWbFUuG4PoTQngeb+khOmJ/1uQX7q++KqkXwScql1TTmQNX3GZgxPGtmYOSMMdHuveBoK 0xtEexL9P5xoQSwZptbC27XJ0EyOOH1IAhRnOCicrm+G4rL7O1x38xQXrbiZ7RffJwVeVr 8QSkXRkJuF3Prm2oUeMwhJyWH2a1HKY= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=KEv1FJcW; dkim=pass header.d=linutronix.de header.s=2020e header.b="G7/CpQCG"; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf02.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685019425; a=rsa-sha256; cv=none; b=dbawLZ2aIkIpKKW2//09XpA9FJMF+3Fz3cSbn8gCBO2+bkl+a76fR9xZ6JQ86W7lUwdpsz NpmINEecmeCb0RV0WikjvZkrbEu8/mDi4RUEs086a8L4Df8GdcMd3zmDmvbLFEHNHDJgTC 85a6c35rlFRsFobILJaGqdGovzH1JBc= Message-ID: <20230525124504.573987880@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1685019423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TpSIWN1012C/jnF1qSmFVH5HMCSWBYgnygUEcktXhkI=; b=KEv1FJcWXGjYuPTissm98dgXrjzYtKGeTchv/bqymtn7Hob0sLHq+4JkooubgNejovd6Pe R2eXOssc13Pl6nnUhqQ6ZhdTTG+tF23Ab7zA9kyggK0cTYAWwlXkBt+xKJFScHc18lkRab vJiLUl3qHF3qIzskn+WwI1ovI8lT2BAE3nKlhBjZsNmEz4TpMU7k9TmOsAoMvYUUHNTKFT pSJKlU5ZB9fd8zjd4ePtQLR9FhoxKm/sYTVPZ7i119ha8p+/4P6MOx1w9eIHKXQ8bBD9+h jBizkB1tQO44Fmw/lUuQsZUqAUw7CFi1f8jEPpBWH6nRPux0TOBJ8Q+DTJeDvA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1685019423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=TpSIWN1012C/jnF1qSmFVH5HMCSWBYgnygUEcktXhkI=; b=G7/CpQCG9iUMYHs06QnhMQl0ubuNCS3SSEqBlEP2RF8a+sc8C8xIfphrOiIz8H1aJxfxB2 iWfaN2YwgXYMZlDw== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [V2 patch 1/6] mm/vmalloc: Prevent stale TLBs in fully utilized blocks References: <20230525122342.109672430@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Date: Thu, 25 May 2023 14:57:03 +0200 (CEST) X-Rspam-User: X-Stat-Signature: 33x64w7tqta9qyn3no3gwx3hw77f1b13 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4B13D80014 X-HE-Tag: 1685019424-676522 X-HE-Meta: U2FsdGVkX197Q+ELY4iMWurhxHLDn0fSBRkJrtcsyYZNpBz2jxrN0kUpEckpvnpePUT1p+eKpKKQToT5XUs4GX3OVf7i20UMTWcPhpNGX2PBd1ZX6aBHYAJtxqoDAvUuwFkqA5oPB/L9gRgbA1zx7oZTA1DSIMTnGOh+0mr2eY4Is/wfaliDejEwr+elDBCtomcmtS/WnCsMdarA3lB9RTi2BGPuBqSrWrxZKm2sZNEtqh2+LfNNd69JihX2zrc7cmQZrpV4UrUT8vbKyXEOpF0a3pI6ra1b4MdbHzc0s7wrNUclpz2ZPQDyFvhByVgRBCAn0BIiVG7zT69sdX4Ryn9yBR2BB6ck1kidSrV6BYGtMDLgSgTBOupq+0kHL4Ij15oREZoL8g7URga9UlKTTQQlxufY7RWBSCXq1ltp5kxovKoZr/jQMPUiuacwMvx+rqfW7TC7DdW2l2TiPgFIBYxRqksCnEgrq0Yo13S59V5wPX9+v2PKEn4RouVZAjXzN2hhF9aYqbpllp86P0zNFZ1FLZZwV/pkcn3ICAhLlfzkxaNuUU+DEFX/GxgAC1oTpn8nOCr829k38EVvUGtVhJRbzV0sMXP5j6sn8/g+xG9Cgm+PSqebw7oHUXjp8MJONTAoKG0GcSvBkbZQXO2hu4Pkds9eZhSTlVAuX8DceOdudWZyelvR4Ii6THnomUvmKrQFq80Y73Tyil3MOSKpn6UJ6xm+HB/nzslo2TmMiAsHHcicyAVemTOUTHL1OoOuhndls4e7GmMU7Rxloc2tpaNxjDyJQFXBV0AQ5nrKWYcFqRxT9zhY9/QJ7i8iauhrC0qO90/Es9lRFv/OFMrT6GFz2a8gAPc2XgDk32uRiqzGJmBlGUfFWRsy/LKHG9Amfu4Brz9XBwAsJLXPU+QgebYJWOAqCjSPLvxJ5H3p6BYGknYK0iaAMNxTG5c0HU4+N8whBUfcWm+rBTYUGZ6 LS85toeX YOteovS6ZEv4NA4FsdX4qHT9RHroPkjFdPov5AjnKkdRQRG9gMCIVrn2vvK+wkkirIQIg7gTA64ipjRKBRAaVtUib0ixBh6/KrYDwpLn+fFgluoDMC8d/eI9vIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: _vm_unmap_aliases() is used to ensure that no unflushed TLB entries for a page are left in the system. This is required due to the lazy TLB flush mechanism in vmalloc. This is tried to achieve by walking the per CPU free lists, but those do not contain fully utilized vmap blocks because they are removed from the free list once the blocks free space became zero. When the block is not fully unmapped then it is not on the purge list either. So neither the per CPU list iteration nor the purge list walk find the block and if the page was mapped via such a block and the TLB has not yet been flushed, the guarantee of _vm_unmap_aliases() that there are no stale TLBs after returning is broken: x = vb_alloc() // Removes vmap_block from free list because vb->free became 0 vb_free(x) // Unmaps page and marks in dirty_min/max range // Block has still mappings and is not put on purge list // Page is reused vm_unmap_aliases() // Can't find vmap block with the dirty space -> FAIL So instead of walking the per CPU free lists, walk the per CPU xarrays which hold pointers to _all_ active blocks in the system including those removed from the free lists. Fixes: db64fe02258f ("mm: rewrite vmap layer") Signed-off-by: Thomas Gleixner Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes Reviewed-by: Uladzislau Rezki (Sony) --- V2: Amended changelog so it's clear that the block is not on purge list either. --- mm/vmalloc.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2236,9 +2236,10 @@ static void _vm_unmap_aliases(unsigned l for_each_possible_cpu(cpu) { struct vmap_block_queue *vbq = &per_cpu(vmap_block_queue, cpu); struct vmap_block *vb; + unsigned long idx; rcu_read_lock(); - list_for_each_entry_rcu(vb, &vbq->free, free_list) { + xa_for_each(&vbq->vmap_blocks, idx, vb) { spin_lock(&vb->lock); if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { unsigned long va_start = vb->va->va_start;