From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EDCBC77B75 for ; Tue, 23 May 2023 14:05:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0790900009; Tue, 23 May 2023 10:05:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB79F900002; Tue, 23 May 2023 10:05:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7EE0900009; Tue, 23 May 2023 10:05:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C9CA8900002 for ; Tue, 23 May 2023 10:05:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 61473A03BA for ; Tue, 23 May 2023 14:05:31 +0000 (UTC) X-FDA: 80821692462.15.DF3974B Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf18.hostedemail.com (Postfix) with ESMTP id 340D71C01F9 for ; Tue, 23 May 2023 14:02:16 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=JKtVAo6p; dkim=pass header.d=linutronix.de header.s=2020e header.b=lzrJhX+3; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf18.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684850538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=4pGoRnJ+fwwVnfQLimOLlNVCz8HekZITmO9TMsFFvE7VLpi7WBKrc8GSUY1SRVLyVR1FLM Lx7OyQL9k4mwfmCu1h9pVSZgZx5PrBqleQRKOAHrdztsTlG/OJZchc/vis7onJ5/y74/52 hskwxBIL6Uynk0X/PnDCZkwV6V2uRX0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=JKtVAo6p; dkim=pass header.d=linutronix.de header.s=2020e header.b=lzrJhX+3; dmarc=pass (policy=none) header.from=linutronix.de; spf=pass (imf18.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684850538; a=rsa-sha256; cv=none; b=mv9wSM5fQjfm0weCWHv6C0XP9fOhYbYmrHyRSQtM/XCkakLQvNGysAOuamOQ2apvdmbosW gO/UsiDCYk7O3oHEvOiXCq5hXJo1JuCCgjzDHyVfRGUKKAYC/q0hrguZNhuucSYv0JM5zJ EA2QghDrrD6v1rLStfJ+6vTddvzo+a8= Message-ID: <20230523140002.747157575@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684850535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=JKtVAo6p0Asbn8XRblzsQCULEGEjNcpx2/hHNDNJb/T3IGaGYt18w1p0hZxzdVp3Nnc7xR oRJhr/FxdQWpf/f5crY59ShV8OPkpWAPTXrOW/u8vkKBg7vFL2NpU2j0BckGRfSdZRc2aI s0IFBXEMBiznw6+55y8L2q4KaYz1NAkEIhN+6sT0rjTl247RTF3szxP5tuvtKKLX5Sd5Cq L6C0YYe4q2i3JywUEfiEe+r2TmjdiZ197yn333H5cf3Yr59+iPZ0eQV2Gj2T0yz3d4h5pg qOH4PlYytyBt/uiLpe3+IhtwsFy5DT8/WGLSqW7k+BLpUM8FS8nJ4T10qb6a7Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684850535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=l4dZ5iSu2A1qB3lFnKbn1ig111WvYDcQRXEAc7kIVCY=; b=lzrJhX+3bK9luFzF+rv2CVBgXzS2NiI1KQDxMKxCAtOIssNPBKkRPdhuXr7WeLH3pPhpJL pSztWK30io/KXADg== From: Thomas Gleixner To: linux-mm@kvack.org Cc: Andrew Morton , Christoph Hellwig , Uladzislau Rezki , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: [patch 4/6] mm/vmalloc: Check free space in vmap_block lockless References: <20230523135902.517032811@linutronix.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Date: Tue, 23 May 2023 16:02:14 +0200 (CEST) X-Rspamd-Queue-Id: 340D71C01F9 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uoqozuq479iec4twkmdpftwnmyi38b5i X-HE-Tag: 1684850536-550971 X-HE-Meta: U2FsdGVkX1+/craFmQynr0/VnvJs2soHyFsz1nLCv2XNdRsMyaod4uAG/zmx1ZvEhbkxDiS5+D4wJEZx0hNuw7QffT0kC09CBe/uPu2/AV6UWJIpj3dYeL6tBd5YFhh1F/z5OhOLfnmOwMeRqPD4vh3XK9g9oLd8IR+iMPUci6+gOuluHwHLT7tDHwp9Tp2DFZP+q+scTRn9QU5N0/SS8MFlybs8ZTvcxKGsrJRnBk1VviDWfjF79Vb9Enlo6nwFgiwYx6zUPUWwh+Bh5CsY6ikk3ihea7wCf34UGEwopzZtCKWZXVqHINqmHsF9B8tVQXD8+uRdA7G52RPHNPhU6hzpQzolHxRo11jaeWjdvd/K2iL0jFtNvCNH2ycuC1MeYyLWTWoGOmlgtVidHVCDaH8Rq2BW0xNO9iwGLU5oLVX2ot9CBOHLxIXloUZV6qwz4lAoOB3nfsmUSEWaInsjiS3DLRC/NbKEK1AaSgGef9vkcB4Ya2531KvPruyf6O4NvkVba0qz87oIqVcvIDVxOEkd8KDMMi46+0QvE6/RvsZNb7qITaUwrmTC8MFchwxb9WMP6R9U1tRvQx4bWMC+2AyDheumGImUsyL1cs0LARlDBZw4QmLdpgWrAe+Q8h8yz00IBY8/vyEJuNjC5YYYs08S/GRyQwGEVqK5jR7kD17iJbsfxyz+R3g9h482r1lph63U/6Fqr6E021OB/eah7Y0ZOS7eoWsSkPFVx8YgYArMGZKlgY1dkHvqc9aRg3YRN7fyjfwp9d04xMOr/xvRUneNSUWqcfjrLZ/+I7g4kjG3HpwVA2fBPbOVCfy1Orxg5kU3NOboT7ZKeuugvctNRUkSFIFZejH0C3k6U1VAWc3iAInETvEz+FD7ohtmTQo+ohRu/t1LxWNjFptwN5dbfFLK9P2fO5MwCuMX066QVUQL8vxTNP5UlVjSheDt15xTvsumyN9EXIhYeTevseH nNhjaeJW hh8XBN3D4T/9OlfP36nnwfEN3nCETMsCahY0XDDgt1kFB/Wc+fxDlcDnn3h7Aa7LBZAyXhKE1/WWgGP45Q8XZr8Kah/WT4gcXox3o3t58ElHlijSBlBCvdR/KRFzCD+vXtfkaEtzaiWbzWis= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vb_alloc() unconditionally locks a vmap_block on the free list to check the free space. This can be done locklessly because vmap_block::free never increases, it's only decreased on allocations. Check the free space lockless and only if that succeeds, recheck under the lock. Signed-off-by: Thomas Gleixner --- mm/vmalloc.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2166,6 +2166,9 @@ static void *vb_alloc(unsigned long size list_for_each_entry_rcu(vb, &vbq->free, free_list) { unsigned long pages_off; + if (READ_ONCE(vb->free) < (1UL << order)) + continue; + spin_lock(&vb->lock); if (vb->free < (1UL << order)) { spin_unlock(&vb->lock); @@ -2174,7 +2177,7 @@ static void *vb_alloc(unsigned long size pages_off = VMAP_BBMAP_BITS - vb->free; vaddr = vmap_block_vaddr(vb->va->va_start, pages_off); - vb->free -= 1UL << order; + WRITE_ONCE(vb->free, vb->free - (1UL << order)); bitmap_set(vb->used_map, pages_off, (1UL << order)); if (vb->free == 0) { spin_lock(&vbq->lock);