From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BA9BC43334 for ; Wed, 22 Jun 2022 06:05:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BE7A8E007E; Wed, 22 Jun 2022 02:05:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 26F676B0114; Wed, 22 Jun 2022 02:05:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 135758E007E; Wed, 22 Jun 2022 02:05:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 04A6A6B0113 for ; Wed, 22 Jun 2022 02:05:18 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id CBF8B61129 for ; Wed, 22 Jun 2022 06:05:17 +0000 (UTC) X-FDA: 79604834274.24.7A09024 Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf18.hostedemail.com (Postfix) with ESMTP id 753431C0004 for ; Wed, 22 Jun 2022 06:05:15 +0000 (UTC) Received: by mail-lf1-f51.google.com with SMTP id a13so16143052lfr.10 for ; Tue, 21 Jun 2022 23:05:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+26yanUYFbs+kQoX0xzkIKd301lMCOIGb1lb+5sasSA=; b=EpFXFh8CzRnMkuXQkhgC+q9r//mSbtWCjbtErlIJMcrWTrmATPgeloBk8/5C7Y+jrQ xsW2zuCVHA+gqM8ly13JXKhpX2UOqrjCSFtc4rL9svF0hBHxvL3j0iCr3+lLdv1WhUht SW0CyXvPeJNwL1seqNwXffEubuFOjtFcH9Y0CgvEXOw64wCKsgY1GH0IpF2QbI3IWMx0 tUvZWxQqQJtMJkKNu42RVoy1sCaY0MDk4cgg17RSD8gj1rIPPjgwbeG5g3oE+9A+Urw8 OMDK0ouRVhtMEjbhfGptAZQpjKcPvPfwOg5ViE+VQ/5ZGHXeWAzS0O7PaN0yjrC73XTt mdVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+26yanUYFbs+kQoX0xzkIKd301lMCOIGb1lb+5sasSA=; b=zqgVea2NZr21HCD9KQmJ8x6QS7sWRlpFCPahsgXebi+XR7NivLvbWotCcU41lx6QPj tYxArwOgkS/XlJ6m8qGQd1P24wUh3PHZ+TMZGjnNT7cikAiGje5iYAW/ENxaFTutOuMU BeNcMJeOtkPz4emHG1xPXx/jHRLzQQuas1GFzZjZnkH4Z6USEM57Q0wI2qgIWBlqMcqd j4W6/6gpgZJ21qzIa/T91p795WQahw+l3J9tJ/9iuM8ttujCljBQgDAvd/qhedKqxy/5 STOoREwr+9nw+SjM+QAYLVGMGy+wY7wqSx1RKv8kX0viM0EITRwM+1b46HaJSh6azE77 607w== X-Gm-Message-State: AJIora/ES0rSg9XxVOzXy7N0KHvoBhpL2H4NvFbwcP8WERowSbzqg5zM Os+s9Wfnu9uFP709T1d1bQ4qNAkUpUTxZCzTd68= X-Google-Smtp-Source: AGRyM1s+ypaQ7hBCb8VPVYLOXsxKYMMorT5FU1sVNsmSt8G4e8aYrYC/fKcIrRmJdLIDFzXZk4ZmC0QHhGDjgQdD8ZY= X-Received: by 2002:a05:6512:ac3:b0:47f:518f:f53e with SMTP id n3-20020a0565120ac300b0047f518ff53emr1164793lfu.520.1655877910609; Tue, 21 Jun 2022 23:05:10 -0700 (PDT) MIME-Version: 1.0 References: <1653447164-15017-1-git-send-email-zhaoyang.huang@unisoc.com> In-Reply-To: From: Zhaoyang Huang Date: Wed, 22 Jun 2022 14:04:41 +0800 Message-ID: Subject: Re: [PATCH] mm: fix racing of vb->va when kasan enabled To: Uladzislau Rezki Cc: "zhaoyang.huang" , Andrew Morton , "open list:MEMORY MANAGEMENT" , LKML , Ke Wang , Christoph Hellwig Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EpFXFh8C; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655877915; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+26yanUYFbs+kQoX0xzkIKd301lMCOIGb1lb+5sasSA=; b=mQGWlnWjsBkA2CmDCxiTtI/toSGOsFDS70xOwdbVcGs7j5nti+m6d02Cpewv5HUmUtaFv5 PUdjSGH9C9vEn2575UlsWjOvpufY/1yUruqTsE8/2Sbjs/ntvhlZZEKLUkZCrbEvChyDHo 80DSo0Y0ZhzaeBKO/VZdCrcoyoAwrbM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655877915; a=rsa-sha256; cv=none; b=hCX+KCHa/+GrTQpaupQMlvrhz/Xu6CXmahGb+L2YrrzPvyuUlRPIy9svVQmslw5dCZf7Jz NuYjSLePryAt7dYwWlc2NTdanEbU9hIdzwpNyJHT8wLWGHIDXqp4SsRZsHDfu0vluPB880 o4BL3QPwOnRrsWk2r01pdAmWKnA8rmE= X-Rspam-User: Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=EpFXFh8C; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com X-Stat-Signature: c5ttu7y97ahirbeik6twt54axih4fxwu X-Rspamd-Queue-Id: 753431C0004 X-Rspamd-Server: rspam12 X-HE-Tag: 1655877915-340763 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000115, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 22, 2022 at 11:15 AM Zhaoyang Huang wrote: > > On Tue, Jun 21, 2022 at 10:29 PM Uladzislau Rezki wrote: > > > > > On Tue, Jun 21, 2022 at 5:27 PM Uladzislau Rezki wrote: > > > > > > > > > On Mon, Jun 20, 2022 at 6:44 PM Uladzislau Rezki wrote: > > > > > > > > > > > > > > > > > > > > > > > Is it easy to reproduce? If so could you please describe the steps? As i see > > > > > > > > the freeing of the "vb" is RCU safe whereas vb->va is not. But from the first > > > > > > > > glance i do not see how it can accessed twice. Hm.. > > > > > > > It was raised from a monkey test on A13_k515 system and got 1/20 pcs > > > > > > > failed. IMO, vb->va which out of vmap_purge_lock protection could race > > > > > > > with a concurrent ra freeing within __purge_vmap_area_lazy. > > > > > > > > > > > > > Do you have exact steps how you run "monkey" test? > > > > > There are about 30+ kos inserted during startup which could be a > > > > > specific criteria for reproduction. Do you have doubts about the test > > > > > result or the solution? > > > > > > > > > > I do not have any doubt about your test results, so if you can trigger it > > > > then there is an issue at least on the 5.4.161-android12 kernel. > > > > > > > > 1. With your fix we get expanded mutex range, thus the worst case of vmalloc > > > > allocation can be increased when it fails and repeat. Because it also invokes > > > > the purge_vmap_area_lazy() that access the same mutex. > > > I am not sure I get your point. _vm_unmap_aliases calls > > > _purge_vmap_area_lazy instead of purge_vmap_area_lazy. Do you have any > > > other solutions? I really don't think my patch is the best way as I > > > don't have a full view of vmalloc mechanism. > > > > > Yep, but it holds the mutex: I still don't get how _purge_vmap_area_lazy hold vmap_purge_lock? > > > > > > mutex_lock(&vmap_purge_lock); > > purge_fragmented_blocks_allcpus(); > > if (!__purge_vmap_area_lazy(start, end) && flush) > > flush_tlb_kernel_range(start, end); > > mutex_unlock(&vmap_purge_lock); > > > > > > I do not have a solution yet. I am trying still to figure out how you can > > trigger it. > > > > > > rcu_read_lock(); > > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > > spin_lock(&vb->lock); > > if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > > unsigned long va_start = vb->va->va_start; > > > > > > so you say that "vb->va->va_start" can be accessed twice. I do not see > > how it can happen. The purge_fragmented_blocks() removes "vb" from the > > free_list and set vb->dirty to the VMAP_BBMAP_BITS to prevent purging > > it again. It is protected by the spin_lock(&vb->lock): > > > > > > spin_lock(&vb->lock); > > if (vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS) { > > vb->free = 0; /* prevent further allocs after releasing lock */ > > vb->dirty = VMAP_BBMAP_BITS; /* prevent purging it again */ > > vb->dirty_min = 0; > > vb->dirty_max = VMAP_BBMAP_BITS; > > > > > > so the VMAP_BBMAP_BITS is set under spinlock. The _vm_unmap_aliases() checks it: > > > > > > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > > spin_lock(&vb->lock); > > if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > > unsigned long va_start = vb->va->va_start; > > unsigned long s, e; > > > > > > if the "vb->dirty != VMAP_BBMAP_BITS". I am missing your point here? > Could the racing be like bellowing scenario? vb->va accessed in [2] > has been freed in [1] > > _vm_unmap_aliases > _vm_unmap_aliases > { > { > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > __purge_vmap_area_lazy > spin_lock(&vb->lock); > merge_or_add_vmap_area > if (vb->dirty) { > > kmem_cache_free(vmap_area_cachep, va)[1] > unsigned long va_start = vb->va->va_start; > [2] reformat the racing graph _vm_unmap_aliases _vm_unmap_aliases { { list_for_each_entry_rcu(vb, &vbq->free, free_list) { __purge_vmap_area_lazy spin_lock(&vb->lock); merge_or_add_vmap_area if (vb->dirty) { kmem_cache_free(vmap_area_cachep, va)[1] unsigned long va_start = vb->va->va_start; [2] > > > > > > > > > > 2. You run 5.4.161-android12 kernel what is quite old. Could you please > > > > retest with latest kernel? I am asking because on the latest kernel with > > > > CONFIG_KASAN i am not able to reproduce it. > > > > > > > > I do a lot of: vm_map_ram()/vm_unmap_ram()/vmalloc()/vfree() in parallel > > > > by 64 kthreads on my 64 CPUs test system. > > > The failure generates at 20s from starting up, I think it is a rare timing. > > > > > > > > Could you please confirm that you can trigger an issue on the latest kernel? > > > Sorry, I don't have an available latest kernel for now. > > > > > Can you do: "gdb ./vmlinux", execute "l *_vm_unmap_aliases+0x164" and provide > > output? > Sorry, I have lost the vmlinux with KASAN enabled and just got some > instructions from logs. > > 0xffffffd010678da8 <_vm_unmap_aliases+0x134>: sub x22, x26, #0x28 > x26 vbq->free > 0xffffffd010678dac <_vm_unmap_aliases+0x138>: lsr x8, x22, #3 > 0xffffffd010678db0 <_vm_unmap_aliases+0x13c>: ldrb w8, [x8,x24] > 0xffffffd010678db4 <_vm_unmap_aliases+0x140>: cbz w8, > 0xffffffd010678dc0 <_vm_unmap_aliases+0x14c> > 0xffffffd010678db8 <_vm_unmap_aliases+0x144>: mov x0, x22 > 0xffffffd010678dbc <_vm_unmap_aliases+0x148>: bl 0xffffffd0106c9a34 > <__asan_report_load8_noabort> > 0xffffffd010678dc0 <_vm_unmap_aliases+0x14c>: ldr x22, [x22] > 0xffffffd010678dc4 <_vm_unmap_aliases+0x150>: lsr x8, x22, #3 > 0xffffffd010678dc8 <_vm_unmap_aliases+0x154>: ldrb w8, [x8,x24] > 0xffffffd010678dcc <_vm_unmap_aliases+0x158>: cbz w8, > 0xffffffd010678dd8 <_vm_unmap_aliases+0x164> > 0xffffffd010678dd0 <_vm_unmap_aliases+0x15c>: mov x0, x22 > 0xffffffd010678dd4 <_vm_unmap_aliases+0x160>: bl 0xffffffd0106c9a34 > <__asan_report_load8_noabort> > > > > > -- > > Uladzislau Rezki