From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7737DC433EF for ; Wed, 22 Jun 2022 03:16:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE1268E007A; Tue, 21 Jun 2022 23:16:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C8FBB8E006E; Tue, 21 Jun 2022 23:16:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B58AE8E007A; Tue, 21 Jun 2022 23:16:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A42398E006E for ; Tue, 21 Jun 2022 23:16:25 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay13.hostedemail.com (Postfix) with ESMTP id 70C7A6107B for ; Wed, 22 Jun 2022 03:16:25 +0000 (UTC) X-FDA: 79604408730.18.6AD2154 Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42]) by imf06.hostedemail.com (Postfix) with ESMTP id D43851800AA for ; Wed, 22 Jun 2022 03:16:23 +0000 (UTC) Received: by mail-lf1-f42.google.com with SMTP id a13so15688940lfr.10 for ; Tue, 21 Jun 2022 20:16:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9MJRbgYqBVvHHWCXDuWt7RjZ0olsk+Nr5RQqp/foNWs=; b=TC9MEj2s14PmEycElP7NtPyJtvkqKsYcVCEV9IYkQfSoHP28/mNoFXESTk53Dl4zPW lgJOdnNeI+bG9bg38LflvT2EZ1E5rYm/8+dvnxMJddy3G57Vi6V/J3Icq/JUvn+ySx/D Z9b9EnPcjx19ZEHGqIVbRMmOm3PMJJQ0gjwRLorxVcML6BpP+r9TNQJ3n5JRqiq2VwoF RAUXKENpQyuBS091oxlurqu2KXvUFQqCzNlcqi2AID1QSfqtoT5ZnBj6zC3VoHU9hVMs vkF2hDX101f8p+NSHctT6SqR6qn7neJb1xBumaBwq5BAys2IcM0GMarTjfh4RahWhsRQ VSJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9MJRbgYqBVvHHWCXDuWt7RjZ0olsk+Nr5RQqp/foNWs=; b=RznrjmjeUkVmbW8HqbmEc1rRy+IaKNLV9UQQSDtRSYtm7RCu4BaXbMFAN4Nc5CvkPE 3Zcgs6GW8Q20cflF3ogBjPfroyRA0BEF4ciwtJVplJdaINM36ny6eXvDhR7Vm0vlW0gS xABMKyFRZ/fVbhE6eqdTSdRdiRDuH1xhi+ZgkXepReWZ3gHiv8xv0L5QALNzOgQrNo6r 9Zkcj66oySl/X9A31kCEENDoeqhcASfCB+qpRB1LA12UHGKdUZalDyDL79aDsObATSod 1r6Q0kJnh8UiZsCf383XIoKz1GBkg4wsXAu0vpthBb6t/+4s7inZYpfVPPCcNSuJdQgr Yy0g== X-Gm-Message-State: AJIora8iV5vQogGtE9DsfvsQQraeqMZWBMAjbUiW6/CH5Ge+KRcTJ/Yg DMci6aP/Mp9S18YZj3HyyWX+FEXmJpLPoA6fEX3EOxF41mo= X-Google-Smtp-Source: AGRyM1s0aA0sF76rxXRjEbyw6HlYUqGZIVF1aEJv8hr6ftczqYq1dQySeO/1QXrqWg2DJpNgQoUybD6Ei344w17J2T0= X-Received: by 2002:ac2:484d:0:b0:47f:7ebf:336 with SMTP id 13-20020ac2484d000000b0047f7ebf0336mr904019lfy.130.1655867782079; Tue, 21 Jun 2022 20:16:22 -0700 (PDT) MIME-Version: 1.0 References: <1653447164-15017-1-git-send-email-zhaoyang.huang@unisoc.com> In-Reply-To: From: Zhaoyang Huang Date: Wed, 22 Jun 2022 11:15:52 +0800 Message-ID: Subject: Re: [PATCH] mm: fix racing of vb->va when kasan enabled To: Uladzislau Rezki Cc: "zhaoyang.huang" , Andrew Morton , "open list:MEMORY MANAGEMENT" , LKML , Ke Wang , Christoph Hellwig Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TC9MEj2s; spf=pass (imf06.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655867783; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9MJRbgYqBVvHHWCXDuWt7RjZ0olsk+Nr5RQqp/foNWs=; b=OEJwWD9ukmfOZqcapxoDWrONuzsbfugwSlsi9nSWxh4FhhE7vsWSv+fA4iMZxnl9wJd5Fi cZbO/crv7o14an/Z3yp5SOChqJaJXrnSMLbpU86Wwxrq2xWi4W420dFXZCfV+T25q7Vifq bt6rSMUou5nFBz2cJPyrJbhscLoFaGM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655867783; a=rsa-sha256; cv=none; b=d8d8z/qpsvH8Y/oJBRZPy9SsqEszJAOlBCT8LnqMxSvpnqKcpFRPAxhfjCohovSPmGnWQG EiJKHaY2cGoC2dw2GcjSor5M4ies5/0d8LDuzAkm59atfk3iaBX1zcWPmYa52dTwp9mtbt 6Af4cjXdEArmkr8/LzcuZezNk0H25BA= X-Stat-Signature: cfdybw9af1dmwj56w43pzuhxfbry9jxm X-Rspam-User: X-Rspamd-Server: rspam07 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=TC9MEj2s; spf=pass (imf06.hostedemail.com: domain of huangzhaoyang@gmail.com designates 209.85.167.42 as permitted sender) smtp.mailfrom=huangzhaoyang@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Queue-Id: D43851800AA X-HE-Tag: 1655867783-505566 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Jun 21, 2022 at 10:29 PM Uladzislau Rezki wrote: > > > On Tue, Jun 21, 2022 at 5:27 PM Uladzislau Rezki wrote: > > > > > > > On Mon, Jun 20, 2022 at 6:44 PM Uladzislau Rezki wrote: > > > > > > > > > > > > > > > > > > > > Is it easy to reproduce? If so could you please describe the steps? As i see > > > > > > > the freeing of the "vb" is RCU safe whereas vb->va is not. But from the first > > > > > > > glance i do not see how it can accessed twice. Hm.. > > > > > > It was raised from a monkey test on A13_k515 system and got 1/20 pcs > > > > > > failed. IMO, vb->va which out of vmap_purge_lock protection could race > > > > > > with a concurrent ra freeing within __purge_vmap_area_lazy. > > > > > > > > > > > Do you have exact steps how you run "monkey" test? > > > > There are about 30+ kos inserted during startup which could be a > > > > specific criteria for reproduction. Do you have doubts about the test > > > > result or the solution? > > > > > > > > I do not have any doubt about your test results, so if you can trigger it > > > then there is an issue at least on the 5.4.161-android12 kernel. > > > > > > 1. With your fix we get expanded mutex range, thus the worst case of vmalloc > > > allocation can be increased when it fails and repeat. Because it also invokes > > > the purge_vmap_area_lazy() that access the same mutex. > > I am not sure I get your point. _vm_unmap_aliases calls > > _purge_vmap_area_lazy instead of purge_vmap_area_lazy. Do you have any > > other solutions? I really don't think my patch is the best way as I > > don't have a full view of vmalloc mechanism. > > > Yep, but it holds the mutex: > > > mutex_lock(&vmap_purge_lock); > purge_fragmented_blocks_allcpus(); > if (!__purge_vmap_area_lazy(start, end) && flush) > flush_tlb_kernel_range(start, end); > mutex_unlock(&vmap_purge_lock); > > > I do not have a solution yet. I am trying still to figure out how you can > trigger it. > > > rcu_read_lock(); > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > spin_lock(&vb->lock); > if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > unsigned long va_start = vb->va->va_start; > > > so you say that "vb->va->va_start" can be accessed twice. I do not see > how it can happen. The purge_fragmented_blocks() removes "vb" from the > free_list and set vb->dirty to the VMAP_BBMAP_BITS to prevent purging > it again. It is protected by the spin_lock(&vb->lock): > > > spin_lock(&vb->lock); > if (vb->free + vb->dirty == VMAP_BBMAP_BITS && vb->dirty != VMAP_BBMAP_BITS) { > vb->free = 0; /* prevent further allocs after releasing lock */ > vb->dirty = VMAP_BBMAP_BITS; /* prevent purging it again */ > vb->dirty_min = 0; > vb->dirty_max = VMAP_BBMAP_BITS; > > > so the VMAP_BBMAP_BITS is set under spinlock. The _vm_unmap_aliases() checks it: > > > list_for_each_entry_rcu(vb, &vbq->free, free_list) { > spin_lock(&vb->lock); > if (vb->dirty && vb->dirty != VMAP_BBMAP_BITS) { > unsigned long va_start = vb->va->va_start; > unsigned long s, e; > > > if the "vb->dirty != VMAP_BBMAP_BITS". I am missing your point here? Could the racing be like bellowing scenario? vb->va accessed in [2] has been freed in [1] _vm_unmap_aliases _vm_unmap_aliases { { list_for_each_entry_rcu(vb, &vbq->free, free_list) { __purge_vmap_area_lazy spin_lock(&vb->lock); merge_or_add_vmap_area if (vb->dirty) { kmem_cache_free(vmap_area_cachep, va)[1] unsigned long va_start = vb->va->va_start; [2] > > > > > > > 2. You run 5.4.161-android12 kernel what is quite old. Could you please > > > retest with latest kernel? I am asking because on the latest kernel with > > > CONFIG_KASAN i am not able to reproduce it. > > > > > > I do a lot of: vm_map_ram()/vm_unmap_ram()/vmalloc()/vfree() in parallel > > > by 64 kthreads on my 64 CPUs test system. > > The failure generates at 20s from starting up, I think it is a rare timing. > > > > > > Could you please confirm that you can trigger an issue on the latest kernel? > > Sorry, I don't have an available latest kernel for now. > > > Can you do: "gdb ./vmlinux", execute "l *_vm_unmap_aliases+0x164" and provide > output? Sorry, I have lost the vmlinux with KASAN enabled and just got some instructions from logs. 0xffffffd010678da8 <_vm_unmap_aliases+0x134>: sub x22, x26, #0x28 x26 vbq->free 0xffffffd010678dac <_vm_unmap_aliases+0x138>: lsr x8, x22, #3 0xffffffd010678db0 <_vm_unmap_aliases+0x13c>: ldrb w8, [x8,x24] 0xffffffd010678db4 <_vm_unmap_aliases+0x140>: cbz w8, 0xffffffd010678dc0 <_vm_unmap_aliases+0x14c> 0xffffffd010678db8 <_vm_unmap_aliases+0x144>: mov x0, x22 0xffffffd010678dbc <_vm_unmap_aliases+0x148>: bl 0xffffffd0106c9a34 <__asan_report_load8_noabort> 0xffffffd010678dc0 <_vm_unmap_aliases+0x14c>: ldr x22, [x22] 0xffffffd010678dc4 <_vm_unmap_aliases+0x150>: lsr x8, x22, #3 0xffffffd010678dc8 <_vm_unmap_aliases+0x154>: ldrb w8, [x8,x24] 0xffffffd010678dcc <_vm_unmap_aliases+0x158>: cbz w8, 0xffffffd010678dd8 <_vm_unmap_aliases+0x164> 0xffffffd010678dd0 <_vm_unmap_aliases+0x15c>: mov x0, x22 0xffffffd010678dd4 <_vm_unmap_aliases+0x160>: bl 0xffffffd0106c9a34 <__asan_report_load8_noabort> > > -- > Uladzislau Rezki