From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E55A3CE835C for ; Mon, 30 Sep 2024 15:22:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D290A6B0195; Mon, 30 Sep 2024 11:22:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD8B66B0197; Mon, 30 Sep 2024 11:22:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B79376B0199; Mon, 30 Sep 2024 11:22:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 9AD366B0195 for ; Mon, 30 Sep 2024 11:22:26 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2ABB81C520C for ; Mon, 30 Sep 2024 15:22:26 +0000 (UTC) X-FDA: 82621771092.11.C6DF7D0 Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) by imf03.hostedemail.com (Postfix) with ESMTP id 21B2D2001A for ; Mon, 30 Sep 2024 15:22:23 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dZxQaVSP; spf=pass (imf03.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1727709577; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=R8XhM/SVrfi8Z4HLnJdUMQS7Li3/arNhnHXWrKzEAUM=; b=WsOsCoJmEjGc5ZM9IuBqK54ZNO15wipCb46aKgbPCGfY9pRkav/MjDjZRENaCtFi0NPH7e /NSUH0iyLerZNoFGyoa/E4SQDESyY84l9/hehQi2OMo3f2Zx89Uyzt330DfOOOIn3L4Q/W VEMgNvWUnMil+ZjPNBBIKbT6PrQKrrg= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dZxQaVSP; spf=pass (imf03.hostedemail.com: domain of urezki@gmail.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=urezki@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1727709577; a=rsa-sha256; cv=none; b=WTAZua0XCKJIS4MiJC2uBJzCosOvlyeg2hYhu3j/iFbwrnM4UQig7cutKiUAMmkLmScKpS n+cbtV7PozZ1SZkdSEj+hCF9bgmJyIKn2aJIRUMSvBLJ74n22ihGz4opUHvan7T8vZZC66 UYFM0ydrqMgIid1Fwyxx178zyXRyiDg= Received: by mail-lf1-f44.google.com with SMTP id 2adb3069b0e04-537a399e06dso5182085e87.1 for ; Mon, 30 Sep 2024 08:22:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1727709742; x=1728314542; darn=kvack.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from:from:to :cc:subject:date:message-id:reply-to; bh=R8XhM/SVrfi8Z4HLnJdUMQS7Li3/arNhnHXWrKzEAUM=; b=dZxQaVSPMjNhhWJ10xrusfx8BRyWQ2kwRYiPhSQr+V7ojkFxs5v/NDU5d0C2cOSFhB OinU63X0nJ1QG+hlnFMplSXWweh6ChiuESg1p+vBiyKx14CZ58ZhL1CyKgCbdL9tumPI w+DgCH4TnCh388nzsfjP/whVqJngqEl6jcihB9eC9hVDyzo0YG15w6q89OaJ1QUducdE XBwgIVYY8HQT/8/OhJmZwT+iASXwDproSgW6giyIXdWhHikKqmpYkqYtiwPblZHROn6f RV9mVYT1dsGjq97E1aA01P8ivsaF0i+AvIdSdSj/L2Vu3KZFArAlKKnCL1ZjTraN3sSd fZtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727709742; x=1728314542; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:date:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R8XhM/SVrfi8Z4HLnJdUMQS7Li3/arNhnHXWrKzEAUM=; b=D1kgVm21vRX47gw9z3sTPDW+qpBOtmV7OkAnkQCU/nxSVlQO/Wy8E+QH9hvlqvGBvY jywqlRJRqAvM3HAvPUs3lp5fbpoMNXFHKumTR34BPfHELky21DSb2EFZG9LHJaXGaU90 Q8/MCDziOAPIYVTfirm/yl3AIqoQmHLkYu8JD7nChI5m/9iXnW7b1OQ2QL+sIuEqkDV3 lnCpDkFD2CzLaUSl9g5osumVqsXBAxnSW5SFEzBM7lhe2/7byPkO4sZJ3T/J+0Bg81oR T2037db9Ex7AEigPUHP7s/gPBFM5axZmzoVd+6h5yagbWYPkrm8CQF6PyDA2Q4cyPJfJ aSAw== X-Forwarded-Encrypted: i=1; AJvYcCWduiJcsuVj+vY4F5Q8oly82N/EzAMhDV7IrItcHiT7q/D7JOrERBBTt25iqKHsiEPhp1imJd/MJA==@kvack.org X-Gm-Message-State: AOJu0YxK4eIzsMHf5HmpUH+Wu0dj1sFUx+BXqGrr+4Y4JvZtv9lVq1nA aJKV/3wZlm1UJNEmqTjXfACkOkpRZJu9D25oie/KuCUdXXZ39rl+ X-Google-Smtp-Source: AGHT+IEF2tMeyV2lmrHGKZUDmwUvk0j2fXTyq8LuUAZzHjRsUzJyQAyKwfaqwb6CHCU/AhJkYC9r4g== X-Received: by 2002:a05:6512:eaa:b0:52e:a68a:6076 with SMTP id 2adb3069b0e04-5389fc6d4d5mr6096930e87.49.1727709741703; Mon, 30 Sep 2024 08:22:21 -0700 (PDT) Received: from pc636 (host-95-193-102-146.mobileonline.telia.com. [95.193.102.146]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5389fd5f5d1sm1255598e87.114.2024.09.30.08.22.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Sep 2024 08:22:21 -0700 (PDT) From: Uladzislau Rezki X-Google-Original-From: Uladzislau Rezki Date: Mon, 30 Sep 2024 17:22:18 +0200 To: Huang Adrian Cc: Uladzislau Rezki , Andrew Morton , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Adrian Huang Subject: Re: [PATCH 1/1] kasan, vmalloc: avoid lock contention when depopulating vmalloc Message-ID: References: <20240925134732.24431-1-ahuang12@lenovo.com> <20240925134706.2a0c2717a41a338d938581ff@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 21B2D2001A X-Stat-Signature: bcwh8qoko3jd56hgt613yixa5k9ubmsy X-Rspam-User: X-HE-Tag: 1727709743-935575 X-HE-Meta: U2FsdGVkX1+pmU6P6PlZXfVOn/4mDOCAQ6U3+EwO00SgH4pp4JyIQ+UbrWGLi8gDYY3c35NDgkuqIObuHNFypHuoCKHljeq67iZZu6tJb3I7vv30yiZuojtKtTmtGM9QGJAGcwtBWrTWJKxm/pZ/OrkUxloMAkN5lDfGYiHJjo1+Wrti7E9m9vv0J6ka98co8UUOtECmls0f+ruwqMMnbV4ibPbbV68HNiIn2kZ+nHnYZyrYMPwQO24iz5G4caKdquc92clLKT0zYdsq/zMcCtJiu/c+6rb9uo1gJvxq+aVhWwg2fHe7oJ8XtkqaoVMZ3oAVSjtE3mBcet7zXIA7WUoEUaueQQnir/jbN2RK9OZ/TkrxIqVtMAoVOqudWj0PlIDPkma905dK0o6j/0C0SL1mZ1d8YteCpWAYd0nrs0UpnPr6zLWnW1tyHVanJ354IqBdmMrEG+gMkQedZmbdU00/QavbDHwcjgvO5lUoQ+4Axgx24fTX8pJB+afE3lWYZvryibFl2oNPyYXN6NzxhxsRsP9Msh7U5sTx+euD+tMjjIrBjbI+j17DXlZ3FE25AFEGh6keUEnsibAG1AKtyWry5vQ3StDuWgm2hLz9Rgmo1q2PnbeCNv8ZZYF3gBJhI2Wxu79Y5CfZICexrQ8c+j38xfAt8eNqM6IEYCkAyHTPYUIL1mQ+SRCJGkdQUqi8e1I0uESiRK9AkbyOlob45/oha34IetIWkOUt9Fo9DT4EUAIHFKiawISZALcFeC/VaUuTuCLE0pMCemNdEkKHQgFLO9WDn6JoRC0F38vK53WyWmRiN5UR6LDM3CQ6VrEugQ53b80RXgfkIpEHkqHYxAYFelwkyFTS8YUE635GPVC9Zdu/dpZt/lixLC669gHJaPsmkzdr5bTYzQuttNZzu/i8qdr//WEVGdvQKMPdv/DNIaRvDXm7lKjfHuQtW78P3qBqTTmKsaS1ArXTt/c loVcYngF rQ6rvHV5hYF7SRxcUwPIYzh475KaZGMOV3AD5sCx7a0PiZi19NwKgFeNP9dekmsCO6K00h/Etx2RGvxbBJ6v7iV9vL1u3PDwA2HIu4J+hrhvcV0r36JfsQNEsuz8wMbONOLEIyiVTxKVMKby0pBngRaodF/ZXbKGosRIbNSngZ4cTxG7Sg+UuKy/NX19qqJfG9S/fp5VeZS8+0P0N9AItjbS4ttNCwQ8jJCYPHW2Hu8nBVNybECutQHkKQ/l2SdU6MqgN+5iCJ9nG03kI1t5c2IaJTLdgbTsazksmroSz53mXPUcpVvDldisqnA1JqEkoxpSxjOVfVmPOgFmthMDD54Td7v3IHwRxPNLsG+crXCnaTVpdBpv/urH+7yiseyVpBA9liz2agwPyL3vpad6aFn2cFxNh6rZBBSL30kIp8YSvTDp8YaL5IVhcr1V9ihSyuB459IqQgK/NLF8c7GtFOvp7tVM1dA8xMRKpo8lwVMiKV1EsGfSe8a19WFIH5NFYjyb0z2j9u3HNd5NGJXc1npMlCgPW6mpNTMuD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, Adrian! > Hello Uladzislau, > > On Fri, Sep 27, 2024 at 12:16 AM Uladzislau Rezki wrote: > > > > Hello, Adrian! > > > > > > > > > > > > From: Adrian Huang > > > > > After re-visiting code path about setting the kasan ptep (pte pointer), > > > > > it's unlikely that a kasan ptep is set and cleared simultaneously by > > > > > different CPUs. So, use ptep_get_and_clear() to get rid of the spinlock > > > > > operation. > > > > > > > > "unlikely" isn't particularly comforting. We'd prefer to never corrupt > > > > pte's! > > > > > > > > I'm suspecting we need a more thorough solution here. > > > > > > > > btw, for a lame fix, did you try moving the spin_lock() into > > > > kasan_release_vmalloc(), around the apply_to_existing_page_range() > > > > call? That would at least reduce locking frequency a lot. Some > > > > mitigation might be needed to avoid excessive hold times. > > > > > > I did try it before. That didn't help. In this case, each iteration in > > > kasan_release_vmalloc_node() only needs to clear one pte. However, > > > vn->purge_list is the long list under the heavy load: 128 cores (128 > > > vmap_nodes) execute kasan_release_vmalloc_node() to clear the corresponding > > > pte(s) while other cores allocate vmalloc space (populate the page table > > > of the vmalloc address) and populate vmalloc shadow page table. Lots of > > > cores contend init_mm.page_table_lock. > > > > > > For a lame fix, adding cond_resched() in the loop of > > > kasan_release_vmalloc_node() is an option. > > > > > > Any suggestions and comments about this issue? > > > > > One question. Do you think that running a KASAN kernel and stressing > > the vmalloc allocator is an issue here? It is a debug kernel, which > > implies it is slow. Also, please note, the synthetic stress test is > > not a real workload, it is tighten in a hard loop to stress it as much > > as we can. > > Totally agree. > > > Can you trigger such splat using a real workload. For example running > > stress-ng --fork XXX or any different workload? > > No, the issue could not be reproduced with stress-ng (over-weekend stress). > > So, please ignore it. Sorry for the noise. > No problem. This is a regular workflow what is normal, IMO :) -- Uladzislau Rezki