From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93FC3C4332F for ; Fri, 23 Dec 2022 17:50:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B189900003; Fri, 23 Dec 2022 12:50:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 93AC9900002; Fri, 23 Dec 2022 12:50:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DC2D900003; Fri, 23 Dec 2022 12:50:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6CD57900002 for ; Fri, 23 Dec 2022 12:50:44 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4EF6140419 for ; Fri, 23 Dec 2022 17:50:44 +0000 (UTC) X-FDA: 80274311208.11.07902A7 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf02.hostedemail.com (Postfix) with ESMTP id C0A628000A for ; Fri, 23 Dec 2022 17:50:40 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671817841; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3OnBXpsIlE2jRqO35gaddQ/u6r/y6RTchJa7AdTvOY0=; b=SpBoDQ52DUesiN8JBMjTcpNz1vQfdGvh8WvmfoEAtgBicq/ff5HfWNn9RDqJckezm3Xjfq RiZSnh9LFb0mmPkmSN9eCRdH0F5BTIejRlxoeyHLex5mNjw7K09WKGSFF4U+2jfWI9NT1f zO6UehnxrFe9JHUscuo2aEX5QgKcE70= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none); spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=cmarinas@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671817841; a=rsa-sha256; cv=none; b=Ao7XK1xPDUWi7cmiMVuTeG/riOWQqO4aYkU8OwYSkOwiH5ViZ57nSRrkIWeoJSilo3Wr7o Xg6BRUBbxLHvwhvW0Ay/K9DKvhWATYUdkneBFl1SqPDOTbU2Yk1ZocKm9LktoUtpzY6QjZ so+FynSQBm22hLWrgkdVcMdN7YTwvvM= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 56A63CE1CFD; Fri, 23 Dec 2022 17:50:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78CD8C433EF; Fri, 23 Dec 2022 17:50:34 +0000 (UTC) Date: Fri, 23 Dec 2022 17:50:31 +0000 From: Catalin Marinas To: Waiman Long Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Muchun Song Subject: Re: [PATCH 2/2] mm/kmemleak: Fix UAF bug in kmemleak_scan() Message-ID: References: <20221210230048.2841047-1-longman@redhat.com> <20221210230048.2841047-3-longman@redhat.com> <7ea28f11-34af-c974-94fb-7e2481942e97@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7ea28f11-34af-c974-94fb-7e2481942e97@redhat.com> X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C0A628000A X-Rspam-User: X-Stat-Signature: xqhpudsd7hi5uknp7qjqbfgbmfkcqs7s X-HE-Tag: 1671817840-234317 X-HE-Meta: U2FsdGVkX19EVy3JYC6M8l8B8tDKeC1AMPqpvHcA/mnzXLKnJ6K/Xt+sAc3iGUQK2lT+LaxVfJlrPhvAL/c05f6wJFMmjYuROhKXK+S1iWPP15ikpLR1r21EW8d05VVUo778YLEXtZ9G8UGolB8o9uFYrb7Bl79so5NmEMMJ/CvDBeIQB/rbtuccp5ZMISW/cWGP/ihL9faJf2wauB1oLniJbs4IC6R9nNb03l+s1G9wOx92jHXSQyiyjnJzyPV9gOHGE7oLP21ianwdBlckfICGpZhAfxCeKgrncIHS12/CO2DuIDCZyI5IgfQcP8+epOEPhdCGmMWCA9qilHO5tyQGwGwkL6VkZ+pcn8bCG/RVnca/Z00khHMO3JCnTufHGX2qXEGPbcopvFRzregkfvwfQ9cpNO9ZF5r59DWuj0+HmJD3idk1leM6cEql8Joa5Qs5lgIQZ2+5AIB1UvHHe5QaOnDVxDoJcapvexF3WGbL2UjNDNimxIiPaMTFTLTClArVZPpo0vBynlqPfm0BwKChLuUcKFC93Y22CfQ/Bva6cViCYrlsZFVAJC9Bcw4mJusWvMMUxko+gEMZtIjAcxZpO/PwjyrgOCYciZhw4/1JJdnrBbFShjO/rE+vGEtLysT286TztesD1Vgp5SAovTE5YnlAwdbnPEebZJfr4+QMlpLEicE7giqAbHK0XsZSJxS0vy5vZC4LYQb6lid2IiL/Ck4g+uEApFjNIHfyT66I6JiWBFam20G/LQrlkqnQw86wJ8rVnvV2V4pteG/QztQLcguSIRncdfIKUyJiT08JkOvWj5V70vevIAlOjV+3sEcRmBbSbXBd1iMrf4PXyv6U2SXMedZrLPgACgRjO9JFAdIpU8ybMYTEXV/EygGaxBbtR7qsntXK3V4Gw9jc9HvmPobQAhr/4jGEfIAHA1K7FVPlak7oPwiu9KApdVk18zWHOfdpxGs63GDqeb1 eUoUof1h kIz1KjFP1XMtL8BO2EbT4n+DaiIH1NGrBhnFRjW2ybryFiknWWJyTrWOLpLH8UzlC0Tc/XYnvC7JobotQDcbwjI6U+pLrcrewgG7aSBZpOcWX/+T4fyIbaJjEGg7KN212irjEvYaFBpwAqP8Qwg3lQln5rqNaDwm1MCcJjT131JBoGO8/B4z7smRfrEW+wDSnx8e2dw1xftA+D3o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Dec 16, 2022 at 11:38:28AM -0500, Waiman Long wrote: > On 12/16/22 05:32, Catalin Marinas wrote: > > On Wed, Dec 14, 2022 at 10:54:28AM -0500, Waiman Long wrote: > > > On 12/14/22 06:16, Catalin Marinas wrote: > > > > On Sat, Dec 10, 2022 at 06:00:48PM -0500, Waiman Long wrote: > > > > > Commit 6edda04ccc7c ("mm/kmemleak: prevent soft lockup in first > > > > > object iteration loop of kmemleak_scan()") fixes soft lockup problem > > > > > in kmemleak_scan() by periodically doing a cond_resched(). It does > > > > > take a reference of the current object before doing it. Unfortunately, > > > > > if the object has been deleted from the object_list, the next object > > > > > pointed to by its next pointer may no longer be valid after coming > > > > > back from cond_resched(). This can result in use-after-free and other > > > > > nasty problem. [...] > > If we get rid of object->lock and just use kmemleak_lock instead, we can > > have a big lock around the scanning, released briefly in > > kmemleak_cond_resched(). A standard list_del() (not _rcu) could be run > > during the resched but it also updates the current object->next. Once > > the lock is re-acquired, the list traversal can continue safely. The > > current object cannot be freed due to get_object(). No need for > > restarting the loop. > > The problem with a big lock (kmemleak_lock) is that we will be disabing > interrupt for an extended period of time which is not ideal. We do this already during scanning - scan_block() takes the kmemleak_lock as this protects the rb tree. We just need to take this lock at a higher level in scan_gray_list() but we can still release it in the loop as before, at least each iteration (even multiple times in an iteration if scan_block() takes longer). > I have posted a v2 patch that drop the idea of restarting the loop. Instead, > I just block the current object from being removed from the object_list to > make sure its next pointer will point to a valid object. I haven't got around to look at that yet. Still trying to see if we can simplify the locking here without a significant effect on latency. > > I don't think we'd miss much in terms of scalability for a debug > > feature. Object freeing already takes the kmemleak_lock, it's just that > > during scanning it will have to wait for the scanning loop to release > > it. We might as well release it within the loop on each iteration. > > > > So my proposal is to replace the rcu list traversal with the classic > > one and kmemleak_lock held (some functions like __find_and_get_object() > > will have to skip the lock). With this in place, we can subsequently > > remove all object->lock instances, just rely on the big lock (we do need > > to run lockdep if we do the latter separately, some nesting is a bit > > weird; my preference would be to remove the object->lock at the same > > time). We still need the rcu freeing in put_object() but for a > > completely different reason: the sl*b allocators don't like being called > > recursively, so we just use the RCU mechanism to free the kmemleak > > structures in a separate thread. > > That was what I thought about when you said you wanted to use a big lock > instead of object->lock in the last email. As I said above, we can't hold > the kmemleak_lock with interrupt disabled for an extended period of time > especially if RT tasks are running. So we may need to release the lock > frequently like per dozen objects or so. I believe we still need > rcu_read_lock() just to be safe. Yes, that's what I had in mind, release the lock very often but use a non-RCU traversal mechanism that updates list_head.next. Yet another option would be to do a quick traversal at the beginning of kmemleak_scan() to only do a get_object(). Once the use_count is increased, they won't be freed. Of course, it needs another walk at the end of the scanning to do the put_object(). I'll have a look at your v2 as well, though most likely early in January. -- Catalin