From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98E3EC433F5 for ; Thu, 6 Oct 2022 03:06:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ACF166B0071; Wed, 5 Oct 2022 23:06:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7EE96B0073; Wed, 5 Oct 2022 23:06:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91FB86B0074; Wed, 5 Oct 2022 23:06:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7FC726B0071 for ; Wed, 5 Oct 2022 23:06:19 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4398D1601A0 for ; Thu, 6 Oct 2022 03:06:19 +0000 (UTC) X-FDA: 79989036078.06.7B5F79E Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf08.hostedemail.com (Postfix) with ESMTP id DC2FE16001B for ; Thu, 6 Oct 2022 03:06:18 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id t10-20020a17090a4e4a00b0020af4bcae10so488521pjl.3 for ; Wed, 05 Oct 2022 20:06:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=KDORDmVJlGAH+2VDPr5/x8Jt3M8M+5CLZ86WDuJmMtQ=; b=XRNtI0RKOybw0AZPuMgzHQ6nwXZmdMO+fdjLU0CJID/MEsvN4UN4GSWoQ19kIarRJ0 OoFyZcjmJ8vff47NqmfklVugFf5abRbNno1bVwJ3LHSCq67q3jKj0ksZNE9mfd1YVbbl b6hQzCEvEKfG1yI/i1Ynw1SIWxy/MjuCItrNo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=KDORDmVJlGAH+2VDPr5/x8Jt3M8M+5CLZ86WDuJmMtQ=; b=Dxr15o+cK+dW9PY4kynmq5R7yyDOHBeLjr+i0mLjoIsn+czIS7+VM20X2bVf9IZGbV mwD4AdHa01XN0v+TaCl/Nzyf2h0PlOUiM9+2rVcIlE6DFU0InbgrwKBqBVoUkrqhadA5 sGcM7jl5GX9RGloPexWQazW1cQwIVwS7KbWenxqiLGqYKYAfUG3PK/teoleMk7a0yGSm nhG0a5DIQfNY+NIiDXgd79fpVIRR6EzmxjAAp0YVuinFCi9LhKBdv7UboOKvevmEgkGs dKQPMn8Rx+Hfz3moVpD4+zMszQ32vkzZh9MsQvSM3Avy3TYtdBYMb6hBYAmb8m3pBgOv xaoA== X-Gm-Message-State: ACrzQf21hDDEKkUYvD8hKWYTmYIIFlK5lhZwtbznNNvAq6+2q6mMa2BZ RDqzs9vndCLoM5Llz25bLkHAYg== X-Google-Smtp-Source: AMsMyM5pnDxtwM6bse0TAOUDjZ+WVFmurkxinUugzQUbn4YATg0uA0iXtH3Qn1ciOgVXjm8wi4zVNQ== X-Received: by 2002:a17:902:6bc7:b0:178:8305:392a with SMTP id m7-20020a1709026bc700b001788305392amr2409484plt.39.1665025577628; Wed, 05 Oct 2022 20:06:17 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id v126-20020a622f84000000b00550724f8ea0sm11590896pfv.128.2022.10.05.20.06.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Oct 2022 20:06:16 -0700 (PDT) Date: Wed, 5 Oct 2022 20:06:15 -0700 From: Kees Cook To: Jorge Merlino , David Howells Cc: Alexander Viro , Eric Biederman , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] Fix race condition when exec'ing setuid files Message-ID: <202210051950.CAF8CDBF@keescook> References: <20220910211215.140270-1-jorge.merlino@canonical.com> <202209131456.76A13BC5E4@keescook> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1665025579; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KDORDmVJlGAH+2VDPr5/x8Jt3M8M+5CLZ86WDuJmMtQ=; b=JpOy9uc3ZOWv3bsGcgOGes0aXT0Bh0P5x80n/5pTESBITrpQg5EdlsoGbixir2LHKikCQN YEmEcTB9N8jVLux04C0RCmD7215YZ8N2Fzq1PEm6F1jSjFRfgmSHbWpAJhftYXb8OgyBpp Mg62oJh7nXfsodFe3L+iCf6onOBDnmY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=XRNtI0RK; spf=pass (imf08.hostedemail.com: domain of keescook@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1665025579; a=rsa-sha256; cv=none; b=l/OBZCP+PMNcjtZKN58iYzl/p+8ZmpmdAP3L2BY1I7ITZNyXGzF+mPwkusJPM7yV8ew1D3 bj7NcPVON8reoulQK6laKumfBeoI7U5xj/kri6ju8+WwT0Pd7ql7hVKCzmmTxYoalphT3U oPtic4C7nHawhC+xKey3cbrKEb6miQs= X-Rspamd-Queue-Id: DC2FE16001B Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=XRNtI0RK; spf=pass (imf08.hostedemail.com: domain of keescook@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=keescook@chromium.org; dmarc=pass (policy=none) header.from=chromium.org X-Rspamd-Server: rspam06 X-Rspam-User: X-Stat-Signature: 54cs7fm6unxjdwt35tcx88htpc6g3quz X-HE-Tag: 1665025578-784882 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Oct 05, 2022 at 01:09:36PM -0300, Jorge Merlino wrote: > On 13/9/22 19:03, Kees Cook wrote: > > I'll want to spend some more time studying this race, but yes, it looks > > like it should get fixed. I'm curious, though, how did you find this > > problem? It seems quite unusual to have a high-load heavily threaded > > process decide to exec. > > I just got a response from our customer regarding the situation where this > race condition occurs: Thanks for getting back details! > Our application is a Rust-based CLI tool that acts as a frontend to > cloud-based testing infrastructure. In one mode of operation it uploads a > large number of test artifacts to cloud storage, spawns compute instances, > and then starts a VPN connection to those instances. The application creates > the VPN connection by executing another setuid-root tool as a subprocess. We > see that this subprocess sometimes fails to setuid. The "high-load heavily > threaded" aspect comes from the fact that we're using the Tokio runtime. > Each upload to cloud storage is a separate Tokio task (i.e. "green thread") > and these are scheduled onto "N" OS-level threads, where N = nproc. In a > large run we may upload a couple thousand artifacts but limit to 50 > concurrent uploads. Once these artifact uploads complete, we typically spawn > the setuid subprocess within 1-2 seconds. Interesting. Seems like the execve might be racing all the threads exiting? > Have you been able to look at this issue? I'll continue looking at this. Dave, this tracks back to commit a6f76f23d297 ("CRED: Make execve() take advantage of copy-on-write credentials") ... any ideas what's happening here? -- Kees Cook