From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A458DC63797 for ; Fri, 13 Jan 2023 02:26:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5E2F8E0002; Thu, 12 Jan 2023 21:26:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D0E428E0001; Thu, 12 Jan 2023 21:26:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BFD158E0002; Thu, 12 Jan 2023 21:26:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B16C18E0001 for ; Thu, 12 Jan 2023 21:26:15 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 7A46580C48 for ; Fri, 13 Jan 2023 02:26:15 +0000 (UTC) X-FDA: 80348186310.11.3DB5DB3 Received: from smtp-fw-6001.amazon.com (smtp-fw-6001.amazon.com [52.95.48.154]) by imf08.hostedemail.com (Postfix) with ESMTP id C438D160006 for ; Fri, 13 Jan 2023 02:26:13 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=ZxdXam82; spf=pass (imf08.hostedemail.com: domain of "prvs=370d619ea=kamatam@amazon.com" designates 52.95.48.154 as permitted sender) smtp.mailfrom="prvs=370d619ea=kamatam@amazon.com"; dmarc=pass (policy=quarantine) header.from=amazon.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673576773; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mYR8JqIg0s6wUvFwzEIDR1VaWz/29iDKbkdZrP40UZA=; b=JyZTA5AANBt4R7oNviaUdB0qd0l8Qv7bgHhM+LlyADzp3pCgdrZQOGIS9xDQcUAkICTbsI HPD12G9XxMtc0KhMUeTtLOJVOhAgoKL7QDRlRL1Y4Gv8gCJJXUW5CeUkZ3QXQ0HfjKd7Ny NveZmFviIiZArWg20F+QTwee0g3ltd4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=ZxdXam82; spf=pass (imf08.hostedemail.com: domain of "prvs=370d619ea=kamatam@amazon.com" designates 52.95.48.154 as permitted sender) smtp.mailfrom="prvs=370d619ea=kamatam@amazon.com"; dmarc=pass (policy=quarantine) header.from=amazon.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673576773; a=rsa-sha256; cv=none; b=jFqIo443ts5R+hTX+INjprGhWKRSweWwQKDnolM/wYMh9LqVorVMWwHWzv0WrERq4RLpBx C0Vi1nmDTGMiGDPOvZYdPWkUXH2lHmRvyLnKE4Fm6DDZefv9e3YlLQMlzp/dbTWMHnfRCn 4c+uTQ7Zj16jXg6q21/8tO8BNR8Wr00= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1673576775; x=1705112775; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mYR8JqIg0s6wUvFwzEIDR1VaWz/29iDKbkdZrP40UZA=; b=ZxdXam82jpV4CPSgrHVsmc8F7GI2H+YvoWz3F6z4QgEObSwWTzvhYk8y PreFwL7eSzooJ0rukQHkcmm3tl9mvRIL67MIHX5BBk/O7pvoI2r7zCQ0d SwLP3XpiOT+GfaR4bGfdPC67gVTO4wxUgUaReOPblRT8j81s5kffBcBPf A=; X-IronPort-AV: E=Sophos;i="5.97,212,1669075200"; d="scan'208";a="287975044" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1e-m6i4x-3e1fab07.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-6001.iad6.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jan 2023 02:26:14 +0000 Received: from EX13MTAUWB002.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-iad-1e-m6i4x-3e1fab07.us-east-1.amazon.com (Postfix) with ESMTPS id A2427836FE; Fri, 13 Jan 2023 02:26:11 +0000 (UTC) Received: from EX19D010UWA004.ant.amazon.com (10.13.138.204) by EX13MTAUWB002.ant.amazon.com (10.43.161.202) with Microsoft SMTP Server (TLS) id 15.0.1497.45; Fri, 13 Jan 2023 02:26:11 +0000 Received: from u9aa42af9e4c55a.ant.amazon.com (10.43.160.120) by EX19D010UWA004.ant.amazon.com (10.13.138.204) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.1118.7; Fri, 13 Jan 2023 02:26:10 +0000 From: Munehisa Kamata To: CC: , , , , , , Subject: Re: another use-after-free in ep_remove_wait_queue() Date: Thu, 12 Jan 2023 18:25:55 -0800 Message-ID: <20230113022555.2467724-1-kamatam@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.43.160.120] X-ClientProxiedBy: EX13D38UWC004.ant.amazon.com (10.43.162.204) To EX19D010UWA004.ant.amazon.com (10.13.138.204) X-Rspamd-Queue-Id: C438D160006 X-Stat-Signature: n6t45fuby8qb77t1hzew8sdtx7h8yupg X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673576773-379551 X-HE-Meta: U2FsdGVkX1+MCQcxRdGGZwCCJ4mWYZVZpfvmEaFU0WICMwgZ4y4NeZ6SYZM14RWUIMPI1yaKIou+fpZ1ZZEHICSqWaGCQT0pSzVU9wkr1jv1xQikOKio7AG+AZ1xKfMjYlcMQNpfUFH+sP1pAk5iFtjGgaxAmAaXvoEimNDFpTjRsdt0mOVSUu9MIlLYIo1OFtBQuizGKtOfJOzTUywGaudZoqYmefwVJY9ye4s0+LBIAhj384rCTz3CJemUb2sAujKkRDHzP6g9/kiV2mAJoqgSXjnr8tZ8GAybJpqbZXk+OMrYOvaLN3wjrBdKoRmgX2tzEXKHf8v77piUNNYLGUo05sguPY7mmdKBvOVlozhU9LGB9NiY0rMdJcFakwGUUOvrPHP6OLUlwn+nnxJguLq7mZ/VIjdcYLWHEQ+W4qRhe7tC4/iYygpV2FjFlGeRlYkTCQQgqoTjsTFTcLOBog842a/8B7ISQc22VHi1JqszMvplKqDYC19/Ay+0ufYyNW+bbr/1scV53j96dFZrj7/+IJnCdZUa2EbloBGs38uoSD0vh5APeiu5x6TJgs/VCqcYx09cehQ7prFogSRXJaO6aeBq/U12G22J+oixsiM8Pl6u9NmVobc9L1OfyK9x/gZnAXeKYDQTf85+mX18cQcC+Ym9jCWkpJquwfWo4fvzl9f3sYYYQFC/CkhFJfb+wEHMp4AGYM+iSNRIOTF/wCif8gQXqcHSARXjyYkQcdtbFSM/W7/VeqhHwUB/td50EhGCgVsBcue3Y6kqysGeEomFNr5S/WEnf2IBfcj7T8s3kKAWoMB1yG6E093exHAk+NnjCWJmtbrqJoMr9M6dQmGPMo6zzWBUlkn+4mvXHG+lTJETORjRTtWsOIZ2Jr+UYure7kZ9yQ8+Pn6KA5c9beAkQYJYwZTWdxjs/nRNA44xV5KkQnDshFa+HIvRY2tOXorauK1ERUqbKcjxeNL a/p+FHRa ByYOoEIA1lCHQ4g8R9RMIe9Eda5psvxZ8UP+IrQ3y69ftHujX6UnUTJbTRR0J9Sq526d70MTOn1HwZpp6R8MCBz13xTxHgoMY28sVqjgj3rgrA30DjAbDhXWcFwyAEkciwBrgjStuJZ+KSSifbQHvlLaDZY1XkbulNmG2L2+i2dr5XIfqdEsA+O6hnEUqtvrs3pGz6XNLWmViEnWB6LvKxTTS8vbhZduPQDz09+pqgWk/Ws9hyTBBc/wUww1zg7BU4jbBO44C4CZiDKEqtoDoR1xNmwbrGNR16UYk2Hj624ocQKd7e+zPJbChZc07gOim2E8WHR6Gl5Gy7Fn8kp2CI/yNDW40AoTmR8WJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000059, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 2023-01-12 22:01:24 +0000, Suren Baghdasaryan wrote: > > On Mon, Jan 9, 2023 at 7:06 PM Suren Baghdasaryan wrote: > > > > On Mon, Jan 9, 2023 at 5:33 PM Suren Baghdasaryan wrote: > > > > > > On Sun, Jan 8, 2023 at 3:49 PM Hillf Danton wrote: > > > > > > > > On 8 Jan 2023 14:25:48 -0800 PM Munehisa Kamata wrote: > > > > > > > > > > That patch survived the repro in my original post, however, the waker > > > > > (rmdir) was getting stuck until a file descriptor of the epoll instance or > > > > > the pressure file got closed. So, if the following modified repro runs > > > > > with the patch, the waker never returns (unless the sleeper gets killed) > > > > > while holding cgroup_mutex. This doesn't seem to be what you expected to > > > > > see with the patch, does it? Even wake_up_all() does not appear to empty > > > > > the queue, but wake_up_pollfree() does. > > > > > > > > Thanks for your testing. And the debugging completes. > > > > > > > > Mind sending a patch with wake_up_pollfree() folded? > > > > > > I finally had some time to look into this issue. I don't think > > > delaying destruction in psi_trigger_destroy() because there are still > > > users of the trigger as Hillf suggested is a good way to go. Before > > > [1] correct trigger destruction was handled using a > > > psi_trigger.refcount. For some reason I thought it's not needed > > > anymore when we placed one-trigger-per-file restriction in that patch, > > > so I removed it. Obviously that was a wrong move, so I think the > > > cleanest way would be to bring back the refcounting. That way the last > > > user of the trigger (either psi_trigger_poll() or psi_fop_release()) > > > will free the trigger. > > > I'll check once more to make sure I did not miss anything and if there > > > are no objections, will post a fix. > > > > Uh, I recalled now why refcounting was not helpful here. I'm making > > the same mistake of thinking that poll_wait() blocks until the call to > > wake_up() which is not the case. Let me think if there is anything > > better than wake_up_pollfree() for this case. > > Hi Munehisa, > Sorry for the delay. I was trying to reproduce the issue but even > after adding a delay before ep_remove_wait_queue() it did not happen. Hi Suren, Thank you for your help here. Just in case, do you have KASAN enabled in your config? If not, this may just silently corrupt a certain memory location and not immediately followed by obvious messages or noticeable event like oops. > One thing about wake_up_pollfree() solution that does not seem right > to me is this comment at > https://elixir.bootlin.com/linux/latest/source/include/linux/wait.h#L253: > > `In the very rare cases where a ->poll() implementation uses a > waitqueue whose lifetime is tied to a task rather than to the 'struct > file' being polled, this function must be called before the waitqueue > is freed...` > > In our case we free the waitqueue from cgroup_pressure_release(), > which is the handler for `release` operation on cgroup psi files. The > other place calling psi_trigger_destroy() is psi_fop_release(), which > is also tied to the lifetime to the psi files. Therefore the lifetime > of the trigger's waitqueue is tied to the lifetime of the files and > IIUC, we should not be required to use wake_up_pollfree(). > Could you please post your .config file? I might be missing some > configuration which prevents the issue from happening on my side. Sure, here is my config. https://gist.github.com/kamatam9/a078bdd9f695e7a0767b061c60e48d50 I confirmed that it's reliably reproducible with v6.2-rc3 as shown below. https://gist.github.com/kamatam9/096a79cf59d8ed8785c4267e917b8675 Regards, Munehisa > Thanks, > Suren. > > > > > > > > > > > [1] https://lore.kernel.org/lkml/20220111232309.1786347-1-surenb@google.com/ > > > > > > Thanks, > > > Suren. > > > > > > > > > > > Hillf > >