From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17F26C433FE for ; Thu, 29 Sep 2022 14:07:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A69DB8D0005; Thu, 29 Sep 2022 10:07:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A19788D0001; Thu, 29 Sep 2022 10:07:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BA2A8D0005; Thu, 29 Sep 2022 10:07:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7CFC78D0001 for ; Thu, 29 Sep 2022 10:07:45 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 44046140CD0 for ; Thu, 29 Sep 2022 14:07:45 +0000 (UTC) X-FDA: 79965301290.28.D7630CC Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf25.hostedemail.com (Postfix) with ESMTP id 60EA5A001A for ; Thu, 29 Sep 2022 14:07:43 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id n7so1376749plp.1 for ; Thu, 29 Sep 2022 07:07:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date; bh=YgqJYszIlI8pskjB/t57QE+szPDZKYoQV8TF2bbwEX8=; b=OrXT1SutETToYdv7j3df9XXvEfwHPS7WQIKr6y7scibKamgKrKdMEJRkXztP74PC7U 0QqdgdDqGd9iG6sywEb+PW/Mw+AEbdjq0v991OuAW55f2b7+0Ynr44ioQY1ctfAu73cN 0D8XUABOV7LgRRfygevpmTihBj1gy9lKCPht2g3g9XEd5i51eD66FSxhdeFVNhrO+Jne 7LeU5g6vIGfDA+YadN+m8UYp+Ll8ejdw7dS5LRfBPRwXua8+TnAlMpotMEK61fb3QEUn eYe3XLyFHCQjRZVwPqt9dICAtbDfQJbfSK1w25l74J35PaiZiyVmc4vk796XoaUS8oVS dXYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=YgqJYszIlI8pskjB/t57QE+szPDZKYoQV8TF2bbwEX8=; b=qrw5CQuunyrhxCIYFgjCs1X6smNyudQa2bPGgVsXKc0wiDmmKxpr7UaaQg6WiFskRI S1RjHBe/TbpoJru5eZ/DO0Ln8Q5XTkk/aC8hXZOgOagV6tPQ+IkY3roZE7/FvYhwz/EH w4Sks+Da9iD1+x/CV7dmaC1ZeXg4L6bHQvQZMDJGG0rk+vwH6paN+P8k21LLq6JJ66Ws SgoIo4lapZPnZxzeiOiK1fweQvAJKndaA5qmLYIO4W324HWjFVz+JGZCHcYH+gwf0m6O ZK2xHsz/bKWg9Q28C5DjpRC4d6r0wdkV6Y1FU2VqBk0+5A7QGUkELYXe2mxaLKeZjr7S NoHQ== X-Gm-Message-State: ACrzQf2ga3ZnB2wWBs+u/UwH7MQBygmLTlF6Yb6T/teyuP3j6SEirKPn LVVgFHHmvSS9Ivm9M6l56Z/oMw== X-Google-Smtp-Source: AMsMyM6cvV2NYLkk14HdJM0NJuqN1eJDwLVwjJsC8krNuktzgpja/UQg8rGDA81BGLv6mrRWtqGYYA== X-Received: by 2002:a17:903:248:b0:172:7520:db04 with SMTP id j8-20020a170903024800b001727520db04mr3708608plh.99.1664460461971; Thu, 29 Sep 2022 07:07:41 -0700 (PDT) Received: from [192.168.1.136] ([198.8.77.157]) by smtp.gmail.com with ESMTPSA id u11-20020a17090341cb00b00174d9bbeda4sm6060863ple.197.2022.09.29.07.07.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 29 Sep 2022 07:07:41 -0700 (PDT) Message-ID: <0f7a2712-5252-260c-3b0f-ec584e1066a3@kernel.dk> Date: Thu, 29 Sep 2022 08:07:39 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.3.0 Subject: Re: [syzbot] inconsistent lock state in kmem_cache_alloc Content-Language: en-US To: Jan Kara , Vlastimil Babka Cc: syzbot , akpm@linux-foundation.org, keescook@chromium.org, linux-kernel@vger.kernel.org, mark.rutland@arm.com, mhiramat@kernel.org, rostedt@goodmis.org, syzkaller-bugs@googlegroups.com, "linux-mm@kvack.org" , Michal Hocko , Amir Goldstein , Matthew Bobrowski , Linux-FSDevel , Sebastian Andrzej Siewior , Matthew Wilcox , io-uring@vger.kernel.org References: <00000000000074b50005e997178a@google.com> <20220929135627.ykivmdks2w5vzrwg@quack3> From: Jens Axboe In-Reply-To: <20220929135627.ykivmdks2w5vzrwg@quack3> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664460463; a=rsa-sha256; cv=none; b=SRe0G3dq7v2wFcAFHg+BnoiGirAQHcCQGnjqZaV5R3ztvyAh6vrx+AzXUpZIVB1LSukho2 mzq9qjboc14mkP3Rgfgf8b8YeTgKyKk8bV519eh1QRzmc+1kqedEFMuiaKwgD+7RYEqn2n ypohoZ2fBjYIZyOGru+5nAYiriJefXs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=OrXT1Sut; dmarc=none; spf=pass (imf25.hostedemail.com: domain of axboe@kernel.dk designates 209.85.214.170 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664460463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YgqJYszIlI8pskjB/t57QE+szPDZKYoQV8TF2bbwEX8=; b=pZP5bt3Wqv4PQJl/1quWPTsapXPa3vhzu8MyOwrcGmjRy9Bj1NemDbRanNTc+500sdT0hw K/bGhhGjhgxYnjQA48jAq+DPfwV7cCKW0EStkHjY+WP8/CPyCtCbJkjIEnyT0qgtAa/QPj u8H5LJgE9VEMnE84ZxssIGuaFrLwubk= X-Rspamd-Queue-Id: 60EA5A001A X-Rspam-User: Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel-dk.20210112.gappssmtp.com header.s=20210112 header.b=OrXT1Sut; dmarc=none; spf=pass (imf25.hostedemail.com: domain of axboe@kernel.dk designates 209.85.214.170 as permitted sender) smtp.mailfrom=axboe@kernel.dk X-Rspamd-Server: rspam03 X-Stat-Signature: mtthtbn1bgm8tr5e6w8wa3hcjn3698qu X-HE-Tag: 1664460463-801332 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 9/29/22 7:56 AM, Jan Kara wrote: > On Thu 29-09-22 15:24:22, Vlastimil Babka wrote: >> On 9/26/22 18:33, syzbot wrote: >>> Hello, >>> >>> syzbot found the following issue on: >>> >>> HEAD commit: 105a36f3694e Merge tag 'kbuild-fixes-v6.0-3' of git://git... >>> git tree: upstream >>> console+strace: https://syzkaller.appspot.com/x/log.txt?x=152bf540880000 >>> kernel config: https://syzkaller.appspot.com/x/.config?x=7db7ad17eb14cb7 >>> dashboard link: https://syzkaller.appspot.com/bug?extid=dfcc5f4da15868df7d4d >>> compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 >>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1020566c880000 >>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=104819e4880000 >>> >>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>> Reported-by: syzbot+dfcc5f4da15868df7d4d@syzkaller.appspotmail.com >> >> +CC more folks >> >> I'm not fully sure what this report means but I assume it's because there's >> a GFP_KERNEL kmalloc() allocation from softirq context? Should it perhaps >> use memalloc_nofs_save() at some well defined point? > > Thanks for the CC. The problem really is that io_uring is calling into > fsnotify_access() from softirq context. That isn't going to work. The > allocation is just a tip of the iceberg. Fsnotify simply does not expect to > be called from softirq context. All the dcache locks are not IRQ safe, it > can even obtain some sleeping locks and call to userspace if there are > suitable watches set up. > > So either io_uring needs to postpone fsnotify calls to a workqueue or we > need a way for io_uring code to tell iomap dio code that the completion > needs to always happen from a workqueue (as it currently does for writes). > Jens? Something like this should probably work - I'll write a test case and vet it. diff --git a/io_uring/rw.c b/io_uring/rw.c index 1ae1e52ab4cb..a25cd44cd415 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -236,14 +236,6 @@ static void kiocb_end_write(struct io_kiocb *req) static bool __io_complete_rw_common(struct io_kiocb *req, long res) { - struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); - - if (rw->kiocb.ki_flags & IOCB_WRITE) { - kiocb_end_write(req); - fsnotify_modify(req->file); - } else { - fsnotify_access(req->file); - } if (unlikely(res != req->cqe.res)) { if ((res == -EAGAIN || res == -EOPNOTSUPP) && io_rw_should_reissue(req)) { @@ -270,6 +262,20 @@ static inline int io_fixup_rw_res(struct io_kiocb *req, long res) return res; } +static void io_req_rw_complete(struct io_kiocb *req, bool *locked) +{ + struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); + + if (rw->kiocb.ki_flags & IOCB_WRITE) { + kiocb_end_write(req); + fsnotify_modify(req->file); + } else { + fsnotify_access(req->file); + } + + io_req_task_complete(req, locked); +} + static void io_complete_rw(struct kiocb *kiocb, long res) { struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb); @@ -278,7 +284,7 @@ static void io_complete_rw(struct kiocb *kiocb, long res) if (__io_complete_rw_common(req, res)) return; io_req_set_res(req, io_fixup_rw_res(req, res), 0); - req->io_task_work.func = io_req_task_complete; + req->io_task_work.func = io_req_rw_complete; io_req_task_work_add(req); } -- Jens Axboe