From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 976D6C433E0 for ; Mon, 1 Mar 2021 12:12:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 23E2764E38 for ; Mon, 1 Mar 2021 12:12:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23E2764E38 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6D6F38D0064; Mon, 1 Mar 2021 07:12:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6891E8D0063; Mon, 1 Mar 2021 07:12:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59D538D0064; Mon, 1 Mar 2021 07:12:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0007.hostedemail.com [216.40.44.7]) by kanga.kvack.org (Postfix) with ESMTP id 44F5D8D0063 for ; Mon, 1 Mar 2021 07:12:06 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EAEB8180AD83E for ; Mon, 1 Mar 2021 12:12:05 +0000 (UTC) X-FDA: 77871192210.03.89D3D92 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf21.hostedemail.com (Postfix) with ESMTP id 12CA3E00044B for ; Mon, 1 Mar 2021 12:12:00 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614600723; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bA6FPiajLwsJsKRo9IkNEVfF/BrE7ZaCRdL4La1+kYI=; b=vUbJUkdNuEka6AAHYqQMFnJiePnYCZgXl0XA74JhC0EHtPLIU8pSgphmTJ4uKCVTqd2HbU cB6FWkXpJ4DRFRoPbvM31jnY8z0LfieQyi2oVxrO20I0Yu84Z57Q1iTvYFtczMeiIOetVU X5EbN1+6xIn/SNoLhuMWKlhM5QI7RwY= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 6F222AE1B; Mon, 1 Mar 2021 12:12:03 +0000 (UTC) Date: Mon, 1 Mar 2021 13:11:57 +0100 From: Michal Hocko To: Shakeel Butt Cc: Mike Kravetz , syzbot , Andrew Morton , LKML , Linux MM , syzkaller-bugs , Eric Dumazet , Mina Almasry Subject: Re: possible deadlock in sk_clone_lock Message-ID: References: <000000000000f1c03b05bc43aadc@google.com> <7b7c4f41-b72e-840f-278a-320b9d97f887@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: cqqk3jjycanexm8owtwahrtnpnbukwak X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 12CA3E00044B Received-SPF: none (suse.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614600720-691631 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri 26-02-21 16:00:30, Shakeel Butt wrote: > On Fri, Feb 26, 2021 at 3:14 PM Mike Kravetz wrote: > > > > Cc: Michal > > > > On 2/26/21 2:44 PM, Shakeel Butt wrote: > > > On Fri, Feb 26, 2021 at 2:09 PM syzbot > > > wrote: > > > > >> other info that might help us debug this: > > >> > > >> Possible interrupt unsafe locking scenario: > > >> > > >> CPU0 CPU1 > > >> ---- ---- > > >> lock(hugetlb_lock); > > >> local_irq_disable(); > > >> lock(slock-AF_INET); > > >> lock(hugetlb_lock); > > >> > > >> lock(slock-AF_INET); > > >> > > >> *** DEADLOCK *** > > > > > > This has been reproduced on 4.19 stable kernel as well [1] and there > > > is a reproducer as well. > > > > > > It seems like sendmsg(MSG_ZEROCOPY) from a buffer backed by hugetlb. I > > > wonder if we just need to make hugetlb_lock softirq-safe. > > > > > > [1] https://syzkaller.appspot.com/bug?extid=6383ce4b0b8ec575ad93 > > > > Thanks Shakeel, > > > > Commit c77c0a8ac4c5 ("mm/hugetlb: defer freeing of huge pages if in non-task > > context") attempted to address this issue. It uses a work queue to > > acquire hugetlb_lock if the caller is !in_task(). > > > > In another recent thread, there was the suggestion to change the > > !in_task to in_atomic. > > > > I need to do some research on the subtle differences between in_task, > > in_atomic, etc. TBH, I 'thought' !in_task would prevent the issue > > reported here. But, that obviously is not the case. > > I think the freeing is happening in the process context in this report > but it is creating the lock chain from softirq-safe slock to > irq-unsafe hugetlb_lock. So, two solutions I can think of are: (1) > always defer the freeing of hugetlb pages to a work queue or (2) make > hugetlb_lock softirq-safe. There is __do_softirq so this should be in the soft IRQ context no? Is this really reproducible with kernels which have c77c0a8ac4c5 applied? Btw. making hugetlb lock irq safe has been already discussed and it seems to be much harder than expected as some heavy operations are done under the lock. This is really bad. Postponing the whole freeing operation into a worker context is certainly possible but I would consider it rather unfortunate. We would have to add some sync mechanism to wait for hugetlb pages in flight to prevent from external observability to the userspace. E.g. when shrinking the pool. -- Michal Hocko SUSE Labs