From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.4 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F6C4C433EF for ; Fri, 10 Sep 2021 10:50:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 970C5611CE for ; Fri, 10 Sep 2021 10:50:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 970C5611CE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C3BB4900002; Fri, 10 Sep 2021 06:50:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEAE66B0072; Fri, 10 Sep 2021 06:50:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB2CB900002; Fri, 10 Sep 2021 06:50:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0040.hostedemail.com [216.40.44.40]) by kanga.kvack.org (Postfix) with ESMTP id 9B7546B0071 for ; Fri, 10 Sep 2021 06:50:54 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4CBF02B36D for ; Fri, 10 Sep 2021 10:50:54 +0000 (UTC) X-FDA: 78571346028.38.DD8F936 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf15.hostedemail.com (Postfix) with ESMTP id BE01FD000097 for ; Fri, 10 Sep 2021 10:50:53 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5AF9021AD9; Fri, 10 Sep 2021 10:50:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1631271052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cLa40tf04nYkoytYOxqmcnFcOjSkFNnyh5QKFWN0cDQ=; b=UmM/PXCv3DjaKbFsOu+/cD98ovCVmoRIUYn1YAMeXxhTCzmGDFwv9po5FdMa5eWu2covb+ WwvwlDwN1xLZcHsxuE2WWehpzw7yCzgYwqJiNiE/LLt+Wnpd8Lgs8VAOHnoy3rmKPfKZTQ 9GnWim+bLE/vjrZlLvv2o1HdAvYM8Ws= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1631271052; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cLa40tf04nYkoytYOxqmcnFcOjSkFNnyh5QKFWN0cDQ=; b=7h9cOBNJV0/tF44d4MDtH45pjx5vGDfQq7wTwZV8ZhT5XifVZXkyrFhUI/1NTlDdemjkSo kvU4CVNX/q3HdoBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 0B64A13D27; Fri, 10 Sep 2021 10:50:52 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id zunNAYw4O2F5HgAAMHmgww (envelope-from ); Fri, 10 Sep 2021 10:50:52 +0000 Message-ID: <1b1569ac-1144-4f9c-6938-b9d79c6743de@suse.cz> Date: Fri, 10 Sep 2021 12:50:51 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.1.0 Subject: Re: [PATCH 0/6] stackdepot, kasan, workqueue: Avoid expanding stackdepot slabs when holding raw_spin_lock Content-Language: en-US To: Shuah Khan , Marco Elver , Andrew Morton Cc: Tejun Heo , Lai Jiangshan , Andrey Konovalov , Walter Wu , Andrey Ryabinin , Alexander Potapenko , Dmitry Vyukov , Vijayanand Jitta , Vinayak Menon , "Gustavo A. R. Silva" , kasan-dev@googlegroups.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Aleksandr Nogikh , Taras Madan , Thomas Gleixner , Peter Zijlstra , Sebastian Andrzej Siewior References: <20210907141307.1437816-1-elver@google.com> <69f98dbd-e754-c34a-72cf-a62c858bcd2f@linuxfoundation.org> From: Vlastimil Babka In-Reply-To: <69f98dbd-e754-c34a-72cf-a62c858bcd2f@linuxfoundation.org> Content-Type: text/plain; charset=UTF-8 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="UmM/PXCv"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7h9cOBNJ; spf=pass (imf15.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: 1e781mqf6gg86qcn1bq9b3wjj8f3f145 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: BE01FD000097 X-HE-Tag: 1631271053-131789 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 9/7/21 22:05, Shuah Khan wrote: > On 9/7/21 8:13 AM, Marco Elver wrote: >> Shuah Khan reported [1]: >> >> =C2=A0 | When CONFIG_PROVE_RAW_LOCK_NESTING=3Dy and CONFIG_KASAN are e= nabled, >> =C2=A0 | kasan_record_aux_stack() runs into "BUG: Invalid wait context= " when >> =C2=A0 | it tries to allocate memory attempting to acquire spinlock in= page >> =C2=A0 | allocation code while holding workqueue pool raw_spinlock. >> =C2=A0 | >> =C2=A0 | There are several instances of this problem when block layer = tries >> =C2=A0 | to __queue_work(). Call trace from one of these instances is = below: >> =C2=A0 | >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0 kblockd_mod_delayed_work_on() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mod_delayed_work_on() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __queue_delay= ed_work() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 _= _queue_work() (rcu_read_lock, raw_spin_lock pool->lock held) >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 insert_work() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 kasan_record_aux_stack() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 kasan_save_stack() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stack_depot_save() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alloc_pages() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __a= lloc_pages() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 get_page_from_freelist() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 rm_queue() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rm_queue_pcplist() >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 local_lock_irqsave(&pagesets.l= ock, flags); >> =C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 [ BUG: Invalid wait context tr= iggered ] >> >> [1] >> https://lkml.kernel.org/r/20210902200134.25603-1-skhan@linuxfoundation= .org >> >> PROVE_RAW_LOCK_NESTING is pointing out that (on RT kernels) the lockin= g >> rules are being violated. More generally, memory is being allocated fr= om >> a non-preemptive context (raw_spin_lock'd c-s) where it is not allowed= . >> >> To properly fix this, we must prevent stackdepot from replenishing its >> "stack slab" pool if memory allocations cannot be done in the current >> context: it's a bug to use either GFP_ATOMIC nor GFP_NOWAIT in certain >> non-preemptive contexts, including raw_spin_locks (see gfp.h and >> ab00db216c9c7). >> >> The only downside is that saving a stack trace may fail if: stackdepot >> runs out of space AND the same stack trace has not been recorded befor= e. >> I expect this to be unlikely, and a simple experiment (boot the kernel= ) >> didn't result in any failure to record stack trace from insert_work(). >> >> The series includes a few minor fixes to stackdepot that I noticed in >> preparing the series. It then introduces __stack_depot_save(), which >> exposes the option to force stackdepot to not allocate any memory. >> Finally, KASAN is changed to use the new stackdepot interface and >> provide kasan_record_aux_stack_noalloc(), which is then used by >> workqueue code. >> >> Marco Elver (6): >> =C2=A0=C2=A0 lib/stackdepot: include gfp.h >> =C2=A0=C2=A0 lib/stackdepot: remove unused function argument >> =C2=A0=C2=A0 lib/stackdepot: introduce __stack_depot_save() >> =C2=A0=C2=A0 kasan: common: provide can_alloc in kasan_save_stack() >> =C2=A0=C2=A0 kasan: generic: introduce kasan_record_aux_stack_noalloc(= ) >> =C2=A0=C2=A0 workqueue, kasan: avoid alloc_pages() when recording stac= k >> >> =C2=A0 include/linux/kasan.h=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 2 += + >> =C2=A0 include/linux/stackdepot.h |=C2=A0 6 +++++ >> =C2=A0 kernel/workqueue.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 |=C2=A0 2 +- >> =C2=A0 lib/stackdepot.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 | 51 ++++++++++++++++++++++++++++++-------- >> =C2=A0 mm/kasan/common.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 |=C2=A0 6 ++--- >> =C2=A0 mm/kasan/generic.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 | 14 +++++++++-- >> =C2=A0 mm/kasan/kasan.h=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 |=C2=A0 2 +- >> =C2=A0 7 files changed, 65 insertions(+), 18 deletions(-) >> >=20 > Thank you. Tested all the 6 patches in this series on Linux 5.14. This = problem > exists in 5.13 and needs to be marked for both 5.14 and 5.13 stable rel= eases. I think if this problem manifests only with CONFIG_PROVE_RAW_LOCK_NESTING then it shouldn't be backported to stable. CONFIG_PROVE_RAW_LOCK_NESTING = is an experimental/development option to earlier discover what will collide with RT lock semantics, without needing the full RT tree. Thus, good to fix going forward, but not necessary to stable backport. > Here is my >=20 > Tested-by: Shuah Khan >=20 > thanks, > -- Shuah >=20