From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4DB9C48BF6 for ; Wed, 21 Feb 2024 09:58:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4EB0A6B00B5; Wed, 21 Feb 2024 04:58:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 49B776B00B6; Wed, 21 Feb 2024 04:58:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33C356B00B7; Wed, 21 Feb 2024 04:58:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 254B76B00B5 for ; Wed, 21 Feb 2024 04:58:33 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ED0151C0A9A for ; Wed, 21 Feb 2024 09:58:32 +0000 (UTC) X-FDA: 81815361264.19.79FB7C0 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf24.hostedemail.com (Postfix) with ESMTP id 9363A180017 for ; Wed, 21 Feb 2024 09:58:29 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eqoWccbd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uHlzl5Mb; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eqoWccbd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uHlzl5Mb; dmarc=none; spf=pass (imf24.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708509510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6CZZLxaCjUniAqfUA8XuBol8bY7o8se1WYeJDD69kWY=; b=nQ7+OxqXR/u9njltm431kVh5cZRg4qrVBNzP1TWSbQ5JzoSo57nvNUPTXyKXtNvaa3HajV nsXseqQjyJABCvjr10Od2wsvM9iXtFKxmb+rJG3iTZuMKn/8+huWb3I84t4azPP7RtmL4+ 6+rtGABfRXyFMm+xZZDDwxEqvi147eA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eqoWccbd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uHlzl5Mb; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eqoWccbd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uHlzl5Mb; dmarc=none; spf=pass (imf24.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708509510; a=rsa-sha256; cv=none; b=VQrRdTRt/OIDbxQKJG2d8Wi9VrVb92INTuLV9yyUehd8F7znFqmg2iQQWOEc6DBjeuZDVh OxQbXWCpqz45EAG+cWEdi5lJKh/jtacmOdwex8ry+WXzL4S02k1WYwWu/yAqRM4SKD7fUt zHYmzXKBBlHVCKzj6qHJ+x0gUPGatvU= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 92DB01FB4C; Wed, 21 Feb 2024 09:58:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1708509506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6CZZLxaCjUniAqfUA8XuBol8bY7o8se1WYeJDD69kWY=; b=eqoWccbdoRy5sD7lxgF7mOoSFHp5C6lhXoEJoGUq9zlS3nrMaTLkHS7eJqywHowi9zOuWv uEvWpF1xEcPNduaRxQqwOzE/poPKvGc8galZRHs2kFwgaGNn0Ytp6kZXaW2ilJzGFkJAr2 OcWkdhacveMY6FiUe3jkaM6laJ+yUlI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1708509506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6CZZLxaCjUniAqfUA8XuBol8bY7o8se1WYeJDD69kWY=; b=uHlzl5MbBIGidM84g/sKNE+UkK+1QMttnvG9eoEGviWkKkZixshbakkWgnUmXJzluiFZwJ XUPLbrrhp99SyzDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1708509506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6CZZLxaCjUniAqfUA8XuBol8bY7o8se1WYeJDD69kWY=; b=eqoWccbdoRy5sD7lxgF7mOoSFHp5C6lhXoEJoGUq9zlS3nrMaTLkHS7eJqywHowi9zOuWv uEvWpF1xEcPNduaRxQqwOzE/poPKvGc8galZRHs2kFwgaGNn0Ytp6kZXaW2ilJzGFkJAr2 OcWkdhacveMY6FiUe3jkaM6laJ+yUlI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1708509506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6CZZLxaCjUniAqfUA8XuBol8bY7o8se1WYeJDD69kWY=; b=uHlzl5MbBIGidM84g/sKNE+UkK+1QMttnvG9eoEGviWkKkZixshbakkWgnUmXJzluiFZwJ XUPLbrrhp99SyzDA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 7856213A69; Wed, 21 Feb 2024 09:58:26 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 23v5HELJ1WUYPgAAD6G6ig (envelope-from ); Wed, 21 Feb 2024 09:58:26 +0000 Message-ID: Date: Wed, 21 Feb 2024 10:58:26 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Stall at page allocations with __GFP_RETRY_MAYFAIL (Re: [PATCH v1] ALSA: memalloc: Fix indefinite hang in non-iommu case) Content-Language: en-US To: Sven van Ashbrook Cc: Takashi Iwai , Karthikeyan Ramasubramanian , LKML , Brian Geffon , stable@vger.kernel.org, Curtis Malainey , Jaroslav Kysela , Takashi Iwai , linux-sound@vger.kernel.org, linux-mm@kvack.org References: <20240214170720.v1.1.Ic3de2566a7fd3de8501b2f18afa9f94eadb2df0a@changeid> <87jzn0ofdb.wl-tiwai@suse.de> <235ab5aa-90a4-4dd7-b2c6-70469605bcfb@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 9363A180017 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: dekar1frjiz99hi5yzya5wcj7d43d4cn X-HE-Tag: 1708509509-14493 X-HE-Meta: U2FsdGVkX19vpmhNGjRORUzbJMpv0RVxAuzLygxLlWHwO9hyaZGqdpIsnP6S/j5G6iXKSEMVuJqIi9HBSmXM1dwmQ2eDzGP4hCQhhgQHH8VbFvesY+dQbZACcMl5IPb+D4QV1hq1AwUHL07vC/Au2mBdbggfG6AAVs/uf4byAGllvCchX/b8KRdUljxqt68K7RA5HmFNfMJ7brTvLZN9Bknkr0pKa7zc4E5OWHbxBSbmOddEu5KfSIzzqyRff6jBO0ugJcIsAfQkMj3kaD1KIfvACUMgBo51Q6cTqpVkoJdkkMkCFW31BsK2gMpMENo0j95w0xPfiyjY9nSr1WG4B7msB6RGkQi+nE1NOSvcWQuuOrMsH7grmNC3CxrnXpbPafv/racY/49xWp8Ixz1ZDpRmGZjk8XNyM/HiXg++sIBYzxw2wJQA70TQ1M5UJaLBAZnOiNSdPoxRsdQxSXZvQhCQTxLpLYrORaaXtzeV+d7A+jzkSxW0VS1SeHvSYoRmiltoit6RkDT0/+aiwyHhZ43l+TMdMGs9N3aLLuMZHrbOxayZ7P3he84oIDsKDDr9dUUdNMbIl7s1rpIf/B/otL5AtgqJ0kHSDfk1bu+noqiRHqiedKBqSv7ZeN57ANXLFx4ygpUBQxlbBpgqTPZ4gMIdUHExrMDfQluSlNb8m+G1CFyGGjFmby5cMv5NiJ9vVAFoap6wq3frO/wISZS5xvfuxwgI226dexxBlBRNbSyfFMemOc7I5NkOHPKsNFSHLlfzeU3ZvzA1QCDZMvTeUwMP9CUOA45MbTiPo7l8tFXMmIPGLq2nGhA4yb1o/YMIO4ulhtdZepmZCNFAakTWNAUkm+FYwfo0c2tL8m5PvfXO9xbBW7/JpXZPizqYhdCKzCo1zw+BEIU/njMLUTtYp52bKxUA+YEQ2+OwfDQYrdjvhh+MWJ+rTrfv6pTkpZ8dEX1P2HRTSkiDo1jcvc4 GO5LPbLR J9RGabMYJw2hiQ9PfkddH1zxgo/5hxwC2+8gx8p4J9hfqRLnYO9nK/E07TAqk8f8u7ufrEpMP5p/KIDKLYH4Q1UKWggF62pspb9smDhUG3Z4iBgwmRqw53VTyQm7yDqGPQ/vTQn0/XCheEOouURL/dxurIQZ/SrbuDJ9zhHEzNjVTWNA675kY2Hxp9mx/pgJ3CpxgxqN+0c04d+mZYfGJ3kFGVss5sq2+Uz0BBMdi6OPYgnnLCnlzKs8VFlTNqkKgddmPFoBRPeER/8aWg17/C8e4xh5d8Eo5weBuopKAWUWlF7z9fVXBFq5ryDZaVdCTdmJxuoTYP+IbXCbxWaTtSYU4izRaxKAx/pDQg1wmEHbqiXHI15WnhMS+avA79X5AVz6TpckSLGr/RfImrWuaFKu7Gv1EQ9te9m6MKTCQbhPsyiM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/20/24 16:52, Sven van Ashbrook wrote: > Takaski, Vlastimil: thanks so much for the engagement! See below. > >> On 2/19/24 12:36, Takashi Iwai wrote: >> > >> > Karthikeyan, Sven, and co: could you guys show the stack trace at the >> > stall? This may give us more clear light. > Here are our notes of the indefinite stall we saw on v5.10 with iommu SoCs. > We did not pursue debugging the stall at the time, in favour of a work-around > with the gfp flags. Therefore we only have partial confidence in the notes > below. Take them with a block of salt, but they may point in a useful direction. > > 1. try to do a "costly" allocation (order > PAGE_ALLOC_COSTLY_ORDER) with > __GFP_RETRY_MAYFAIL set. > > 2. page alloc's __alloc_pages_slowpath [1] tries to get a page from > the freelist. > This fails because there is nothing free of that costly order. > > 3. page alloc tries to reclaim by calling __alloc_pages_direct_reclaim, which > bails out [2] because a zone is ready to be compacted; it pretends > to have made > a single page of progress. > > 4. page alloc tries to compact, but this always bails out early [3] > because __GFP_IO is not set > (it's not passed by the snd allocator, and even if it were, we are > suspending so the > __GFP_IO flag would be cleared anyway). > > 5. page alloc believes reclaim progress was made (because of the > pretense in item 3) and > so it checks whether it should retry compaction. The compaction > retry logic [4] thinks > it should try again, because: > a) reclaim is needed because of the early bail-out in item 4 > b) a zonelist is suitable for compaction > > 6. goto 2. indefinite stall. Thanks a lot, seems this can indeed happen even in 6.8-rc5. We're mishandling the case where compaction is skipped due to lack of __GFP_IO, which is indeed cleared in suspend/resume. I'll create a fix. Please don't hesitate to report such issues the next time, even if not fully debugged :) >> >> > Also, Vlastimil suggested that tracepoints would be helpful if that's >> > really in the page allocator, too. >> > > > We might be able to generate traces by bailing out of the indefinite > stall using a timer, > which should hopefully give us a device that's "alive enough" to read > the traces. > > Can you advise which tracepoints you'd like to see? Is trace-cmd [5] > suitable to capture > this? > > [1] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/third_party/kernel/v5.10/mm/page_alloc.c;l=4654;drc=a16293af64a1f558dab9a5dd7fb05fdbc2b7c5c0 > [2] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/third_party/kernel/v5.10/mm/vmscan.c;drc=44452e4236561f6e36ec587805a52b683e2804c9;l=6177 > [3] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/third_party/kernel/v5.10/mm/compaction.c;l=2479;drc=d7b105aa1559e6c287f3f372044c21c7400b7784 > [4] https://source.chromium.org/chromiumos/chromiumos/codesearch/+/main:src/third_party/kernel/v5.10/mm/page_alloc.c;l=4171;drc=a16293af64a1f558dab9a5dd7fb05fdbc2b7c5c0 > [5] https://chromium.googlesource.com/chromiumos/docs/+/HEAD/kernel_development.md#ftrace-debugging