From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6EBFC433F5 for ; Thu, 9 Sep 2021 11:54:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3474461BB5 for ; Thu, 9 Sep 2021 11:54:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3474461BB5 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=suse.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 96A81900002; Thu, 9 Sep 2021 07:54:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 919D26B0072; Thu, 9 Sep 2021 07:54:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E124900002; Thu, 9 Sep 2021 07:54:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0055.hostedemail.com [216.40.44.55]) by kanga.kvack.org (Postfix) with ESMTP id 6D4376B0071 for ; Thu, 9 Sep 2021 07:54:34 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E348618259E3A for ; Thu, 9 Sep 2021 11:54:33 +0000 (UTC) X-FDA: 78567877626.02.376DD6D Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf05.hostedemail.com (Postfix) with ESMTP id 7F4755051BD3 for ; Thu, 9 Sep 2021 11:54:33 +0000 (UTC) Received: from relay2.suse.de (relay2.suse.de [149.44.160.134]) by smtp-out2.suse.de (Postfix) with ESMTP id 1BDBA1FDEA; Thu, 9 Sep 2021 11:54:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1631188472; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bnU+HtTYKZVwaE80A++rJfB4IJVmmPy41Bjr0LW+UBk=; b=q2SbhZ0A68UeuUHoQJhZFlAC/dQtGULYwguqWPb5BCrAFM0VNKmRU5aIYFYXCAc+35lB9S jgXR4ZnEOb5a5ieqf4ChqPNoVAvYjJmSZdnuuj0t+QoT5x+5L21OKadwWXs9RYKtL1lrwS 43kIF58v8EpO+s0zxt/PszKVUVtg83Q= Received: from suse.cz (unknown [10.100.201.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by relay2.suse.de (Postfix) with ESMTPS id A23FDA3D64; Thu, 9 Sep 2021 11:54:31 +0000 (UTC) Date: Thu, 9 Sep 2021 13:54:31 +0200 From: Michal Hocko To: Mike Kravetz Cc: Hillf Danton , Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH RESEND 0/8] hugetlb: add demote/split page functionality Message-ID: References: <20210816162749.22b921a61156a091f3e1d14d@linux-foundation.org> <20210816184611.07b97f4c26b83090f5d48fab@linux-foundation.org> <10d86c18-f0cf-395f-4209-17ac71b9fc03@oracle.com> <2d826470-d345-0196-1359-b79ed08dfc66@oracle.com> <02a1a50f-4e7c-4eb7-519c-35b26ec2c6af@oracle.com> <20210907085001.3773-1-hdanton@sina.com> <6c42bed7-d4dd-e5eb-5a74-24cf64bf52d3@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6c42bed7-d4dd-e5eb-5a74-24cf64bf52d3@oracle.com> Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=q2SbhZ0A; spf=pass (imf05.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7F4755051BD3 X-Stat-Signature: n74k3o9phm1bpikgat5ymcjib3asibni X-HE-Tag: 1631188473-506781 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed 08-09-21 14:00:19, Mike Kravetz wrote: > On 9/7/21 1:50 AM, Hillf Danton wrote: > > On Mon, 6 Sep 2021 16:40:28 +0200 Vlastimil Babka wrote: > >> On 9/2/21 20:17, Mike Kravetz wrote: > >>> > >>> Here is some very high level information from a long stall that was > >>> interrupted. This was an order 9 allocation from alloc_buddy_huge_page(). > >>> > >>> 55269.530564] __alloc_pages_slowpath: jiffies 47329325 tries 609673 cpu_tries 1 node 0 FAIL > >>> [55269.539893] r_tries 25 c_tries 609647 reclaim 47325161 compact 607 > >>> > >>> Yes, in __alloc_pages_slowpath for 47329325 jiffies before being interrupted. > >>> should_reclaim_retry returned true 25 times and should_compact_retry returned > >>> true 609647 times. > >>> Almost all time (47325161 jiffies) spent in __alloc_pages_direct_reclaim, and > >>> 607 jiffies spent in __alloc_pages_direct_compact. > >>> > >>> Looks like both > >>> reclaim retries > MAX_RECLAIM_RETRIES > >>> and > >>> compaction retries > MAX_COMPACT_RETRIES > >>> > >> Yeah AFAICS that's only possible with the scenario I suspected. I guess > >> we should put a limit on compact retries (maybe some multiple of > >> MAX_COMPACT_RETRIES) even if it thinks that reclaim could help, while > >> clearly it doesn't (i.e. because somebody else is stealing the page like > >> in your test case). > > > > And/or clamp reclaim retries for costly orders > > > > reclaim retries = MAX_RECLAIM_RETRIES - order; > > > > to pull down the chance for stall as low as possible. > > Thanks, and sorry for not replying quickly. I only get back to this as > time allows. > > We could clamp the number of compaction and reclaim retries in > __alloc_pages_slowpath as suggested. However, I noticed that a single > reclaim call could take a bunch of time. As a result, I instrumented > shrink_node to see what might be happening. Here is some information > from a long stall. Note that I only dump stats when jiffies > 100000. > > [ 8136.874706] shrink_node: 507654 total jiffies, 3557110 tries > [ 8136.881130] 130596341 reclaimed, 32 nr_to_reclaim > [ 8136.887643] compaction_suitable results: > [ 8136.893276] idx COMPACT_SKIPPED, 3557109 Can you get a more detailed break down of where the time is spent. Also How come the number of reclaimed pages is so excessive comparing to the reclaim target? There is something fishy going on here. -- Michal Hocko SUSE Labs