From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29A98C43461 for ; Thu, 22 Apr 2021 08:56:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 93F656144E for ; Thu, 22 Apr 2021 08:56:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 93F656144E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CCCB26B0070; Thu, 22 Apr 2021 04:56:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C54DF6B0071; Thu, 22 Apr 2021 04:56:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A590E6B0072; Thu, 22 Apr 2021 04:56:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0249.hostedemail.com [216.40.44.249]) by kanga.kvack.org (Postfix) with ESMTP id 8411A6B0070 for ; Thu, 22 Apr 2021 04:56:32 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2249D1E1C for ; Thu, 22 Apr 2021 08:56:32 +0000 (UTC) X-FDA: 78059397024.22.D0AB86D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 01DC6C0007D6 for ; Thu, 22 Apr 2021 08:56:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1619081791; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wHQB1IPbhElZKGgSLLBWBMmqCT2h+h35Ym1IEByfZ34=; b=NDG9DrXfCo4zlV7axS7MDvv+WJBqTA4MrZbJSxpZmA7eO0FkP1N+jVj4TyPHYd3MlBzEW6 /BdX4LK51NDpMCy3zO3xOK412TBgCR/0u5gKImrUd/kIFlXg0Fgqd/f+zCslZcEphRaJ+F KTUnYB+B0fiqRUiNZEBFaP0wG4qs9Zc= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-133-6ATtXob5MBymQ0UfKuZYMg-1; Thu, 22 Apr 2021 04:56:29 -0400 X-MC-Unique: 6ATtXob5MBymQ0UfKuZYMg-1 Received: by mail-wm1-f69.google.com with SMTP id a71-20020a1c984a0000b029013841a43c3dso1069618wme.3 for ; Thu, 22 Apr 2021 01:56:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:cc:references:from:organization:subject :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=wHQB1IPbhElZKGgSLLBWBMmqCT2h+h35Ym1IEByfZ34=; b=Qt70vdwr0vBrGYN6uQpk/gKXujHMOmjw3/IGt6O8xH+ZmFZyP/tiRBegJCmSequZrG Rl5iGvoIa3FIksOQ1VSwOjiEV0Vn4cIM82mM531PHiku62QWgp/5IFBi8ltOwaEsblSA siYtP396gopVv5UH7KksVojsMNDnY2yB4DI+SsXGHvXt3buYgKxO+y1/c4TilnrmlLCB 95uATwGVqN85YUrj3CknyhkGOCVbcKi6Rq9gRE/+B8FJ7zRxmteOrlfwoY8JVFGu8F9I pjnYrnUgaltIgJLBECbsIAxC3XkGKx30rlmJUpqqGjmbcqsE+uxI6ryjrggGeiAdEcL3 F6LQ== X-Gm-Message-State: AOAM533ey76XPM83LdCurIRkmVifG/UlHwtoBsVotWCASzI9PpFhb8hB OhiFrBqFxUcNtExeS2yZWeG07+q7H1b7ts4LjkLjCACtXFtcdDDmpqBU5r3g+sWED0h3VNofmAc SuzS92MDPVMp0IgnG5865vf7c5t0jiepvJOgRvKAQtLIsI0Pb9YzWk7NhOY0= X-Received: by 2002:a1c:7315:: with SMTP id d21mr2654912wmb.155.1619081788205; Thu, 22 Apr 2021 01:56:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwsPHfelfld6YqOw0tWP+F88dcipkz+DIt+3hCnhdfScPHT+MwTYgYvwxcA6S6SOCRu/NXMbA== X-Received: by 2002:a1c:7315:: with SMTP id d21mr2654871wmb.155.1619081787876; Thu, 22 Apr 2021 01:56:27 -0700 (PDT) Received: from [192.168.3.132] (p4ff23eb0.dip0.t-ipconnect.de. [79.242.62.176]) by smtp.gmail.com with ESMTPSA id o17sm2600591wrg.80.2021.04.22.01.56.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 22 Apr 2021 01:56:27 -0700 (PDT) To: Michal Hocko , Florian Fainelli Cc: Vlastimil Babka , Mel Gorman , Minchan Kim , Johannes Weiner , l.stach@pengutronix.de, LKML , linux-mmc@vger.kernel.org, Jaewon Kim , Michal Nazarewicz , Joonsoo Kim , Oscar Salvador , "linux-mm@kvack.org" References: From: David Hildenbrand Organization: Red Hat Subject: Re: alloc_contig_range() with MIGRATE_MOVABLE performance regression since 4.9 Message-ID: <8919b724-ce5b-a80f-bbea-98b99af97357@redhat.com> Date: Thu, 22 Apr 2021 10:56:26 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 01DC6C0007D6 X-Stat-Signature: 6qrfi4gyewdhr3xqkrc1qjseks41iegx Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619081786-929958 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 22.04.21 09:49, Michal Hocko wrote: > Cc David and Oscar who are familiar with this code as well. >=20 > On Wed 21-04-21 11:36:01, Florian Fainelli wrote: >> Hi all, >> >> I have been trying for the past few days to identify the source of a >> performance regression that we are seeing with the 5.4 kernel but not >> with the 4.9 kernel on ARM64. Testing something newer like 5.10 is a b= it >> challenging at the moment but will happen eventually. >> >> What we are seeing is a ~3x increase in the time needed for >> alloc_contig_range() to allocate 1GB in blocks of 2MB pages. The syste= m >> is idle at the time and there are no other contenders for memory other >> than the user-space programs already started (DHCP client, shell, etc.= ). Hi, If you can easily reproduce it might be worth to just try bisecting;=20 that could be faster than manually poking around in the code. Also, it would be worth having a look at the state of upstream Linux.=20 Upstream Linux developers tend to not care about minor performance=20 regressions on oldish kernels. There has been work on improving exactly the situation you are=20 describing -- a "fail fast" / "no retry" mode for alloc_contig_range().=20 Maybe it tackles exactly this issue. https://lkml.kernel.org/r/20210121175502.274391-3-minchan@kernel.org Minchan is already on cc. (next time, please cc linux-mm on core-mm questions; maybe you tried,=20 but ended up with linux-mmc :) ) >> >> I have tried playing with the compact_control structure settings but >> have not found anything that would bring us back to the performance of >> 4.9. More often than not, we see test_pages_isolated() returning an >> non-zero error code which would explain the slow down, since we have >> some logic that re-tries the allocation if alloc_contig_range() return= s >> -EBUSY. If I remove the retry logic however, we don't get -EBUSY and w= e >> get the results below: >> >> 4.9 shows this: >> >> [ 457.537634] allocating: size: 1024MB avg: 59172 (us), max: 137306 >> (us), min: 44859 (us), total: 591723 (us), pages: 512, per-page: 115 (= us) >> [ 457.550222] freeing: size: 1024MB avg: 67397 (us), max: 151408 (us)= , >> min: 52630 (us), total: 673974 (us), pages: 512, per-page: 131 (us) >> >> 5.4 show this: >> >> [ 222.388758] allocating: size: 1024MB avg: 156739 (us), max: 157254 >> (us), min: 155915 (us), total: 1567394 (us), pages: 512, per-page: 306= (us) >> [ 222.401601] freeing: size: 1024MB avg: 209899 (us), max: 210085 (us= ), >> min: 209749 (us), total: 2098999 (us), pages: 512, per-page: 409 (us) >> >> This regression is not seen when MIGRATE_CMA is specified instead of >> MIGRATE_MOVABLE. >> >> A few characteristics that you should probably be aware of: >> >> - There is 4GB of memory populated with the memory being mapped into t= he >> CPU's address starting at space at 0x4000_0000 (1GB), PAGE_SIZE is 4KB >> >> - there is a ZONE_DMA32 that starts at 0x4000_0000 and ends at >> 0xE480_0000, from there on we have a ZONE_MOVABLE which is comprised o= f >> 0xE480_0000 - 0xfdc00000 and another range spanning 0x1_0000_0000 - >> 0x1_4000_0000 >> >> Attached is the kernel configuration. >> >> Thanks! >> --=20 >> Florian >=20 >=20 >=20 --=20 Thanks, David / dhildenb