From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.0 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE, SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21F5CC433B4 for ; Thu, 22 Apr 2021 17:50:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 739F4613F5 for ; Thu, 22 Apr 2021 17:50:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 739F4613F5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A24BC6B007D; Thu, 22 Apr 2021 13:50:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D36D6B0080; Thu, 22 Apr 2021 13:50:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D7DB6B0081; Thu, 22 Apr 2021 13:50:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 452916B007D for ; Thu, 22 Apr 2021 13:50:21 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 02BEC5DDA for ; Thu, 22 Apr 2021 17:50:21 +0000 (UTC) X-FDA: 78060742242.26.88673EC Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf15.hostedemail.com (Postfix) with ESMTP id 0FA1EA0003BF for ; Thu, 22 Apr 2021 17:50:17 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id u7so22102168plr.6 for ; Thu, 22 Apr 2021 10:50:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=mHPTQE74w3DtNKeE2TE1/pjpjT0CSeqdv53WlGy0hDQ=; b=mILJom+SKX4TA5KYv0mFQTTG17Kln7uTFsbB5oyHTJPzxDm9CMv0VIszYPsjLdJLMQ MFbE9/OiZMWGR0SAUou1hmtDXXcaJ01WAomfESUsJLytRd7tGFZcK1jq8zv4ns41Y27h ZY8Hz+Swd/ZYnyq38fLJqHi+2z0r6O0MLdMtzHGWimoFZL4DwlGJWtzV/JtENH23ihRI dAIytvEzffbKeOIr8oDXhqKqTBzPpeKjBxUsl4/4s/ByhkPOw2GRTMMSiNNRH0yCcVml 7Ridpgnno2gmdrCDUyqJy1Q1QXzEu0vieaeZ/Oqj04synlY0xsYhUYOSel1UIhmM3gwX 1RBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=mHPTQE74w3DtNKeE2TE1/pjpjT0CSeqdv53WlGy0hDQ=; b=iSkzhhj+bjSOo2LkZOt+Fko9jqHVivpd/tYx94HlNS1XUX7Ep3fO8LLFoF5GdE87w2 7CzQ10K8mcqp5pzGFc2Vh9QjgVh8Nokx4eZwH3pVKBu9k5Lwa30jtX54o00mSUAvjJMX 4/1EgAHGLUERxSXxudguo5c6BjWhkUj8NjHzttS3045Yad5IwTN64DBBhKe17qNG36FZ qp0cvmcdAVYF/8QuAAIAX0w24Ghnk8TJoGIJmId2wNn8mHK8X9PXoEW+t9iOTG6/GpDa 2a+b2W9Mpr/ZuRCFV2y5AfA+Yc9x/3nMQbkV8cG07zgmdg8fsbBh/WRJdCop4y9UXlL3 CBLw== X-Gm-Message-State: AOAM532agfr3SiAnuWYwY7TgW4pme0Ly6nnhZnXjVBFMFFypoksIhN78 Q9y1V4lwq49dxVF8egDUP4LAj0MPOU4= X-Google-Smtp-Source: ABdhPJxtYCEOdLx6RKAsSSL3wbA+A9IrX2IKUBGe4YZLDiAdgHYYFHj3MB1IuDMyASrkQonPSHhBeA== X-Received: by 2002:a17:90a:a60b:: with SMTP id c11mr5168848pjq.125.1619113819245; Thu, 22 Apr 2021 10:50:19 -0700 (PDT) Received: from [10.230.29.202] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id t21sm2577207pfg.211.2021.04.22.10.50.17 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 22 Apr 2021 10:50:18 -0700 (PDT) Subject: Re: alloc_contig_range() with MIGRATE_MOVABLE performance regression since 4.9 To: David Hildenbrand , Michal Hocko Cc: Vlastimil Babka , Mel Gorman , Minchan Kim , Johannes Weiner , l.stach@pengutronix.de, LKML , Jaewon Kim , Michal Nazarewicz , Joonsoo Kim , Oscar Salvador , "linux-mm@kvack.org" References: <8919b724-ce5b-a80f-bbea-98b99af97357@redhat.com> From: Florian Fainelli Message-ID: <58726a6b-5468-a6b4-7c26-371ef5d71ee2@gmail.com> Date: Thu, 22 Apr 2021 10:50:16 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <8919b724-ce5b-a80f-bbea-98b99af97357@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 0FA1EA0003BF X-Stat-Signature: 6a5jkrgjwsx5gotyy99d4epdu8zgaxjk Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=mail-pl1-f175.google.com; client-ip=209.85.214.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619113817-654433 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/22/2021 1:56 AM, David Hildenbrand wrote: > On 22.04.21 09:49, Michal Hocko wrote: >> Cc David and Oscar who are familiar with this code as well. >> >> On Wed 21-04-21 11:36:01, Florian Fainelli wrote: >>> Hi all, >>> >>> I have been trying for the past few days to identify the source of a >>> performance regression that we are seeing with the 5.4 kernel but not >>> with the 4.9 kernel on ARM64. Testing something newer like 5.10 is a = bit >>> challenging at the moment but will happen eventually. >>> >>> What we are seeing is a ~3x increase in the time needed for >>> alloc_contig_range() to allocate 1GB in blocks of 2MB pages. The syst= em >>> is idle at the time and there are no other contenders for memory othe= r >>> than the user-space programs already started (DHCP client, shell, etc= .). >=20 > Hi, >=20 > If you can easily reproduce it might be worth to just try bisecting; > that could be faster than manually poking around in the code. >=20 > Also, it would be worth having a look at the state of upstream Linux. > Upstream Linux developers tend to not care about minor performance > regressions on oldish kernels. This is a big pain point here and I cannot agree more, but until we bridge that gap, this is not exactly easy to do for me unfortunately and neither is bisection :/ >=20 > There has been work on improving exactly the situation you are > describing -- a "fail fast" / "no retry" mode for alloc_contig_range(). > Maybe it tackles exactly this issue. >=20 > https://lkml.kernel.org/r/20210121175502.274391-3-minchan@kernel.org >=20 > Minchan is already on cc. This patch does not appear to be helping, in fact, I had locally applied this patch from way back when: https://lkml.org/lkml/2014/5/28/113 which would effectively do this unconditionally. Let me see if I can showcase this problem a x86 virtual machine operating in similar conditions to ours. >=20 > (next time, please cc linux-mm on core-mm questions; maybe you tried, > but ended up with linux-mmc :) ) Yes that was the intent, thanks for correcting that. >=20 >>> >>> I have tried playing with the compact_control structure settings but >>> have not found anything that would bring us back to the performance o= f >>> 4.9. More often than not, we see test_pages_isolated() returning an >>> non-zero error code which would explain the slow down, since we have >>> some logic that re-tries the allocation if alloc_contig_range() retur= ns >>> -EBUSY. If I remove the retry logic however, we don't get -EBUSY and = we >>> get the results below: >>> >>> 4.9 shows this: >>> >>> [=C2=A0 457.537634] allocating: size: 1024MB avg: 59172 (us), max: 13= 7306 >>> (us), min: 44859 (us), total: 591723 (us), pages: 512, per-page: 115 >>> (us) >>> [=C2=A0 457.550222] freeing: size: 1024MB avg: 67397 (us), max: 15140= 8 (us), >>> min: 52630 (us), total: 673974 (us), pages: 512, per-page: 131 (us) >>> >>> 5.4 show this: >>> >>> [=C2=A0 222.388758] allocating: size: 1024MB avg: 156739 (us), max: 1= 57254 >>> (us), min: 155915 (us), total: 1567394 (us), pages: 512, per-page: >>> 306 (us) >>> [=C2=A0 222.401601] freeing: size: 1024MB avg: 209899 (us), max: 2100= 85 (us), >>> min: 209749 (us), total: 2098999 (us), pages: 512, per-page: 409 (us) >>> >>> This regression is not seen when MIGRATE_CMA is specified instead of >>> MIGRATE_MOVABLE. >>> >>> A few characteristics that you should probably be aware of: >>> >>> - There is 4GB of memory populated with the memory being mapped into = the >>> CPU's address starting at space at 0x4000_0000 (1GB), PAGE_SIZE is 4K= B >>> >>> - there is a ZONE_DMA32 that starts at 0x4000_0000 and ends at >>> 0xE480_0000, from there on we have a ZONE_MOVABLE which is comprised = of >>> 0xE480_0000 - 0xfdc00000 and another range spanning 0x1_0000_0000 - >>> 0x1_4000_0000 >>> >>> Attached is the kernel configuration. >>> >>> Thanks! >>> --=C2=A0 >>> Florian >> >> >> >=20 >=20 --=20 Florian