From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FBF2C433FE for ; Tue, 30 Nov 2021 12:38:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BDCB6B0073; Tue, 30 Nov 2021 07:38:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76D206B0074; Tue, 30 Nov 2021 07:38:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 634FB6B0075; Tue, 30 Nov 2021 07:38:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 5430A6B0073 for ; Tue, 30 Nov 2021 07:38:00 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1DD5F18215F90 for ; Tue, 30 Nov 2021 12:37:50 +0000 (UTC) X-FDA: 78865548300.21.B6BE9D2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 7ADC9F000095 for ; Tue, 30 Nov 2021 12:37:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1638275868; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RZr+a5tL4DADMraKmRynBX7SEB4EyUgSbj6iPNQh77U=; b=NmDUdTg21RalCpzvy5Ud5/XzmbCM5iWcyntc4TaTrbmMw2tCxorSW6LuG7ULhm4oYoG77b juL3V7SAgh3oyNruDY3PojEQayh/q8YOhwUl7dnCvtQiLohQHNlP0Q31yr6hQox5jRjkGn WV/ABc3+wc+BqBiJYZOhkBdP8VvHkHc= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-384-I2iMO5JhOKK5UJXgd-O1Ag-1; Tue, 30 Nov 2021 07:37:45 -0500 X-MC-Unique: I2iMO5JhOKK5UJXgd-O1Ag-1 Received: by mail-wr1-f69.google.com with SMTP id o4-20020adfca04000000b0018f07ad171aso3554006wrh.20 for ; Tue, 30 Nov 2021 04:37:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent :content-language:to:references:from:organization:subject :in-reply-to:content-transfer-encoding; bh=RZr+a5tL4DADMraKmRynBX7SEB4EyUgSbj6iPNQh77U=; b=v9C2Km7fKBpXCbwt8FdM+1t98iciwhuRG0hHDlawACBubnmX1NmOcsyJut0uv1BNKX K/fr66f0n1gZqYiwtbq+vtBANEQVXs21OTrqWUBrYssMlCpisdW0zLdrhq0isCXGhWql JbUtiHT0fyYdWSIwXeIMU/s5VZLTy3kTzL033okbgSXVi51IHA/qvZKr4yhbXdQfhloB g1EXSa6+q+9l2MX/8VSxooRXYQ0w5HslkwexGPfKp8kQICUxjMKbY1uZQPichF+Kli4u MO86+nn/yr9LBWKJCaHKuvSiMecd5+SyAhTXoH6n+btNZgRiNiHak/9svWpb7prCgzwm PzDg== X-Gm-Message-State: AOAM532/fg3KeZl0W76M+4uCkKOy3aKLFyBH1skSkL//W+fIwoUDxgUb C294YnYjANKa+9WVoZJI8ydCvS4vW4vDgNxYSjRUb2bCjUvmSwoDmUHv2tUlPHJ5N4vdxEXL2bj F1nY3zzgbddc= X-Received: by 2002:adf:9cc2:: with SMTP id h2mr40862472wre.464.1638275864384; Tue, 30 Nov 2021 04:37:44 -0800 (PST) X-Google-Smtp-Source: ABdhPJyy6eeuTBeD0ytMVUE9M8KnlLzuCplmKmOaRzW3VFWKt/xCaYs/qnSy5O6dRe1MHyIUsAfulA== X-Received: by 2002:adf:9cc2:: with SMTP id h2mr40862454wre.464.1638275864192; Tue, 30 Nov 2021 04:37:44 -0800 (PST) Received: from [192.168.3.132] (p5b0c68ec.dip0.t-ipconnect.de. [91.12.104.236]) by smtp.gmail.com with ESMTPSA id x1sm16535357wru.40.2021.11.30.04.37.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 30 Nov 2021 04:37:43 -0800 (PST) Message-ID: Date: Tue, 30 Nov 2021 13:37:43 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.2.0 To: "liupeng (DM)" , "akpm@linux-foundation.org" , "vbabka@suse.cz" , "linux-mm@kvack.org" References: <3c6349ddd9a34732a251467b7fa4fe93@huawei.com> From: David Hildenbrand Organization: Red Hat Subject: =?UTF-8?B?UmU6IFtRVUVTVElPTl0g4oCccGxhY2UgcGFnZXMgdG8gdGFpbOKAnSBy?= =?UTF-8?Q?egress_memory_read_bandwidth_about_10=25_under_our_test_cases?= In-Reply-To: <3c6349ddd9a34732a251467b7fa4fe93@huawei.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7ADC9F000095 X-Stat-Signature: widbtywbnegjqynxr4chkxeor64ye99u Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NmDUdTg2; spf=none (imf16.hostedemail.com: domain of david@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1638275863-809968 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 30.11.21 12:51, liupeng (DM) wrote: > Hi David, >=20 > =C2=A0 Hi! >=20 > We met a performance regression(lmbench bw_mmap_rd/bw_mem) on our serve= r > with >=20 > patch 7fef431be9c9(mm/page_alloc: place pages to tail in > __free_pages_core()), >=20 > =C2=A0 >=20 > Case A.=C2=A0 ./bw_mmap_rd -P 1 512m mmap_only out.file >=20 > Case B.=C2=A0 ./bw_mem -P 1 512m frd >=20 > =C2=A0 >=20 > mode0: IMC Interleave =3D auto, Channel Interleave =3D auto, Rank Inter= leave > =3D auto >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0without=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 with=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 regression >=20 > case A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A01053= 5.14=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9645.02=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 8.4% >=20 > case B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A01052= 6.88=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 9797.63=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 6.9% >=20 > also we found different memory interleaving have different results >=20 > =C2=A0 >=20 > mode1: IMC Interleave =3D 1-way, Channel Interleave =3D 2-way, Rank > Interleave =3D2-way >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0without=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 with=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > regression >=20 > case A=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A01054= 3.01=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010643.12=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 -0.9% >=20 > case B=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A01097= 1.33=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010853.99=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1% >=20 > =C2=A0 >=20 > The 7fef431be9c9 changes the different memory layout,=C2=A0 it seems th= at the > issue It doesn't change the physical memory layout. It changes the order in which physical memory will get allocated. >=20 > is related with memory interleave. But the patch does lead to > performance regression. >=20 > =C2=A0 >=20 > Any suggestion about this regression? Does it happen only on specific HW? Commit 7fef431be9c9 discovered a bunch of BUGs already, including physical memory that has different performance characteristics due to firmware bugs, which indicated some physical memory e.g., as uncached to the OS. That commit triggers allocation of physical memory in a different order, which sometimes makes "bad physical memory" getting allocated earlier than later in some benchmarks. Such effects are usually only visible on a freshly started system; once the system was busy for some time and the free page lists are nondeterministic, you might observe similar results. I wrote a kernel module that was able to identify bad physical memory regions in some affected setups: https://github.com/davidhildenbrand/kstream/blob/main/README.md You can try running it after a freshly booted system to eventually identify some "bad" physical memory areas. But it might not be the root cause. Maybe it's just some caching effects? --=20 Thanks, David / dhildenb