From: "黄朝阳 (Zhaoyang Huang)" <zhaoyang.huang@unisoc.com>
To: Breno Leitao <leitao@debian.org>
Cc: "Andrew Morton" <akpm@linux-foundation.org>,
"Matthew Wilcox" <willy@infradead.org>,
"Suren Baghdasaryan" <surenb@google.com>,
"Minchan Kim" <minchan@kernel.org>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"Zhaoyang Huang" <huangzhaoyang@gmail.com>,
"王科 (Ke Wang)" <Ke.Wang@unisoc.com>,
"usamaarif642@gmail.com" <usamaarif642@gmail.com>,
"riel@surriel.com" <riel@surriel.com>,
"hannes@cmpxchg.org" <hannes@cmpxchg.org>,
"nphamcs@gmail.com" <nphamcs@gmail.com>
Subject: reply: [PATCHv5] mm: skip CMA pages when they are not available
Date: Tue, 13 Aug 2024 09:58:32 +0000 [thread overview]
Message-ID: <a74760bd1d81467db2a03b77d3aef7d3@BJMBX01.spreadtrum.com> (raw)
>
>On Wed, May 31, 2023 at 10:51:01AM +0800, zhaoyang.huang wrote:
>> From: Zhaoyang Huang <zhaoyang.huang@unisoc.com>
>>
>> This patch fixes unproductive reclaiming of CMA pages by skipping them
>> when they are not available for current context. It is arise from
>> bellowing OOM issue, which caused by large proportion of MIGRATE_CMA
>pages among free pages.
>
>Hello,
>
>I've been looking into a problem with high memory pressure causing OOMs in
>some of our workloads, and it seems that this change may have introduced lock
>contention when there is high memory pressure.
>
>I've collected some metrics for my specific workload that suggest this change
>has increased the lruvec->lru_lock waittime-max by 500x and the
>waittime-avg by 20x.
>
>Experiment
>==========
>
>The experiment involved 100 hosts, each with 64GB of memory and a single
>Xeon 8321HC CPU. The experiment ran for over 80 hours.
>
>Half of the hosts (50) were configured with the patch reverted and lock stat
>enabled, while the other half was run against the upstream version.
>All machines had hugetlb_cma=6G set as a command-line argument.
>
>In this context, "upstream" refers to kernel release 6.9 with some minor
>changes that should not impact the results.
>
>Workload
>========
>
>The workload is a Java based application that fully utilized the memory, in fact,
>the JVM runs with `-Xms50735m -Xmx50735m` arguments.
>
>Results:
>=======
>
>A few values from lockstat:
>
> waittime-max waittime-total waittime-avg
>holdtime-max
>6.9: 242889 15618873933 715
>17485
>6.9-with-revert: 487 688563299 34
>464
>
>The full data could be seen at:
>https://docs.google.com/spreadsheets/d/1Dl-8ImlE4OZrfKjbyWAIWWuQtgD3f
>wEEl9INaZQZ4e8/edit?usp=sharing
>
>Possible causes:
>================
>
>I've been discussing this with colleagues and we're speculating that the high
>contention might be linked to the fact that CMA regions are now being skipped.
>This could potentially extend the duration of the
>isolate_lru_folios() 'while' loop, resulting in increased pressure on the lock.
>
>However, I want to emphasize that I'm not an expert in this area and I am
>simply sharing the data I collected.
Could you please try below patch which could be helpful
https://lore.kernel.org/linux-mm/CAOUHufa7OBtNHKMhfu8wOOE4f0w3b0_2KzzV7-hrc9rVL8e=iw@mail.gmail.com/
next reply other threads:[~2024-08-13 10:00 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-08-13 9:58 黄朝阳 (Zhaoyang Huang) [this message]
2024-08-16 17:20 ` Breno Leitao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a74760bd1d81467db2a03b77d3aef7d3@BJMBX01.spreadtrum.com \
--to=zhaoyang.huang@unisoc.com \
--cc=Ke.Wang@unisoc.com \
--cc=akpm@linux-foundation.org \
--cc=hannes@cmpxchg.org \
--cc=huangzhaoyang@gmail.com \
--cc=leitao@debian.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=minchan@kernel.org \
--cc=nphamcs@gmail.com \
--cc=riel@surriel.com \
--cc=surenb@google.com \
--cc=usamaarif642@gmail.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox