From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A62A3CDB474 for ; Sat, 21 Oct 2023 03:25:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 076288D0009; Fri, 20 Oct 2023 23:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 026CD8D0008; Fri, 20 Oct 2023 23:25:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E583B8D0009; Fri, 20 Oct 2023 23:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D625A8D0008 for ; Fri, 20 Oct 2023 23:25:51 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9146C1A05CF for ; Sat, 21 Oct 2023 03:25:51 +0000 (UTC) X-FDA: 81368029302.21.E9C76F3 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) by imf12.hostedemail.com (Postfix) with ESMTP id 22D8F40009 for ; Sat, 21 Oct 2023 03:25:47 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697858749; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QXbC6FrxDbD717aFdr5U9sjjSqFg7vrogQN6Cbtw+BM=; b=lopY83vfDEFsDPJ1kb8oijMJ/J7WGQO+4C9xJiAbQAp9g/Wb3KmI710UuWnxnhOemI3xQJ /U/wFi12P9R1+MAx4PRItdgcLNoAjLBVkNqD/A3CuesXaUE6I9TQhfVPiho9cMMfZssu4v fmd/4/cdJZYLPKx9knleqqfxiPZwFPc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697858749; a=rsa-sha256; cv=none; b=kXiIBNOmvzGLr7WXDymDucW7e4sRgdJZxj7tFAJ1leHomWbMwv+L9WGeN/0swnWQt+Lg4V CH48gBjqGXx+YbgMxfDdyGOCmPg0nEdpiQeSy/dQ1ysWE0XmejhcWV20zN3qRM+WAIfhm+ 8NsH2zTembmWCWrX5OULve22e6Ptzzg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.119 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0VuYY98S_1697858742; Received: from 192.168.43.40(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VuYY98S_1697858742) by smtp.aliyun-inc.com; Sat, 21 Oct 2023 11:25:43 +0800 Message-ID: Date: Sat, 21 Oct 2023 11:25:57 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.15.1 Subject: Re: [PATCH v2] mm: migrate: record the mlocked page status to remove unnecessary lru drain To: "Huang, Ying" Cc: akpm@linux-foundation.org, mgorman@techsingularity.net, hughd@google.com, vbabka@suse.cz, ziy@nvidia.com, fengwei.yin@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <87jzriez8h.fsf@yhuang6-desk2.ccr.corp.intel.com> From: Baolin Wang In-Reply-To: <87jzriez8h.fsf@yhuang6-desk2.ccr.corp.intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 22D8F40009 X-Rspam-User: X-Stat-Signature: e1zc5sgnanpcncg6yo9j3k3y8rks6urq X-Rspamd-Server: rspam03 X-HE-Tag: 1697858747-517428 X-HE-Meta: U2FsdGVkX19QgJT2Z7kkCrYYXJSVXzLS57FgvhpbFWtohXK6YARrk+CQKgV4w0VGtWjXt4YXLvf5rRleH/P0W4DNEY3UfeQU86umdVb5w59sgY/ZdYwmq/ANj06xAzTVXGF9MnIPbal9xr5mpjU7EdRuuW17C24wkr85hDHEkPYrReW0RS47YdTQ+mS6h3B70mu429dV+OHAHbTXdgdYP3UnJfnKp9M5sOBV7UQLIMzZTmr95JK65QcofRU/cjxvJ4bX0VkdXrdkUpDDbHz7QudSR73mg567VE19aMjQj4Uoaqc+lQtRDd4l1lYkx9kEkFNoA7lEfHwg0Ww0plzvP0C/GLBh+mhFJ8kJUwMJHmnvhmB9uWm3XmS3WLY/olVdvwpWvyIysmwc9XfmHRwozWzOGLTxRBYq/1Zp1aEKBQpx842AcJZou/2YB1+forW2+1j2WVWKNQ8otl1C454U29zXvLlmC7GJ+fnlMs9t40YuTw0LyQ2z/yUU17hMGdg/z35NtB8foFTglwXWp73g0Fo+heiC4mwvLkoJ5BUkcOY55m8S4H3AY7cBHXYFGtlxFEmB6oRtTIJDK8d3LVtQewexI6tHZy8uf0WQMJ4DiD2af02gzWH4Y8CCXuUh6s17dZJE6HO77YxDqvGNu/KXMfr/ltXCDYG3tElyAoJQbov3dBGcLMBtPidYBf0in7xxi4Dhicye3g+OqrCOfnyXZeyPAIeEY2QmHo91Q3svX0Kb4QtrrtLBdbTdWjnxWmd/tAq5E8UuYHByd4xO9Q5t0osIm6m7kuEq8pFQajR2xYZ6RVHU4Y5xMyKuIqfKgvwJqEA0pZ1zDSuYEZ+y0w54y4ym+U1H9FRUxWYJ7sAUeAFZWEHobQyYs0g+6Qh99Uf0vfhZJI8HDXfNAx11uvv/5HAvVLa3QRNVvq2dgLg2KvW5U0ZLQPaDI7l5VmjQPXiUDWRZQ0S2o5oavkstjEr MTH/VwgT O/Z9ElfstZTn3tznasARBH/hgZnahcP+RmsYw+ww6zpmBFc3U5y4fXuJrHPiw5QabYF4LkVTGJNVQ7w0hZ40BZNAVfg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 10/20/2023 11:42 AM, Huang, Ying wrote: > Baolin Wang writes: > >> When doing compaction, I found the lru_add_drain() is an obvious hotspot >> when migrating pages. The distribution of this hotspot is as follows: >> - 18.75% compact_zone >> - 17.39% migrate_pages >> - 13.79% migrate_pages_batch >> - 11.66% migrate_folio_move >> - 7.02% lru_add_drain >> + 7.02% lru_add_drain_cpu >> + 3.00% move_to_new_folio >> 1.23% rmap_walk >> + 1.92% migrate_folio_unmap >> + 3.20% migrate_pages_sync >> + 0.90% isolate_migratepages >> >> The lru_add_drain() was added by commit c3096e6782b7 ("mm/migrate: >> __unmap_and_move() push good newpage to LRU") to drain the newpage to LRU >> immediately, to help to build up the correct newpage->mlock_count in >> remove_migration_ptes() for mlocked pages. However, if there are no mlocked >> pages are migrating, then we can avoid this lru drain operation, especailly >> for the heavy concurrent scenarios. >> >> So we can record the source pages' mlocked status in migrate_folio_unmap(), >> and only drain the lru list when the mlocked status is set in migrate_folio_move(). >> In addition, the page was already isolated from lru when migrating, so checking >> the mlocked status is stable by folio_test_mlocked() in migrate_folio_unmap(). >> >> After this patch, I can see the hotpot of the lru_add_drain() is gone: >> - 9.41% migrate_pages_batch >> - 6.15% migrate_folio_move >> - 3.64% move_to_new_folio >> + 1.80% migrate_folio_extra >> + 1.70% buffer_migrate_folio >> + 1.41% rmap_walk >> + 0.62% folio_add_lru >> + 3.07% migrate_folio_unmap >> >> Meanwhile, the compaction latency shows some improvements when running >> thpscale: >> base patched >> Amean fault-both-1 1131.22 ( 0.00%) 1112.55 * 1.65%* >> Amean fault-both-3 2489.75 ( 0.00%) 2324.15 * 6.65%* >> Amean fault-both-5 3257.37 ( 0.00%) 3183.18 * 2.28%* >> Amean fault-both-7 4257.99 ( 0.00%) 4079.04 * 4.20%* >> Amean fault-both-12 6614.02 ( 0.00%) 6075.60 * 8.14%* >> Amean fault-both-18 10607.78 ( 0.00%) 8978.86 * 15.36%* >> Amean fault-both-24 14911.65 ( 0.00%) 11619.55 * 22.08%* >> Amean fault-both-30 14954.67 ( 0.00%) 14925.66 * 0.19%* >> Amean fault-both-32 16654.87 ( 0.00%) 15580.31 * 6.45%* >> >> Signed-off-by: Baolin Wang >> --- >> Chages from v1: >> - Use separate flags in __migrate_folio_record() to avoid to pack flags >> in each call site per Ying. >> --- >> mm/migrate.c | 47 +++++++++++++++++++++++++++++++++++------------ >> 1 file changed, 35 insertions(+), 12 deletions(-) >> >> diff --git a/mm/migrate.c b/mm/migrate.c >> index 125194f5af0f..fac96139dbba 100644 >> --- a/mm/migrate.c >> +++ b/mm/migrate.c >> @@ -1027,22 +1027,39 @@ union migration_ptr { >> struct anon_vma *anon_vma; >> struct address_space *mapping; >> }; >> + >> +enum { >> + PAGE_WAS_MAPPED = 1 << 0, > > PAGE_WAS_MAPPED = BIT(0) ? Sure, will do. > >> + PAGE_WAS_MLOCKED = 1 << 1, >> +}; >> + >> static void __migrate_folio_record(struct folio *dst, >> - unsigned long page_was_mapped, >> + unsigned int page_was_mapped, >> + unsigned int page_was_mlocked, >> struct anon_vma *anon_vma) >> { >> union migration_ptr ptr = { .anon_vma = anon_vma }; >> + unsigned long page_flags = 0; > > page_flags wasn't a good name, it can be confused with page->flags. Agree. > May be something like "page_attrs"? OK, I prefer to the 'old_page_state' suggested by Hugh :) >> + >> + if (page_was_mapped) >> + page_flags |= PAGE_WAS_MAPPED; >> + if (page_was_mlocked) >> + page_flags |= PAGE_WAS_MLOCKED; >> dst->mapping = ptr.mapping; >> - dst->private = (void *)page_was_mapped; >> + dst->private = (void *)page_flags; >> } >> >> static void __migrate_folio_extract(struct folio *dst, >> int *page_was_mappedp, >> + int *page_was_mlocked, > > Better to use the same naming convention. Either both have "p" suffix, > or both not. OK. > > Otherwise looks good to me. Thanks for reviewing.