From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 029B1CDB47E for ; Fri, 20 Oct 2023 04:48:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C5BC8D01C0; Fri, 20 Oct 2023 00:48:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 74F1C8D001C; Fri, 20 Oct 2023 00:48:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F2968D01C0; Fri, 20 Oct 2023 00:48:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 4D68A8D001C for ; Fri, 20 Oct 2023 00:48:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 187FFC06AF for ; Fri, 20 Oct 2023 04:48:47 +0000 (UTC) X-FDA: 81364609494.09.0407FFF Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) by imf20.hostedemail.com (Postfix) with ESMTP id 3E0D11C0003 for ; Fri, 20 Oct 2023 04:48:45 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BWDPzpft; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697777325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MuXc7Q2Jk4MNdQDQDTBuYE9+U+hZgbTmv6xP9Jj7WR4=; b=o8C4ZOCEZYlCQCauY+VXZ8lHvmCmLlfB/lbITQu4UqHN8brojXTqoGWMr+I6gWSykbOasb WvRpDDdrkjJc0iJXQCMsnemFSYvWUbD1rAhI9uyaSJDmfxdQ/cbfAivtg3KeuJU4YMAJAm lJ2GJjSzLNpAxY8IHMVknqKu1QOA5zw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BWDPzpft; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697777325; a=rsa-sha256; cv=none; b=TEKk/czg/Dk0Pp7DB9QnGGhdWObulRXra8AmAaPpYmGAKV1kyOEym39jphzPFD25PSY4jH xTt4CImG4aouhtknL30/br53/cBtj8uWJ5c5gB2p6AUu3wiVu2/9Kzt/p6ui+PDwXfbBre VfFe7KCP3MkH+XmIxhFHWr/ipBumLs0= Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-5a7eef0b931so4395907b3.0 for ; Thu, 19 Oct 2023 21:48:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697777324; x=1698382124; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=MuXc7Q2Jk4MNdQDQDTBuYE9+U+hZgbTmv6xP9Jj7WR4=; b=BWDPzpftPopFRwliSAv6ngpSM87ln3d68XQGccAoxKjBC/yU9ykA90peEjtEt9qtW/ EhwTI09JGBkmloFPQpTVDmpdFzyUtiWZ+P+X1pZQPXbMLnF0SB2sqEzTRrvpvUAjFsid PIIs9RIQHyM/49Prvs8X8qRtzZMD9UfAhNijwALTsVeGa+5WQmlAzfd+LHcGJhl7cGcU 6mSumC0AOyVIGCZP15+/jE77y/uhAvj+iAYZnnrhEXsW3i3h/33JcVL80w9uWUFn5f2s IX5Lw1er2Y2/G7Zwzg1NHphsu4QJy5mRaaEF11itb6MLZhBQJtxpYbUTFyZXOmP2G7GP 6P+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697777324; x=1698382124; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MuXc7Q2Jk4MNdQDQDTBuYE9+U+hZgbTmv6xP9Jj7WR4=; b=vNeej9tdJWHXHU1D/QwD34F9uWET6a96bZnwceCSJz2CV6njZznc7KF98hBS9gnS2e p7PkC/7g0XRAbEv/AQUuMGZESbNE6uLyhDeqzIo2Pjw4IxssiulacZQpa+KhTSDW9g5m YtiJrgoM6QWjxtAARA/IoG2uL0xBzAhludqPjb4ZsrTBxUpqSKHWRZieycgxmpWr8WRN a44mFiDMI+pf4Arjzh1c6BJ+peuNUwFK95nhlmithaWiX70IoUnrgHmI9S8WPyukB0RJ g69bLZ/M0vFw+Jh+V3GmiSkZmmJYwWmwH0nhRdTyKAvKkMZCRoLvJ0Bv6Ue7e0W/L3Du j8Rw== X-Gm-Message-State: AOJu0YxC0tIaobmvMWRxYbyLMJWyE9JQYy0o1ppM+1Q6eiUX4lhJjYxk 0MMnl1uznzCaMSPVGXtB+c8CUQ== X-Google-Smtp-Source: AGHT+IGMrFhifeFApRSzmFsxrFUtbTf5MSdzrLHSQeV80jPchA1ZgiSdLi9vV2A+G2Sl/AxTQrWwAw== X-Received: by 2002:a05:690c:10c:b0:5a7:c1f1:24b with SMTP id bd12-20020a05690c010c00b005a7c1f1024bmr839645ywb.22.1697777324220; Thu, 19 Oct 2023 21:48:44 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id n8-20020a817208000000b005925c896bc3sm416914ywc.53.2023.10.19.21.48.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Oct 2023 21:48:43 -0700 (PDT) Date: Thu, 19 Oct 2023 21:48:32 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Baolin Wang cc: akpm@linux-foundation.org, mgorman@techsingularity.net, hughd@google.com, vbabka@suse.cz, ying.huang@intel.com, ziy@nvidia.com, fengwei.yin@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v2] mm: migrate: record the mlocked page status to remove unnecessary lru drain In-Reply-To: Message-ID: <163ce2c0-9c8a-3db3-26a7-4d115fb95802@google.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 3E0D11C0003 X-Stat-Signature: nneq5x89hopm8sg8m4i8p4s33x16rend X-Rspam-User: X-HE-Tag: 1697777325-307024 X-HE-Meta: U2FsdGVkX1/0YBwHsOQlOPNbsTbpItkI2FXyp1Pc5aVK6bYLCi4ICye36saqtaMrJI9euWtI6DX3pWXOkopC/sukSUhm1LHftu5HdqMbVEOLJQa/FtHzRa1+4K1VrHlysYIbAgUnUqIx+dHMUfV4ZBrnE8NqUkftecJQ0afs5MyonaM8NHgohtOzIOSuF2X76E972IqDqxi4mn3xDyasyK5Rdvj5kqT6Lw8vgBLFcH38TYVuy5I/PXboW1ZacacXk/3NublKoacjyVirSlT6lJ+qOhPRWDAiy4jm3YoE+TG2glgMCDc0dVWLGlad5BqUhSR1yEOo0PWn1pO8Mzud16XDf1WYc2HEOBiNAhMjYUv81riLPvXiqsIPqJLiIazxjKwVAsN53UaU+BEbZDPS80mOejzZLbPMuiDvTKq123DTlngpUNIhb+swwbvBmHVCr7PHQgVTqstbwq2dOkfaqKEyg/8FKVE13F6saHTEGnrZAGx0nwNHEtL7ys/gVqTF37lD28IVc+AGFeSe+5dnWEEvrap8Q+UI7k1SAZblOjyUvgefCY+FHdz9vwCejSnN1DYx4BZ5uZHEUBtg818b+QbHOEmqmxWCwAteRZ6CCutmU9UF97Q8K8x2eHif1e/SYupbiL+z8b5HcyyLNE/HVvP2YT8waf6lmkmdgcI9tCxeE8u0PK0pfyhn5UzOlJWqt9Fep3E7hT6MKV7kuCygunDriqzUo7w2pd9/fF2sv1hbYv6OPDBx8E4gouhzyCwgbXRtx9TbOwezhRUEmA7pUDMAfEegSEZknmDH0En6ypw23/sQZH9dw1G22R5a+6l9akVPWvmklmVwM+KCfdtd76N1FgDJUJmHQf0lx0bF2ZzHGYZ7IfVYIJzjl8zKDcIxlZPykZxQsp4vdjyj+PZD3j2r15Wa+7CwYdC1LO6Wzbh77UpbBXs1Az18T540J85ebhZSKCUQzDoU+BAl8zB LduO9E+S r1gu+t60HAmPXOW0efNnEsGQaRW1NuEsBJn7+4NcLXD8q8HwL9b40UaI1J1GAqib8h9bXINkV5bFyNUbHM6JO2DdSydGBZ9qVhARRq5ta/kz+jBQv7EblNqLB/Gs/3AhfnTUAQPMLHVF8yNxL8k7nZZOM343RoGrXobQYqXvyWbMB+Xz/1xn3jXWFL/OBBdWwTMAAe1XITzIqsqD4XVRxARGOVHy8oGUMpWn7GSz6nnEBC7DMnvglq9KnHcSe84rFQFkubesSaBq3MQD/xVklM9nfE24hPbal1KpztfZ6UUoSHGPcYMna0DUtwTyYP9RwBgIic0bZGWFS63OPBo8B1pmj2JVVgUY6ShDwTFfXNbAzKM1fictOTlJLXWtiNRZtAhjl X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 20 Oct 2023, Baolin Wang wrote: > When doing compaction, I found the lru_add_drain() is an obvious hotspot > when migrating pages. The distribution of this hotspot is as follows: > - 18.75% compact_zone > - 17.39% migrate_pages > - 13.79% migrate_pages_batch > - 11.66% migrate_folio_move > - 7.02% lru_add_drain > + 7.02% lru_add_drain_cpu > + 3.00% move_to_new_folio > 1.23% rmap_walk > + 1.92% migrate_folio_unmap > + 3.20% migrate_pages_sync > + 0.90% isolate_migratepages > > The lru_add_drain() was added by commit c3096e6782b7 ("mm/migrate: > __unmap_and_move() push good newpage to LRU") to drain the newpage to LRU > immediately, to help to build up the correct newpage->mlock_count in > remove_migration_ptes() for mlocked pages. However, if there are no mlocked > pages are migrating, then we can avoid this lru drain operation, especailly > for the heavy concurrent scenarios. > > So we can record the source pages' mlocked status in migrate_folio_unmap(), > and only drain the lru list when the mlocked status is set in migrate_folio_move(). > In addition, the page was already isolated from lru when migrating, so checking > the mlocked status is stable by folio_test_mlocked() in migrate_folio_unmap(). > > After this patch, I can see the hotpot of the lru_add_drain() is gone: > - 9.41% migrate_pages_batch > - 6.15% migrate_folio_move > - 3.64% move_to_new_folio > + 1.80% migrate_folio_extra > + 1.70% buffer_migrate_folio > + 1.41% rmap_walk > + 0.62% folio_add_lru > + 3.07% migrate_folio_unmap > > Meanwhile, the compaction latency shows some improvements when running > thpscale: > base patched > Amean fault-both-1 1131.22 ( 0.00%) 1112.55 * 1.65%* > Amean fault-both-3 2489.75 ( 0.00%) 2324.15 * 6.65%* > Amean fault-both-5 3257.37 ( 0.00%) 3183.18 * 2.28%* > Amean fault-both-7 4257.99 ( 0.00%) 4079.04 * 4.20%* > Amean fault-both-12 6614.02 ( 0.00%) 6075.60 * 8.14%* > Amean fault-both-18 10607.78 ( 0.00%) 8978.86 * 15.36%* > Amean fault-both-24 14911.65 ( 0.00%) 11619.55 * 22.08%* > Amean fault-both-30 14954.67 ( 0.00%) 14925.66 * 0.19%* > Amean fault-both-32 16654.87 ( 0.00%) 15580.31 * 6.45%* > Seems a sensible change with good results (I'll conceal how little of the stats I understand, I expect everyone else understands them: in my naivety, I'm mainly curious why rmap_walk's 1.23% didn't get a + on it). > Signed-off-by: Baolin Wang > --- > Chages from v1: > - Use separate flags in __migrate_folio_record() to avoid to pack flags > in each call site per Ying. > --- > mm/migrate.c | 47 +++++++++++++++++++++++++++++++++++------------ > 1 file changed, 35 insertions(+), 12 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 125194f5af0f..fac96139dbba 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1027,22 +1027,39 @@ union migration_ptr { > struct anon_vma *anon_vma; > struct address_space *mapping; > }; > + > +enum { > + PAGE_WAS_MAPPED = 1 << 0, > + PAGE_WAS_MLOCKED = 1 << 1, > +}; > + I was whispering to myself "I bet someone will suggest BIT()"; and indeed that someone has turned out to be Huang, Ying. > static void __migrate_folio_record(struct folio *dst, > - unsigned long page_was_mapped, > + unsigned int page_was_mapped, > + unsigned int page_was_mlocked, > struct anon_vma *anon_vma) > { > union migration_ptr ptr = { .anon_vma = anon_vma }; > + unsigned long page_flags = 0; Huang, Ying preferred a different name, me too: old_page_state? > + > + if (page_was_mapped) > + page_flags |= PAGE_WAS_MAPPED; > + if (page_was_mlocked) > + page_flags |= PAGE_WAS_MLOCKED; What's annoying me about the patch is all this mix of page_was_mapped and page_was_mlocked variables, then the old_page_state bits. Can't it be done with PAGE_WAS_ bits in old_page_state throughout, without any page_was_mapped and page_was_mlocked variables? Hugh