From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 401E1C36005 for ; Tue, 25 Mar 2025 21:56:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49461280047; Tue, 25 Mar 2025 17:56:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4437E280005; Tue, 25 Mar 2025 17:56:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 30B66280047; Tue, 25 Mar 2025 17:56:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 128E3280005 for ; Tue, 25 Mar 2025 17:56:12 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 39C8C14103F for ; Tue, 25 Mar 2025 21:56:13 +0000 (UTC) X-FDA: 83261432226.01.3CA2FD1 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf27.hostedemail.com (Postfix) with ESMTP id 3737440002 for ; Tue, 25 Mar 2025 21:56:10 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Inb4HghO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yuanchu@google.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=yuanchu@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742939771; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GawijQeSK3EpSZkeGcHDhjBfafNJ3MVAaKRkLHgLy3g=; b=z/Ufp892d+gZiXiGBQ7qBwSqVMojIbc+tVcBukgEMVafcC+s8Pfg92TDFAijE+fEwI0FF+ l1Jsz7o01uG9qlKSgeqKdQGyiy/4JF7D9be6xYHyY7yG8sCO/ZIxX3lLcpk5bfdW4QW349 26lF4mudvphoJnM4XGZ+40pllLQhQ3Y= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Inb4HghO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yuanchu@google.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=yuanchu@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742939771; a=rsa-sha256; cv=none; b=di4SVnZ67smmJsUN1jLgSvYY5Wx0FrHSWlVPhs01ZPqyJmNIiDaBqrXhseKY7PG3xgKSIo KqYaBSasP3yy4KDpqgnuO0lzblcF1Ygg/IxvF8jmiAnTBWRr2YqRs4fuqg8HRHLB097aKA E4q+YEm9N4xVTe0dPM014YYTP1iWWig= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2240aad70f2so74795ad.0 for ; Tue, 25 Mar 2025 14:56:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1742939770; x=1743544570; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=GawijQeSK3EpSZkeGcHDhjBfafNJ3MVAaKRkLHgLy3g=; b=Inb4HghOYL9tRE37mXhTQMhWLjF+i9BSTJ8gyJ6NV/D0QbBIvOehgrIHMrSDqghfUJ qx1y8F0jmzO/d89MrjIb+MHgSQkUcIso0xdjFLWZvQ8i0YpUvv+42UEG6ge03WD/pliS xa7BPgwjwEe+ArBYBG1Wv5gWvTe/TWaBXmyA/CnsRKr4DspbRN/9aL105/oUYK4kI4jF Pm3bYVNdKz/t9AB4Do1e+vHmiH5CUTCYmplqiA8FZldUfFaYypnE/s2ObVRRBhK8wO6L uLHO6cMnTQ8x0PEAtCehdOnlyNL06XbVcmUherP+DJlH8xFhw7urTName9nP0xkaZZRV n6qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742939770; x=1743544570; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GawijQeSK3EpSZkeGcHDhjBfafNJ3MVAaKRkLHgLy3g=; b=Tv2hwUS1tfKAeTPx1VW8thZDxgMdaz5iNb02MqSwhYiTuj9VJX3w8DK6HOrBtiz+c5 svX7OoE74T46rn+AlcQxh1zVg/3ojwmMeJIPDg7QgU0Z/FzO0z+hXHotRB8XnnTeTgVZ cwUxYigMlC7YWc4Vhe6rFwOOlSPYVoWVc1n2fexNDnLn1vGnok18gUhB390Xp12fOk2a vmiEe1czq4Bs75h4fGZQIlsWvNTXVWnIpz/o5xYAi82NJrtd7oP0jVSj43IM2NGd02Zc ITxP2m1GvG294j2jOVKcSLyXhEdY1ENKdf9zoAbAABJeReWgsoGUhRx6RTcwAxsUkdEi rUzw== X-Forwarded-Encrypted: i=1; AJvYcCXggj9M1Knen7y+JxrZDcY0R7oLKKHi+tfVGNMhVc/N4MZ02zt1lpw4Bu1M7JRsB4W7E9w1U9lW7w==@kvack.org X-Gm-Message-State: AOJu0YwvZFE8ZJsx2oUsoVBG3Xij0mVbVtjdbh8zM3d82Lvw1iCvEC2V RqcSwT9xDOr4KfYen91TWBWYAaA7uNcRCqNusTW1Zndtys/dLYT1LniCgNRX1wcs98QFGdJewvg xmQ4qg9CLMwOD8cIxoaEbFiQWxQrx4twi1f6b X-Gm-Gg: ASbGnctei8+nMz6pywXogoDYr6Y7B6HdLV+I19bIorCMmYh1on/DveGP9po5jyGBo6l xNALOSxTooH6rHTzLBoP9jF6TXAkvaSq990WQiNoYRZg4sTTTnSgpUl3T6WL8ouugHJYm5ZicIK PcCcKuFWw2kRQHoEc/j5WnCBeETYLP2PkQNSj3t48KQ9q/WDGFxhqzj0ra3ICnxk0Q X-Google-Smtp-Source: AGHT+IGp8JXeHefat2sYKtXMfrAIHQLr7hYftj+JashDB/csUXNinqTUEneXZfdmYVqR0BdM1+g7I+KawBYu30LRXEo= X-Received: by 2002:a17:902:f64c:b0:215:7ced:9d67 with SMTP id d9443c01a7336-227f36a5585mr171955ad.24.1742939769655; Tue, 25 Mar 2025 14:56:09 -0700 (PDT) MIME-Version: 1.0 References: <20250324220301.1273038-1-kinseyho@google.com> In-Reply-To: From: Yuanchu Xie Date: Tue, 25 Mar 2025 14:55:52 -0700 X-Gm-Features: AQ5f1JqSK_zS-V30gVeRAqs4kb5kgLUolENQOggIiCEscFmsmuMw79jGcxPABzA Message-ID: Subject: Re: [RFC PATCH v1 0/2] mm: multi-gen LRU scanning for page promotion To: Bharata B Rao Cc: Kinsey Ho , linux-mm@kvack.org, linux-kernel@vger.kernel.org, AneeshKumar.KizhakeVeetil@arm.com, Hasan.Maruf@amd.com, Jonathan.Cameron@huawei.com, Michael.Day@amd.com, akpm@linux-foundation.org, dave.hansen@intel.com, david@redhat.com, feng.tang@intel.com, gourry@gourry.net, hannes@cmpxchg.org, honggyu.kim@sk.com, hughd@google.com, jhubbard@nvidia.com, k.shutemov@gmail.com, kbusch@meta.com, kmanaouil.dev@gmail.com, leesuyeon0506@gmail.com, leillc@google.com, liam.howlett@oracle.com, mgorman@techsingularity.net, mingo@redhat.com, nadav.amit@gmail.com, nphamcs@gmail.com, peterz@infradead.org, raghavendra.kt@amd.com, riel@surriel.com, rientjes@google.com, rppt@kernel.org, shivankg@amd.com, shy828301@gmail.com, sj@kernel.org, vbabka@suse.cz, weixugc@google.com, willy@infradead.org, ying.huang@linux.alibaba.com, ziy@nvidia.com, dave@stgolabs.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 3737440002 X-Stat-Signature: 8aqqwux17yo4fu5439n6wts9i7bou1mt X-Rspam-User: X-HE-Tag: 1742939770-826631 X-HE-Meta: U2FsdGVkX199DHIeYSUNL9HJiax1VZruBdzUOHfDVJKyRK+kHbqSnt9y+XrwHsOeUzanJY/4z3gQB1yFfOoUQ6bwcO06hBq2lKHj0nBiWA3XuothpyxdD0YtEYeGzjasqnBm+fn5ulgJNdzHJz/SZp2mq+uC8fcEaogEYE0Pi/hE2rWaFV65XUySUdSjNeJfXrmWdWt5ePrt4pjuSKG3CDnb7y6qi4t7i3FWgCblkSYPbBmRv2v0sm7J2EPnU1HViMXdLWS3QzoNGsY7a9NeBmX2ldLOUZXqzcUp8Jr3lwgoE6ddlpKYETRgEEn0/BllTcOW7Fd7y9a43DDBSbmfB6Gkx7rOJxJVujP9NAecRfIc+7mC5AsSxULImg+Veji60wxWuOqzevQ12nvB2kTwdWjNLLeKhXPmaMK70uONBIeNw5xS3kx0+orsmbVTLBw00pxHr8ux5VjhHVX8YDvUH2p4lArp5nBXeDe/FkjGNdJRLAZAGA79JiFziE/8Zkr8hSV2z8PEbDFMGjAAwlskEXMJ4bGLzSy10qcYOvqy95TzCdaJFNVm3i8jtj/2BDdm9VVh1L84/ADxOrz9q2cm+LzhHXBvynu4As+clLXnjshk3H8q1lgSTwpc7lcy3RXVegRUbWtOIxpU7iSxoaWiSzM+VG4yIrMkyNvGe2MSVXMXmqbMt/U946O829v+nVq0/jN55NjsHpjkcvkyRPd2YbpgTFEtjIDkPk0gyhJvSzBWByZYjc1lVE+CmzKZjqzdFQM31IiLVCvIACdGjhFU9/oShcFHp1f+d691fAC0sKEKoW1xfy6zUJhVujmzj+A0pq1qhgnHu1dwR1pVSAl5+hTxqYDlQlo05Bd2Iu/c6Lvj2rYheyBQ4DOhGqBxT4yem6wkS73eN+whEtO0vcuNHyuBojEutEjLHpsi8My+9GbeOPxew6whmYNG7wBoPM5O+pQvqY3mp8A7kuSpt5F CcS3Vlpn WJ46uTJhhSG0Qd7S8AjwqROK/oOtfJG26LYVytcrXg9QQ5bjagDbzMjAlQzUw/z+NhN1qZw1M8vf+yrmpE4SVq/IsewUM4W2h6Mjma1H9T1EHo0MW6emvvFiGZj85avBloHJI9c0n0XA6LT4AO2s2k71H2GjF5efeBEgAJdU+mVac73RVG0fg62RXrtXBJLPZHlaV2Q/nA9UJtsG8e9UOpQVXWQmJbwk+wGSfn46a5zupHCn6HSDKxKCeH2obna7XmnCcYtW8KwryE1PKRggly/0J68ipwYVi4R5NRMQvKyFMZnIoG7Tyssri8A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Mar 25, 2025 at 4:56=E2=80=AFAM Bharata B Rao wro= te: > > Thanks for your patchset. I haven't looked at the patches in detail yet, > but gave it a quick try with the micro-benchmark that I have been using. Thanks for running the numbers. Unfortunately neither of us can attend LSF/MM in person, but we're excited about this opportunity for collaboration. > > The below numbers can be compared with the base numbers that I have > posted here > (https://lore.kernel.org/linux-mm/20250325081832.209140-1-bharata@amd.com= /). > Test 2 in the above link is the one I tried with this patchset. > > kernel.numa_balancing =3D 0 > demotion=3Dtrue > cpufreq governor=3Dperformance > > Benchmark run configuration: > Compute-node =3D 1 > Memory-node =3D 2 > Memory-size =3D 206158430208 > Hot-region-size =3D 1073741824 > Nr-hot-regions =3D 192 > Access pattern =3D random > Access granularity =3D 4096 > Delay b/n accesses =3D 0 > Load/store ratio =3D 50l50s > THP used =3D no > Nr accesses =3D 25769803776 > Nr repetitions =3D 512 > > Benchmark completed in 605983205.0 us The benchmark does seem to complete in less time, but I'm not sure why especially given the small number of pages promoted. I think it would also be useful to see the usage breakdown of DRAM/CXL over time. > > numa_hit 63621437 > numa_miss 2721737 > numa_foreign 2721737 > numa_interleave 0 > numa_local 48243292 > numa_other 18099882 > pgpromote_success 0 > pgpromote_candidate 0 > pgdemote_kswapd 15409682 > pgdemote_direct 0 > pgdemote_khugepaged 0 > numa_pte_updates 0 > numa_huge_pte_updates 0 > numa_hint_faults 0 > numa_hint_faults_local 0 > numa_pages_migrated 19596 > pgmigrate_success 15429278 > pgmigrate_fail 256 > > kpromoted_recorded_accesses 27647687 > kpromoted_recorded_hwhints 0 > kpromoted_recorded_pgtscans 27647687 > kpromoted_record_toptier 0 Makes sense, we skip toptier scanning > kpromoted_record_added 17184209 > kpromoted_record_exists 10463478 > kpromoted_mig_right_node 0 > kpromoted_mig_non_lru 404308 > kpromoted_mig_cold_old 6417567 > kpromoted_mig_cold_not_accessed 10342825 > kpromoted_mig_promoted 19509 Compared to 611077 (IBS number) this is a lot lower. > kpromoted_mig_dropped 17164700 > > When I try to get the same benchmark numbers for kpromoted driven by > kmmscand, kpromoted gets overwhelmed with the amount of data that > kmmdscand provides while no such issues with the amount of accesses > reported by this patchset. The scan interval in this series is 4 seconds, while the kmmscand's pause between scanning is 16ms. So there're definitely some gaps here. The MGLRU page table walk also has a bunch of optimizations, and some of them are more focused on reclaim, so we might need to tweak some things there too. Yuanchu