From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD59DC25B4F for ; Mon, 13 May 2024 01:49:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 50D1D6B020C; Sun, 12 May 2024 21:49:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 494926B020D; Sun, 12 May 2024 21:49:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 335286B020E; Sun, 12 May 2024 21:49:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0ECC86B020C for ; Sun, 12 May 2024 21:49:43 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BE66E1C12F6 for ; Mon, 13 May 2024 01:49:42 +0000 (UTC) X-FDA: 82111691004.09.5B82FF6 Received: from tiger.tulip.relay.mailchannels.net (tiger.tulip.relay.mailchannels.net [23.83.218.248]) by imf27.hostedemail.com (Postfix) with ESMTP id 4DF9140005 for ; Mon, 13 May 2024 01:49:40 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=stgolabs.net header.s=dreamhost header.b=UGi6UEH0; arc=pass ("mailchannels.net:s=arc-2022:i=1"); dmarc=none; spf=pass (imf27.hostedemail.com: domain of dave@stgolabs.net designates 23.83.218.248 as permitted sender) smtp.mailfrom=dave@stgolabs.net ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715564980; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I9cm9Fno6jR0pegWY+BeTAe9x7lI8fPb7sLCR6GCBe4=; b=UH6lvGNH/6gsp5WeBEXpL1YOpus3zorzhvzNOFjmAQVC05qLuHUQVIW1sjRLIVG0TibdXI mlWp+xVIwj0h82uJugdLiYvQ9bptP8VBZnqtIJbnEaRLCpOy3wFiwW8iV+u+KY4Xeh83OK KUTujfWEk/diRFREy9bXA9CYZSQR5Qw= ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1715564980; a=rsa-sha256; cv=pass; b=XVZZicSzT6wHDFQ7sUVXIiSpC/2AcC/FrtDu8uNbBG6BzE1O7qg8nAbkvT8T7fSOZOe67B K66cm05EnyUIe8nxYd/1N4swNIr21u5oV0S5naZyVdgT3UhihmXFosJFvORIh6OMLndbm6 GOlOJgngpe5Hk5xVdAfwnmLMT388Dv0= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=stgolabs.net header.s=dreamhost header.b=UGi6UEH0; arc=pass ("mailchannels.net:s=arc-2022:i=1"); dmarc=none; spf=pass (imf27.hostedemail.com: domain of dave@stgolabs.net designates 23.83.218.248 as permitted sender) smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id EF207363903; Mon, 13 May 2024 01:49:38 +0000 (UTC) Received: from pdx1-sub0-mail-a312.dreamhost.com (unknown [127.0.0.6]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 4AC003638F2; Mon, 13 May 2024 01:49:38 +0000 (UTC) ARC-Seal: i=1; s=arc-2022; d=mailchannels.net; t=1715564978; a=rsa-sha256; cv=none; b=L2+LE4HUl1Ax4WOfjm/c0UrvTJd1HYmFuVsrh8+4hONY1eFMSI+pC7c9DvMlmN4skCht7P ttsJMd5T7sNFETj2Me59ojBz+GNWX9oeNuCNVdrCntYm/WnkLzUQPIcte+wjyf5WWa1Lka wT4zHPl9GZCofMZAYaTRv0Y1bXjZuGOgzcla1rO/gfbtoWuuGlwr8afJeZKDrcFxgFAz8o 2TQe/fiFQlu6ncijhdwoYn6za90RE++RLxjgvbQMy7uJsbe44jIyLxsURR0djzo1xDVqD8 GLfK0oabYxbtEoQUhrtU7V3zQceroVulyK9+6WqJxJWlzs/VoVjCjNuh382D7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=mailchannels.net; s=arc-2022; t=1715564978; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I9cm9Fno6jR0pegWY+BeTAe9x7lI8fPb7sLCR6GCBe4=; b=C2GkvcdbbolD2DYmhdEqG5jG+3wDm56DrEfFB7T8B3TCVEKhRGfjjGnGPMVziKRAk3EueM jn4gk8vx6n4ZmH1i5WtSRlyn7Sc69x7AsKDEgg7GDLHuYDQW9UG4cAOm2yBKwwy3eqlZ7g JFtM8Q3uGovS8FOmuVaclSNtpymPkHXjhYjattj+UVy9qor/Q0ket8EkdNg3S3lGedCF2f yLIkvFEurDXEiixgBblDaBiBO+EYiv1fOPRFlJ6sK02Y7CJoctLYNZdJo7Pg5xf5sDicyU y9SakghDp2IRuiJhmrBXWz4c0XUF6oZpMN9S12WOYt0xH4JoF/1lp77vCnXw9A== ARC-Authentication-Results: i=1; rspamd-68bbddc7f5-92d4k; auth=pass smtp.auth=dreamhost smtp.mailfrom=dave@stgolabs.net X-Sender-Id: dreamhost|x-authsender|dave@stgolabs.net X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|dave@stgolabs.net X-MailChannels-Auth-Id: dreamhost X-Illegal-Society: 24818bf47bcf4a8c_1715564978814_3143677094 X-MC-Loop-Signature: 1715564978814:4049928017 X-MC-Ingress-Time: 1715564978813 Received: from pdx1-sub0-mail-a312.dreamhost.com (pop.dreamhost.com [64.90.62.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384) by 100.109.254.205 (trex/6.9.2); Mon, 13 May 2024 01:49:38 +0000 Received: from offworld (unknown [172.56.181.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: dave@stgolabs.net) by pdx1-sub0-mail-a312.dreamhost.com (Postfix) with ESMTPSA id 4Vd2V828tGzF4; Sun, 12 May 2024 18:49:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=stgolabs.net; s=dreamhost; t=1715564978; bh=I9cm9Fno6jR0pegWY+BeTAe9x7lI8fPb7sLCR6GCBe4=; h=Date:From:To:Cc:Subject:Content-Type; b=UGi6UEH05smMD12dtErf1cHUaY5XPedd2D758G085wFMf47ujGrQap5rFZDdz9BiC EptRyQ/Ds9dUlA2NtpnkA2X8o+PN7P6uePMBKTQlMo3PQfmvi6c59s4gmHTz6Iutwp us/Fm6/6U2Fme+36kFY/heT4+JchqkR88k4AaFxnxv1JEXZb9GZSrZR++beJg112AB PZcpkpiFtpZTJKbAn2fG7Da6qHTAUdSuKVYQ5C06fRCW4mNZRvoaLAXp+MtH12rAUp kU4lyuIdfRSE3M7Go6QNgWh6IclMU/ZkaZU98p+CacbKi/M4W4AmNBFC0KG7+ohrQN KcsvS1x5m//4g== Date: Sun, 12 May 2024 18:49:25 -0700 From: Davidlohr Bueso To: "Huang, Ying" Cc: David Rientjes , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Michal Hocko , Dan Williams , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , "Aneesh Kumar K.V" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Wei Xu , Johannes Weiner , SeongJae Park , David Hildenbrand , peterz@infradead.org, a.manzanares@samsung.com Subject: Re: [LSF/MM/BPF TOPIC] Locally attached memory tiering Message-ID: Mail-Followup-To: "Huang, Ying" , David Rientjes , lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, Michal Hocko , Dan Williams , John Hubbard , Zi Yan , Bharata B Rao , Dave Jiang , "Aneesh Kumar K.V" , Alistair Popple , Christoph Lameter , Andrew Morton , Linus Torvalds , Dave Hansen , Mel Gorman , Jon Grimm , Gregory Price , Wei Xu , Johannes Weiner , SeongJae Park , David Hildenbrand , peterz@infradead.org, a.manzanares@samsung.com References: <20240508213918.7ndnrjs6pxnklbpi@offworld> <87pltviwv5.fsf@yhuang6-desk2.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <87pltviwv5.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: NeoMutt/20240416 X-Stat-Signature: fg4bifotc3syf47a5pmqxq8mauny83df X-Rspamd-Queue-Id: 4DF9140005 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1715564980-272485 X-HE-Meta: U2FsdGVkX19SWpWYarX529cKsT8NTXYTWMn+dUmkPI9ROh9/OibkkrDF+It4HDc1ReSZJP/LAqj3xctfHTUFVIFMbNb5t92f29oQncLQxMIAr2BI5P2tEwa93alghPsFJ3UvtIH2JB/va34ZRSQUZUkpaF66MJTedcRyUZfZ9EkZubFSIE+G9Wheu4At168uF+l0xzE00STr0y8OHItqu20jtSdIp5GwoAryCJ8WVzZLsjBEbGE1KjRJwA+dAjopIBuIvAN12SiRzXlAPmLK/+NEqs/bj/g36rp1iyKo6hpminnsomtsxJrY1ocdRgVJmT+HZV38nIi917Ho9DB4f7TRK59yVV8hnm5fTpyfrqv1Zvv+X+QNmOZxrEJvd4DRu/guWCHbaZUOZmYCqGZdoWDIIFGdZS1iRn7dfpHcSLZqNe2knkKmQoHH/lpChciUXwa2nhO5j/r5NJ29BqAKpDtI1k3Ad/Z+2mNxAGFSKY10iTodR8Coh3QYY1GsVmieiMVAdUMAuehAA3Hys03WyI3VynJruqtyeMlP+08v8AkE27Egtnr5XkWnU6TvSl+G80Un9pbIwauDp/5htCDxn9ywpc/6OEVG22rEonjKo/nDuhxpf2l0TgvFrrwBzTF4qW2KhGSydmFhGLVP0CBtdo6nW23Uy05Paepuo7tts6pyGJLbGo/CtHPPLiJLtcTQ/sh2sCiFlNHbjs8cDOtbdwW4nkITOeqLO6hWFdpe63X08OoQhithNMWvt+3FL9iqfNREDG/M2eQQTxfTw8hx+c+FzX5bEZmrHFOl1DgJE7HjOe/4Hc+wlBo5UQcN4NDXX3bL4GYh09aiyG/laeYNaZxSBXbT8uFZjsgQMd1ATNU1RfGnvYHpU5iPgjbRe7O4piNl68QMWG1cC3UaAy8xJap+e6kqQ+cmN4zMBFR2LcXqcgbKKPTlWwTPVlVri8e/Da+hruMDKy7sr7LqnGu Hyn9HuVz g80451+q0Uy7fEe1a2GyvnYr4TGOczbpeoce00ZyGWAePd194YRz8TPwyZ1VnL6FpJRzvBuV51TCGysnGgXWzL427aLJV/sGupgd251pOepr0gdslDXn6F0j+Djb4RbmrbGAstq+5GQZzKX+6cbPRBlEiswOhupgFjUobGvMDHB8TORoqYhGuZr2h3pb7VhfH2d4TMPlunMlnFvEDfI2GQrNHxvNSg/19ifd6lXZO0QATQE29nNM9JrsWHTW0qRFtoO8V4xsrMsur2wTe6nVFAZ7fTCnrA0k7xkrhavw8JBOcdEMBk1ee1dZL6cPPPUjS431yXsjRmAR4MT4kHp5cmwfAFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, 09 May 2024, Huang, Ying wrote: >With the default configuration, current NUMA balancing based promotion >solution will almost try to promote any faulting pages. To select hot >pages to promote and control thrashing between NUMA nodes, the promote >rate limit needs to be configured. For example, via, > >echo 200 > /proc/sys/kernel/numa_balancing_promote_rate_limit_MBps > >200MB hot pages will be selected and promoted every second. Can you try it? Yes, I've played with this tunnable and, just like the LRU approach, it shows nice micro wins (less amount of promotions/demotions) but little for actual benchmark improvements at a higher level, merely noise level or very sublte wins. In fact, the actual data from that series for this parameter was a ~2% pmbench win with the rate limiting, but a 69% promotion rate descrease. And this is really my point, how much effort do we want to put in optimizing software mechanisms for hot page detection? Are there other benchmarks we should be using? And perhaps doing the async promotion and not incurring in the numa balancing overhead and comparing the cost of migration before promoting would yield some better numbers, but that also might be easy to get wrong when compared to the relative hotness of the page. Thanks, Davidlohr