From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA0ABC4167B for ; Tue, 5 Dec 2023 20:46:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 642F16B0083; Tue, 5 Dec 2023 15:46:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5F3816B0085; Tue, 5 Dec 2023 15:46:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BBD66B0087; Tue, 5 Dec 2023 15:46:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3CCDD6B0083 for ; Tue, 5 Dec 2023 15:46:19 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 12FF9140266 for ; Tue, 5 Dec 2023 20:46:19 +0000 (UTC) X-FDA: 81533947278.26.ABC38F4 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf11.hostedemail.com (Postfix) with ESMTP id 959C340011 for ; Tue, 5 Dec 2023 20:46:16 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=EXszuCb6; spf=pass (imf11.hostedemail.com: domain of akpm@linux-foundation.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701809177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VyILmcLfJVapOgAFpHNihPSkuzmDzkRLXzq7cnRbeZE=; b=IbVWs7xACwjf54cl9uz62YSB3RdOy3F3L6VYl4b1bqAa7W3URqvrAeQIDk8RWtuMtjWnGI gAea2OzAlCs4y2I+wubR14n5+ZrERxZ5z0tDB/6K25TKg/FCiPOqoFr8+WHKnLZdvyunOG a953pWgH/adpvCR54nCb0p6g76Bvkys= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701809177; a=rsa-sha256; cv=none; b=C5buapnFJdB4cCZsrWeMbRmIzISL4/TcmvluE3t0cfHURH0FFtCS1yx0vsnaC8zVoAF3Sw rkU7EU7UHrvYPve/6Qc0f9dWkY/aeR4yg/dTtLqO6T1qvz39qKHZVYYpjkI08mQHpez9Xh znNue7UP3GuVvFoyog7AvmrMhpVEbVw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=linux-foundation.org header.s=korg header.b=EXszuCb6; spf=pass (imf11.hostedemail.com: domain of akpm@linux-foundation.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=akpm@linux-foundation.org; dmarc=none Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id DA2EFCE185A; Tue, 5 Dec 2023 20:46:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E2D39C433C7; Tue, 5 Dec 2023 20:46:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1701809171; bh=QRoUE19qRtHescz0H5jxE4IfSkWb5vJkxf2yeRy6dfA=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=EXszuCb6OBuuMGlVtrbb7H9IgZB/NfwSd67Af3xwiN0RBK8jeiR8Bg+F58HSIP9Bw ugyXPDaobPyndakmDYXScizcLiAKJACf9l8cHCwKxgn5n2um578reCuFjeSIKsAigX 1aiQp2EELa6yvoDKmIDD8W0o7qUk+sOTwh/vjpFc= Date: Tue, 5 Dec 2023 12:46:10 -0800 From: Andrew Morton To: Stefan Roesch Cc: kernel-team@fb.com, david@redhat.com, hannes@cmpxchg.org, riel@surriel.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3 0/4] mm/ksm: Add ksm advisor Message-Id: <20231205124610.ca70c47e880daa81a0f21f07@linux-foundation.org> In-Reply-To: <20231204234906.1237478-1-shr@devkernel.io> References: <20231204234906.1237478-1-shr@devkernel.io> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Stat-Signature: j8wdgxsmgjhay6zffipwzewua5c3cft6 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 959C340011 X-Rspam-User: X-HE-Tag: 1701809176-837925 X-HE-Meta: U2FsdGVkX1+eyxFSlhYQSQHVc+7/68y7s+zGMmEP9n+CRvF7G1B303Z3ivf+5FfHZbk6u5vO3jB6yWCXFC+jLT7JkYnmNzff02R4Xnec+EuONqZWPYwUe5zYE8Z2h0i5U2M+3kORM3fUBIOWJt/1du5r9zi5JNUQzrecuZHnSZCDsv6hiihFoulugyWfVsdu7m9dI6Md6owRFlNqGYKgpc2SNy/pwWQYV9+g+6SVUOMSM8hlY2sjWt2LV/70GQA8FWcIdbe7Ln1t1AUlvKZdAyzbv4xCdYQv5auYx3PPRATGqalYoKwnf0baKF0a0kySoQahhSJTcN9am3hjuU8LvQLySSEynAoHbjExLXJCh6coPV0MybqKA3INK+zvxE7BIPEhBvTlRhEGA/UrNmCs3hBaYrblhYbVOaEmUzuwfbaj8nl/guDyEOo5u0E1hysjpQlmJAyVnNyCS+GvOwsUoazIS+G9kh6y0wEp4dguIVrZLwcIGqnnXvRB3vkNq8rGSVcZQCuQDCAGor7Xs0GlLxR/PtxtSleal+iRa3LBUWP5/ibnjci0u9MvfS0JjXqcHsrPuEt+hOIeH8RZ+FHahWd5bPWuwASnPa8uJmcEnP1dVGpoiMY/gAyMQST4NO98813Wp8OCDR+yLyWZ49vdjb4CGtcFFsCGSXaGAbNRFdxqoqynIfmdaazq3z3FoSZCIc/jVtqt3sjbLT2unoARbzGvinEoSD/wQYMKI/jhs+fdFBaF0lL8JwY4i4nUxEWV1dQH78PdpykKZ5s9GDnqdt8JP5Ux2FgxQUlskinEKNFwXNyjk9iSYrXoaD6+b5P1Uo5Rar6ePrHrirnaJleL9SZWnFA/CXVmxOMxS/IK0SryZRLZ7QZIg6eE46BCxIip4GSVRBeST2eucJNOroAXeyPl/d0qOl/XXJAiLO0SgoX+WW0f2Q5R5wJbzOymhyrhu9VzDTHy9guq8xSxIPg /9iIXmmy 9qUohgoNqtdiX83z6nN5wF3pEZa62DgYzMMegHEBbFDRCxbhvGOsfZajbcUL+wCs+gz56Ooc3lNPbjFq25LOCt1Mup22L1sCjL9S14jERUQHBqM5jFW9YBL76Dw0MZfdJ4qLSWHgsuFwOW3YCvv6RUSJ5JGZCU8MmGnr+KisALRZd4uGnzBFtNwr4jTpt81FSaLHX95eBC56GavTU5zKmjYQ4J/d7Ne4kj7Bo/3lNjw2zWALtIWLMCB5mRSbnIatheeC914vGtn9XB5JQaL1yn3+MnGk0kPxSTAedAJVwQR2ABL1S6kJXsd5JZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, 4 Dec 2023 15:49:02 -0800 Stefan Roesch wrote: > What is the KSM advisor? > ========================= > The ksm advisor automatically manages the pages_to_scan setting to > achieve a target scan time. The target scan time defines how many seconds > it should take to scan all the candidate KSM pages. In other words the > pages_to_scan rate is changed by the advisor to achieve the target scan > time. Dumb question time. Can this be done in userspace? Presumably this will require exposing some additional kernel state to userspace. > Why do we need a KSM advisor? > ============================== > The number of candidate pages for KSM is dynamic. It can often be observed > that during the startup of an application more candidate pages need to be > processed. Without an advisor the pages_to_scan parameter needs to be > sized for the maximum number of candidate pages. With the scan time > advisor the pages_to_scan parameter based can be changed based on demand. > > Algorithm > ========== > The algorithm calculates the change value based on the target scan time > and the previous scan time. To avoid pertubations an exponentially > weighted moving average is applied. > > The algorithm has a max and min > value to: > - guarantee responsiveness to changes > - to limit CPU resource consumption > > Parameters to influence the KSM scan advisor > ============================================= > The respective parameters are: > - ksm_advisor_mode > 0: None (default), 1: scan time advisor > - ksm_advisor_target_scan_time > how many seconds a scan should of all candidate pages take > - ksm_advisor_max_cpu > upper limit for the cpu usage in percent of the ksmd background thread > > The initial value and the max value for the pages_to_scan parameter can > be limited with: > - ksm_advisor_min_pages > minimum value for pages_to_scan per batch > - ksm_advisor_max_pages > maximum value for pages_to_scan per batch Should these be called ksm_advisor_min_pages_to_scan? > The default settings for the above two parameters should be suitable for > most workloads. > > The parameters are exposed as knobs in /sys/kernel/mm/ksm. By default the > scan time advisor is disabled. Disabling it will reduce the effectiveness of testing. What are the risks of defaulting to "on"? > Currently there are two advisors: > - none and > - scan-time. > > Resource savings > ================= > Tests with various workloads have shown considerable CPU savings. Most > of the workloads I have investigated have more candidate pages during > startup. Once the workload is stable in terms of memory, the number of > candidate pages is reduced. Without the advisor, the pages_to_scan needs > to be sized for the maximum number of candidate pages. So having this > advisor definitely helps in reducing CPU consumption. > > For the instagram workload, the advisor achieves a 25% CPU reduction. 25% of what? What is the overall affect on machine resource consumption? > Once the memory is stable, the pages_to_scan parameter gets reduced to > about 40% of its max value. > > The new advisor works especially well if the smart scan feature is also > enabled. > > How is defining a target scan time better? > =========================================== > For an administrator it is more logical to set a target scan time.. The > administrator can determine how many pages are scanned on each scan. > Therefore setting a target scan time makes more sense. > > In addition the administrator might have a good idea about the memory > sizing of its respective workloads. > > Setting cpu limits is easier than setting The pages_to_scan parameter. The > pages_to_scan parameter is per batch. For the administrator it is difficult > to set the pages_to_scan parameter. > > Tracing > ======= > A new tracing event has been added for the scan time advisor. The new > trace event is called ksm_advisor. It reports the scan time, the new > pages_to_scan setting and the cpu usage of the ksmd background thread. > > Other approaches > ================= > > Approach 1: Adapt pages_to_scan after processing each batch. If KSM > merges pages, increase the scan rate, if less KSM pages, reduce the > the pages_to_scan rate. This doesn't work too well. While it increases > the pages_to_scan for a short period, but generally it ends up with a > too low pages_to_scan rate. > > Approach 2: Adapt pages_to_scan after each scan. The problem with that > approach is that the calculated scan rate tends to be high. The more > aggressive KSM scans, the more pages it can de-duplicate. > > There have been earlier attempts at an advisor: > propose auto-run mode of ksm and its tests > (https://marc.info/?l=linux-mm&m=166029880214485&w=2)