From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6EDDC02182 for ; Thu, 23 Jan 2025 20:33:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2715F280019; Thu, 23 Jan 2025 15:33:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 220F5280017; Thu, 23 Jan 2025 15:33:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E91E280019; Thu, 23 Jan 2025 15:33:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id E5651280017 for ; Thu, 23 Jan 2025 15:33:47 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 74A44805FE for ; Thu, 23 Jan 2025 20:33:47 +0000 (UTC) X-FDA: 83039867694.28.6DBC451 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf24.hostedemail.com (Postfix) with ESMTP id 91FB7180008 for ; Thu, 23 Jan 2025 20:33:45 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=dubeyko-com.20230601.gappssmtp.com header.s=20230601 header.b="TwVZfvG/"; spf=pass (imf24.hostedemail.com: domain of slava@dubeyko.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=slava@dubeyko.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737664425; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=FWKEcRh71BAg0M2WtgM0lmWRruTWaXBmuyXlhb/Yq0c=; b=4Togj+whkJI61VVQYczwD5SZ+uxI1wA/9DjRDM+A+ID0UGKMkKFhkDs7dqLXwuH4b9F2qR Ph8eIoTY0I5bApcCvkIoVBze4hA7GBIpD3Ov6ZYF97SqzvOdKSoUFPpr07PrVR1Wf1Wrzn PfQP4Ikue0R1zyRd6319ZdwGf4QvLuE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737664425; a=rsa-sha256; cv=none; b=NZvkpjqOlALsNlR+zdiAI7H5hhBNwG7RAtiQqvLohU2iPD/GOK0iE9IOu3L5i5WunS9AKA eEQqo5lLeIF/RDBXSRJFuNXGZhDuNXAR5npW+7beD54UhZho0Q5QiDa34dn72dJbuQ0CRR u3PKc5YRo37H1NVH1WLz11aOw+uRepo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=dubeyko-com.20230601.gappssmtp.com header.s=20230601 header.b="TwVZfvG/"; spf=pass (imf24.hostedemail.com: domain of slava@dubeyko.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=slava@dubeyko.com; dmarc=none Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2ee46851b5eso2027855a91.1 for ; Thu, 23 Jan 2025 12:33:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dubeyko-com.20230601.gappssmtp.com; s=20230601; t=1737664424; x=1738269224; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=FWKEcRh71BAg0M2WtgM0lmWRruTWaXBmuyXlhb/Yq0c=; b=TwVZfvG/mSNrQuqCJO2w1AMvwTYFTnUxeFXaxv9ZnFgPx4/PtFnwL1ba/7RR7yGdRp QBznrj/n6ahfok6x7919kE4UDQCLCym3Vrp72oqfou+Yr3IK0ukzgKtcxUAV9JWBEj4U IsI3xftzA2qJdW10HBME4ntwYA6PM5tzu1MVxmuZ589lpZ+y2O40t/lz56dSNEKKs4UY XRHR0gDYctfNnkxqeBP/awGj4d/KhRXvE+wnqxOEKH7s2CkkT3/nAysFVE3RrUJaIDEG +rDq9TdQbFP7pexImstszLaEb2/0ECTIlt9XyaWGknsMHhyFb3y4E1zcmjAU7fWLrx4b 7Liw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737664424; x=1738269224; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FWKEcRh71BAg0M2WtgM0lmWRruTWaXBmuyXlhb/Yq0c=; b=lXqH5HBVeAJvuNiwkrsSYaj5Fe/UhsugRR58FOC09mGYpDJ0WPkpEQ00Wk6eUwNVRn M3MFNVyW+YKYl6uRGZgkSjUmfL8rrf9ByKgGG8IzuVM561xQgKC88w/fcB5E1wY+xzWN dTohSXAbk+2F4V7VY5SBe43i/LTt1CbJ96A6OLClYzxuNExAESbsJJLOstdsHzJe3EWX T+dNVYpF/zbaR8ACxHL10wEDf4Xwy2A8bzoLYr6aLhs51s3I5ZU6sUeTZedY72Hfu3/r x4KQqBixrffkK/4V6qnRjux5zhL5Tx1zztRWG1JQ2rHddmgnVlDhNte1I2qyX18Q+F6g 0xmQ== X-Forwarded-Encrypted: i=1; AJvYcCXJjP/6ryTTvKNYXEw2GjHooLUoqWNJYa1/dylax/ordJ9q6jyvY2BYTDoD75Qsl//ucqoLAsUgAw==@kvack.org X-Gm-Message-State: AOJu0Yzj9OcDLUUsKJ501T+qLIPfM5i8Wxt9IvtuvfKVA1EmDaMOY015 hKFhJUVkdwteVgqg9wpIdSU/B0xYbz7eDGKiQKKZXVHFBouFvS9FxwDqzx37pxI= X-Gm-Gg: ASbGncvZYgkmYyVLP54p/YjpLzYPVQ2+7zdzOX7apZcVYJ3OjV6THvFJpcaDoeLN3sQ nJXZEQzQTuvJw7GLla3ltHeG2LX68xFm+8uJ2GPVYNvExpYS1Jf76XsGlbH/Dpg7lRDapWOwEVI gWLujoXdllahncPpFXLPytyE8Ks/2QSEADmSj83GDe+q47KECrUEp7NioHaZl3s4LUjvhIeRk94 a3wq28/6ZQ3n5KR9cVoZh+asd2cgB2N9o9HCr6b4I1uQUIQ047zR+0gykfSoTOp4FZ0KCyWMxFq 65Lb0vZWRsc06KM3ycU= X-Google-Smtp-Source: AGHT+IHruNaV/fe3ZJuXQt7dVzgbe6RJhqsGmZcMMO0ypaCWrIi0BmI2NW+jNcrCG6nHd+Ztk82R7A== X-Received: by 2002:a17:90a:c2c5:b0:2ee:8008:b583 with SMTP id 98e67ed59e1d1-2f782c9c88fmr43158452a91.16.1737664424284; Thu, 23 Jan 2025 12:33:44 -0800 (PST) Received: from system76-pc.attlocal.net ([2600:1700:6476:1430:3505:6c:7825:7b9]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2f7ffa56fbbsm143335a91.16.2025.01.23.12.33.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jan 2025 12:33:43 -0800 (PST) From: Viacheslav Dubeyko To: lsf-pc@lists.linux-foundation.org Cc: linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-mm@kvack.org, javier.gonz@samsung.com, Slava.Dubeyko@ibm.com, gfarnum@ibm.com, Viacheslav Dubeyko Subject: [LSF/MM/BPF TOPIC] Generalized data temperature estimation framework Date: Thu, 23 Jan 2025 12:33:19 -0800 Message-ID: <20250123203319.11420-1-slava@dubeyko.com> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 5ncg17ngoqjd37s66q11x94zk4ph48u7 X-Rspamd-Queue-Id: 91FB7180008 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1737664425-516798 X-HE-Meta: U2FsdGVkX19uFhZ03puxeKtt7z8AO6NUNdnOb//chwBEbRujA/YZ/lsxBxZtIuOQQKUvdLr/KLGOEnR8hpfcyvvnxsaoTDUNw//G1RpZ1Yq1vYPBCpTYBxBn89oF5cF2BLjBXM1uwdMDicwx8njBdc/zBnQcslkhz3DX2/RCkE+oXUaOp4+qsmU4QZubbUtvwu1I3rv8GeRMMMI2DjI1NvXX0+ipdDJipkmkPUlKRO5KTzuEijdmBvv7/PunfA5+m2qKpb07yRwUtIStlw1ihnsntWLFAkkWKz8KAIrm0exQQSybQf27aIwiwvLI2OF29qSW8I6xJ1odKlHOzR5gbF/25y2vKqKU4N9PEVIIgZd2AcoBhCVx/boX2YMPV1xIlkWg74RQ7fepzMYM6frIGDa7r8XbIVDAwkSepYQTrQ3sY3e4UhGE6asyetLgF5NjKUQH1AsHFJysOSjFaFwBA9qdaGILU6De6uEzQB9aP4o/itzhesOd8XxQVGELBeDzCTv2ScZLZYSKEbIlea8YmVZ6U69XdaoWyZoGDYpxEUJAE6h0Pt3AApt4yAZkn8IpFd7Bat3l/3l12uzB6dgisX0tFVwe+eWF4SSKm4BNvS0i6ci3jGjKIirHmwjOVNPsHOH20pTSs6y+QuT/5L2kTNV8esISyRa30K9cxbcnoMnVch0jUFZu3cbjFALF1GCteceVJPMJCxzY2Jq8yH4y+y1t8C+7gPhb05rJNz3eaHP4Ah3YRFjm6fDFsUbXoTS4z1AypMxZeiei2c1j8nynQhqUjWmVF3UIhq6f14w4YpPRJKiU4yW/2cTL8tX70jJTQ07f4MKid5E6fNrXiLIQjBLynjTFOBciodonrZBTtmNAOJA972WX7m8GL321YkxRrWeDsGkE3yMy8ogfC8JHZPuhXMCNcIZOWN6dwS3r/YLKfhGeBUzPTiolLFXBxoRAyjfDDwQds3trWE687sr u+66Rx+k mu03H4853mEin0f9SH5qrwgtycCvNMPT/Bm2xALtFAl5qvzFqw+CnTDhtXuYXmco6YEydGoczOw8tWcIZeBw/6033tBfO2RacZE9XZbwtaOSBNoEXP4GJbUQwgYH/g94D7CLNDM+O169aZnXMLLi9R9QHEHKIDpuST9pJMQB9HmMGDmsKPNDSYu5NjEnjdep31Ky4hn1emDkt86lXxvpijAST7tlhzFJV2HLwPtNgFKloNM6lZTm7WizGVeeNB50cz/il3ffDy4mja9+ci7WlvKhB/i5MLHkh06+3L0tMTQlbOU+V77r41NQEnD7pGFAvWzwDInQKa0p+h9a4cVxLJRzq8j0t9GbSaSHL4PdZQBhcRGLiFxFguWuXUu7LMcDy0herv6SvbukQ20s= X-Bogosity: Unsure, tests=bogofilter, spamicity=0.462361, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hello, I would like to discuss a generalized data "temperature" estimation framework. [PROBLEM DECLARATION] Efficient data placement policy is a Holy Grail for data storage and file system engineers. Achieving this goal is equally important and really hard. Multiple data storage and file system technologies have been invented to manage the data placement policy (for example, COW, ZNS, FDP, etc). But these technologies still require the hints related to nature of data from application side. [DATA "TEMPERATURE" CONCEPT] One of the widely used and intuitively clear idea of data nature definition is data "temperature" (cold, warm, hot data). However, data "temperature" is as intuitively sound as illusive definition of data nature. Generally speaking, thermodynamics defines temperature as a way to estimate the average kinetic energy of vibrating atoms in a substance. But we cannot see a direct analogy between data "temperature" and temperature in physics because data is not something that has kinetic energy. [WHAT IS GENERALIZED DATA "TEMPERATURE" ESTIMATION] We usually imply that if some data is updated more frequently, then such data is more hot than other one. But, it is possible to see several problems here: (1) How can we estimate the data "hotness" in quantitative way? (2) We can state that data is "hot" after some number of updates. It means that this definition implies state of the data in the past. Will this data continue to be "hot" in the future? Generally speaking, the crucial problem is how to define the data nature or data "temperature" in the future. Because, this knowledge is the fundamental basis for elaboration an efficient data placement policy. Generalized data "temperature" estimation framework suggests the way to define a future state of the data and the basis for quantitative measurement of data "temperature". [ARCHITECTURE OF FRAMEWORK] Usually, file system has a page cache for every inode. And initially memory pages become dirty in page cache. Finally, dirty pages will be sent to storage device. Technically speaking, the number of dirty pages in a particular page cache is the quantitative measurement of current "hotness" of a file. But number of dirty pages is still not stable basis for quantitative measurement of data "temperature". It is possible to suggest of using the total number of logical blocks in a file as a unit of one degree of data "temperature". As a result, if the whole file was updated several times, then "temperature" of the file has been increased for several degrees. And if the file is under continous updates, then the file "temperature" is growing. We need to keep not only current number of dirty pages, but also the number of updated pages in the near past for accumulating the total "temperature" of a file. Generally speaking, total number of updated pages in the nearest past defines the aggregated "temperature" of file. And number of dirty pages defines the delta of "temperature" growth for current update operation. This approach defines the mechanism of "temperature" growth. But if we have no more updates for the file, then "temperature" needs to decrease. Starting and ending timestamps of update operation can work as a basis for decreasing "temperature" of a file. If we know the number of updated logical blocks of the file, then we can divide the duration of update operation on number of updated logical blocks. As a result, this is the way to define a time duration per one logical block. By means of multiplying this value (time duration per one logical block) on total number of logical blocks in file, we can calculate the time duration of "temperature" decreasing for one degree. Finally, the operation of division the time range (between end of last update operation and begin of new update operation) on the time duration of "temperature" decreasing for one degree provides the way to define how many degrees should be subtracted from current "temperature" of the file. [HOW TO USE THE APPROACH] The lifetime of data "temperature" value for a file can be explained by steps: (1) iget() method sets the data "temperature" object; (2) folio_account_dirtied() method accounts the number of dirty memory pages and tries to estimate the current temperature of the file; (3) folio_clear_dirty_for_io() decrease number of dirty memory pages and increases number of updated pages; (4) folio_account_dirtied() also decreases file's "temperature" if updates hasn't happened some time; (5) file system can get file's temperature and to share the hint with block layer; (6) inode eviction method removes and free the data "temperature" object. Thanks, Slava.