From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26FF8C25B74 for ; Thu, 16 May 2024 05:16:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FE0C6B0394; Thu, 16 May 2024 01:16:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AD4D6B0395; Thu, 16 May 2024 01:16:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 375566B0396; Thu, 16 May 2024 01:16:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1B4B76B0394 for ; Thu, 16 May 2024 01:16:41 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 844E3140263 for ; Thu, 16 May 2024 05:16:40 +0000 (UTC) X-FDA: 82123098960.23.ECADE7A Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf15.hostedemail.com (Postfix) with ESMTP id B4385A000A for ; Thu, 16 May 2024 05:16:38 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOkIB18x; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715836598; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KoXgcVY4N+e9r2pMN8P7bYq7SwU78Nqi08mMaWxF6xQ=; b=UpjVmel2WZ2YgrsH4s89JvO+76om1h8Xcpa8erc+zEXPw6fOScYGyYKMZ7sDKNIa0U9qky FuLRQphVL/eLk8kcU4EPIaqS8Rqc5VtWZ5oFh29j+c94FV/ESjZSEu9S8M9XZNTWRafMb2 UEzT9E4nU33z9p3NneVJzr5L/Fi4Puo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=VOkIB18x; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf15.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715836598; a=rsa-sha256; cv=none; b=6jsHiay0EYN9d9KWBso2Jch+5ATKAodX4U1tkg0G6h2Y+soX6wG2BG4dWJiRMqOzhIE1Bp v3FzIC2zNlbHxR6AteuKTCuEARCYnvOzVB5CGcGlqHwHGEIilKBYbxyAowHXUSt9f5u6MQ HgVUCBkOexhZQCaJEyHj1cnnDpGZ2M0= Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-420107286ecso379735e9.0 for ; Wed, 15 May 2024 22:16:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715836597; x=1716441397; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=KoXgcVY4N+e9r2pMN8P7bYq7SwU78Nqi08mMaWxF6xQ=; b=VOkIB18xQwFO7toB9yZukxvkpjtFRYUxVGsY14UMX/tfHtY5wwXIDwtGNhbqGaHWKB Ry3v3TYp8MWw2S607IFsLZyiKXrcusg2Cj2oWjQZ7jbzYj3XQNawfBSPc/qG8SgeTW5x Fe+xVrUpSs+Y+2i7MQxS29kNMI+3wuSg2FGe3qt70SgFzQ1XHPyItp9DYAJGUSYkzvzV 5OEzBA/3lqrA4bVKQMzGIw20i/XbYm5w71oQ7QPm7PgN3YkUc3rcIiys9gL99hKQeFqr RTrwyuCdTYJaqiY1IrBdoXyBKBi/v+q8LQje53u5SrwJLHvyfSCt8g5c229e95JEbl7f lVhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715836597; x=1716441397; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KoXgcVY4N+e9r2pMN8P7bYq7SwU78Nqi08mMaWxF6xQ=; b=PG+4JImIP9pnX9hTU1I5AcQNivZVCMAGXIUQ6uLVNzP7PiU25eX/qFPwyysIwhF2ce 7ifVTZ45iZKjb/xU5pm9zkoBu7FRrrBPLMVZuLUU1tAhJfS3LaicXag1bhnjKS4X+Vu3 y2KBGVfjCe6qoZZR/I+6JFxLeJjpYIGKPQLxuBmKKld9hlpxq1DE/VDfTLm5FEZPxaJP W1YOgaHO9OBDcEvRvbO38i0s0nTZkxTsmQJY6iQpFeLUSAPYHarjNlUTzO3ROS3Ds+6R dIj/eke65TLg//q0wscXFItvYm1xvgDWpmD60y7mZ6e9Vc2SoKQenzC3sVAbIQj0G0tz N8DQ== X-Forwarded-Encrypted: i=1; AJvYcCUCmsqO35YscdnTyot22B9J1qXCzDeAuM1yZtbhRvrg5NmWog4zDpS7g/sPbKlDvL1s0qRDEmMcXLArcUnXlBc48aY= X-Gm-Message-State: AOJu0YwA9fMgQso8TxB2pksj3+3FrzT0YGRiSdBxGxYRBOdlpG3NHscX t/mdOfBHQHx0EGocHF+kU6clo2UlwEfIGORhyAZpuhVqUYrex/Pq3nBD9+YjgrnNBkUyEf483Ac n9knvo3R0Yke+5ITL2k1ETs/R28XadIaMgFpO X-Google-Smtp-Source: AGHT+IGl6JA3apmhA5+1wz63KlREjdBXsa4woxLLESUPu12IA5lR2Ns83wfiKirMFSk2oX8yY7FK/Ckoh93rZ9xrmps= X-Received: by 2002:a05:600c:c0c:b0:419:fa3c:fb46 with SMTP id 5b1f17b1804b1-4200f8c08eemr10463945e9.5.1715836596728; Wed, 15 May 2024 22:16:36 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yu Zhao Date: Wed, 15 May 2024 23:15:58 -0600 Message-ID: Subject: Re: [LSFMM] automating measuring memory fragmentation To: Luis Chamberlain Cc: Michal Hocko , Dan Williams , John Hubbard , Daniel Gomez , linux-mm , lsf-pc@lists.linux-foundation.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B4385A000A X-Stat-Signature: serd1kd8xtnscmryadxxoj8a5e3hdbrf X-HE-Tag: 1715836598-590648 X-HE-Meta: U2FsdGVkX1/oREpTvRGLW/lpnaU+TIRl276FhbbinO3E3Mj2nWQyIRBqLkMYDhMmVEbDk+/4XPivVxz/OCX6h2IOy/lYbHcXkqcuPvIBIdGr9ohzwlFuIWVqgEIcU0rA3szp8DGo8FI/azMYbLcbTI2R7DID5VY1kZhyzc81ChjKH76CppIk8RVutPCKaPhKiE2RlGSO0ih0qghNHPxAoCOmXTJNEnPlpwVDxWeD+zisDcZ/DUWn1uhzY89nZ/BC8N2dc2RjHI8EAbW2wkfRz4YokvwGr7qS6tXIGFxi2lU98NzBNeYjjhO1B4poAK+YklgObWgCip5b21PS7iwlPBYbblLeJDSw0TL84fl7lJsP+b6hvZ7h616oyeHmUcNALhxmxVnTtG08qcwtJmuxfcebHB/QCb0j9QHeToVMixuOsdzNawXs/4hCgrPykMinWxU0UfZLgKrVrjImqfb9QoibYMCMV01+Etv0UVHEEsf28IIafy7stHSL1DNaYsvtfZWK4HQmNXXvbOPnWFaeHChgbZzT/LR9UGfd/y1HBtrEqyjcCTH52VnHTlgN5Lhk0jWfxOVdaMeDP00EUFgHiJ+Dz1wuTZS0OdMSzguBN4DMu0WZSLf1p0QLmLpOKQthm5LwGMKfiZKXFIQuY3ykzuuiPz46m1vVWiBOIRR5/79kNjLlTls94XBTQ7n+xsQaqf0gsOy3UfgA6VC/Kxmgs0it72o50SABX3178JSopp+wzj3sHQNDNOtce4dbftHIT9+Omv6MaQLQK6oE1riruaargXhgQx4iqim8okJmzAgBFpWSpsLATxAk7NHNhoZ+o+nbiVTBXdZ+UG3VUiPvY+duGWTPHnY/YJh9pDxuWTEe0x151VOitGG8TNjCMfbH0UoVmAd6Muc63QDni/Abdny1HutR9/vi5uMdmcrTPVtUWZBYPJqtF9x/hF9odm4fLbblBQMnIsNm4VcJTns 2sOmxLBK 9hSOT/YSq/t5vV/HPACcUuGKFOv2HfX3LZ2aiRbYzFzVWfQkwl6YS3/0MUp8ohkfVVXjRyowvfCk5uN9WeNE4rAY8CvWJ9DX6iXKuIVDzelGswRLZJgT4lfD0MBITmNZyXgKQVcAJzjXVNf8qihylLgD1hv20x/fdRYQ7f+1RWLC1eSj38j6aZIzdHBlcwnR5x99RHcdOxlspwt5DJMvLktpGv1pHr4Jcw+C5TjDNqsSxIF13H9+YYFXnTzxMTNZ004y3GpBT3Hwj7VVcHVgxQAV2OPy54r9jYgpgSEYGvaIdx4tV+fRY9MREhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, May 15, 2024 at 1:34=E2=80=AFPM Luis Chamberlain wrote: > > RFC to see if we have a breakout session today at LSFMM. > > After the TAO talk today it occurred to me that it might make sense > to review how we're measuring memory fragmentation today. We're looking > to add automation support into kdevops for this to help compare and > contrast memory fragmentation behaviour with one kernel against another. > A while ago, while mTHP was being evaluated I asked genearlly how we > could measure fragmentation with a simple one value, and John Hubbard > had one recommendation [0], working that proved we could simplify things > [1] but we also could just use the existing fragmentation index and only > consider the values where this is concerned for fragmentation and not > lack of memory. It begs the question of how folks are measuring memory > fragmentation today in production, and if they have any desirable > changes. The first approach being considered is to reproduce the > workloads Mel Gorman had written and used for mmtests and leverage those > on kdevops, perhaps modernize them, but before we do so it seems > reviewing how we measure fragmentation today might be useful to others > too. > > As for mmtests integration into kdevops, first order of business are > just a few distro-friendly updates [2], for the next steps after that > though it would be great to review the above. > > [0] https://lore.kernel.org/all/5ac6a387-0ca7-45ca-bebc-c3bdd48452cb@nvid= ia.com/T/#u > [1] https://lkml.kernel.org/r/20240314005710.2964798-1-mcgrof@kernel.org > [2] https://lore.kernel.org/kdevops/20240319044621.2682968-1-mcgrof@kerne= l.org/ Please correct me if I'm wrong -- I don't think we can use a single measure to describe fragmentation in an actionable way. IMO, we would need at least multiple values, e.g., fragmentation index for each non-zero order, to describe how fragmented the memory is with respect to the order of interest. Of course we could encode multiple fragmentation indices into a single value, but that's not really one measure. Fragmentation index of an order can tell whether reclaim+compaction can theoretically result in a free area of that order. As an average, fragmentation index can't tell which actionable unit area, e.g., pageblock, would be the best candidate for reclaim and/or compaction. That would require a ranking model, e.g., a cost function and weights for reclaim and compaction operations, and calculations of the cost to produce a free area of a requested order for each pageblock, i.e., a 2-dimensional measure costs_to_produce_free_area[NR_non_zero_orders][NR_pageblocks].