From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D23C4725D for ; Mon, 22 Jan 2024 05:46:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1B1E6B0078; Mon, 22 Jan 2024 00:46:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECAF56B007B; Mon, 22 Jan 2024 00:46:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D92346B007D; Mon, 22 Jan 2024 00:46:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C9C196B0078 for ; Mon, 22 Jan 2024 00:46:39 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 700E1A0908 for ; Mon, 22 Jan 2024 05:46:39 +0000 (UTC) X-FDA: 81705862518.26.6F13E84 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf30.hostedemail.com (Postfix) with ESMTP id 8BFFD8000D for ; Mon, 22 Jan 2024 05:46:37 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VvOmkui0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of cuiyangpei@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=cuiyangpei@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705902397; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IflNS6Etc8g12+qLnuO+gXfIx3jLJ1iHCnSoNRmsAEY=; b=6h+e2sOGG1BUfHhcz6rCO03iBM0FvMOha7fdwMZUT5DWejY8fdOtwiVUbpL/v/+/lyUDGQ C5/xD5GFTkA0sUlTq1Od33x3J+gkgmbVQejCxOmtUf14eSS01vD9bo8ukhImKzWQZPnbvB o6E4FEzRraLZaHleQ9vf1wsGe8i3FZg= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VvOmkui0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of cuiyangpei@gmail.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=cuiyangpei@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705902397; a=rsa-sha256; cv=none; b=RTS3C/g10UKnQ/LZk76X1wRGzKTyrrm1fEAN8nAFmo6azm1ElkWgq6NyRIb+r3PrieRXPH rth4hINa3Cx9BzJRaAmPA2HyuTU7e3xl/JmADF3Wo7ML1XxpuAS0W9h/PhrEZL7gc2jGk/ sRMNi6Li/j4AUYJsS2lEMrpcqYs1T4U= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-1d748d43186so1681075ad.0 for ; Sun, 21 Jan 2024 21:46:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705902396; x=1706507196; darn=kvack.org; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=IflNS6Etc8g12+qLnuO+gXfIx3jLJ1iHCnSoNRmsAEY=; b=VvOmkui0doKpSDmz9Lad4ofSHKwN85JeMW02nOoHySCOaug7EG90UfynZ1fWsxY1qn 5Zt4LaRVqUZaXWrDSkc0um9nUSj99MbO8/7d2R/wGnOQqZmtSHBVU9GLwUKBSDVfpKTU 2kQu8yvvwWuhqxB7gjzf3cWg0x7LHSz45fXN72C6rkZtYW13brZ/cGNm4cs2PMg6AjG2 pVk9D3VZaJYPh6tQ03gnPubVmfxlwKiz/JJ0LvhhYFBgT2RJi/VqYK/1AnxzYIFyx+mA GeoGu7UlPvHloVfXc3GYhmJAuLItfY4bWLn3TRmb3f0efX492uV9vEDPc80EhZ3E1i0U RL2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705902396; x=1706507196; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IflNS6Etc8g12+qLnuO+gXfIx3jLJ1iHCnSoNRmsAEY=; b=iiP1+u6tXk8I3Y3hFS1u97W3SkoERAzFQndGTK11f1al6sAc31yk9SzC9cTpDU31Ba mYuzVKlaG4X9D9OZuRDItlt6s6/mt4zA0FpbCAQr25/bJlaPEuQq4Qf6DpHBsKtB0Erx OF+xVVjQ+AuCC0vE4DohN5g4Td53h26eX78ZhvYcL90JgQZjk0Yd3M+OlsGKpNPUy0/J rB3CCWDqGlHXFFWBrxmAi6x8claAqB6r7iS5dGoo+oJYhsP98u8VAKML7sxe9Ch1lerD ckOOhBF4Lv40zn2Y558aqqQAGi/79n4ACVRMD/wSIhOYkBHdtqvGh6UK5yYtVNd+buvP 8xvw== X-Gm-Message-State: AOJu0YxHUXHzz9Q6HkVm3/mwjWFplKrFWPBH2frW33/VnDJUNtZKV7nq szGhTtu6nJKLCviqxKQ/aivHKfKOS2qw5YDvPwBx/CwlNhJMc1yu X-Google-Smtp-Source: AGHT+IHG1N155t3rfeAWb6fP/BEBG5N33Go4BNyWCFzghW9M1uDKo4TALaThs42HCSbSIuc5tQvc6w== X-Received: by 2002:a17:902:7794:b0:1d0:6ffd:6e85 with SMTP id o20-20020a170902779400b001d06ffd6e85mr1471333pll.125.1705902396245; Sun, 21 Jan 2024 21:46:36 -0800 (PST) Received: from cuiyangpei ([43.224.245.227]) by smtp.gmail.com with ESMTPSA id kl5-20020a170903074500b001d72d445778sm3233018plb.204.2024.01.21.21.46.34 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Sun, 21 Jan 2024 21:46:35 -0800 (PST) Date: Mon, 22 Jan 2024 13:46:31 +0800 From: cuiyangpei To: SeongJae Park , akpm@linux-foundation.org, damon@lists.linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: xiongping1@xiaomi.com Subject: Re: [PATCH 1/2] mm/damon/sysfs: Implement recording feature Message-ID: <20240122054631.GA7880@cuiyangpei> References: <20231203054313.GA14316@cuiyangpei> <20231203193745.57035-1-sj@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231203193745.57035-1-sj@kernel.org> User-Agent: Mutt/1.9.4 (2018-02-28) X-Rspamd-Queue-Id: 8BFFD8000D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: x8nrj5ykr9oc3bprrjnkytbuumkgnrap X-HE-Tag: 1705902397-284329 X-HE-Meta: U2FsdGVkX1/gAe/Yicz8aRVHdqU1sCBFWHbrstaP4CXPnogeRpnj1shbNSmRoH+f1oHtesMcrrnze12+oVcAORCAmouLPQSC09thLCpvsi7DqQYK56eSi+KuDb36zGV9Zj+Le1dFF7b2c7e/QwctKdQCuFFi1qYGXHjE4k+gJeF0ftpDJS+O7hnCUoCCrn+3VGg1BKsXQFYNzKVpr3OghkSj3goqZk8j9c8EMTT3stLv/xqCU3iieWbAxZFmaT6HrfROs9hxV+LEOEzkeaUnLHmbaWSGb/HLyxzYncSzcnEpNyOri4CqA21WlUF1DbpT9UgFa0R1AbkLRDXnl25mMPPNcuyu3NMll/3o5UhAXr26XSv9PjTbr24dHwR5SFkfYQ6J/eUlezJY7gGoHzlBo5fmTHUJqzd0vojUP/SMVBaRwzvb3xPazY1rUD9WCMBHUbaJGJ2g+coeu+Mvxk5a0d35xFpWRL/svg7qhZVXZzzO/s3iCyKJoSEqdyeCczfwr+ErboVFt1jezVX6GmMD/o6TRM45U16gckmIJVqwZfNKf75dnf64tfkjaFMVXRiOxnDNXqnT58ZIfeYEL/0lt4ySzkC3+SdpwIxxbBeZzLCbD+FrTceDdNsYwKdGpeY/U2b5SV3xcaYSjuyqBGg9vg3om4z5Qa3Vpvxpb3y6ng8KpYlqL8Y0fsgQNGnr8oytQwiGeR+UAm/cYrx06GCBlIBHUMfNylEYd1rmHYtBci6zfbxdwbQkBEnPxHMobD9M8Ul1Swu+JJkZhu4aNpXosRc/Ad7EMeQCN+YDTKxDMbi/Ea1lYzmSdb5frixUG7wkmvQ+aik2E0QmK/YnhKv/sS767ePnmagsss+XGuUtZDAGK00fSFEMylo46gYoMuCVNeytzzCPk8+WL88XSQNFoVnFlwRCm4bRYaEF8g6pnbv+mce8nSiwLYkQH/ddALgu7gh8MX06F4ypcrDhTfo twhw8K+z yMED43jPZxytkIeEuzkBnhrJpvdpnoLNQdwr87ioSKF2XSQGb4VJ/bBIbGgZ9r+BUP0382C3/uAszBPNDo8yNJC98F4cN8EOQLrriK48MdhqWAhqh9b1FNrepvovc4Lx35hlck1GwbCFJCb1OeVj1NCvOR8rA2Bc06oHrCOFuCH3ziv/xd6eRuohQ6OB4wx2amb1LYz8vsHr2SFTTjsarpfBuUkZgyV3SGX2kagayEFuEKsmTXgA9S8zVovv+3Bf4DtwQoGww7uSxb73iC8aV8mzQP5OgkoUzzADPkTp53CVzRVdn/FUdJAk0ub3iwigsTedZ/x12ger5R321W7+lLK/GT+pREOc0UOpp374VQyAKRkA7uTy07gPI/oFHpWDBeMWfN36boOrBeIKEjxV3mjrK+SOdfXBkYeKpjeKXd8eOo6N4W2YnBshPJj8i0PYp13Yhq4DqaP9u3rF3AdjgGBomzzJ4ZY4KeqXSbedPOT1hcY9oR6J34SUBoCOSRL9Ri7GK1dXzC1vd1eM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Dec 03, 2023 at 07:37:45PM +0000, SeongJae Park wrote: > On 2023-12-03T13:43:13+08:00 cuiyangpei wrote: > > > On Fri, Dec 01, 2023 at 05:31:12PM +0000, SeongJae Park wrote: > > > Hi Cuiyangpei, > > > > > > On Fri, 1 Dec 2023 20:25:07 +0800 cuiyangpei wrote: > > > > > > > On Thu, Nov 30, 2023 at 07:44:20PM +0000, SeongJae Park wrote: > > > > > Hi Cuiyangpei, > > > > > > > > > > On Thu, 30 Nov 2023 17:14:26 +0800 cuiyangpei wrote: > > > > > > > > > > > Hi SeongJae, > > > > > > > > > > > > We also investigated the operation schemes you mentioned, but we don't > > > > > > think it can fit our needs. > > > > > > > > > > > > On android, user will open many apps and switch between these apps as > > > > > > needs. We hope to monitor apps' memory access only when they are on > > > > > > foreground and record the memory access pattern when they are switched > > > > > > to the background. > > > > > > > > > > > > When avaliable memory reaches a threshold, we will use these access > > > > > > patterns with some strategies to recognize those memory that will have > > > > > > little impact on user experience and to reclaim them proactively. > > > > > > > > > > > > I'm not sure I have clarified it clearly, if you still have questions > > > > > > on this, please let us know. > > > > > > > > > > So, to my understanding, you expect applications may keep similar access > > > > > pattern when they are in foreground, but have a different, less aggressive > > > > > access pattern in background, and therefore reclaim memory based on the > > > > > foreground-access pattern, right? > > > > > > > > > > > > > Different apps may have different access pattern. On android, the apps will > > > > join in freeze cgroup and be frozen after switch to the background. So we > > > > monitor apps' memory access only when they are in foreground. > > > > > > Thank you for this enlightening me :) > > > > > > > > Very interesting idea, thank you for sharing! > > > > > > > > > > Then, yes, I agree current DAMOS might not that helpful for the situation, and > > > > > this record feature could be useful for your case. > > > > > > > > > > That said, do you really need full recording of the monitoring results? If > > > > > not, DAMOS provides DAMOS tried regions feature[1], which allows users get the > > > > > monitoring results snapshot that include both frequency and recency of all > > > > > regions in an efficient way. If single snapshot is not having enough > > > > > information for you, you could collect multiple snapshots. > > > > > > > > > > You mentioned absence of Python on Android as a blocker of DAMOS use on the > > > > > previous reply[2], but DAMOS tried regions feature is not depend on tracepoints > > > > > or Python. > > > > > > > > > > Of course, I think you might already surveyed it but found some problems. > > > > > Could you please share that in detail if so? > > > > > > > > > DAMOS tried regions feature you mentioned is not fully applicable. It needs to > > > > apply schemes on regions. There is no available scheme we can use for our use > > > > case. What we need is to return regions with access frequency and recency to > > > > userspace for later use. > > > > > > > > > Thank you for the answer, I understand your concern. One of the available > > > DAMOS action is 'stat'[1], which does nothing but just count the statistic. > > > Using DAMOS scheme for any access pattern with 'stat' action, you can extract > > > the access pattern via DAMOS tried regions feature of DAMON sysfs interface, > > > without making any unnecessary impact to the workload. > > > > > > Quote from [2]: > > > > > > The expected usage of this directory is investigations of schemes' behaviors, > > > and query-like efficient data access monitoring results retrievals. For the > > > latter use case, in particular, users can set the action as stat and set the > > > access pattern as their interested pattern that they want to query. > > > > > > For example, you could > > > > > > # cd /sys/kernel/mm/damon/admin > > > # > > > # # populate directories > > > # echo 1 > kdamonds/nr_kdamonds; echo 1 > kdamonds/0/contexts/nr_contexts; > > > # echo 1 > kdamonds/0/contexts/0/schemes/nr_schemes > > > # cd kdamonds/0/contexts/0/schemes/0 > > > # > > > # # set the access pattern for any case (max as 2**64 - 1), and action as stat > > > # echo 0 > access_pattern/sz/min > > > # echo 18446744073709551615 > access_pattern/sz/max > > > # echo 0 > access_pattern/nr_accesses/min > > > # echo 18446744073709551615 > access_pattern/nr_accesses/max > > > # echo 0 > access_pattern/age/min > > > # echo 18446744073709551615 > access_pattern/age/max > > > # echo stat > action > > > > > > And this is how DAMON user-space tool is getting the snapshot with 'damo show' > > > command[3]. > > > > > > Could this be used for your case? Please ask any question if you have :) > > > > > > [1] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n > > > [2] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n-tried-regions, > > > [3] https://github.com/awslabs/damo/blob/next/USAGE.md#damo-show > > > > Thank you for your detailed response, it is very helpful to us. We will look into it > > and contact you if we have any questions. > > So glad to hear this. Please let me know if you have any questions or need any > help :) > > > Thanks, > SJ > > > > > > > > > > > > Thanks, > > > SJ > > > > > > > > [1] https://docs.kernel.org/admin-guide/mm/damon/usage.html#schemes-n-tried-regions > > > > > [2] https://lore.kernel.org/damon/20231129131315.GB12957@cuiyangpei/ > > > > > > > > > > > > > > > Thanks, > > > > > SJ > > > > > > > > > > > > > > > > > Thanks. Hi SeongJae, We set 'access_pattern' and 'stat' action in schemes when apps are on foreground, record apps' memory access pattern when they are switched to the background with 'update_schemes_tried_regions' state. But it catch the snapshot after next aggregation interval. DAMON is still sampling during the app switches to the background and the next aggregation time, which can cause the value of "age" to change. The sampling results during this period cannot accurately reflect the app's foreground access pattern. Is there any way to catch sampling result immediately after setting the "update_schemes_tried_regions" state? Alternatively, can it return the "last_nr_accesses" and "last_age" values in tried_regions/ directory? Do you have any other suggestions? Thanks.