From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.6 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79199C433E2 for ; Wed, 15 Jul 2020 16:36:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 38B7920658 for ; Wed, 15 Jul 2020 16:36:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="izvgTdok" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38B7920658 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C7086B0006; Wed, 15 Jul 2020 12:36:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 976AD6B0007; Wed, 15 Jul 2020 12:36:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 864D16B0008; Wed, 15 Jul 2020 12:36:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id 705786B0006 for ; Wed, 15 Jul 2020 12:36:55 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D0F4D1EE6 for ; Wed, 15 Jul 2020 16:36:54 +0000 (UTC) X-FDA: 77040864348.28.fold10_580709626efb Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin28.hostedemail.com (Postfix) with ESMTP id A57346C09 for ; Wed, 15 Jul 2020 16:36:54 +0000 (UTC) X-HE-Tag: fold10_580709626efb X-Filterd-Recvd-Size: 6667 Received: from mail-lj1-f194.google.com (mail-lj1-f194.google.com [209.85.208.194]) by imf43.hostedemail.com (Postfix) with ESMTP for ; Wed, 15 Jul 2020 16:36:54 +0000 (UTC) Received: by mail-lj1-f194.google.com with SMTP id q4so3360783lji.2 for ; Wed, 15 Jul 2020 09:36:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=eCQSS5Z/XAuJJ0ObjR5nRfl8x7vpuq0gBBiMlc5n388=; b=izvgTdokwsQbPkEHKbGYg+vgL3PLbT710ndTtF+lw0QVNhZPDQQjkr8aN4juyq1ITY yQaBTbw4G262maEKv7mI6zKBmKapyRz7IN5soCyNo1VZQX2zOEPVC7DSLWN/JmKEwdA5 lI8Oeb9+QNbM582nfIuOILLRmXR0wXZ8dtVMWeD5p77C+CgAO3F/4N3Oz+huCpu3K9Vc OTktHziUeHCrwlIub3z5XnFu/xigyh3HdjfxCrAXVzW9vuSRzKqfZ0DvNEKYGC28/++q gYM+2CZgooNe3NtWog1NoDMOmCYr8FS9mjj8RmfHzJ1TEOktmvCLP6r3NgZXZhU/gB35 sxGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=eCQSS5Z/XAuJJ0ObjR5nRfl8x7vpuq0gBBiMlc5n388=; b=REhLUziFpoORuTTpa3lAcwIUDdI0DaIvh0lAoKMSrhBmDyZT/EL5A/MBIOXrX+aoAS yNENcCBIGZDErLGPKuJ0ZyTTqTM2CpMugHYejBkH3eK6pjHnqnVQSJkFK+YB/BVSVSnQ I4fUsNqUOFw2J1QlvEbQaeREBS5e1tglWfWRn+U+2b7eD1qosINnW1nTFE/ya0uGC9di yMLJiS0ZBGD5c/fiDThWRpqL1+sXQ+uMF/3geT34PGFeY7QHPIO67DkJy4kfiwdll6kX m9veZgQp1Z+/0pOc5nTOF0G+5In/zB9ZWMT0Tel9gUkU/Ustvk6Ho/ZJVi9LKqO6c0mM CXkQ== X-Gm-Message-State: AOAM533R6Q8BhhkU09eBv8vBjklr38rrXXX0dRBfaIS6LrgOi1/UMFoc ji4skvdHmFnCrGZ0YF6dqAqVS7j8wA7zVOgO0hpTtA== X-Google-Smtp-Source: ABdhPJxkvCC8/QXGjTsKModznq9Sitpzv8TpZrU2xepInx6yXDiYQJ55n1/Rgdh/Jkbnh3Eg86yNuwgRgkS6wU5FZEc= X-Received: by 2002:a2e:9585:: with SMTP id w5mr24703ljh.58.1594831012200; Wed, 15 Jul 2020 09:36:52 -0700 (PDT) MIME-Version: 1.0 References: <1585649077-10896-1-git-send-email-laoar.shao@gmail.com> In-Reply-To: <1585649077-10896-1-git-send-email-laoar.shao@gmail.com> From: Shakeel Butt Date: Wed, 15 Jul 2020 09:36:41 -0700 Message-ID: Subject: Re: [PATCH v2 0/2] psi: enhance psi with the help of ebpf To: Yafang Shao Cc: Johannes Weiner , "Peter Zijlstra (Intel)" , Andrew Morton , Linux MM , "open list:BLOCK LAYER" , LKML Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: A57346C09 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000060, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Yafang, On Tue, Mar 31, 2020 at 3:05 AM Yafang Shao wrote: > > PSI gives us a powerful way to anaylze memory pressure issue, but we can > make it more powerful with the help of tracepoint, kprobe, ebpf and etc. > Especially with ebpf we can flexiblely get more details of the memory > pressure. > > In orderc to achieve this goal, a new parameter is added into > psi_memstall_{enter, leave}, which indicates the specific type of a > memstall. There're totally ten memstalls by now, > MEMSTALL_KSWAPD > MEMSTALL_RECLAIM_DIRECT > MEMSTALL_RECLAIM_MEMCG > MEMSTALL_RECLAIM_HIGH > MEMSTALL_KCOMPACTD > MEMSTALL_COMPACT > MEMSTALL_WORKINGSET_REFAULT > MEMSTALL_WORKINGSET_THRASH > MEMSTALL_MEMDELAY > MEMSTALL_SWAPIO > With the help of kprobe or tracepoint to trace this newly added agument we > can know which type of memstall it is and then do corresponding > improvement. I can also help us to analyze the latency spike caused by > memory pressure. > > But note that we can't use it to build memory pressure for a specific type > of memstall, e.g. memcg pressure, compaction pressure and etc, because it > doesn't implement various types of task->in_memstall, e.g. > task->in_memcgstall, task->in_compactionstall and etc. > > Although there're already some tracepoints can help us to achieve this > goal, e.g. > vmscan:mm_vmscan_kswapd_{wake, sleep} > vmscan:mm_vmscan_direct_reclaim_{begin, end} > vmscan:mm_vmscan_memcg_reclaim_{begin, end} > /* no tracepoint for memcg high reclaim*/ > compcation:mm_compaction_kcompactd_{wake, sleep} > compcation:mm_compaction_begin_{begin, end} > /* no tracepoint for workingset refault */ > /* no tracepoint for workingset thrashing */ > /* no tracepoint for use memdelay */ > /* no tracepoint for swapio */ > but psi_memstall_{enter, leave} gives us a unified entrance for all > types of memstall and we don't need to add many begin and end tracepoints > that hasn't been implemented yet. > > Patch #2 gives us an example of how to use it with ebpf. With the help of > ebpf we can trace a specific task, application, container and etc. It also > can help us to analyze the spread of latencies and whether they were > clustered at a point of time or spread out over long periods of time. > > To summarize, with the pressure data in /proc/pressure/memroy we know that > the system is under memory pressure, and then with the newly added tracing > facility in this patchset we can get the reason of this memory pressure, > and then thinks about how to make the change. > The workflow can be illustrated as bellow. > > REASON ACTION > | compcation | improve compcation | > | vmscan | improve vmscan | > Memory pressure -| workingset | improve workingset | > | etc | ... | > I have not looked at the patch series in detail but I wanted to get your thoughts if it is possible to achieve what I am trying to do with this patch series. At the moment I am only interested in global reclaim and I wanted to enable alerts like "alert if there is process stuck in global reclaim for x seconds in last y seconds window" or "alert if all the processes are stuck in global reclaim for some z seconds". I see that using this series I can identify global reclaim but I am wondering if alert or notifications are possible. Android is using psi monitors for such alerts but it does not use cgroups, so, most of the memstalls are related to global reclaim stall. For cgroup environment, do we need for add support to psi monitor similar to this patch series? thanks, Shakeel