From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00704C2D0F1 for ; Wed, 1 Apr 2020 01:23:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B881B208E4 for ; Wed, 1 Apr 2020 01:23:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hy3d+uVW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B881B208E4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 487C26B0032; Tue, 31 Mar 2020 21:23:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 410406B0037; Tue, 31 Mar 2020 21:23:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8516B006C; Tue, 31 Mar 2020 21:23:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 0F2526B0032 for ; Tue, 31 Mar 2020 21:23:03 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B8279181AEF00 for ; Wed, 1 Apr 2020 01:23:02 +0000 (UTC) X-FDA: 76657537404.03.owl89_1a85af5e8c015 X-HE-Tag: owl89_1a85af5e8c015 X-Filterd-Recvd-Size: 7981 Received: from mail-io1-f68.google.com (mail-io1-f68.google.com [209.85.166.68]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Wed, 1 Apr 2020 01:23:02 +0000 (UTC) Received: by mail-io1-f68.google.com with SMTP id x9so17443069iom.10 for ; Tue, 31 Mar 2020 18:23:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Z2QHlAG4R2a3GsBY1Sh9yeXpBxrWZUKwyn4p8dfZUCs=; b=hy3d+uVWn9fBJjcef03m/QD3vQIZruQ5Af9+40C1Mcybj3hYYW6OtnmNQ6K0Hv1aAf F3+do+pXWHYNusCQv7vpnxT3natkYX6QxYaCc7zJu1EC1skVIIx90eEQlw/t2z/kMYjR HYRxlMlJNZwltiYoW/WwlzDoxi9elcsxQFbXCM+II/ON9tahA72/PLhZ0w3w03oWYLKx +Zzj/SzRSANY0Kh1WciHZiZxfrHZNUhubzoE6IqAugYnttlQXeT0p6Q9SQ04z9EUDMEf DB6CmvhhLRnG1gK2nUHGV/iZdOVvVgDJY4BVON6sTQ5dQbxVnix+1S1cJmZj8qJApNiv gexA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Z2QHlAG4R2a3GsBY1Sh9yeXpBxrWZUKwyn4p8dfZUCs=; b=UKuA8BRHj9Rk85b7/0A+r+5H7JiRXvRp07ZpC3uOhAMVTTF3H7OQgg18kTrNt8PxQm uWSJSJP2jLxTzpIIJYHxY309uVJ919kQgMjV91xpIssggDydPe6TAywqCdFNSrgbKa4A jcSPEH+z3QD9qw0pI50eqgQYEDULwSy3avNjtTJ3P0sZaEBgyfCGUzA+kkEcAuGKS2IL CW/Zw34CxGaZdlUJkhLyNJE69VmU7a8FualAnGIBMHgnpsSO1eKf/Dp7LwOvAJQawTZj HZEfSV+M3dlwk56c/nxADImvB+iP6VciEBdMNXptyWXxkAAqYM7rILHCAjEFr84UtgtN hPJg== X-Gm-Message-State: ANhLgQ1qIRyrpF+dlrg5CG7MspxkCEX/XBgaWq9QszpLS+OycfRYonCF v/D7UjDb+QoNCgboASwrGKGA94g5bmdRKbZVfZs= X-Google-Smtp-Source: ADFU+vua/Mghg3wP2uz1d4+N70EOBTxAQsg9C+j/cD+lu2bR+th4xx/nLCk8XZ+P+5+zM8C5J7zXhb5i/jj3kAVC0OI= X-Received: by 2002:a6b:7903:: with SMTP id i3mr9107365iop.66.1585704181384; Tue, 31 Mar 2020 18:23:01 -0700 (PDT) MIME-Version: 1.0 References: <1585221127-11458-1-git-send-email-laoar.shao@gmail.com> <20200326143102.GB342070@cmpxchg.org> <20200331151103.GB2089@cmpxchg.org> In-Reply-To: <20200331151103.GB2089@cmpxchg.org> From: Yafang Shao Date: Wed, 1 Apr 2020 09:22:24 +0800 Message-ID: Subject: Re: [PATCH 0/2] psi: enhance psi with the help of ebpf To: Johannes Weiner Cc: Peter Zijlstra , Andrew Morton , Michal Hocko , Jens Axboe , mgorman@suse.de, Steven Rostedt , mingo@redhat.com, Linux MM , linux-block@vger.kernel.org, LKML Content-Type: text/plain; charset="UTF-8" X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Mar 31, 2020 at 11:11 PM Johannes Weiner wrote: > > On Fri, Mar 27, 2020 at 09:17:59AM +0800, Yafang Shao wrote: > > On Thu, Mar 26, 2020 at 10:31 PM Johannes Weiner wrote: > > > > > > On Thu, Mar 26, 2020 at 07:12:05AM -0400, Yafang Shao wrote: > > > > PSI gives us a powerful way to anaylze memory pressure issue, but we can > > > > make it more powerful with the help of tracepoint, kprobe, ebpf and etc. > > > > Especially with ebpf we can flexiblely get more details of the memory > > > > pressure. > > > > > > > > In orderc to achieve this goal, a new parameter is added into > > > > psi_memstall_{enter, leave}, which indicates the specific type of a > > > > memstall. There're totally ten memstalls by now, > > > > MEMSTALL_KSWAPD > > > > MEMSTALL_RECLAIM_DIRECT > > > > MEMSTALL_RECLAIM_MEMCG > > > > MEMSTALL_RECLAIM_HIGH > > > > MEMSTALL_KCOMPACTD > > > > MEMSTALL_COMPACT > > > > MEMSTALL_WORKINGSET_REFAULT > > > > MEMSTALL_WORKINGSET_THRASHING > > > > MEMSTALL_MEMDELAY > > > > MEMSTALL_SWAPIO > > > > > > What does this provide over the events tracked in /proc/vmstats? > > > > > > > /proc/vmstat only tells us which events occured, but it can't tell us > > how long these events take. > > Sometimes we really want to know how long the event takes and PSI can > > provide us the data > > For example, in the past days when I did performance tuning for a > > database service, I monitored that the latency spike is related with > > the workingset_refault counter in /proc/vmstat, and at that time I > > really want to know the spread of latencies caused by > > workingset_refault, but there's no easy way to get it. Now with newly > > added MEMSTALL_WORKINGSET_REFAULT, I can get the latencies caused by > > workingset refault. > > Okay, but how do you use that information in practice? > With the newly added facility, we can know when these events occur and how long each event takes. Then we can use these datas to generate a Latency Heat Map[1] and to compare whether these latencies match with the application latencies recoreded in its log - for example the slow query log in mysql. If the refault latencies match with the slow query log, then these slow queries are caused by these workingset refault. If the refault latencies don't match with slow query log, IOW much smaller than the slow query log, then the slow query log isn't caused by the workingset refault. High rate of refaults may not cause high pressure, if the backing device is fast enough. While the latencies of refaults give us a direct relationship with memory pressure. [1]. http://www.brendangregg.com/HeatMaps/latency.html > > > Can you elaborate a bit how you are using this information? It's not > > > quite clear to me from the example in patch #2. > > > > > > > From the traced data in patch #2, we can find that the high latencies > > of user tasks are always type 7 of memstall , which is > > MEMSTALL_WORKINGSET_THRASHING, and then we should look into the > > details of wokingset of the user tasks and think about how to improve > > it - for example, by reducing the workingset. > > That's an analyses we run frequently as well: we see high pressure, > and then correlate it with the events. > > High rate of refaults? The workingset is too big. > > High rate of compaction work? Somebody is asking for higher order > pages under load; check THP events next. > > etc. > > This works fairly reliably. I'm curious what the extra per-event > latency breakdown would add and where it would be helpful. > > I'm not really opposed to your patches it if it is, I just don't see > the usecase right now. > As I explained above, the rate of these events can't give us a full view of the memory pressure. High memory pressure may not caused by high rate of vmstat events, while it can be caused by low rate of events but with high latencies. Latencies are the application really concerns, that's why PSI is very useful for us. Furthermore, there're some events are not recored in vmstat. e.g. typf of memstall vmstat event MEMSTALL_KSWAPD pageoutrun, {pgscan, pgsteal}_kswapd MEMSTALL_RECLAIM_DIRECT {pgscan,steal}_direct MEMSTALL_RECLAIM_MEMCG /* no event */ MEMSTALL_RECLAIM_HIGH /* no event */ MEMSTALL_KCOMPACTD compact_daemon_wake MEMSTALL_COMPACT compact_{stall, fail, success} MEMSTALL_WORKINGSET_REFAULT workingset_refault MEMSTALL_WORKINGSET_THRASH /* no event */ MEMSTALL_MEMDELAY /* no event */ MEMSTALL_SWAPIO pswpin After we add these types of memstall, we don't need to add these missed events one by one. Thanks Yafang