Re: [Ksummit-discuss] [TECH TOPIC] Kernel tracing and end-to-end performance breakdown

[Jan Kara <jack@suse.cz>]

On Thu 21-07-16 09:54:53, Chris Mason wrote:
> On 07/21/2016 06:00 AM, Jan Kara wrote:
> >
> >So I think improvements in performance analysis are always welcome but
> >current proposal seems to be somewhat handwavy so I'm not sure what outcome
> >you'd like to get from the discussion... If you have a more concrete
> >proposal how you'd like to achieve what you need, then it may be worth
> >discussion.
> >
> >As a side note I know that Google (and maybe Facebook, not sure here) have
> >out-of-tree patches which provide really neat performance analysis
> >capabilities. I have heard they are not really upstreamable because they
> >are horrible hacks but maybe they can be a good inspiration for this work.
> >If we could get someone from these companies to explain what capabilities
> >they have and how they achieve this (regardless how hacky the
> >implementation may be), that may be an interesting topic.
> 
> At least for facebook, we're moving most things to bpf.  The most
> interesting part of our analysis isn't so much from the tool used to record
> it, it's from being able to aggregate over the fleet and making comparisons
> at scale.
> 
> For example, Josef setup the off-cpu flame graphs such that we can record
> stack traces for a latency higher than N, and then sum up the most expensive
> stack traces over a large number of machines.  It makes it much easier to
> find those happens-once-a-day problems.

By latency higher than N, do you mean that e.g. a syscall took more than N,
or just that a process is sleeping for more than N in some place?

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR