Re: [Ksummit-discuss] [TECH TOPIC] Kernel tracing and end-to-end performance breakdown

[Chris Mason <clm@fb.com>]

On 07/21/2016 11:45 AM, Jan Kara wrote:
> On Thu 21-07-16 09:54:53, Chris Mason wrote:
>> On 07/21/2016 06:00 AM, Jan Kara wrote:
>>>
>>> So I think improvements in performance analysis are always welcome but
>>> current proposal seems to be somewhat handwavy so I'm not sure what outcome
>>> you'd like to get from the discussion... If you have a more concrete
>>> proposal how you'd like to achieve what you need, then it may be worth
>>> discussion.
>>>
>>> As a side note I know that Google (and maybe Facebook, not sure here) have
>>> out-of-tree patches which provide really neat performance analysis
>>> capabilities. I have heard they are not really upstreamable because they
>>> are horrible hacks but maybe they can be a good inspiration for this work.
>>> If we could get someone from these companies to explain what capabilities
>>> they have and how they achieve this (regardless how hacky the
>>> implementation may be), that may be an interesting topic.
>>
>> At least for facebook, we're moving most things to bpf.  The most
>> interesting part of our analysis isn't so much from the tool used to record
>> it, it's from being able to aggregate over the fleet and making comparisons
>> at scale.
>>
>> For example, Josef setup the off-cpu flame graphs such that we can record
>> stack traces for a latency higher than N, and then sum up the most expensive
>> stack traces over a large number of machines.  It makes it much easier to
>> find those happens-once-a-day problems.
>
> By latency higher than N, do you mean that e.g. a syscall took more than N,
> or just that a process is sleeping for more than N in some place?

Single sleep longer than N.  It would be a little more involved to track 
all the sleeps in a single syscall, but we haven't needed to (yet).

-chris