[Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Mathieu Desnoyers]

Our attempts at extending the Linux scheduler Tracepoints [1,2,3] to cover sched
rt and deadline policies went fairly well until we reached the point where
we had to extend the tracing ABI exposed to user-space by Ftrace through the
DebugFS tracing sub-directory. We aim to expose more accurate and complete
scheduler information (based on the scheduling class), and eventually deprecate
implementation-specific fields that should never have been exposed to user-space
in the first place.

Example of problems we are facing:

  * Humans on the receiving end of a kernel ABI

    Should we limit the design of kernel ABIs based on their direct
    use by humans through echo and redirections, or can we aim design
    of those interfaces at scripts and user-space programs/libraries ?

    A simple and extensible kernel ABI is rarely an easy to use end-user
    interface.

  * How can we deprecate, remove, or re-purpose a field in an
    event ? For instance, the "prio" field in the scheduler
    instrumentation is an internal implementation detail.

Perhaps it would be good to use the opportunity to meet at the
Kernel Summit and discuss this issue.

Thanks,

Mathieu and Julien


[1] [RFC PATCH 5/6] tracing: extend scheduling tracepoints
    http://lkml.kernel.org/r/1474060148-13171-5-git-send-email-jdesfossez@efficios.com
[2] [RFC PATCH v2 0/5] Additional scheduling information in tracepoints
    http://lkml.kernel.org/r/1474649375-28056-1-git-send-email-jdesfossez@efficios.com
[3] [RFC PATCH v3 0/2] Extend scheduling tracepoints
    http://lkml.kernel.org/r/1484327993-5036-1-git-send-email-jdesfossez@efficios.com

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 21:20:54 +0000 (UTC)
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:

> Our attempts at extending the Linux scheduler Tracepoints [1,2,3] to cover sched
> rt and deadline policies went fairly well until we reached the point where
> we had to extend the tracing ABI exposed to user-space by Ftrace through the
> DebugFS tracing sub-directory. We aim to expose more accurate and complete
> scheduler information (based on the scheduling class), and eventually deprecate
> implementation-specific fields that should never have been exposed to user-space
> in the first place.
> 
> Example of problems we are facing:
> 
>   * Humans on the receiving end of a kernel ABI
> 
>     Should we limit the design of kernel ABIs based on their direct
>     use by humans through echo and redirections, or can we aim design
>     of those interfaces at scripts and user-space programs/libraries ?

Well, that human is mainly Peter ;-)

But we also can't break tools that user the sched_switch tracepoint,
namely trace-cmd and perf, as well as powertop.

> 
>     A simple and extensible kernel ABI is rarely an easy to use end-user
>     interface.
> 
>   * How can we deprecate, remove, or re-purpose a field in an
>     event ? For instance, the "prio" field in the scheduler
>     instrumentation is an internal implementation detail.

One way is to fix all tools that use it and make sure they get out to
the distros before making the change.

> 
> Perhaps it would be good to use the opportunity to meet at the
> Kernel Summit and discuss this issue.

Agreed. In this topic, I would like to focus more on extending, or
implementations on overlaying tracepoints where one tracepoint exists
in the code, but there's multiple ways to display it.

-- Steve


> 
> Thanks,
> 
> Mathieu and Julien
> 
> 
> [1] [RFC PATCH 5/6] tracing: extend scheduling tracepoints
>     http://lkml.kernel.org/r/1474060148-13171-5-git-send-email-jdesfossez@efficios.com
> [2] [RFC PATCH v2 0/5] Additional scheduling information in tracepoints
>     http://lkml.kernel.org/r/1474649375-28056-1-git-send-email-jdesfossez@efficios.com
> [3] [RFC PATCH v3 0/2] Extend scheduling tracepoints
>     http://lkml.kernel.org/r/1484327993-5036-1-git-send-email-jdesfossez@efficios.com
> 

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 4:55 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>>   * How can we deprecate, remove, or re-purpose a field in an
>>     event ? For instance, the "prio" field in the scheduler
>>     instrumentation is an internal implementation detail.
>
> One way is to fix all tools that use it and make sure they get out to
> the distros before making the change.

OR DO THE THING THAT PEOPLE HAVE BEEN TOLD TO DO AT LEAST THREE KERNEL
SUMMITS NOW: LEAVE THE DAMN FIELD ALONE, AND FILL IT WITH ZERO. OR
ONE. OR BRAN MUFFINS. I DON'T CARE. BUT DON'T REMOVE IT, AND STOP
USING IT AS AN EXCUSE FOR WHY NOTHING CAN EVER BE DONE.

Really. I don't want to have this stupid tracing discussion one more
time. We've had it. Several times. This exact issue has come up.
Several times.

So stop wasting everybodys time one more year.  I'm going to walk out
if people start discussing this thing again.

             Linus "grumpy"

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 17:03:05 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Thu, Jun 29, 2017 at 4:55 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> >>
> >>   * How can we deprecate, remove, or re-purpose a field in an
> >>     event ? For instance, the "prio" field in the scheduler
> >>     instrumentation is an internal implementation detail.  
> >
> > One way is to fix all tools that use it and make sure they get out to
> > the distros before making the change.  
> 
> OR DO THE THING THAT PEOPLE HAVE BEEN TOLD TO DO AT LEAST THREE KERNEL
> SUMMITS NOW: LEAVE THE DAMN FIELD ALONE, AND FILL IT WITH ZERO. OR
> ONE. OR BRAN MUFFINS. I DON'T CARE. BUT DON'T REMOVE IT, AND STOP
> USING IT AS AN EXCUSE FOR WHY NOTHING CAN EVER BE DONE.

Well, we actually were able to in the past remove a field after getting
the one user up to date (powertop) remember? I fixed powertop, waited a
few years until the fix was in Debian stable, and then removed the
field. Nobody noticed.  I thought that was the point. If user space
breaks, and nobody is around to complain about it, did it really break?

The reason that was important to remove, is that it was a field in
*every* tracepoint. It was only 4 bytes, but when you have 4 million
tracepoints in the buffers, that's 4 megs of memory wasted (a normal
tracepoint is about 24 bytes, which makes 4 bytes a big percentage).
It's similar to wasted fields in the page struct. It bloats up fast.

> 
> Really. I don't want to have this stupid tracing discussion one more
> time. We've had it. Several times. This exact issue has come up.
> Several times.

This is actually something quite different, and new. It sounds similar,
but its not.

I should have been the one to post the topic, because what Mathieu
wrote, makes it sound very much like what we've discussed to death in
the past.

What we use to talk about at ksummit was about stable ABIs and such.
How to get new tracepoints into the kernel subsystems like the file
system and not worry that these tracepoints will cause harm later to
development. THAT IS NOT WHAT WE ARE TALKING ABOUT NOW. (just to get
your attention ;-)

> 
> So stop wasting everybodys time one more year.  I'm going to walk out
> if people start discussing this thing again.

Here's what the new issue is. We have a single tracepoint in the
scheduler that denotes sched switch. It currently looks like this:

name: sched_switch
ID: 287
format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;	offset:3;	size:1;	signed:0;
	field:int common_pid;	offset:4;	size:4;	signed:1;

	field:char prev_comm[16];	offset:8;	size:16;	signed:1;
	field:pid_t prev_pid;	offset:24;	size:4;	signed:1;
	field:int prev_prio;	offset:28;	size:4;	signed:1;
	field:long prev_state;	offset:32;	size:8;	signed:1;
	field:char next_comm[16];	offset:40;	size:16;	signed:1;
	field:pid_t next_pid;	offset:56;	size:4;	signed:1;
	field:int next_prio;	offset:60;	size:4;	signed:1;

The issue is that we now have a new scheduling class called
SCHED_DEADLINE, were prio is completely useless. We would like to add
the dynamic fields of "remaining runtime", "next deadline", "next
period".

Now sched_switch is also one of the most commonly used tracepoints, as
it lets a user see what preempts their process, what system services
are running and for how long, etc etc. The thing is, we don't want to
bloat that tracepoint. Adding fields for a scheduling class that is
used by a very small niche class, is a waste for everyone else.

One of the ideas I've had is to allow for "overlays". That is, we don't
want to add another trace_sched_switch() in the scheduler, as that will
add a little more overhead to the normal non tracing case. Thus, since
we already have that hook (the trace_sched_switch) it would be good to
tap into it, and have another way to extract more data from the
tracepoint. That is, the overlay.

The problem we have is how to implement it?

We could make one tracepoint hook location have several different
"tracepoints" in the tracefs directory letting the user choose how much
information they want to trace. Have different tracepoints that can be
enabled for a single location, where it may show extended fields.

I know people would like to have a way to cut down some fields, as
real-estate in the ring buffer is of high value, and the smaller the
events are, the more data one can collect. People who use tracing
really do care about any wasted space (which is why we like to avoid
writing zeros in fields no longer valued, it makes it harder to get the
data you are after).

In summary, this is not another beat the dead horse how to do stable
tracepoints. The focus is, how to make tracepoints more user
customizable for their use cases.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 5:32 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> This is actually something quite different, and new. It sounds similar,
> but its not.

No, I remember the particular scheduler tracepoint discussion.

And the answer remains exactly the same. If some internal detail no
longer exists, just zero out the field.

And no, don't add new, even more specialized fields.

Just get over this discussion already.

              Linus

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 17:41:29 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Thu, Jun 29, 2017 at 5:32 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > This is actually something quite different, and new. It sounds similar,
> > but its not.  
> 
> No, I remember the particular scheduler tracepoint discussion.
> 
> And the answer remains exactly the same. If some internal detail no
> longer exists, just zero out the field.
> 
> And no, don't add new, even more specialized fields.
> 
> Just get over this discussion already.
> 

How does this solve adding key information that lots of people want
added. SCHED_DEADLINE is new, and lots of people want to use the
tracing infrastructure to analyze their applications. I know when I was
at Red Hat, we had our own tracepoints that we added to our kernel to
do just that. Right now, everyone has to patch their kernel to get the
necessary information. Red Hat isn't the only one. Mathieu is doing it
too. Perhaps SuSE is as well.

It doesn't solve anything.

Basically, what you are saying is the opposite of what we feared might
happen. People have always been afraid that a tracepoint might inhibit
kernel development because of some key information it exposes. Now you
are saying, don't touch the tracepoint, and just let it rot. Now the
usefulness of a tracepoint has become limited due to advancements in
the development of the kernel.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 20:32:24 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:


> The problem we have is how to implement it?
> 
> We could make one tracepoint hook location have several different
> "tracepoints" in the tracefs directory letting the user choose how much
> information they want to trace. Have different tracepoints that can be
> enabled for a single location, where it may show extended fields.
> 
> I know people would like to have a way to cut down some fields, as
> real-estate in the ring buffer is of high value, and the smaller the
> events are, the more data one can collect. People who use tracing
> really do care about any wasted space (which is why we like to avoid
> writing zeros in fields no longer valued, it makes it harder to get the
> data you are after).
> 
> In summary, this is not another beat the dead horse how to do stable
> tracepoints. The focus is, how to make tracepoints more user
> customizable for their use cases.

Just to explain what Mathieu was talking about with echo and such, is
that Peter Zijlstra has been against multiple tracepoints for that one
sched switch location. He would prefer a dynamic case, instead of
having to enable all tracepoints, to get full functionality, as he
tends to use echo / cat to interact with ftrace than by using the
tools. He doesn't want the hassle of enabling more than one tracepoint
for sched_switch. One could call him lazy ;) but he's also the
maintainer of that tracepoint!

What would be really nice is a way to have sched_switch be a dynamic
tracepoint where it can trace differently depending on what is being
scheduled in and out. I'm sure other tracepoints locations may want a
similar feature.

But the devil is in the details of implementation, and we need to do
all this without breaking the existing tools that use the tracing
system. (See I have been listening to you!)

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 5:52 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Just to explain what Mathieu was talking about with echo and such, is
> that Peter Zijlstra has been against multiple tracepoints for that one
> sched switch location.

I am too.

Dammit, if somebody cares about one partiocular scheduler, then that
person can add dynamic tracepoints.

Leave the existing one alone. Really. Zero out any fields that no
longer make sense. Really. Don't beat this damn horse again. It's been
dead for three years, and it's not just smelling bad, it's bloating in
some scary ways.

The only reason for static tracepoints are for major tools like
powertop. There is no way in hell such a tool will care about fields
that only exist for one particular scheduler implementation. Don't add
new random crap.

If somebody is interested in *one* particular odd low-level scheduler,
he damn well can add the dynamic points.

This is the last I want to ever hear about it, and I particularly do
not want to have this be a kernel summit discussion. We've had it
before. Get over it.

                Linus

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 18:00:42 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Thu, Jun 29, 2017 at 5:52 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Just to explain what Mathieu was talking about with echo and such, is
> > that Peter Zijlstra has been against multiple tracepoints for that one
> > sched switch location.  
> 
> I am too.

I didn't realize you even used the tracepoints.

> 
> Dammit, if somebody cares about one partiocular scheduler, then that
> person can add dynamic tracepoints.

You mean kprobes? Or perhaps eBPF?

The information about SCHED_DEADLINE is not trivial enough to extract
with them.

> 
> Leave the existing one alone. Really. Zero out any fields that no
> longer make sense. Really. Don't beat this damn horse again. It's been
> dead for three years, and it's not just smelling bad, it's bloating in
> some scary ways.
> 
> The only reason for static tracepoints are for major tools like
> powertop. There is no way in hell such a tool will care about fields
> that only exist for one particular scheduler implementation. Don't add
> new random crap.

Well, the world does have people that use tools besides powertop.

> 
> If somebody is interested in *one* particular odd low-level scheduler,
> he damn well can add the dynamic points.

Again, not a trivial task. It's much easier to patch the kernel. Which,
I guess is what will be needed from now on.

> 
> This is the last I want to ever hear about it, and I particularly do
> not want to have this be a kernel summit discussion. We've had it
> before. Get over it.

OK, this is my last email on the subject. Too bad you feel this way.

Just one last note. I've tried very hard to keep tracing as contained
as possible. That is, not to let implementation details and such creep
into the rest of the kernel. I worked on making it as robust as
possible. All solutions to this had one major requirement. That is, it
must be contained, and not something that would cause any disruption in
any other part of the kernel or even other tracepoints. But oh well,
this idea is now dead. Well, at least until the demand for it boils up
into your visibility.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

I know I said that was going to be my last email, but I just noticed
something by re-reading the thread.

On Thu, 29 Jun 2017 21:16:41 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> > On Thu, Jun 29, 2017 at 5:52 PM, Steven Rostedt <rostedt@goodmis.org> wrote:  
> > >
> > > Just to explain what Mathieu was talking about with echo and such, is
> > > that Peter Zijlstra has been against multiple tracepoints for that one
> > > sched switch location.    
> > 
> > I am too.  
> > 
> > Dammit, if somebody cares about one partiocular scheduler, then that
> > person can add dynamic tracepoints.  
> 
> You mean kprobes? Or perhaps eBPF?

I missed this because you cut it out. What I originally wrote:

  Just to explain what Mathieu was talking about with echo and such, is
  that Peter Zijlstra has been against multiple tracepoints for that one
  sched switch location. He would prefer a dynamic case, instead of
  having to enable all tracepoints, to get full functionality, as he
  tends to use echo / cat to interact with ftrace than by using the
  tools. He doesn't want the hassle of enabling more than one tracepoint
  for sched_switch.


Where I mentioned that he prefers a dynamic tracepoint. Is that what
you mean? Because what he prefers doesn't exist yet. It's not kprobes
nor eBPF. It is something we've been talking about implementing, and we
were going to discuss the implementation at ksummit.

Are you OK with that?

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 6:27 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Where I mentioned that he prefers a dynamic tracepoint. Is that what
> you mean? Because what he prefers doesn't exist yet. It's not kprobes
> nor eBPF. It is something we've been talking about implementing, and we
> were going to discuss the implementation at ksummit.
>
> Are you OK with that?

I'm not going to be at that discussion, and I don't think it should be
a ksummit thing, at least not the maintainership part. It's a
technical detail that should be discussed with the people who are
interested in that technical detail. Maybe a BOF?

But yes, I was talking about something very similar to what I think
Peter is talking about - the ability to attach a ebpf script to
kprobes and extract data dynamically. We've supported ebpf tracepoints
for years afaik, what is actually missing from using that for whatever
particular extension people want to use?

Wasn't that the whole promise (and premise) of using ebpf programs in
tracing? Exactly the ability to generate sane statistics and traces
dynamically. I know that was what I was sold on, even if it might not
actually have worked out that way.

                 Linus

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 18:51:14 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Thu, Jun 29, 2017 at 6:27 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Where I mentioned that he prefers a dynamic tracepoint. Is that what
> > you mean? Because what he prefers doesn't exist yet. It's not kprobes
> > nor eBPF. It is something we've been talking about implementing, and we
> > were going to discuss the implementation at ksummit.
> >
> > Are you OK with that?  
> 
> I'm not going to be at that discussion, and I don't think it should be
> a ksummit thing, at least not the maintainership part. It's a
> technical detail that should be discussed with the people who are
> interested in that technical detail. Maybe a BOF?

The thread was posted as "TECH TOPIC" which was suppose to be only for
those that are interested. It wasn't posted for "MAINTAINERS SUMMIT".
Thus, we are fine with that.

IIUC, there's two things happening for "ksummit". There's the
"Maintainer's summit" which is your hand picked group. And then there's
the extension of individual technical topics for a larger body of
people (also this helps get people to the location that may need to be
available for anything that comes out of you Maintainer's Summit, but
were not invited). This mailing list was for both, with the subject tags
to differentiate which is for which.

> 
> But yes, I was talking about something very similar to what I think
> Peter is talking about - the ability to attach a ebpf script to
> kprobes and extract data dynamically. We've supported ebpf tracepoints
> for years afaik, what is actually missing from using that for whatever
> particular extension people want to use?

Well, I don't want to put words in his mouth, but as he's probably
currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We
were talking about making the static tracepoints more "dynamic". I'm not
sure he's ever used eBPF with tracing.

> 
> Wasn't that the whole promise (and premise) of using ebpf programs in
> tracing? Exactly the ability to generate sane statistics and traces
> dynamically. I know that was what I was sold on, even if it might not
> actually have worked out that way.

eBPF is still very limited in tracing. Currently it is only implemented
for perf. Although, it has been on my todo list to get it working for
ftrace as well, and implementing eBPF for ftrace can also be on the
agenda. The issue is eBPF still needs easier users tools to get it
working, as it needs to be compiled down. But that's all details that
can be sorted out.

I still like to have a TECH TOPIC on all these issues at Kernel Summit.
You don't need to be there, and it would only be for those that are
interested in the discussion.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 7:12 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Well, I don't want to put words in his mouth, but as he's probably
> currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We
> were talking about making the static tracepoints more "dynamic". I'm not
> sure he's ever used eBPF with tracing.

I don't know how else you would make them dynamic, though.
Realistically, ebpf seems to be working really well for the networking
people, and seems to be the obvious solution.

Now, the networking people have obviously *made* it work for them. So
it's not like it's some kind of "ebpf automatically solves all
problems" thing. ebpf needs some infrastructure too, to be able to get
to the interesting data sanely (and safely).

> eBPF is still very limited in tracing. Currently it is only implemented
> for perf. Although, it has been on my todo list to get it working for
> ftrace as well, and implementing eBPF for ftrace can also be on the
> agenda.

Oh, I thought it worked outside of perf already. My bad. I'm actually
surprised it doesn't interact with ftrace, since it seems like the
perfect use case.

                Linus

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 19:34:53 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:


> Oh, I thought it worked outside of perf already. My bad. I'm actually
> surprised it doesn't interact with ftrace, since it seems like the
> perfect use case.

Agreed. I need to push that up in priority in my todo list.

Getting ebpf functionality into tracing could also help filtering and
such, which is still done with tree logic.

Anyway, I would like to get people together in discussing some of these
enhancements at ksummit as well. How to load ebpf into ftrace is another
ABI issue that we need to do right, otherwise that could bite us years
from now.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Alexei Starovoitov]

On Thu, Jun 29, 2017 at 07:34:53PM -0700, Linus Torvalds wrote:
> On Thu, Jun 29, 2017 at 7:12 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > Well, I don't want to put words in his mouth, but as he's probably
> > currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We
> > were talking about making the static tracepoints more "dynamic". I'm not
> > sure he's ever used eBPF with tracing.
> 
> I don't know how else you would make them dynamic, though.
> Realistically, ebpf seems to be working really well for the networking
> people, and seems to be the obvious solution.
> 
> Now, the networking people have obviously *made* it work for them. So
> it's not like it's some kind of "ebpf automatically solves all
> problems" thing. ebpf needs some infrastructure too, to be able to get
> to the interesting data sanely (and safely).
> 
> > eBPF is still very limited in tracing. Currently it is only implemented
> > for perf. Although, it has been on my todo list to get it working for
> > ftrace as well, and implementing eBPF for ftrace can also be on the
> > agenda.
> 
> Oh, I thought it worked outside of perf already. My bad. I'm actually
> surprised it doesn't interact with ftrace, since it seems like the
> perfect use case.

If Steven has a use case for bpf in ftrace, I don't mind, but I don't see
a value yet. Everything we wanted to see inside the kernel we can already
do with perf_events and bpf scripts.
Here is one scheduler related script:
https://github.com/iovisor/bcc/blob/master/tools/runqlen.py#L79
it builds a histogram of task->se.cfs_rq->nr_running like
# ./runqlat 
Tracing run queue latency... Hit Ctrl-C to end.
^C
     usecs               : count     distribution
         0 -> 1          : 233      |***********                             |
         2 -> 3          : 742      |************************************    |
         4 -> 7          : 203      |**********                              |
         8 -> 15         : 173      |********                                |
        16 -> 31         : 24       |*                                       |
        32 -> 63         : 0        |                                        |

This particular script just samples the whole system at given frequency
and uses probe_read() to walk kernel internal data strctures.
It's obviously unstable interace and scripts break from time to time.
The worst offender over the last years was constantly changing
internals of 'struct request' on block side, so the biosnoop.py bpf script
has ugly code like this:
#ifdef REQ_WRITE
    data.rwflag = !!(req->cmd_flags & REQ_WRITE);
#elif defined(REQ_OP_SHIFT)
    data.rwflag = !!((req->cmd_flags >> REQ_OP_SHIFT) == REQ_OP_WRITE);
#else
    data.rwflag = !!((req->cmd_flags & REQ_OP_MASK) == REQ_OP_WRITE);
#endif
to be able to run with different kernel versions, but that's fine.
When one wants to look inside kernel structures they need to
be ready to change their scripts with every kernel version
and that is well understood.

Re-reading Mathieu's original email I don't really understand
what he's trying to solve that is not solved already.

Also I'm not planning to fly to Prague just for tracing discussion.
There is netdev2.2 right after in Seoul.
And tracing microconf at plumbers in September which is imo better
suited to discuss tracing related topics.

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 19:58:54 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:


> Also I'm not planning to fly to Prague just for tracing discussion.
> There is netdev2.2 right after in Seoul.
> And tracing microconf at plumbers in September which is imo better
> suited to discuss tracing related topics.

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not
moving at all. If it is to be accepted, it needs some talk proposals,
and fast!

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 23:02:51 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Thu, 29 Jun 2017 19:58:54 -0700
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
> 
> 
> > Also I'm not planning to fly to Prague just for tracing discussion.
> > There is netdev2.2 right after in Seoul.
> > And tracing microconf at plumbers in September which is imo better
> > suited to discuss tracing related topics.  
> 
> Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not
> moving at all. If it is to be accepted, it needs some talk proposals,
> and fast!

Also note, Mathieu has stated he wont be attending Plumbers, and I'm
not sure Peter will be either as he has smaller things to attend to.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Mathieu Desnoyers]

----- On Jun 29, 2017, at 11:20 PM, rostedt rostedt@goodmis.org wrote:

> On Thu, 29 Jun 2017 23:02:51 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
>> On Thu, 29 Jun 2017 19:58:54 -0700
>> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>> 
>> 
>> > Also I'm not planning to fly to Prague just for tracing discussion.
>> > There is netdev2.2 right after in Seoul.
>> > And tracing microconf at plumbers in September which is imo better
>> > suited to discuss tracing related topics.
>> 
>> Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not
>> moving at all. If it is to be accepted, it needs some talk proposals,
>> and fast!
> 
> Also note, Mathieu has stated he wont be attending Plumbers, and I'm
> not sure Peter will be either as he has smaller things to attend to.

I take it back. Work permit delays postpone my conflicting house
renovation work, so I will likely be able to make it to LPC finally. :)

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 27 Jul 2017 14:35:45 +0000 (UTC)
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:


> > Also note, Mathieu has stated he wont be attending Plumbers, and I'm
> > not sure Peter will be either as he has smaller things to attend to.  
> 
> I take it back. Work permit delays postpone my conflicting house
> renovation work, so I will likely be able to make it to LPC finally. :)

Great! I see you updated the Wiki that you are attending as well.

 http://wiki.linuxplumbersconf.org/2017:tracing

Thanks, looking forward in seeing you there.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Josef Bacik]

On 6/29/17, 11:02 PM, "Steven Rostedt" <rostedt@goodmis.org> wrote:

On Thu, 29 Jun 2017 19:58:54 -0700
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:


> Also I'm not planning to fly to Prague just for tracing discussion.
> There is netdev2.2 right after in Seoul.
> And tracing microconf at plumbers in September which is imo better
> suited to discuss tracing related topics.

Which reminds me. The LPC Tracing Microconf WIKI has been stale, and not
moving at all. If it is to be accepted, it needs some talk proposals,
and fast!

Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,

Josef

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Fri, 30 Jun 2017 18:24:12 +0000
Josef Bacik <jbacik@fb.com> wrote:

> Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,

I shouldn't have used the term "talk", as it really is all about
discussions. In fact, if you need more than one slide, you have too
many.

That said, I could probably come up with a few things, starting with
this trace event issue. But it will be pointless if Peter Zijlstra and
Mathieu are not there.

But having ideas about dynamic fields in tracepoints is always
interesting. Not to mention talking about Tom Zanussi's latest
histogram work. It may be pretty much completed, but I would like to
discuss where we go from there.

One last thing. I don't want to have too many responsibilities, as I'm
on the LPC program committee and I need to make sure I have time to
fulfill any action items I'm responsible for during the conference.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

[ I forgot to add Tom to the Cc list. Sending again. ]

On Fri, 30 Jun 2017 14:29:56 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 30 Jun 2017 18:24:12 +0000
> Josef Bacik <jbacik@fb.com> wrote:
> 
> > Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,  
> 
> I shouldn't have used the term "talk", as it really is all about
> discussions. In fact, if you need more than one slide, you have too
> many.
> 
> That said, I could probably come up with a few things, starting with
> this trace event issue. But it will be pointless if Peter Zijlstra and
> Mathieu are not there.
> 
> But having ideas about dynamic fields in tracepoints is always
> interesting. Not to mention talking about Tom Zanussi's latest
> histogram work. It may be pretty much completed, but I would like to
> discuss where we go from there.
> 
> One last thing. I don't want to have too many responsibilities, as I'm
> on the LPC program committee and I need to make sure I have time to
> fulfill any action items I'm responsible for during the conference.
> 
> -- Steve
> 

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Josef Bacik]

[ I forgot to add Tom to the Cc list. Sending again. ]

On Fri, 30 Jun 2017 14:29:56 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 30 Jun 2017 18:24:12 +0000
> Josef Bacik <jbacik@fb.com> wrote:
> 
> > Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,  
> 
> I shouldn't have used the term "talk", as it really is all about
> discussions. In fact, if you need more than one slide, you have too
> many.
> 
> That said, I could probably come up with a few things, starting with
> this trace event issue. But it will be pointless if Peter Zijlstra and
> Mathieu are not there.
> 
> But having ideas about dynamic fields in tracepoints is always
> interesting. Not to mention talking about Tom Zanussi's latest
> histogram work. It may be pretty much completed, but I would like to
> discuss where we go from there.
> 
> One last thing. I don't want to have too many responsibilities, as I'm
> on the LPC program committee and I need to make sure I have time to
> fulfill any action items I'm responsible for during the conference.
> 

Yeah plumbers is a weird venue for tracing, I always hope that we are going to have people like Brendan or other sysadmin-y people show up and say “this is what sucks about tracing, please fix it”, and then we can go fix it.  It doesn’t really seem to happen that way tho, and for things like tracing ABI there just aren’t the right people in the room to have that kind of discussion.  My proposal was just going to be a laundry list of things that would make my life easier, but it doesn’t really warrant a full micro-conference to listen to me bitch for an hour.  If it turns out nobody else has much to talk about then we can just declare tracing is feature complete and we can talk about something else ;).  Thanks,

Josef

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Fri, 30 Jun 2017 18:37:59 +0000
Josef Bacik <jbacik@fb.com> wrote:

> [ I forgot to add Tom to the Cc list. Sending again. ]
> 
> On Fri, 30 Jun 2017 14:29:56 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > On Fri, 30 Jun 2017 18:24:12 +0000
> > Josef Bacik <jbacik@fb.com> wrote:
> >   
> > > Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll put up my proposal in the next day or two, I think Brendan has something he’s going to talk about.  Thanks,    
> > 
> > I shouldn't have used the term "talk", as it really is all about
> > discussions. In fact, if you need more than one slide, you have too
> > many.
> > 
> > That said, I could probably come up with a few things, starting with
> > this trace event issue. But it will be pointless if Peter Zijlstra and
> > Mathieu are not there.
> > 
> > But having ideas about dynamic fields in tracepoints is always
> > interesting. Not to mention talking about Tom Zanussi's latest
> > histogram work. It may be pretty much completed, but I would like to
> > discuss where we go from there.
> > 
> > One last thing. I don't want to have too many responsibilities, as I'm
> > on the LPC program committee and I need to make sure I have time to
> > fulfill any action items I'm responsible for during the conference.
> >   
> 
> Yeah plumbers is a weird venue for tracing, I always hope that we are
> going to have people like Brendan or other sysadmin-y people show up
> and say “this is what sucks about tracing, please fix it”, and then
> we can go fix it.  It doesn’t really seem to happen that way tho, and
> for things like tracing ABI there just aren’t the right people in the
> room to have that kind of discussion.  My proposal was just going to
> be a laundry list of things that would make my life easier, but it
> doesn’t really warrant a full micro-conference to listen to me bitch
> for an hour.  If it turns out nobody else has much to talk about then
> we can just declare tracing is feature complete and we can talk about
> something else ;).  Thanks,
> 

At this rate, I'm guessing that Tracing is not going to be on the
Plumbers' agenda.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Mathieu Desnoyers]

----- On Jul 6, 2017, at 3:10 PM, rostedt rostedt@goodmis.org wrote:

> On Fri, 30 Jun 2017 18:37:59 +0000
> Josef Bacik <jbacik@fb.com> wrote:
> 
>> [ I forgot to add Tom to the Cc list. Sending again. ]
>> 
>> On Fri, 30 Jun 2017 14:29:56 -0400
>> Steven Rostedt <rostedt@goodmis.org> wrote:
>> 
>> > On Fri, 30 Jun 2017 18:24:12 +0000
>> > Josef Bacik <jbacik@fb.com> wrote:
>> >   
>> > > Yup I’ll start bugging people to submit talk proposals, starting with you!  I’ll
>> > > put up my proposal in the next day or two, I think Brendan has something he’s
>> > > going to talk about.  Thanks,
>> > 
>> > I shouldn't have used the term "talk", as it really is all about
>> > discussions. In fact, if you need more than one slide, you have too
>> > many.
>> > 
>> > That said, I could probably come up with a few things, starting with
>> > this trace event issue. But it will be pointless if Peter Zijlstra and
>> > Mathieu are not there.
>> > 
>> > But having ideas about dynamic fields in tracepoints is always
>> > interesting. Not to mention talking about Tom Zanussi's latest
>> > histogram work. It may be pretty much completed, but I would like to
>> > discuss where we go from there.
>> > 
>> > One last thing. I don't want to have too many responsibilities, as I'm
>> > on the LPC program committee and I need to make sure I have time to
>> > fulfill any action items I'm responsible for during the conference.
>> >   
>> 
>> Yeah plumbers is a weird venue for tracing, I always hope that we are
>> going to have people like Brendan or other sysadmin-y people show up
>> and say “this is what sucks about tracing, please fix it”, and then
>> we can go fix it.  It doesn’t really seem to happen that way tho, and
>> for things like tracing ABI there just aren’t the right people in the
>> room to have that kind of discussion.  My proposal was just going to
>> be a laundry list of things that would make my life easier, but it
>> doesn’t really warrant a full micro-conference to listen to me bitch
>> for an hour.  If it turns out nobody else has much to talk about then
>> we can just declare tracing is feature complete and we can talk about
>> something else ;).  Thanks,
>> 
> 
> At this rate, I'm guessing that Tracing is not going to be on the
> Plumbers' agenda.

Since the Kernel Summit and Plumbers do not seem like a good fit to have
discussions involving both tracing end users and developers, we have
adapted the Tracing Summit schedule to have half day of the usual
presentations, and half day dedicated to such discussions. Steven has
volunteered to run the discussion part.

The Tracing Summit will take place on October 27th in Prague, on the
Friday right after Kernel Summit.

So if you have tracing topics that you would like to discuss at this
event, the CFP/CFD and all the information are available here:

http://tracingsummit.org/wiki/TracingSummit2017

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[James Bottomley]

On Fri, 2017-07-21 at 21:45 +0000, Mathieu Desnoyers wrote:
> 
> ----- On Jul 6, 2017, at 3:10 PM, rostedt rostedt@goodmis.org wrote:
> 
> > 
> > On Fri, 30 Jun 2017 18:37:59 +0000
> > Josef Bacik <jbacik@fb.com> wrote:
> > 
> > > 
> > > [ I forgot to add Tom to the Cc list. Sending again. ]
> > > 
> > > On Fri, 30 Jun 2017 14:29:56 -0400
> > > Steven Rostedt <rostedt@goodmis.org> wrote:
> > > 
> > > > 
> > > > On Fri, 30 Jun 2017 18:24:12 +0000
> > > > Josef Bacik <jbacik@fb.com> wrote:
> > > >   
> > > > > 
> > > > > Yup I’ll start bugging people to submit talk proposals,
> > > > > starting with you!  I’ll put up my proposal in the next day
> > > > > or two, I think Brendan has something he’s going to talk
> > > > > about.  Thanks,
> > > > 
> > > > I shouldn't have used the term "talk", as it really is all
> > > > about discussions. In fact, if you need more than one slide,
> > > > you have too many.
> > > > 
> > > > That said, I could probably come up with a few things, starting
> > > > with this trace event issue. But it will be pointless if Peter
> > > > Zijlstra and Mathieu are not there.
> > > > 
> > > > But having ideas about dynamic fields in tracepoints is always
> > > > interesting. Not to mention talking about Tom Zanussi's latest
> > > > histogram work. It may be pretty much completed, but I would
> > > > like to discuss where we go from there.
> > > > 
> > > > One last thing. I don't want to have too many responsibilities,
> > > > as I'm on the LPC program committee and I need to make sure I
> > > > have time to fulfill any action items I'm responsible for
> > > > during the conference.
> > > >   
> > > 
> > > Yeah plumbers is a weird venue for tracing, I always hope that we
> > > are going to have people like Brendan or other sysadmin-y people
> > > show up and say “this is what sucks about tracing, please fix
> > > it”, and then we can go fix it.  It doesn’t really seem to happen
> > > that way tho, and for things like tracing ABI there just aren’t
> > > the right people in the room to have that kind of discussion.  My
> > > proposal was just going to be a laundry list of things that would
> > > make my life easier, but it doesn’t really warrant a full micro-
> > > conference to listen to me bitch for an hour.  If it turns out
> > > nobody else has much to talk about then we can just declare
> > > tracing is feature complete and we can talk about something else
> > > ;).  Thanks,
> > > 
> > 
> > At this rate, I'm guessing that Tracing is not going to be on the
> > Plumbers' agenda.
> 
> Since the Kernel Summit and Plumbers do not seem like a good fit to
> have discussions involving both tracing end users and developers

First the disclaimer: being on the Plumbers Programme Committee, I'm
biased.  However, I have to say that the design of Plumbers is to bring
together everyone interested in the plumbing of Linux.  That means end
users as well, so it's not correct to say it's not a good fit.

It also looks like there's been some renewed interest in having a
Tracing MC at Plumbers, so my best guess now is that it will happen.
 That's not to say the two events can't easily co-exist: being on
different continents means better opportunities for attendees with
international travel restrictions.

James

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

Actually, Brendan Gregg got enough proposals together and there will be a tracing MC at Plumbers this year.

-- Steve


On July 21, 2017 5:45:57 PM EDT, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>
>
>----- On Jul 6, 2017, at 3:10 PM, rostedt rostedt@goodmis.org wrote:
>
>> On Fri, 30 Jun 2017 18:37:59 +0000
>> Josef Bacik <jbacik@fb.com> wrote:
>> 
>>> [ I forgot to add Tom to the Cc list. Sending again. ]
>>> 
>>> On Fri, 30 Jun 2017 14:29:56 -0400
>>> Steven Rostedt <rostedt@goodmis.org> wrote:
>>> 
>>> > On Fri, 30 Jun 2017 18:24:12 +0000
>>> > Josef Bacik <jbacik@fb.com> wrote:
>>> >   
>>> > > Yup I’ll start bugging people to submit talk proposals, starting
>with you!  I’ll
>>> > > put up my proposal in the next day or two, I think Brendan has
>something he’s
>>> > > going to talk about.  Thanks,
>>> > 
>>> > I shouldn't have used the term "talk", as it really is all about
>>> > discussions. In fact, if you need more than one slide, you have
>too
>>> > many.
>>> > 
>>> > That said, I could probably come up with a few things, starting
>with
>>> > this trace event issue. But it will be pointless if Peter Zijlstra
>and
>>> > Mathieu are not there.
>>> > 
>>> > But having ideas about dynamic fields in tracepoints is always
>>> > interesting. Not to mention talking about Tom Zanussi's latest
>>> > histogram work. It may be pretty much completed, but I would like
>to
>>> > discuss where we go from there.
>>> > 
>>> > One last thing. I don't want to have too many responsibilities, as
>I'm
>>> > on the LPC program committee and I need to make sure I have time
>to
>>> > fulfill any action items I'm responsible for during the
>conference.
>>> >   
>>> 
>>> Yeah plumbers is a weird venue for tracing, I always hope that we
>are
>>> going to have people like Brendan or other sysadmin-y people show up
>>> and say “this is what sucks about tracing, please fix it”, and then
>>> we can go fix it.  It doesn’t really seem to happen that way tho,
>and
>>> for things like tracing ABI there just aren’t the right people in
>the
>>> room to have that kind of discussion.  My proposal was just going to
>>> be a laundry list of things that would make my life easier, but it
>>> doesn’t really warrant a full micro-conference to listen to me bitch
>>> for an hour.  If it turns out nobody else has much to talk about
>then
>>> we can just declare tracing is feature complete and we can talk
>about
>>> something else ;).  Thanks,
>>> 
>> 
>> At this rate, I'm guessing that Tracing is not going to be on the
>> Plumbers' agenda.
>
>Since the Kernel Summit and Plumbers do not seem like a good fit to
>have
>discussions involving both tracing end users and developers, we have
>adapted the Tracing Summit schedule to have half day of the usual
>presentations, and half day dedicated to such discussions. Steven has
>volunteered to run the discussion part.
>
>The Tracing Summit will take place on October 27th in Prague, on the
>Friday right after Kernel Summit.
>
>So if you have tracing topics that you would like to discuss at this
>event, the CFP/CFD and all the information are available here:
>
>http://tracingsummit.org/wiki/TracingSummit2017
>
>Thanks,
>
>Mathieu

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Josef Bacik]

Do we want to talk about ABI at the micro conference?  Facebook uses tracing everywhere in production so I can talk about it from both a user and maintainer standpoint.  Thanks,

Josef

Sent from my iPhone

> On Jul 23, 2017, at 11:49 AM, Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> Actually, Brendan Gregg got enough proposals together and there will be a tracing MC at Plumbers this year.
> 
> -- Steve
> 
> 
>> On July 21, 2017 5:45:57 PM EDT, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>> 
>> 
>> ----- On Jul 6, 2017, at 3:10 PM, rostedt rostedt@goodmis.org wrote:
>> 
>>> On Fri, 30 Jun 2017 18:37:59 +0000
>>> Josef Bacik <jbacik@fb.com> wrote:
>>> 
>>>> [ I forgot to add Tom to the Cc list. Sending again. ]
>>>> 
>>>> On Fri, 30 Jun 2017 14:29:56 -0400
>>>> Steven Rostedt <rostedt@goodmis.org> wrote:
>>>> 
>>>>> On Fri, 30 Jun 2017 18:24:12 +0000
>>>>> Josef Bacik <jbacik@fb.com> wrote:
>>>>> 
>>>>>> Yup I’ll start bugging people to submit talk proposals, starting
>> with you!  I’ll
>>>>>> put up my proposal in the next day or two, I think Brendan has
>> something he’s
>>>>>> going to talk about.  Thanks,
>>>>> 
>>>>> I shouldn't have used the term "talk", as it really is all about
>>>>> discussions. In fact, if you need more than one slide, you have
>> too
>>>>> many.
>>>>> 
>>>>> That said, I could probably come up with a few things, starting
>> with
>>>>> this trace event issue. But it will be pointless if Peter Zijlstra
>> and
>>>>> Mathieu are not there.
>>>>> 
>>>>> But having ideas about dynamic fields in tracepoints is always
>>>>> interesting. Not to mention talking about Tom Zanussi's latest
>>>>> histogram work. It may be pretty much completed, but I would like
>> to
>>>>> discuss where we go from there.
>>>>> 
>>>>> One last thing. I don't want to have too many responsibilities, as
>> I'm
>>>>> on the LPC program committee and I need to make sure I have time
>> to
>>>>> fulfill any action items I'm responsible for during the
>> conference.
>>>>> 
>>>> 
>>>> Yeah plumbers is a weird venue for tracing, I always hope that we
>> are
>>>> going to have people like Brendan or other sysadmin-y people show up
>>>> and say “this is what sucks about tracing, please fix it”, and then
>>>> we can go fix it.  It doesn’t really seem to happen that way tho,
>> and
>>>> for things like tracing ABI there just aren’t the right people in
>> the
>>>> room to have that kind of discussion.  My proposal was just going to
>>>> be a laundry list of things that would make my life easier, but it
>>>> doesn’t really warrant a full micro-conference to listen to me bitch
>>>> for an hour.  If it turns out nobody else has much to talk about
>> then
>>>> we can just declare tracing is feature complete and we can talk
>> about
>>>> something else ;).  Thanks,
>>>> 
>>> 
>>> At this rate, I'm guessing that Tracing is not going to be on the
>>> Plumbers' agenda.
>> 
>> Since the Kernel Summit and Plumbers do not seem like a good fit to
>> have
>> discussions involving both tracing end users and developers, we have
>> adapted the Tracing Summit schedule to have half day of the usual
>> presentations, and half day dedicated to such discussions. Steven has
>> volunteered to run the discussion part.
>> 
>> The Tracing Summit will take place on October 27th in Prague, on the
>> Friday right after Kernel Summit.
>> 
>> So if you have tracing topics that you would like to discuss at this
>> event, the CFP/CFD and all the information are available here:
>> 
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__tracingsummit.org_wiki_TracingSummit2017&d=DwIFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=sDzg6MvHymKOUgI8SFIm4Q&m=i0WUwxGtMNqWf0sUfcJGu8mjy4EmALhzGj4FSSAj_10&s=uSqVNCjkvfgDy8m4bV0fRhLxXZzd2b5MOnzIs5uAugM&e= 
>> 
>> Thanks,
>> 
>> Mathieu
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Sun, 23 Jul 2017 16:24:09 +0000
Josef Bacik <jbacik@fb.com> wrote:

> Do we want to talk about ABI at the micro conference?  Facebook uses
> tracing everywhere in production so I can talk about it from both a
> user and maintainer standpoint.  Thanks,

Yes, please add that to the wiki.

-- Steve

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Peter Zijlstra]

Yay, tracing fight!! :/

On Thu, Jun 29, 2017 at 10:12:45PM -0400, Steven Rostedt wrote:
> On Thu, 29 Jun 2017 18:51:14 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:

> > But yes, I was talking about something very similar to what I think
> > Peter is talking about - the ability to attach a ebpf script to
> > kprobes and extract data dynamically. We've supported ebpf tracepoints
> > for years afaik, what is actually missing from using that for whatever
> > particular extension people want to use?
> 
> Well, I don't want to put words in his mouth, but as he's probably
> currently putting mush in a baby's mouth, so I'll do it anyway. ;-) We
> were talking about making the static tracepoints more "dynamic". I'm not
> sure he's ever used eBPF with tracing.

So my concerns/objections are two-fold:

 - I want only a single static tracepoint in the code.

 - I want only a single 'event' associated with this in userspace.

   (in particular I only see confusion happening when we have:
    sched_switch_fair, sched_switch_rt, sched_switch_deadline events for
    the exact same event; people will forget to enable one or more and
    wonder WTF they have holes in their traces)

These are not strange constraints / demands in my book. Just turns out
its 'difficult' to pull off or something.  I'm in fact fine with simply
adding bits to the one tracepoint we have; although others (that'd be
you Steve) are not because expensive.

Further complications seem to stem from the fact that I use the tracefs
interface exclusively. I don't know how to use perf or trace-cmd or any
of that new fangled stuff to do tracing -- nor do I really care, it
works for me (same why I'm happy with sysvinit, I don't _want_ to have
to relearn my 20+ year old sysadmin skillz, there's better things in live
to spend time on, that baby you mentioned for example).

So on that same vein, I'd be entirely helpless using eBPF to do tracing,
that's even more complicated. That said, I don't typically need this
crud anyway, I just change my kernel and rebuild, reboot and am happy,
that's far easier than trying to figure out how eBPF works.


In any case, baby vomit is more fun that this subject :-)

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Linus Torvalds]

On Thu, Jun 29, 2017 at 6:16 PM, Steven Rostedt <rostedt@goodmis.org> wrote:
>
> I didn't realize you even used the tracepoints.

I don't.

But I absolutely refuse to see this kind of idiotic churn, and even
more the idiotic "let's discuss it at ksummit" when we've had those
tracing discussions for years and they apparently didn't result in
anything workable.

Dammit, this stupid tracing issue really has been discussed for too
long, and the answer hasn't changed. You can have a static tracepoint,
but only for GENERIC stuff. Not random internal stuff that only makes
sense to one scheduler implementation. We've made that mistake too
many times before, we're not making it again.

Didn't that useless 'prio' field teach you anything? Instead, now you
want to add new fields that are not generic, but specific to one very
particular scheduling class. EXACTLTY like "prio" was/is.

> You mean kprobes? Or perhaps eBPF?
>
> The information about SCHED_DEADLINE is not trivial enough to extract
> with them.

You clearly shouldn't extract it unless you are a scheduler developer,
and then you can damn well instrument that thing yourself with your
own private tracepoints that you don't try to claim are generic and
useful to anybody else.

> Well, the world does have people that use tools besides powertop.

I said "like powertop".

I did not say "only powertop".

The fact is, no generic scheduler tracing tool that is run by real
people is ever going to care about some esoteric field that isn't even
relevant to most schedulers out there.

> Just one last note. I've tried very hard to keep tracing as contained
> as possible. That is, not to let implementation details and such creep
> into the rest of the kernel.

Yeah, and apparently it was a complete failure, if extending a single
breakpoint isn't possible without breaking stuff, and if people STILL
talk about the idiotic "oh, but it has that argument that doesn't make
sense" issue.

And dammit, having that same stupid argument _again_ isn't going to
improve things.

Why can't you just attach some eBPF script to the one tracepoint you
already have? I know that has been at least discussed, and it seems to
be the only reasonable way forward, since the existing thing clearly
isn't working.

                  Linus

Re: [Ksummit-discuss] [TECH TOPIC] Pulling away from the tracing ABI quicksands

[Steven Rostedt]

On Thu, 29 Jun 2017 18:38:11 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> Why can't you just attach some eBPF script to the one tracepoint you
> already have? I know that has been at least discussed, and it seems to
> be the only reasonable way forward, since the existing thing clearly
> isn't working.

I like the eBPF idea, unfortunately it suffers the same fate. It only
has access to fields that have been exported via the tracepoint. Now we
are back to what fields are we going to make visible, and what happens
when they no longer make sense?

I guess you can argue that we need to keep modifying that eBPF script,
for each kernel release.

-- Steve