Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Hillf Danton]

On 3 Oct 2022 23:20:32 +0000 John Stultz <jstultz@google.com>
> +#ifdef CONFIG_RT_SOFTIRQ_OPTIMIZATION
> +#define __use_softirq_opt 1
> +/*
> + * Return whether the given cpu is currently non-preemptible
> + * while handling a potentially long softirq, or if the current
> + * task is likely to block preemptions soon because it is a
> + * ksoftirq thread that is handling slow softirq.
> + */
> +static bool cpu_busy_with_softirqs(int cpu)
> +{
> +	u32 softirqs = per_cpu(active_softirqs, cpu) |
> +		       __cpu_softirq_pending(cpu);
> +	struct task_struct *cpu_ksoftirqd = per_cpu(ksoftirqd, cpu);
> +	struct task_struct *curr;
> +	struct rq *rq = cpu_rq(cpu);
> +	int ret;
> +
> +	rcu_read_lock();
> +	curr = READ_ONCE(rq->curr); /* unlocked access */
> +	ret = (softirqs & LONG_SOFTIRQ_MASK) &&
> +		 (curr == cpu_ksoftirqd ||
> +		  preempt_count() & SOFTIRQ_MASK);
> +	rcu_read_unlock();
> +	return ret;
> +}
> +#else
> +#define __use_softirq_opt 0
> +static bool cpu_busy_with_softirqs(int cpu)
> +{
> +	return false;
> +}
> +#endif /* CONFIG_RT_SOFTIRQ_OPTIMIZATION */
> +
> +static bool rt_task_fits_cpu(struct task_struct *p, int cpu)
> +{
> +	return !cpu_busy_with_softirqs(cpu) && rt_task_fits_capacity(p, cpu);
> +}

On one hand, RT task is not layency sensitive enough if it fails to preempt
ksoftirqd. On the other, deferring softirq to ksoftirqd barely makes sense
in 3/3 if it preempts the current RT task.

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[John Stultz]

On Mon, Oct 3, 2022 at 6:56 PM Hillf Danton <hdanton@sina.com> wrote:
> On 3 Oct 2022 23:20:32 +0000 John Stultz <jstultz@google.com>
> > +#ifdef CONFIG_RT_SOFTIRQ_OPTIMIZATION
> > +#define __use_softirq_opt 1
> > +/*
> > + * Return whether the given cpu is currently non-preemptible
> > + * while handling a potentially long softirq, or if the current
> > + * task is likely to block preemptions soon because it is a
> > + * ksoftirq thread that is handling slow softirq.
> > + */
> > +static bool cpu_busy_with_softirqs(int cpu)
> > +{
> > +     u32 softirqs = per_cpu(active_softirqs, cpu) |
> > +                    __cpu_softirq_pending(cpu);
> > +     struct task_struct *cpu_ksoftirqd = per_cpu(ksoftirqd, cpu);
> > +     struct task_struct *curr;
> > +     struct rq *rq = cpu_rq(cpu);
> > +     int ret;
> > +
> > +     rcu_read_lock();
> > +     curr = READ_ONCE(rq->curr); /* unlocked access */
> > +     ret = (softirqs & LONG_SOFTIRQ_MASK) &&
> > +              (curr == cpu_ksoftirqd ||
> > +               preempt_count() & SOFTIRQ_MASK);
> > +     rcu_read_unlock();
> > +     return ret;
> > +}
> > +#else
> > +#define __use_softirq_opt 0
> > +static bool cpu_busy_with_softirqs(int cpu)
> > +{
> > +     return false;
> > +}
> > +#endif /* CONFIG_RT_SOFTIRQ_OPTIMIZATION */
> > +
> > +static bool rt_task_fits_cpu(struct task_struct *p, int cpu)
> > +{
> > +     return !cpu_busy_with_softirqs(cpu) && rt_task_fits_capacity(p, cpu);
> > +}
>
> On one hand, RT task is not layency sensitive enough if it fails to preempt
> ksoftirqd. On the other, deferring softirq to ksoftirqd barely makes sense
> in 3/3 if it preempts the current RT task.

Apologies, I'm not sure I'm following you here. Why would ksoftirqd
preempt the rt task?

thanks
-john

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Hillf Danton]

On 3 Oct 2022 19:29:36 -0700 John Stultz <jstultz@google.com>
> 
> Why would ksoftirqd preempt the rt task?
> 
For example the kthread becomes sensitive to latency.

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[John Stultz]

On Tue, Oct 4, 2022 at 5:22 PM Hillf Danton <hdanton@sina.com> wrote:
> On 3 Oct 2022 19:29:36 -0700 John Stultz <jstultz@google.com>
> >
> > Why would ksoftirqd preempt the rt task?
> >
> For example the kthread becomes sensitive to latency.

Again, my apologies, I'm still not sure I am following. Could you
expand a bit on the case you're concerned about? Is it the case where
the ksoftirqd thread is configured to run at higher rtprio?

thanks
-john

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Hillf Danton]

On 4 Oct 2022 18:13:52 -0700 John Stultz <jstultz@google.com>
> On Tue, Oct 4, 2022 at 5:22 PM Hillf Danton <hdanton@sina.com> wrote:
> > On 3 Oct 2022 19:29:36 -0700 John Stultz <jstultz@google.com>
> > >
> > > Why would ksoftirqd preempt the rt task?
> > >
> > For example the kthread becomes sensitive to latency.
> 
> Is it the case where
> the ksoftirqd thread is configured to run at higher rtprio?
> 
Yes, you are right.

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Qais Yousef]

On 10/05/22 14:01, Hillf Danton wrote:
> On 4 Oct 2022 18:13:52 -0700 John Stultz <jstultz@google.com>
> > On Tue, Oct 4, 2022 at 5:22 PM Hillf Danton <hdanton@sina.com> wrote:
> > > On 3 Oct 2022 19:29:36 -0700 John Stultz <jstultz@google.com>
> > > >
> > > > Why would ksoftirqd preempt the rt task?
> > > >
> > > For example the kthread becomes sensitive to latency.
> > 
> > Is it the case where
> > the ksoftirqd thread is configured to run at higher rtprio?
> > 
> Yes, you are right.

I don't see a problem here. If a sys-admin configures their ksoftirqds to be
a higher priority RT tasks than the audio threads, then they better know what
they're doing :-)

The issue at hand here is that the softirqs boundedness is hard to control. And
the scheduling delays ensued are hard to deal with by any sys-admin.

Networking has actually introduced some knobs to help control that - but the
tricky bit of still being able to deliver high throughput networking while
keeping the softirq bounded to minimize scheduling delays/latencies. I think
even for PREEMPT_RT, high performance networking could be impacted to achieve
the required low latency.

See this paper which explores this duality:

	https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.702.7571&rep=rep1&type=pdf


With WiFi 6 and 5G mobile networks, phones are actually expected to deliver
multi-gigabit network throughputs.


Cheers

--
Qais Yousef

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Hillf Danton]

On 10 Oct 2022 16:42:16 +0100 Qais Yousef <qais.yousef@arm.com>
> On 10/05/22 14:01, Hillf Danton wrote:
> > On 4 Oct 2022 18:13:52 -0700 John Stultz <jstultz@google.com>
> > > On Tue, Oct 4, 2022 at 5:22 PM Hillf Danton <hdanton@sina.com> wrote:
> > > > On 3 Oct 2022 19:29:36 -0700 John Stultz <jstultz@google.com>
> > > > >
> > > > > Why would ksoftirqd preempt the rt task?
> > > > >
> > > > For example the kthread becomes sensitive to latency.
> > > 
> > > Is it the case where
> > > the ksoftirqd thread is configured to run at higher rtprio?
> > > 
> > Yes, you are right.
> 
> I don't see a problem here. If a sys-admin configures their ksoftirqds to be
> a higher priority RT tasks than the audio threads, then they better know what
> they're doing :-)
> 
Thanks for taking a look.

> The issue at hand here is that the softirqs boundedness is hard to control. And
> the scheduling delays ensued are hard to deal with by any sys-admin.
> 
Given "The RT scheduler is for tasks that require strick scheduling
requirements over all else, including performance." [1], I would invite
Steve to shed some more light on the relation between RT scheduler and
performance/network throughputs.

Bonus question, why softirq took no care of RT tasks, given strick
scheduling requirements above?

[1] https://lore.kernel.org/lkml/257E96C2-6ABD-4DD6-9B7F-771DA3D1BAAA@goodmis.org/

> Networking has actually introduced some knobs to help control that - but the
> tricky bit of still being able to deliver high throughput networking while
> keeping the softirq bounded to minimize scheduling delays/latencies. I think
> even for PREEMPT_RT, high performance networking could be impacted to achieve
> the required low latency.
> 
> See this paper which explores this duality:
> 
> 	https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.702.7571&rep=rep1&type=pdf
> 
> 
> With WiFi 6 and 5G mobile networks, phones are actually expected to deliver
> multi-gigabit network throughputs.

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Qais Yousef]

On 10/11/22 19:18, Hillf Danton wrote:

[...]

> > The issue at hand here is that the softirqs boundedness is hard to control. And
> > the scheduling delays ensued are hard to deal with by any sys-admin.
> > 
> Given "The RT scheduler is for tasks that require strick scheduling
> requirements over all else, including performance." [1], I would invite
> Steve to shed some more light on the relation between RT scheduler and
> performance/network throughputs.
> 
> Bonus question, why softirq took no care of RT tasks, given strick
> scheduling requirements above?
> 
> [1] https://lore.kernel.org/lkml/257E96C2-6ABD-4DD6-9B7F-771DA3D1BAAA@goodmis.org/

I think you're mixing the two patches up. That patch is to help describe some
latency requirements for CFS tasks. It has nothing to do with RT. Your
suggestion to use RT scheduler is not valid as these tasks can't be converted
to RT. Which is what Steve was trying to say IIUC.

Generally converting normal application tasks into RT is a recipe for disaster
because:

	1. Misbehaving RT tasks (busy looping indefinitely) can hog the system
	   to a halt.
	2. How do you manage priorities of all these pseudo-random RT tasks
	   each application spawns so you end up with correct resource sharing?

ie: using RT policy is a privileged operation for a reason :-)

The target audience for latency_nice is the average Joe task from any
application that might have some latency requirements to deliver a better user
experience. ie: it's not strict latency requirement. We just want to handle
delays within _CFS_ part of the scheduler.

By the way, your emails don't seem to make it to LKML for some reason; they
show as '[not found]' on lore.

	https://lore.kernel.org/lkml/20221010154216.6mw7fszdaoajurvm@wubuntu/#r


Thanks

--
Qais Yousef

Re: [RFC PATCH v4 2/3] sched: Avoid placing RT threads on cores handling long softirqs

[Hillf Danton]

On 12 Oct 2022 15:10:37 +0100 Qais Yousef <qais.yousef@arm.com>
> On 10/11/22 19:18, Hillf Danton wrote:
> > > The issue at hand here is that the softirqs boundedness is hard to control. And
> > > the scheduling delays ensued are hard to deal with by any sys-admin.
> > > 
> > Given "The RT scheduler is for tasks that require strick scheduling
> > requirements over all else, including performance." [1], I would invite
> > Steve to shed some more light on the relation between RT scheduler and
> > performance/network throughputs.
> > 
> > Bonus question, why softirq took no care of RT tasks, given strick
> > scheduling requirements above?
> > 
> > [1] https://lore.kernel.org/lkml/257E96C2-6ABD-4DD6-9B7F-771DA3D1BAAA@goodmis.org/
> 
> I think you're mixing the two patches up.

Nope, see below.

> That patch is to help describe some
> latency requirements for CFS tasks. It has nothing to do with RT. Your

Correct.

> suggestion to use RT scheduler is not valid as these tasks can't be converted
> to RT.

My suggestion [2,3] to that patch is add a mild change to tick preempt.

[2] https://lore.kernel.org/all/20221008103439.107-1-hdanton@sina.com/
[3] https://lore.kernel.org/all/20220920113238.1176-1-hdanton@sina.com/

> Which is what Steve was trying to say IIUC.

No followup.
> 
> Generally converting normal application tasks into RT is a recipe for disaster
> because:
> 
> 	1. Misbehaving RT tasks (busy looping indefinitely) can hog the system
> 	   to a halt.
> 	2. How do you manage priorities of all these pseudo-random RT tasks
> 	   each application spawns so you end up with correct resource sharing?

Thanks for good material.
> 
> ie: using RT policy is a privileged operation for a reason :-)

Take note.

Hillf