Re: [RFC PATCH] mm, oom: oom ratelimit auto tuning

[Michal Hocko <mhocko@kernel.org>]

On Sat 11-04-20 05:36:14, Yafang Shao wrote:
> Recently we find an issue that when OOM happens the server is almost
> unresponsive for several minutes. That is caused by a slow serial set
> with "console=ttyS1,19200". As the speed of this serial is too slow, it
> will take almost 10 seconds to print a full OOM message into it. And
> then all tasks allocating pages will be blocked as there is almost no
> pages can be reclaimed. At that time, the memory pressure is around 90
> for a long time. If we don't print the OOM messages into this serial,
> a full OOM message only takes less than 1ms and the memory pressure is
> less than 40.

Which part of the oom report takes the most time? I would expect this to
be the dump_tasks part which can be pretty large when there is a lot of
eligible tasks to kill.
 
> We can avoid printing OOM messages into slow serial by adjusting
> /proc/sys/kernel/printk to fix this issue, but then all messages with
> KERN_WARNING level can't be printed into it neither, that may loss some
> useful messages when we want to collect messages from the it for
> debugging purpose.

A large part of the oom report is printed with KERN_INFO log level. So
you can reduce a large part of the output while not losing other
potentially important information.

> So it is better to decrease the ratelimit. We can introduce some sysctl
> knobes similar with printk_ratelimit and burst, but it will burden the
> amdin. Let the kernel automatically adjust the ratelimit, that would be
> a better choice.

No new knobs for ratelimiting. Admin shouldn't really care about these
things. Besides that I strongly suspect that you would be much better of
by disabling /proc/sys/vm/oom_dump_tasks which would reduce the amount
of output a lot. Or do you really require this information when
debugging oom reports?

> The OOM ratelimit starts with a slow rate, and it will increase slowly
> if the speed of the console is rapid and decrease rapidly if the speed
> of the console is slow. oom_rs.burst will be in [1, 10] and
> oom_rs.interval will always greater than 5 * HZ.

I am not against increasing the ratelimit timeout. But this patch seems
to be trying to be too clever.  Why cannot we simply increase the
parameters of the ratelimit? I am also interested whether this actually
works. AFAIR ratelimit doesn't really work reliably when the ratelimited
operation takes a long time because the internals have no way to see
when the operation finished.

> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> ---
>  mm/oom_kill.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 48 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index dfc357614e56..23dba8ccf313 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -954,8 +954,10 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>  {
>  	struct task_struct *victim = oc->chosen;
>  	struct mem_cgroup *oom_group;
> -	static DEFINE_RATELIMIT_STATE(oom_rs, DEFAULT_RATELIMIT_INTERVAL,
> -					      DEFAULT_RATELIMIT_BURST);
> +	static DEFINE_RATELIMIT_STATE(oom_rs, 20 * HZ, 1);
> +	int delta;
> +	unsigned long start;
> +	unsigned long end;
>  
>  	/*
>  	 * If the task is already exiting, don't alarm the sysadmin or kill
> @@ -972,8 +974,51 @@ static void oom_kill_process(struct oom_control *oc, const char *message)
>  	}
>  	task_unlock(victim);
>  
> -	if (__ratelimit(&oom_rs))
> +	if (__ratelimit(&oom_rs)) {
> +		start = jiffies;
>  		dump_header(oc, victim);
> +		end = jiffies;
> +		delta = end - start;
> +
> +		/*
> +		 * The OOM messages may be printed to a serial with very low
> +		 * speed, e.g. console=ttyS1,19200. It will take long
> +		 * time to print these OOM messages to this serial, and
> +		 * then processes allocating pages will all be blocked due
> +		 * to it can hardly reclaim pages. That will case high
> +		 * memory pressure and the system may be unresponsive for a
> +		 * long time.
> +		 * In this case, we should decrease the OOM ratelimit or
> +		 * avoid printing OOM messages into the slow serial. But if
> +		 * we avoid printing OOM messages into the slow serial, all
> +		 * messages with KERN_WARNING level can't be printed into
> +		 * it neither, that may loss some useful messages when we
> +		 * want to collect messages from the console for debugging
> +		 * purpose. So it is better to decrease the ratelimit. We
> +		 * can introduce some sysctl knobes similar with
> +		 * printk_ratelimit and burst, but it will burden the
> +		 * admin. Let the kernel automatically adjust the ratelimit
> +		 * would be a better chioce.
> +		 * In bellow algorithm, it will decrease the OOM ratelimit
> +		 * rapidly if the console is slow and increase the OOM
> +		 * ratelimit slowly if the console is fast. oom_rs.burst
> +		 * will be in [1, 10] and oom_rs.interval will always
> +		 * greater than 5 * HZ.
> +		 */
> +		if (delta < oom_rs.interval / 10) {
> +			if (oom_rs.interval >= 10 * HZ)
> +				oom_rs.interval /= 2;
> +			else if (oom_rs.interval > 6 * HZ)
> +				oom_rs.interval -= HZ;
> +
> +			if (oom_rs.burst < 10)
> +				oom_rs.burst += 1;
> +		} else if (oom_rs.burst > 1) {
> +			oom_rs.burst = 1;
> +			oom_rs.interval = 4 * delta;
> +		}
> +
> +	}
>  
>  	/*
>  	 * Do we need to kill the entire memory cgroup?
> -- 
> 2.18.2

-- 
Michal Hocko
SUSE Labs