Re: [PATCH -mmotm 0/7] memcg: move charge at task migration (04/Dec)

[Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>]

[-- Attachment #1: Type: text/plain, Size: 3296 bytes --]

On Fri, 4 Dec 2009 16:00:42 +0900, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> On Fri, 4 Dec 2009 15:53:17 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > On Fri, 4 Dec 2009 14:46:09 +0900
> > Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp> wrote:
> 
> > > In this version:
> > >        |  252M  |  512M  |   1G
> > >   -----+--------+--------+--------
> > >    (1) |  0.15  |  0.30  |  0.60
> > >   -----+--------+--------+--------
> > >    (2) |  0.15  |  0.30  |  0.60
> > >   -----+--------+--------+--------
> > >    (3) |  0.22  |  0.44  |  0.89
> > > 
> > Nice !
> > 
> 
> Ah. could you clarify...
> 
>  1. How is fork()/exit() affected by this move ?
I measured using unixbench(./Run -c 1 spawn execl). I used the attached script to do
task migration infinitly(./switch3.sh /cgroup/memory/01 /cgroup/memory/02 [pid of bash]).
The script is executed on a different cpu from the unixbench's by taskset.

(1) no task migration(run on /01)

Execl Throughput                                192.7 lps   (29.9 s, 2 samples)
Process Creation                                475.5 lps   (30.0 s, 2 samples)

Execl Throughput                                191.2 lps   (29.9 s, 2 samples)
Process Creation                                463.4 lps   (30.0 s, 2 samples)

Execl Throughput                                191.0 lps   (29.9 s, 2 samples)
Process Creation                                474.9 lps   (30.0 s, 2 samples)


(2) under task migration between /01 and /02 w/o setting move_charge_at_immigrate

Execl Throughput                                150.2 lps   (29.8 s, 2 samples)
Process Creation                                344.1 lps   (30.0 s, 2 samples)

Execl Throughput                                146.9 lps   (29.8 s, 2 samples)
Process Creation                                337.7 lps   (30.0 s, 2 samples)

Execl Throughput                                150.5 lps   (29.8 s, 2 samples)
Process Creation                                345.3 lps   (30.0 s, 2 samples)


(3) under task migration between /01 and /02 w/ setting move_charge_at_immigrate

Execl Throughput                                142.9 lps   (29.9 s, 2 samples)
Process Creation                                323.1 lps   (30.0 s, 2 samples)

Execl Throughput                                146.6 lps   (29.8 s, 2 samples)
Process Creation                                332.0 lps   (30.0 s, 2 samples)

Execl Throughput                                150.9 lps   (29.8 s, 2 samples)
Process Creation                                344.2 lps   (30.0 s, 2 samples)


(those values seem terrible :(  I run them on KVM guest...)
(2) seems a bit better than (3), but the impact of task migration itself is
far bigger.


>  2. How long cpuset's migration-at-task-move requires ?
>     I guess much longer than this.
I measured in the same environment using fakenuma. It took 1.17sec for 256M,
2.33sec for 512M, and 4.69sec for 1G.


>  3. If need to reclaim memory for moving tasks, can this be longer ?
I think so.

>     If so, we may need some trick to release cgroup_mutex in task moving.
> 
hmm, I see your concern but I think it isn't so easy.. IMHO, we need changes
in cgroup layer and should take care not to cause dead lock.


Regards,
Daisuke Nishimura.

[-- Attachment #2: switch3.sh --]
[-- Type: text/x-sh, Size: 451 bytes --]

#!/bin/bash

SRC=$1
DST=$2
EXCLUDE_LIST="$3"

move_task()
{
	local ret

	for pid in $1
	do
		echo "${EXCLUDE_LIST}" | grep -w -q $pid
		if [ $? -eq 0 ]; then
			continue
		fi
		echo -n "$pid "
		/bin/echo $pid >$2/tasks 2>/dev/null
	done
	echo ""
}

stopflag=0

interrupt()
{
	stopflag=1
}

trap interrupt INT

while [ $stopflag -ne 1 ]
do
	echo "----- `date` -----"
	move_task "`cat ${SRC}/tasks`" ${DST}
	TMP=${SRC}
	SRC=${DST}
	DST=${TMP}
done