Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[bill4carson]

Hi, all

I'm playing with memory cgroup, I'm a bit confused why
memory.usage in bytes is steadily increasing at 4K page pace
after every mmap/dirty/unmap sequence.

On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:

a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes  <-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat


And when the test case exit, memory.usage_in_bytes is not *ZERO*, but
the previous increased value.

I'm puzzled about what I saw, can anyone please give me some tips
to understand this?


Thanks in advance.

-- 
Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[KAMEZAWA Hiroyuki]

(2012/03/23 17:04), bill4carson wrote:

> Hi, all
> 
> I'm playing with memory cgroup, I'm a bit confused why
> memory.usage in bytes is steadily increasing at 4K page pace
> after every mmap/dirty/unmap sequence.
> 
> On linux-3.6.34.10/linux-3.3.0-rc5
> A simple test case does following:
> 
> a) mmap 128k memory in private anonymous way
> b) dirty all 128k to demand physical page
> c) print memory.usage_in_bytes  <-- increased at 4K after every loop
> d) unmap previous 128 memory
> e) goto a) to repeat

In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes

For efficiency, as other kernel components, memory cgroup uses some optimization
to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
value for efficient access. (Of course, when necessary, it's synchronized.)
If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
value in memory.stat(see 5.2).
==

In current implementation, memcg tries to charge resource in size of 32 pages.
So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
This is affected by caches in other cpus and other flushing operations caused
by some workload in other cgroups. memcg's usage_in_bytes is not precise in
128k degree.

- How memory.stat changes ?
- What happens when you do test with 4M alloc/free ?

Thanks,
-Kame








--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[bill4carson]

On 2012a1'03ae??23ae?JPY 16:19, KAMEZAWA Hiroyuki wrote:
> (2012/03/23 17:04), bill4carson wrote:
>
>> Hi, all
>>
>> I'm playing with memory cgroup, I'm a bit confused why
>> memory.usage in bytes is steadily increasing at 4K page pace
>> after every mmap/dirty/unmap sequence.
>>
>> On linux-3.6.34.10/linux-3.3.0-rc5
>> A simple test case does following:
>>
>> a) mmap 128k memory in private anonymous way
>> b) dirty all 128k to demand physical page
>> c) print memory.usage_in_bytes<-- increased at 4K after every loop
>> d) unmap previous 128 memory
>> e) goto a) to repeat
>
> In Documentation/cgroup/memory.txt
> ==
> 5.5 usage_in_bytes
>
> For efficiency, as other kernel components, memory cgroup uses some optimization
> to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
> method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
> value for efficient access. (Of course, when necessary, it's synchronized.)
> If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
> value in memory.stat(see 5.2).
> ==
>
> In current implementation, memcg tries to charge resource in size of 32 pages.
> So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
> This is affected by caches in other cpus and other flushing operations caused
> by some workload in other cgroups. memcg's usage_in_bytes is not precise in
> 128k degree.

Thanks for the internal design details.
I noticed on 2.6.34, it's checked on every 512 Kbytes

See http://lxr.linux.no/#linux+v2.6.34/mm/memcontrol.c#L571

And I haven't see the 3.3.0 changes.


>
> - How memory.stat changes ?


root@localhost:/sys/fs/cgroup/memory/a> cat memory.stat;cat 
memory.usage_in_bytes
cache 0
rss 131072            <------ when mmap/dirty/
mapped_file 0
pgpgin 1278
pgpgout 1246
inactive_anon 0
active_anon 131072
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 9223372036854775807
total_cache 0
total_rss 131072
total_mapped_file 0
total_pgpgin 1278
total_pgpgout 1246
total_inactive_anon 0
total_active_anon 131072
total_inactive_file 0
total_active_file 0
total_unevictable 0


root@localhost:/sys/fs/cgroup/memory/a> cat memory.stat;cat 
memory.usage_in_bytes
cache 0
rss 4096            <------ when mmap/dirty/unmap
mapped_file 0
pgpgin 1278
pgpgout 1277
inactive_anon 0
active_anon 4096
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 9223372036854775807
total_cache 0
total_rss 4096
total_mapped_file 0
total_pgpgin 1278
total_pgpgout 1277
total_inactive_anon 0
total_active_anon 4096
total_inactive_file 0
total_active_file 0
total_unevictable 0


> - What happens when you do test with 4M alloc/free ?
>

I tried on 2.6.34, it's the same behavior.


> Thanks,
> -Kame
>
>
>
>
>
>
>
>
>

-- 
Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[bill4carson]

On 2012a1'03ae??23ae?JPY 16:19, KAMEZAWA Hiroyuki wrote:
> (2012/03/23 17:04), bill4carson wrote:
>
>> Hi, all
>>
>> I'm playing with memory cgroup, I'm a bit confused why
>> memory.usage in bytes is steadily increasing at 4K page pace
>> after every mmap/dirty/unmap sequence.
>>
>> On linux-3.6.34.10/linux-3.3.0-rc5
>> A simple test case does following:
>>
>> a) mmap 128k memory in private anonymous way
>> b) dirty all 128k to demand physical page
>> c) print memory.usage_in_bytes<-- increased at 4K after every loop
>> d) unmap previous 128 memory
>> e) goto a) to repeat
>
> In Documentation/cgroup/memory.txt
> ==
> 5.5 usage_in_bytes
>
> For efficiency, as other kernel components, memory cgroup uses some optimization
> to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
> method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
> value for efficient access. (Of course, when necessary, it's synchronized.)
> If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
> value in memory.stat(see 5.2).
> ==
>
> In current implementation, memcg tries to charge resource in size of 32 pages.
> So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
> This is affected by caches in other cpus and other flushing operations caused
> by some workload in other cgroups. memcg's usage_in_bytes is not precise in
> 128k degree.
>
Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes
reached 128k, it rolls back to 4k again. So it doesn't hurt any more.

I haven't found the code regarding to this behavior.


> - How memory.stat changes ?
> - What happens when you do test with 4M alloc/free ?
>
> Thanks,
> -Kame
>
>
>
>
>
>
>
>
>

-- 
Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[KAMEZAWA Hiroyuki]

(2012/03/23 17:59), bill4carson wrote:

> 
> 
> On 2012a1'03ae??23ae?JPY 16:19, KAMEZAWA Hiroyuki wrote:
>> (2012/03/23 17:04), bill4carson wrote:
>>
>>> Hi, all
>>>
>>> I'm playing with memory cgroup, I'm a bit confused why
>>> memory.usage in bytes is steadily increasing at 4K page pace
>>> after every mmap/dirty/unmap sequence.
>>>
>>> On linux-3.6.34.10/linux-3.3.0-rc5
>>> A simple test case does following:
>>>
>>> a) mmap 128k memory in private anonymous way
>>> b) dirty all 128k to demand physical page
>>> c) print memory.usage_in_bytes<-- increased at 4K after every loop
>>> d) unmap previous 128 memory
>>> e) goto a) to repeat
>>
>> In Documentation/cgroup/memory.txt
>> ==
>> 5.5 usage_in_bytes
>>
>> For efficiency, as other kernel components, memory cgroup uses some optimization
>> to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
>> method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
>> value for efficient access. (Of course, when necessary, it's synchronized.)
>> If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
>> value in memory.stat(see 5.2).
>> ==
>>
>> In current implementation, memcg tries to charge resource in size of 32 pages.
>> So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
>> This is affected by caches in other cpus and other flushing operations caused
>> by some workload in other cgroups. memcg's usage_in_bytes is not precise in
>> 128k degree.
>>
> Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes
> reached 128k, it rolls back to 4k again. So it doesn't hurt any more.


rolls back before unmap() ?

> 
> I haven't found the code regarding to this behavior.


Could you post your test program ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[bill4carson]

On 2012a1'03ae??23ae?JPY 17:04, KAMEZAWA Hiroyuki wrote:
> (2012/03/23 17:59), bill4carson wrote:
>
>>
>>
>> On 2012a1'03ae??23ae?JPY 16:19, KAMEZAWA Hiroyuki wrote:
>>> (2012/03/23 17:04), bill4carson wrote:
>>>
>>>> Hi, all
>>>>
>>>> I'm playing with memory cgroup, I'm a bit confused why
>>>> memory.usage in bytes is steadily increasing at 4K page pace
>>>> after every mmap/dirty/unmap sequence.
>>>>
>>>> On linux-3.6.34.10/linux-3.3.0-rc5
>>>> A simple test case does following:
>>>>
>>>> a) mmap 128k memory in private anonymous way
>>>> b) dirty all 128k to demand physical page
>>>> c) print memory.usage_in_bytes<-- increased at 4K after every loop
>>>> d) unmap previous 128 memory
>>>> e) goto a) to repeat
>>>
>>> In Documentation/cgroup/memory.txt
>>> ==
>>> 5.5 usage_in_bytes
>>>
>>> For efficiency, as other kernel components, memory cgroup uses some optimization
>>> to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
>>> method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
>>> value for efficient access. (Of course, when necessary, it's synchronized.)
>>> If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
>>> value in memory.stat(see 5.2).
>>> ==
>>>
>>> In current implementation, memcg tries to charge resource in size of 32 pages.
>>> So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
>>> This is affected by caches in other cpus and other flushing operations caused
>>> by some workload in other cgroups. memcg's usage_in_bytes is not precise in
>>> 128k degree.
>>>
>> Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes
>> reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
>
>
> rolls back before unmap() ?
>
After unmap

>>
>> I haven't found the code regarding to this behavior.
>
>
> Could you post your test program ?
>
Yes, it's a bit of messy, you can mock at me:)

-------------------------------------------------------------
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/eventfd.h>
#include <fcntl.h>
#include <limits.h>

#define MMAP_SIZE (128*1024)

int * ptr_array[1024];
int depth=0;

char cmd;
int pagesize;
int mapsize;

void get_key(void)
{

	cmd = getchar();
	putchar('\n');
	getchar();
}

int is_this_key(char key)
{
	return (cmd == key) ? 1:0;
}

void getpage(int *ptr)
{
	int i;
	for (i = 0; i < mapsize/4098; i++){
		*ptr = 4;/* alloc a physical page */
		ptr += 1024; /*move to next page*/
	}
}
void show_stat(void)
{
	int i;

	for (i = depth; i--; i < 0){
		printf("[%2d]:%8p\n", i, ptr_array[i]);
	}

}
int main(int argc, char ** argv)
{

	int *ptr;
	int i;
     char usage_in_bytes_path[PATH_MAX];
	char          tasks_path[PATH_MAX];
     char                 tmp[PATH_MAX];
     int usage_in_bytes = -1;
	int fd_tasks = -1;

	char *root_path;
	int ret;

	if (argc > 2)
		printf("Usage: oomtst [map size]\n");

	mapsize = MMAP_SIZE;


	strcpy(tmp, argv[1]);
	root_path = dirname(tmp);
	printf("root_path:%s\n", root_path);

	ret = snprintf(usage_in_bytes_path, PATH_MAX, 
"%s/memory.usage_in_bytes", root_path);
	if (ret >= PATH_MAX) {
		fputs("Path to memory.usage_in_bytes is too long\n", stderr);
		goto out;
	}
	puts(usage_in_bytes_path);

	ret = snprintf(tasks_path, PATH_MAX, "%s/tasks", root_path);
	if (ret >= PATH_MAX) {
		fputs("Path to memory.usage_in_bytes is too long\n", stderr);
		goto out;
	}
	puts(tasks_path);
	
	fd_tasks = open(tasks_path, O_WRONLY);
	if (fd_tasks == -1) {
		fprintf(stderr, "Cannot open %s: %s\n", tasks_path,
				strerror(errno));
		goto out;
	}

	printf("Using PID:%u\n", getpid());
	{
		char tasks_str[32];

		ret = sprintf(tasks_str, "%d", getpid());
		ret = write(fd_tasks, tasks_str, strlen(tasks_str));

	}


	while(1){
		char used_bytes[64];
		uint64_t tmp;

		usage_in_bytes = open(usage_in_bytes_path, O_RDWR);
		if (usage_in_bytes == -1) {
			fprintf(stderr, "Cannot open %s: %s\n", usage_in_bytes_path,
				strerror(errno));
			goto out;
		}



		printf("Enter a command (m: malloc f: free ?: exit):");
		get_key();

		if (is_this_key('m')) {	
			ptr_array[depth] = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, 
MAP_PRIVATE|MAP_ANON, 0, 0);
			if (ptr_array[depth] == NULL){
				perror("msg: cannot malloc\n");
				exit(2);
			}

			printf("malloc %d Kbytes at %p\n", mapsize/1024, ptr_array[depth]);
			getpage(ptr_array[depth]);
			depth++;

		}else if (is_this_key('f')) {
			if (depth == 0)
				break;

			printf("free %d Kbytes at %p\n", mapsize/1024, ptr_array[depth -1]);
			munmap(ptr_array[depth -1], mapsize);
			--depth;

		}else
			break;

		show_stat();


		memset(&used_bytes, 0, 33/*sizeof(used_bytes)*/);
		ret = read(usage_in_bytes, &used_bytes, 32);
		if (ret == -1) {
			perror("Cannot read from usage_in_bytes");
			break;
		}
		tmp = atoll(used_bytes);
         printf("used_bytes:%llu Kbytes\n", tmp/1024);

		if (usage_in_bytes >= 0)
			close(usage_in_bytes);

	}
	
	out:
		for (i = 0; i < depth; i++){
			printf("free %d Kbytes at %p\n", mapsize/1024, ptr_array[i]);
			munmap(ptr_array[i], mapsize);
		}


	if (usage_in_bytes >= 0)
		close(usage_in_bytes);

	return 0;
}



> Thanks,
> -Kame
>
>

-- 
Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[KAMEZAWA Hiroyuki]

(2012/03/23 18:15), bill4carson wrote:

> 
> 
> On 2012a1'03ae??23ae?JPY 17:04, KAMEZAWA Hiroyuki wrote:
>> (2012/03/23 17:59), bill4carson wrote:
>>
>>>
>>>
>>> On 2012a1'03ae??23ae?JPY 16:19, KAMEZAWA Hiroyuki wrote:
>>>> (2012/03/23 17:04), bill4carson wrote:
>>>>
>>>>> Hi, all
>>>>>
>>>>> I'm playing with memory cgroup, I'm a bit confused why
>>>>> memory.usage in bytes is steadily increasing at 4K page pace
>>>>> after every mmap/dirty/unmap sequence.
>>>>>
>>>>> On linux-3.6.34.10/linux-3.3.0-rc5
>>>>> A simple test case does following:
>>>>>
>>>>> a) mmap 128k memory in private anonymous way
>>>>> b) dirty all 128k to demand physical page
>>>>> c) print memory.usage_in_bytes<-- increased at 4K after every loop
>>>>> d) unmap previous 128 memory
>>>>> e) goto a) to repeat
>>>>
>>>> In Documentation/cgroup/memory.txt
>>>> ==
>>>> 5.5 usage_in_bytes
>>>>
>>>> For efficiency, as other kernel components, memory cgroup uses some optimization
>>>> to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the
>>>> method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz
>>>> value for efficient access. (Of course, when necessary, it's synchronized.)
>>>> If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP)
>>>> value in memory.stat(see 5.2).
>>>> ==
>>>>
>>>> In current implementation, memcg tries to charge resource in size of 32 pages.
>>>> So, if you get 32 pages and free 32pages, usage_in_bytes may not change.
>>>> This is affected by caches in other cpus and other flushing operations caused
>>>> by some workload in other cgroups. memcg's usage_in_bytes is not precise in
>>>> 128k degree.
>>>>
>>> Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes
>>> reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
>>
>>
>> rolls back before unmap() ?
>>
> After unmap
> 

>>>
>>> I haven't found the code regarding to this behavior.
>>
>>
>> Could you post your test program ?
>>
> Yes, it's a bit of messy, you can mock at me:)


Thank you. I tried.

I checked /proc/<pid>/smaps and compares it before I press 'm' and after I press 'm'.
So, anonymous should be increased 128k.

Then...
==
[root@bluextal kamezawa]# diff -Nru before.txt after.txt | grep Anonymous
 Anonymous:             0 kB
-Anonymous:            12 kB
+Anonymous:           140 kB   <============= 128kbytes you allocated
-Anonymous:             8 kB
+Anonymous:            12 kB   <============= maybe some library's heap.

==

Then, 4k remains even if you free all allocated area by your program, I think.

Thanks,
-Kame









--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

[Glauber Costa]

On 03/23/2012 09:59 AM, bill4carson wrote:
>>
> Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes
> reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
>
> I haven't found the code regarding to this behavior.

That's actually quite annoying, IMHO.
I personally think that everytime one tries to read from usage, we 
should flush the caches and show the correct figures, or at least as 
correct as we can.

That's specially bad because under load, this is wrong by O(#cpus)...

For just reading the file, this might be okay because there is an 
alternative for it (although not that intuitive), but for the threshold 
code, we are probably hitting them a lot more than we should in big 
machines.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>