slow hugetlb from 2.6.15

slow hugetlb from 2.6.15

[stanojr]

hello

look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html
i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on that web
(in comparing to standard 4kb page)
its feature or bug ? 
i am just interested where can be hugepages used, but if they are slower than normal pages
its pointless to use it :) 

stanojr

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: slow hugetlb from 2.6.15

[Badari Pulavarty]

On Tue, 2006-06-27 at 20:23 +0200, stanojr@blackhole.websupport.sk
wrote:
> hello
> 
> look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html
> i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on that web
> (in comparing to standard 4kb page)
> its feature or bug ? 

Most likely, its due to new feature - demand paging for large pages :)
Doing mlock() on mmaped area help ?

Thanks,
Badari

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: slow hugetlb from 2.6.15

[Chen, Kenneth W]

Badari Pulavarty wrote on Tuesday, June 27, 2006 11:48 AM
> On Tue, 2006-06-27 at 20:23 +0200, stanojr@blackhole.websupport.sk wrote:
> > hello
> > 
> > look at this benchmark http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html
> > i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16
> > on that web (in comparing to standard 4kb page)
> > its feature or bug ? 
> 
> Most likely, its due to new feature - demand paging for large pages :)
> Doing mlock() on mmaped area help ?


The original code measures not only the access time, but also page fault
path, that explains the huge difference with hugetlb between 2.6.12 and
2.6.16.  The former kernel prefaults, thus fault time is all done at mmap
call and is not counted at all in the timing measurement, while the latter
measurement includes faulting of hugetlb page.  Though it is a mystery to
see that faulting on hugetlb page is significantly longer than faulting a
normal page.

Yes, mlock() would take the variation out of the equation (if such call is
made outside the measurement).

- Ken

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: slow hugetlb from 2.6.15

[Dave Hansen]

On Tue, 2006-06-27 at 12:23 -0700, Chen, Kenneth W wrote:
>   Though it is a mystery to
> see that faulting on hugetlb page is significantly longer than
> faulting a
> normal page.

There's an awful lot more data to zero when allocating a page which is
1000 times bigger.  It would be really interesting to see kernel
profiles, but my money is on clear_huge_page().

-- Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: slow hugetlb from 2.6.15

[Chen, Kenneth W]

Dave Hansen wrote on Tuesday, June 27, 2006 2:51 PM
> On Tue, 2006-06-27 at 12:23 -0700, Chen, Kenneth W wrote:
> >   Though it is a mystery to
> > see that faulting on hugetlb page is significantly longer than
> > faulting a normal page.
> 
> There's an awful lot more data to zero when allocating a page which is
> 1000 times bigger.  It would be really interesting to see kernel
> profiles, but my money is on clear_huge_page().


I was under the impression that the test code will touch equal amount of
memory for both hugetlb page and normal pages.  Yes, faulting one hugetlb
page will require zeroing 1024 times more memory than a normal page, but
yet it will be 1024 times less of number of page fault.  I was referring
to time required to fault 1 hugetlb page at 4MB versus 1024 normal page
fault at 4KB. I wasn't expecting the former to be longer than the latter.

- Ken

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: slow hugetlb from 2.6.15

[Nish Aravamudan]

On 6/27/06, stanojr@blackhole.websupport.sk
<stanojr@blackhole.websupport.sk> wrote:
> hello
>
> look at this benchmark
> http://www-unix.mcs.anl.gov/~kazutomo/hugepage/note.html
> i try benchmark it on latest 2.6.17.1 (x86 and x86_64) and it slow like 2.6.16 on
> that web
> (in comparing to standard 4kb page)
> its feature or bug ?
> i am just interested where can be hugepages used, but if they are slower than
> normal pages its pointless to use it :)

I believe your benchmark is measuring the time in such a way to make
current kernels look worse than older ones.

Basically, newer kernels (the ones that have the performance issue
you're seeing) use demand faulting of hugepages.

Thus, timing only the bench() call, as is done now, causes the newer
kernels to appear to take longer, as the pages must be zero'd, and the
page tables must be set up, as part of the bench() invocation (first
use).

In contrast, the older kernels would have done that up front, and thus
that time was not being accounted for in the bench() run.

There are a few ways to make sure this is the case:

1) Time the mmap, bench and unmap calls all together.

2) Add memset(addr, 1, LENGTH) and memset(addr, 0, LENGTH) calls
*before* bench(), to fault in the hugepages before running bench().

If the numbers return to normal after this, then the analysis above
should be accurate. If not, we do have a problem.

Thanks to Andy Whitcroft and Mel Gorman for insight into the potential problem.

Thanks,
Nish

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>