howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Kacper Wysocki]

On 7/25/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Rene Herman <rene.herman@gmail.com> wrote:
>
> > Nick Piggin is the person to convince it seems and if I've read things
> > right (I only stepped into this thing at the updatedb mention, so
> > maybe I haven't) his main question is _why_ the hell it helps
> > updatedb. [...]
[snip howto get a patch merged]
> But a "here is a solution, take it or leave it" approach, before having
> communicated the problem to the maintainer and before having debugged
> the problem is the wrong way around. It might still work out fine if the
> solution is correct (especially if the patch is small and obvious), but
> if there are any non-trivial tradeoffs involved, or if nontrivial amount
> of code is involved, you might see your patch at the end of a really
> long (and constantly growing) waiting list of patches.

Is that what happened with swap prefetch these two years? The approach
has been wrong?

In this particular instance, there are lots of people who have looked
at, tried, tested, reported on, advocated, and most importantly -been
using for several years- swap prefetch. That's a lot of people, it's
hard to coordinate a unified approach, eh?

You are the gatekeepers. The dudes with the keys. The guys who must
claim technical and moral superiority over the rabble, the rest of us.

I always believed that the linux development model consisted of
collaboration with technical merit as the prime goal in mind. But the
sheer volume of patches and the small number of keyholders precludes
such a scenario. So the gatekeepers have trusted advisors, and it's
quicker to look at numbers than to try out all the random (and often
crappy) patches that flow in.

It's easier to ask for more testing, more feedback, more "unbiased",
easier to just drop it rather than to look at the new code itself.
It's easier and less risky to try and understand a problem yourself,
and write your own code. After all, you trust your own code, right?

Life is so much simpler without any alien objects around.
However, cool things usually come from outside such a stable environment.

The problem isn't debugging the wrong way around. The problem is no
matter how many people read it, test it, use it, brag about it,
measure it, it doesn't matter at all unless they can convince one of
these few dudes that he should really turn that key.

0K

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Rene Herman]

On 07/25/2007 05:30 PM, Kacper Wysocki wrote:

>>> main question is _why_ the hell it helps updatedb.

> Is that what [ ... ]

And there we go again -- off into blabber-land. Why does swap-prefetch help 
updatedb? Or doesn't it? And if it doesn't, why should anyone trust anything 
else someone who said it does says?

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Ingo Molnar]

* Kacper Wysocki <kacperw@online.no> wrote:

> [snip howto get a patch merged]

> > But a "here is a solution, take it or leave it" approach, before 
> > having communicated the problem to the maintainer and before having 
> > debugged the problem is the wrong way around. It might still work 
> > out fine if the solution is correct (especially if the patch is 
> > small and obvious), but if there are any non-trivial tradeoffs 
> > involved, or if nontrivial amount of code is involved, you might see 
> > your patch at the end of a really long (and constantly growing) 
> > waiting list of patches.
> 
> Is that what happened with swap prefetch these two years? The approach 
> has been wrong?

i dont know - but one of the maintainers of the code (Nick) says that he 
asked for but did not get debug feedback:

> > > And yet despite my repeated pleas, none of those people has yet 
> > > spent a bit of time with me to help analyse what is happening.

Con, the maintainer of -ck, certainly has (or had, when he maintained 
it) enough clout to coordinate such an effort between non-developer -ck 
users and the MM maintainers. Maybe he attempted to do that and has 
tried to provide debug feedback to MM maintainers?

	Ingo

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [ck] Re: howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Michael Chang]

On 7/25/07, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Kacper Wysocki <kacperw@online.no> wrote:
>
> > [snip howto get a patch merged]
>
> > > But a "here is a solution, take it or leave it" approach, before
> > > having communicated the problem to the maintainer and before having
> > > debugged the problem is the wrong way around. It might still work
> > > out fine if the solution is correct (especially if the patch is
> > > small and obvious), but if there are any non-trivial tradeoffs
> > > involved, or if nontrivial amount of code is involved, you might see
> > > your patch at the end of a really long (and constantly growing)
> > > waiting list of patches.
> >
> > Is that what happened with swap prefetch these two years? The approach
> > has been wrong?
>
> i dont know - but one of the maintainers of the code (Nick) says that he
> asked for but did not get debug feedback:

Perhaps this is unimportant now, I don't know, but who did he ask?
Where did he ask? Where should the feedback have gone? (For example,
is he subscribed to -ck?) To be perfectly honest, I find this very
surprising, considering the number of people that appear to be
supporting this patch. I can only wonder whether it's possible this
request never got to any of them.

-- 
Michael Chang

Please avoid sending me Word or PowerPoint attachments. Send me ODT,
RTF, or HTML instead.
See http://www.gnu.org/philosophy/no-word-attachments.html
Thank you.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Robert Deaton]

On 7/25/07, Rene Herman <rene.herman@gmail.com> wrote:
> And there we go again -- off into blabber-land. Why does swap-prefetch help
> updatedb? Or doesn't it? And if it doesn't, why should anyone trust anything
> else someone who said it does says?

I don't think anyone has ever argued that swap-prefetch directly helps
the performance of updatedb in any way, however, I do recall people
mentioning that updatedb, being a ram intensive task, will often cause
things to be swapped out while it runs on say a nightly cronjob. If a
person is not at their computer, updatedb will run, cause all their
applications to be swapped out, finish its work, and exit, leaving all
the other applications that would have otherwise been left in RAM for
when the user returns to his/her computer in swap. Thus, when someone
returns, you have to wait for all your applications to be swapped back
in.

Swap prefetch, on the other hand, would have kicked in shortly after
updatedb finished, leaving the applications in swap for a speedy
recovery when the person comes back to their computer.

-- 
--Robert Deaton

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

updatedb

[Rene Herman]

On 07/25/2007 07:15 PM, Robert Deaton wrote:

> On 7/25/07, Rene Herman <rene.herman@gmail.com> wrote:

>> And there we go again -- off into blabber-land. Why does swap-prefetch 
>> help updatedb? Or doesn't it? And if it doesn't, why should anyone 
>> trust anything else someone who said it does says?

> I don't think anyone has ever argued that swap-prefetch directly helps 
> the performance of updatedb in any way

People have argued (claimed, rather) that swap-prefetch helps their system 
after updatedb has run -- you are doing so now.

> however, I do recall people mentioning that updatedb, being a ram
> intensive task, will often cause things to be swapped out while it runs
> on say a nightly cronjob.

Problem spot no. 1.

RAM intensive? If I run updatedb here, it never grows itself beyond 2M. Yes, 
two. I'm certainly willing to accept that me and my systems are possibly not 
the reference but assuming I'm _very_ special hasn't done much for me either 
in the past.

The thing updatedb does do, or at least has the potential to do, is fill 
memory with cached inodes/dentries but Linux does not swap to make room for 
caches. So why will updatedb "often cause things to be swapped out"?

[ snip ]

> Swap prefetch, on the other hand, would have kicked in shortly after
> updatedb finished, leaving the applications in swap for a speedy
> recovery when the person comes back to their computer.

Problem spot no. 2.

If updatedb filled all of RAM with inodes/dentries, that RAM is now used 
(ie, not free) and swap-prefetch wouldn't have anywhere to prefetch into so 
would _not_ have kicked in.

So what's happening? If you sit down with a copy op "top" in one terminal 
and updatedb in another, what does it show?

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Andika Triwidada]

On 7/26/07, Rene Herman <rene.herman@gmail.com> wrote:
> On 07/25/2007 07:15 PM, Robert Deaton wrote:
>
> > On 7/25/07, Rene Herman <rene.herman@gmail.com> wrote:
>
> >> And there we go again -- off into blabber-land. Why does swap-prefetch
> >> help updatedb? Or doesn't it? And if it doesn't, why should anyone
> >> trust anything else someone who said it does says?
>
> > I don't think anyone has ever argued that swap-prefetch directly helps
> > the performance of updatedb in any way
>
> People have argued (claimed, rather) that swap-prefetch helps their system
> after updatedb has run -- you are doing so now.
>
> > however, I do recall people mentioning that updatedb, being a ram
> > intensive task, will often cause things to be swapped out while it runs
> > on say a nightly cronjob.
>
> Problem spot no. 1.
>
> RAM intensive? If I run updatedb here, it never grows itself beyond 2M. Yes,
> two. I'm certainly willing to accept that me and my systems are possibly not
> the reference but assuming I'm _very_ special hasn't done much for me either
> in the past.

Might be insignificant, but updatedb calls find (~2M) and sort (~26M).
Definitely not RAM intensive though (RAM is 1GB).

>
> The thing updatedb does do, or at least has the potential to do, is fill
> memory with cached inodes/dentries but Linux does not swap to make room for
> caches. So why will updatedb "often cause things to be swapped out"?
>

[ snip ]

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Bongani Hlope]

On Thursday 26 July 2007 05:59:53 Rene Herman wrote:
>
> Problem spot no. 1.
>
> RAM intensive? If I run updatedb here, it never grows itself beyond 2M.
> Yes, two. I'm certainly willing to accept that me and my systems are
> possibly not the reference but assuming I'm _very_ special hasn't done much
> for me either in the past.
>
> The thing updatedb does do, or at least has the potential to do, is fill
> memory with cached inodes/dentries but Linux does not swap to make room for
> caches. So why will updatedb "often cause things to be swapped out"?
>
> [ snip ]
>
> > Swap prefetch, on the other hand, would have kicked in shortly after
> > updatedb finished, leaving the applications in swap for a speedy
> > recovery when the person comes back to their computer.
>
> Problem spot no. 2.
>
> If updatedb filled all of RAM with inodes/dentries, that RAM is now used
> (ie, not free) and swap-prefetch wouldn't have anywhere to prefetch into so
> would _not_ have kicked in.
>
> So what's happening? If you sit down with a copy op "top" in one terminal
> and updatedb in another, what does it show?
>
> Rene.

Just tested that, there's a steady increase in the useage of buff

<start updatedb>
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  1      0 1279412 201160 234720    0    0   193    29  558  657  5  1 89  5
 0  1      0 1276624 203436 234872    0    0  2276     0 1638 2456  4  2 48 48
 1  1      0 1273372 206292 235012    0    0  2852     0 1773 2755  3  3 48 46
 2  1      0 1270128 208376 235360    0    0  2084     0 1545 2168  5  2 47 46

8<

 0  1      0 1228004 237288 237836    0    0  2192     0 1669 2941  6  3 47 44
 1  1      0 1223424 239228 238020    0    0  1932   272 1580 2881  9  4 44 44
 1  1      0 1219692 241600 238208    0    0  2372     0 1719 2881 10  4 45 43
 0  1      0 1217296 243372 238312    0    0  1772     0 1526 2320  4  2 49 46

8<

 0  1      0 1166852 277912 240840    0    0  2244     0 1699 3037  7  2 48 43
 0  1      0 1164016 279528 241016    0    0  1608   824 1512 2364  7  2 47 44
 1  1      0 1161256 281860 241264    0    0  2332     0 1709 2769  7  2 49 43
 1  1      0 1155632 284792 241452    0    0  2932     0 1835 3084  8  4 46 42

8< 

 0  1      0 1104568 324788 243616    0    0  3500     4 1879 3054  5  4 46 46
 1  1      0 1099596 328524 243768    0    0  3736     0 1990 3257  7  4 48 43
 1  1      0 1093976 332516 244060    0    0  3984   572 2013 3348  6  3 48 43
 0  1      0 1090320 335396 244340    0    0  2880     0 1760 2925  5  3 47 46

8<

 1  1      0 1025212 384380 248224    0    0  2940     0 1763 2864  6  3 46 46
 0  1      0 1022196 386444 248328    0    0  2064     8 1527 2543  5  2 45 47
 0  1      0 1018620 389476 248404    0    0  3032     0 1798 2988  6  3 47 45
 0  1      0 1014800 392364 248552    0    0  2888     0 1738 2821  5  2 48 45

8<

 0  1      0 425200 839828 273392    0    0  1744     0 1441 2248  9  2 44 46
 0  1      0 423360 841220 273544    0    0  1384   368 1374 2144  3  1 48 48
 0  1      0 421288 842868 273576    0    0  1648     0 1400 2141  4  2 46 48
 0  1      0 418252 845172 273676    0    0  2300     0 1570 2492  3  1 49 48
 0  0      0 417300 846100 273776    0    0   928     0 1232 1837  3  2 72 24 

<updatedb finished>

 0  0      0 416724 846100 273776    0    0     0     0 1025 1579  5  1 94  0
 0  0      0 417012 846100 273776    0    0     0     0 1002 1474  3  1 97  0
 1  0      0 417220 846100 273776    0    0     0     0 1026 1414  2  0 98  0

So 32 percent of free memory went to the buffers.

5 minutes later it's still not freed

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 2  0      0 409500 846652 277320    0    0   286    31  585  766  6  1 83 10
 1  0      0 409328 846652 277320    0    0     0     0 1003 1442  3  1 97  0

/proc/slabinfo
ext3_inode_cache  176198 176200    816    5    1 : tunables   54   27    8 : 
slabdata  35240  35240      0
dentry            233054 233054    208   19    1 : tunables  120   60    8 : 
slabdata  12266  12266      0
buffer_head       228303 228327    104   37    1 : tunables  120   60    8 : 
slabdata   6171   6171      0

run OpenOffice

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 403664 847056 277460    0    0   235    26  577  766  6  1 85  8
 0  0      0 403656 847056 277460    0    0     0     0 1003 1385  5  0 96  0
 0  0      0 403888 847056 277460    0    0     0     0 1237 1968  3  1 96  0

8<

<starts openoffice>
 0  0      0 400708 847088 277620    0    0     0     0 1058 1259  4  0 95  0
 0  0      0 400584 847088 277620    0    0     0     0 1246 1647  7  1 93  0
 1  1      0 389796 847164 284100    0    0  6528   116 1215 2663 10  4 71 14

8<

 0  0      0 307000 847464 361384    0    0     0     0 1031 1398  5  1 95  0
 0  0      0 307000 847464 361384    0    0     0     0 1003 1369  3  1 95  0
 0  0      0 307124 847464 361384    0    0     0     0 1025 1535  4  1 95  0

8<

 1  1      0 301920 847516 363176    0    0  1780   132 1092 1705 11  2 77 10
 1  1      0 289296 847588 367620    0    0  4608   152 1221 2280 31  4 48 18
 1  0      0 285672 847612 369572    0    0  1936     0 1061 3545 14  3 72 10
<open office loaded>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/26/2007 08:39 AM, Bongani Hlope wrote:

> On Thursday 26 July 2007 05:59:53 Rene Herman wrote:

>> So what's happening? If you sit down with a copy op "top" in one terminal
>> and updatedb in another, what does it show?

> Just tested that, there's a steady increase in the useage of buff

Great. Now concentrate on the "swpd" column, as it's the only thing relevant 
here. The fact that an updatedb run fills/replaces caches is completely and 
utterly unsurprising and not something swap-prefetch helps with. The only 
thing it does is bring back stuff from _swap_.

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Bongani Hlope]

On Thursday 26 July 2007 08:56:59 Rene Herman wrote:
> On 07/26/2007 08:39 AM, Bongani Hlope wrote:
> > On Thursday 26 July 2007 05:59:53 Rene Herman wrote:
> >> So what's happening? If you sit down with a copy op "top" in one
> >> terminal and updatedb in another, what does it show?
> >
> > Just tested that, there's a steady increase in the useage of buff
>
> Great. Now concentrate on the "swpd" column, as it's the only thing
> relevant here. The fact that an updatedb run fills/replaces caches is
> completely and utterly unsurprising and not something swap-prefetch helps
> with. The only thing it does is bring back stuff from _swap_.
>

;)

I have 2Gb of RAM and I never ever touched swap on all my work loads. I was 
just showing the behavior of updatedb on my desktop. I have never even looked 
at the swap-prefetch patch (for obvious reasons). I think people should also 
look at their /proc/sys/vm/overcommit_ratio

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/26/2007 08:23 AM, Andika Triwidada wrote:

> On 7/26/07, Rene Herman <rene.herman@gmail.com> wrote:

>> RAM intensive? If I run updatedb here, it never grows itself beyond 2M.
>> Yes, two. I'm certainly willing to accept that me and my systems are 
>> possibly not the reference but assuming I'm _very_ special hasn't done
>> much for me either in the past.
> 
> Might be insignificant, but updatedb calls find (~2M) and sort (~26M).

It does? My updatedb certainly doesn't seem to (slackware 12.0). It's using 
"Secure Locate". Different distributions using different versions of locate 
it seems?

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/26/2007 09:08 AM, Bongani Hlope wrote:

> On Thursday 26 July 2007 08:56:59 Rene Herman wrote:

>> Great. Now concentrate on the "swpd" column, as it's the only thing 
>> relevant here. The fact that an updatedb run fills/replaces caches is 
>> completely and utterly unsurprising and not something swap-prefetch
>> helps with. The only thing it does is bring back stuff from _swap_.
> 
> ;)
> 
> I have 2Gb of RAM and I never ever touched swap on all my work loads. I
> was just showing the behavior of updatedb on my desktop. I have never
> even looked at the swap-prefetch patch (for obvious reasons).

I see... thanks for the report :)

> I think people should also look at their /proc/sys/vm/overcommit_ratio

In the sense that current stuff might be evicted earlier with no or little 
overcommit?

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Andika Triwidada]

On 7/26/07, Rene Herman <rene.herman@gmail.com> wrote:
> On 07/26/2007 08:23 AM, Andika Triwidada wrote:
>
> > On 7/26/07, Rene Herman <rene.herman@gmail.com> wrote:
>
> >> RAM intensive? If I run updatedb here, it never grows itself beyond 2M.
> >> Yes, two. I'm certainly willing to accept that me and my systems are
> >> possibly not the reference but assuming I'm _very_ special hasn't done
> >> much for me either in the past.
> >
> > Might be insignificant, but updatedb calls find (~2M) and sort (~26M).
>
> It does? My updatedb certainly doesn't seem to (slackware 12.0). It's using
> "Secure Locate". Different distributions using different versions of locate
> it seems?
>
> Rene.
>


I'm using Debian Sid, updatedb is from package findutils v4.2.31

-- 
andika

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Björn Steinbrink]

On 2007.07.26 08:56:59 +0200, Rene Herman wrote:
> On 07/26/2007 08:39 AM, Bongani Hlope wrote:
>
>> On Thursday 26 July 2007 05:59:53 Rene Herman wrote:
>
>>> So what's happening? If you sit down with a copy op "top" in one terminal
>>> and updatedb in another, what does it show?
>
>> Just tested that, there's a steady increase in the useage of buff
>
> Great. Now concentrate on the "swpd" column, as it's the only thing 
> relevant here. The fact that an updatedb run fills/replaces caches is 
> completely and utterly unsurprising and not something swap-prefetch helps 
> with. The only thing it does is bring back stuff from _swap_.

But that's with a system that has plenty of RAM available.

The following vmstat output is from a run for which I ran a memory hog
to simulate a box with just 1GB of RAM (didn't want to reboot ;-)). That
(or even less) is probably a more likely amount of RAM for a majority of
users.

Other than the memory hog, there's a relatively small Firefox process
(just about 150MB RSS), Xorg, mutt an apache and some other stuff,
leaving about 128MB of RAM "free".

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0    696  16360  47608  74600    0    0     7    13    4   30  0  0 99  0
 0  0    696  16352  47608  74600    0    0     0    48  213  530  0  0 100  0
 0  1    796  16024  45516  74548    0   17   882   160  515 1698  1  3 58 38
 0  1   1092  16124  41752  74164    0   43  1931    43  660 2219  1  4 50 45
 1  1   1548  35096  24224  69036    0  107  1115   571  473 1616  1  4 50 45
 2  1   8980  45560  18552  58580    0 1324  1069  1324  453 1705  1  4 50 45
 2  1  12460  44840  21048  56588    0 1160   831  1345  403 1351  0  1 51 48
 2  1  14348  44220  23016  55408    0  629   661   947  353 1140  0  2 50 48
 [snip]
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  1  88904  72160  55368  38908    0 1377   836  1576  424 1403  0  3 50 47
 0  1  96080  74084  57600  38660    0 2373   747  2559  412 1312  0  2 48 49
 1  1 100036  74816  61544  38660    0 1319  1312  1547  524 1605  1  3 50 47
 0  1 107032  72996  64728  37780    4 2332  1065  2341  461 1686  1  5 50 45
 2  1 115036  68944  75908  36768    0 2660  3731  2941 1133 3721  1  6 49 44
 3  0 125160  58768  90548  36628    0 3375  4883  3798 1458 4606  1  6 50 43
 2  1 125176  48560 102364  36536    0    5  3973  1377 1342 3701  1  4 50 46
 [snip]
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  1 360628 101444 191420  34496    0  748  1927   760  670 2322  3  3 48 46
 1  0 362064 100996 191972  34520    0  479   184   479  226  654 50  1 41  8
 1  0 362064  99752 191980  34520    0    0     0     9  182  594 50  0 50  0
 4  0 362064  98728 191980  34520   11    0    11     5  179  588 49  0 50  0
 2  0 362064  97528 191988  34520    0    0     0    15  188  603 50  0 50  0
 2  0 362064  95876 191988  34520   43    0    43    13  190  603 50  0 49  1
 1  0 362064  95008 191996  34520   21    0    21    12  183  604 50  0 50  0
 2  0 364900  63516 193212  63456    0  947   408  1281  368 1163 16  3 50 31
 0  0 364868 139108 193284  39220    0    0    69 11213  383 15413 25  8 61  6
 1  0 364868 139116 193312  39220    0    0     0  1284  224  595  0  0 98  1
 2  0 364868 139240 193320  39220    0    0     0     9  182  553  0  0 100  0


Note that the total RSS usage of updatedb+sort was just about 50MB,
nevertheless swap grew to more than 300MB. It's also interesting that
swapping is so aggressive, that the amount of free memory is constantly
growing. I'm a missing something or wouldn't it be smarter to use that
free memory for buffers and cache first? (x86_64 system, so even if
highmem on x86 could be responsible, it's not the case here.)

Will now go and see what happens if I play with swappiness.

Bjorn

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Björn Steinbrink]

On 2007.07.26 11:58:29 +0200, Bjorn Steinbrink wrote:
> Note that the total RSS usage of updatedb+sort was just about 50MB,
> nevertheless swap grew to more than 300MB. It's also interesting that
> swapping is so aggressive, that the amount of free memory is constantly
> growing. I'm a missing something or wouldn't it be smarter to use that
> free memory for buffers and cache first? (x86_64 system, so even if
> highmem on x86 could be responsible, it's not the case here.)
> 
> Will now go and see what happens if I play with swappiness.

Hm, swappiness set to 0 looks even more weird to me, especially the
beginning, where (AFAICT) basically buffers and caches are dropped just
to get a pretty huge amount of free RAM.

With swappiness set to 100, you basically get what you expect: swapping.
But at least to me, that swapping looks _a lot_ smarter than what it did
for the default swappiness of 60 or the 0 swappiness. Swap is growing,
but so are the buffers, and the cache also only shrinks at a single
point, probably when the "sort" process starts to grow. Plus, the amount
of free memory isn't growing to insane sizes like in the other cases.

vmstat output for both cases to be found below.

Bjorn

vmstat output for swappiness = 0

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  25140  37712  65304    1    1     7    14    4   31  0  0 99  0
 4  0      0  25132  37736  65312    0    0     0    38  212  604  1  0 98  1
 2  0      0  25132  37736  65312    0    0     0     0  177  479  0  0 100  0
 3  1      0  17252  42568  65312    0    0  1516    23  641 2332  1  3 54 41
 3  2      0  15172  45412  59908    0    0  1567   482  585 2051  1  4 50 45
 2  2      0  20436  44320  54524    0    0   743     9  368 1196  0  1 50 49
 3  2      0  28416  36580  54520    0    0   533     4  312 1016  0  1 50 49
 1  2      4  44316  22780  54464    0    0   128   191  240  786  1  1 43 54
 3  0   3884  55256  18112  45772    0    0   356   277  318  937  0  3 39 57
 1  2   3924  57276  19004  40700    0    0   631   117  347 1160  0  1 50 48
 0  1   4108  61328  15348  40344    0    0   768   301  378 1218  0  1 50 49
 1  1   4276  60812  15612  40216    0  300   585   359  344 1045  0  1 50 49
 0  2   4276  62184  17484  39624    0  995   703  1208  370 1188  0  2 50 48
 0  2   4296  68244  14616  36268    0    0   360    11  275  973  0  1 50 49
 2  2   4296  75292   6976  36016    0  137   667   480  424 1315  0  3 50 47
 3  1   4300  78984   6344  33392    0    0   639   635  517 1142  0  3 49 47
 2  1   4300  80816   5520  32520    0    1   992   587  683 1479  0  7 48 45
 1  2   4556  82244   3704  32504    0   85   607   659  452 1040  0  3 50 47
 0  1   4556  81600   4972  32420    0    0   665    44  362 1040  0  1 50 49
 0  2   4940  80364   4588  32508    0  128   552   539  375  995  0  3 50 47
 0  1   5004  83116   3576  32420    0   21   416   591  405  862  0  3 49 47
 0  1   5004  80196   5388  32440    0    0   604     0  328 1014  0  2 50 48
 0  1   5004  82976   2548  32544    0    0   461   343  363  923  0  4 49 47
 1  1   5004  81528   3780  32472    0    0  1124   157  542 1524  1  4 49 47
 0  1   5516  81940   4000  32488    0  171   865   575  489 1356  0  4 49 46
 0  2   5804  82412   2380  32516    0   96   660   423  385 1160  1  6 42 51
 0  2   5804  81476   2868  32480    0    0   864   204  493 1242  0  5 50 45
 0  1   6268  83124   2088  32508    0  155   678   551  409 1107  0  7 46 47
 0  2   6268  83216   1576  32380    0    0   771   153  450 1330  1 11 43 46
 0  3  59420  97888    736  32224    0 17717   300 18035  375  869  0 27 12 60
 0  4 176160 214288    800  32316    0 38913    23 38917  347  502  0  6 30 64
 1  2 176212 242608   2752  32716    0   17   683    95  441 1256  0  4 41 54
 1  1 176212 237464   5492  32568    0    0   883   188  452 1488  1  2 50 48
 1  0 176212 232296   9628  32636    0    0  1368   263  533 1690  1  2 50 47
 0  1 176212 225852  13264  32652    0    0  1212     0  480 1818  1  3 50 46
 0  1 176212 202076  31708  32660    0    0  6143   348 1723 5654  2  7 50 41
 0  1 176212 177196  49952  32560    0    0  6081     0 1698 5413  1  6 50 43
 0  1 176212 147744  58332  32652    0    0  2791   467  884 3565  1  5 50 44
 0  1 176212 130604  63788  32664    0    0  1813   407  645 2637  1  5 50 44
 0  1 176212 119076  68012  32584    0    0  1408     0  529 2239  0  4 50 45
 2  1 176212  74112  83696  32600    0    0  5225   667 1496 7206  5 13 50 33
 1  1 176212  29296  99788  32620    0    0  5360    16 1524 7009  4 15 49 32
 0  1 176212  16012 105112  32660    0    0  1773  2901  891 2417  1  4 50 45
 1  1 194444  16656 117236  32624    0 6077  4036  6337 1228 6040  6  9 49 36
 1  1 194572  15996 122056  32568    0   43  1607    43  579 2740  1  5 50 44
 0  1 209788  15888 125520  32636    0 5072  1149  5520  503 1836  1  4 49 46
 0  1 231568  17304 133968  32884   21 7260  2933  7260  922 3912  2  9 46 43
 0  1 231568  16300 139004  32980    0    0  1695   523  636 2077  1  3 49 46
 1  1 238440  15900 145208  33072    0 2291  2102  2722  736 2482  1  5 49 46
 2  0 246140  15940 147788  32980    0 2545   860  2545  407 1353 21  3 39 36
 4  0 246140  16712 147680  32980    0    0     0   247  190  544 50  0 50  0
 3  0 246140  15392 147680  32980    0    0     0     0  177  508 50  1 49  0
 3  0 277644  18180 147688  32980    0 10501     0 10511  229  518 50  0 47  2
 4  0 277644  15940 147688  32980    0    0     0  1747  361  514 50  0 49  1
 4  0 277644  16880 147700  33212    0    0    79    15  221  519 49  0 50  1
 0  2 309192  13900 147560  61856    0 10516    11 10523  186  528 46  3 49  2
 4  0 315176  70264 149128  62880    0 1995   411 11668  506 2372  5  3 38 53
 0  0 315176  95912 149212  38104    0    0   132    68  197 16128 23  6 69  2
 0  0 315176  95920 149232  38104    0    0     0  1568  216  488  0  0 98  1
 0  0 315176  95920 149240  38104    0    0     0     9  201  662  0  1 99  0


------------------------


vmstat output for swappiness = 100

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  17120 150356  45104    1    1     8    14    4   32  0  0 99  0
 1  0      0  17072 150360  45160    0    0     1     0  190  700  0  0 100  0
 0  0      0  17072 150380  45160    0    0     0    44  196  610  0  1 98  1
 0  1   6944  15320 154124  45304    0 2315  1211  2315  582 2141  0  4 54 41
 3  1  17752  16780 159000  45164    0 3603  1614  4039  618 2062  0  2 48 50
 2  1  17876  15700 163044  45228    0   41  1348    41  514 2013  0  2 50 48
 3  1  32372  16872 166124  45180    0 4832  1021  5104  491 1888  1  4 50 45
 0  1  39964  16640 168620  45184    0 2531   844  2564  427 1331  0  4 50 46
 0  1  39964  15868 170580  45216    0    0   651   331  351 1096  0  1 50 49
 0  1  47312  16164 172184  45240    0 2449   529  2653  336  915  0  1 50 49
 3  1  55180  15392 174656  45140    0 2623   825  2623  397 1212  1  1 50 48
 0  1  63256  15648 177380  45168    0 2692   904  3085  433 1527  1  4 50 45
 2  1  68672  15408 180608  45136    0 1805  1076  2199  516 1506  0  3 50 47
 2  1  73804  16628 182420  45156    0 1711   599  2144  352 1066  1  1 50 48
 2  1  77396  16412 184192  45152    0 1197   588  1559  337 1066  0  0 50 50
 2  1  79668  16492 185944  45112    0  757   581   773  334 1058  0  2 50 48
 3  1  88964  15588 187452  45092    0 3099   500  3480  331 1046  0  1 49 50
 2  1  89040  16700 188940  45072    0   25   495    55  313 1135  0  1 50 49
 4  1  97700  15716 192584  45108    0 2887  1212  3197  503 1706  0  3 49 48
 3  1  99952  15564 195144  45056    0  751   851  2420  650 1289  0  3 50 47
 2  1 108624  15988 198256  45084    0 2891  1035  3045  455 1449  0  2 50 48
 2  1 117380  16664 200972  45080    0 2919   911  3257  430 1368  1  3 50 46
 3  1 175564  44768 180728  43316   11 19383   321 19412  370 2982  4 22 43 31
 5  0 175664  53424 160468  43048   11   33  1617   189  586 2834  1  8 50 41
 2  1 175744  54132 153648  42920    0   27  1163  1571  495 2032  1  6 49 44
 2  1 176116  41212 144836  32460    0  103  5997   756 1692 8567  7 18 49 27
 4  0 197192  40988 150000  19180   11 5592  3716  5592 1133 5697  3  9 47 41
 2  1 212884  45396 156332  19144   11 5231  2119  5573  746 2952  0  5 49 46
 3  1 226452  25416 166504  19132    0 4523  4388  4527 1301 6233  4 14 49 33
 3  1 233632  26628 168816  19188   11 2393   779  2676  394 1376  0  2 50 48
 2  1 242304  25748 171292  19192   21 2891  1189  3339  496 1856  1  4 50 45
 3  0 259492  24100 176708  19192    0 5729  1871  5739  678 2703  1  5 50 44
 1  1 259512  16628 184492  18864    0    7  2591  1047  969 3416  1  7 50 42
 2  1 270404  16984 190096  18920   11 3631  2035  3643  712 2367  1  4 50 45
 2  1 276056  17440 191416  18860   43 1884   656  4713  575 1212  0  2 50 48
 3  0 280704  16040 192140  18932    0 1549   439  1561  301  950 41  1 38 20
 4  0 280704  16200 192124  18932   21    0    21  1384  348  531 50  0 49  0
 3  0 280704  16352 192132  18932   11    0    11     9  183  524 50  0 50  0
 3  0 280704  15396 192132  18932   11    0    11     0  180  519 50  0 50  0
 3  0 286300  15740 191632  18932   11 1865    11  1877  194  529 49  0 50  1
 3  0 286300  16216 191644  19032   85    0   128     0  183  539 50  0 49  1
 3  0 324104  15824 191784  48024   21 12601   108 15856  299  678 22  4 45 29
 4  0 323904  62648 193092  51844   11    0   375    28  347 16416 26  6 43 25
 2  0 323904  92140 193156  23880   11    0    82  1609  220 1439  1  2 95  3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/26/2007 11:58 AM, Bjorn Steinbrink wrote:

> Will now go and see what happens if I play with swappiness.

I in fact managed to overlook _all_ of swappiness (*) and was quite frankly 
under the impression that Linux would simply never swap anything out to make 
room for cache. Which is basic enough a misunderstanding that I'll go sulk 
in a corner now.

Rene.

(*)

$ grep -ri swappiness Documentation
$

Sigh...

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Re: howto get a patch merged (WAS: Re: -mm merge plans for 2.6.23)

[Jos Poortvliet]

[-- Attachment #1.1: Type: text/plain, Size: 2765 bytes --]

On 7/25/07, Robert Deaton <false.hopes@gmail.com> wrote:
>
> On 7/25/07, Rene Herman <rene.herman@gmail.com> wrote:
> > And there we go again -- off into blabber-land. Why does swap-prefetch
> help
> > updatedb? Or doesn't it? And if it doesn't, why should anyone trust
> anything
> > else someone who said it does says?
>
> I don't think anyone has ever argued that swap-prefetch directly helps
> the performance of updatedb in any way, however, I do recall people
> mentioning that updatedb, being a ram intensive task, will often cause
> things to be swapped out while it runs on say a nightly cronjob. If a
> person is not at their computer, updatedb will run, cause all their
> applications to be swapped out, finish its work, and exit, leaving all
> the other applications that would have otherwise been left in RAM for
> when the user returns to his/her computer in swap. Thus, when someone
> returns, you have to wait for all your applications to be swapped back
> in.
>
> Swap prefetch, on the other hand, would have kicked in shortly after
> updatedb finished, leaving the applications in swap for a speedy
> recovery when the person comes back to their computer.


Note that the updatedb scenario should actually be properly fixed some other
way: updatedb, touching everything only once, shouldn't dirty the caches
like it does.

But the same thing happens when you open a 10 megapixel picture for editing
in Krita, or start OpenOffice. After closing them, a lot of ram is freed.
Yet the data which is pushed to swap when you used these apps will remain
there, until you start using them. Swap-prefetch will gently get this data
back (while keeping it also in swap, to ensure a quick response when the ram
is needed for another big app - so you have the advantages, but not the
disadvantages!).

I haven't heard anyone claim this scenario can be 'fixed' in a 'more proper'
way than with swap prefetch. And nobody has been able to prove that
swap-prefetch has any bad sideeffects. So it IS a net-gain. But only for
desktop users who hit swap. So it's good for those doing video and photo
editing, for example. Or low-mem systems with OpenOffice. Or ppl doing heavy
compiles while low on ram.

It doesn't help nor hinder laptop users (it is automatically turned off on
laptops to save power), and it doesn't help nor hinder big 16-gb-ram systems
(they probably don't hit swap often, quick responses might not be important,
they're mostly busy so swap-prefetch doesn't run and most importantly: they
won't have it turned on anyway).

--
> --Robert Deaton
> _______________________________________________
> http://ck.kolivas.org/faqs/replying-to-mailing-list.txt
> ck mailing list - mailto: ck@vds.kolivas.org
> http://vds.kolivas.org/mailman/listinfo/ck
>

[-- Attachment #1.2: Type: text/html, Size: 3532 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: updatedb

[Bongani Hlope]

On Thursday 26 July 2007 10:01:11 Rene Herman wrote:
> On 07/26/2007 09:08 AM, Bongani Hlope wrote:
> > On Thursday 26 July 2007 08:56:59 Rene Herman wrote:
> >> Great. Now concentrate on the "swpd" column, as it's the only thing
> >> relevant here. The fact that an updatedb run fills/replaces caches is
> >> completely and utterly unsurprising and not something swap-prefetch
> >> helps with. The only thing it does is bring back stuff from _swap_.
> >
> > ;)
> >
> > I have 2Gb of RAM and I never ever touched swap on all my work loads. I
> > was just showing the behavior of updatedb on my desktop. I have never
> > even looked at the swap-prefetch patch (for obvious reasons).
>
> I see... thanks for the report :)
>
> > I think people should also look at their /proc/sys/vm/overcommit_ratio
>
> In the sense that current stuff might be evicted earlier with no or little
> overcommit?
>

Oops, that was suppose to be /proc/sys/vm/swappiness Sorry about the 
confusion.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Jesper Juhl]

On 26/07/07, Andika Triwidada <andika@gmail.com> wrote:
> On 7/26/07, Rene Herman <rene.herman@gmail.com> wrote:
> > On 07/25/2007 07:15 PM, Robert Deaton wrote:
> >
> > > On 7/25/07, Rene Herman <rene.herman@gmail.com> wrote:
> >
> > >> And there we go again -- off into blabber-land. Why does swap-prefetch
> > >> help updatedb? Or doesn't it? And if it doesn't, why should anyone
> > >> trust anything else someone who said it does says?
> >
> > > I don't think anyone has ever argued that swap-prefetch directly helps
> > > the performance of updatedb in any way
> >
> > People have argued (claimed, rather) that swap-prefetch helps their system
> > after updatedb has run -- you are doing so now.
> >
> > > however, I do recall people mentioning that updatedb, being a ram
> > > intensive task, will often cause things to be swapped out while it runs
> > > on say a nightly cronjob.
> >
> > Problem spot no. 1.
> >
> > RAM intensive? If I run updatedb here, it never grows itself beyond 2M. Yes,
> > two. I'm certainly willing to accept that me and my systems are possibly not
> > the reference but assuming I'm _very_ special hasn't done much for me either
> > in the past.
>
> Might be insignificant, but updatedb calls find (~2M) and sort (~26M).
> Definitely not RAM intensive though (RAM is 1GB).
>

That doesn't match my box at all :

root@dragon:/home/juhl# free
             total       used       free     shared    buffers     cached
Mem:       2070856    1611548     459308          0      59312     740760
-/+ buffers/cache:     811476    1259380
Swap:       987988          0     987988

root@dragon:/home/juhl# updatedb

root@dragon:/home/juhl# free
             total       used       free     shared    buffers     cached
Mem:       2070856    1724204     346652          0     144708     745328
-/+ buffers/cache:     834168    1236688
Swap:       987988          0     987988


This is a Slackware Linux 12.0 system.


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/27/2007 02:46 AM, Jesper Juhl wrote:

> On 26/07/07, Andika Triwidada <andika@gmail.com> wrote:

>> Might be insignificant, but updatedb calls find (~2M) and sort (~26M). 
>> Definitely not RAM intensive though (RAM is 1GB).
> 
> That doesn't match my box at all :

[ ... ]

> This is a Slackware Linux 12.0 system.

Yes, already identified that there are more updatedb's around. We are using 
"Secure Locate" and others simply the locate from the GNU findutils. Either 
version does not itself use significant memory though and seems irrelevant 
to the orginal swap-prefetch issue -- if updatedb filled memory with inodes 
and dentries the memory is no longer free and swap-prefetch can't prefetch 
anything.

The remaining issue of updatedb unnecessarily blowing away VFS caches is 
being discussed (*) in a few thread-branches still running.

Rene.

(*) I so much wanted to say "buried".

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Mike Galbraith]

On Fri, 2007-07-27 at 08:00 +0200, Rene Herman wrote:

> The remaining issue of updatedb unnecessarily blowing away VFS caches is 
> being discussed (*) in a few thread-branches still running.

If you solve that, the swap thing dies too, they're one and the same
problem.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/27/2007 09:54 AM, Mike Galbraith wrote:

> On Fri, 2007-07-27 at 08:00 +0200, Rene Herman wrote:
> 
>> The remaining issue of updatedb unnecessarily blowing away VFS caches is 
>> being discussed (*) in a few thread-branches still running.
> 
> If you solve that, the swap thing dies too, they're one and the same
> problem.

I still wonder what the "the swap thing" is though. People just kept saying 
that swap-prefetch helped which would seem to indicate their problem didnt 
have anything to do with updatedb.

Also, I know shit about the VFS so this may well be not very educated but to 
me something like FADV_NOREUSE on a dirfd sounds like a much more promising 
approach than the convoluted userspace schemes being discussed, if only 
because it'll actually be implemented/used.

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Mike Galbraith]

On Fri, 2007-07-27 at 10:28 +0200, Rene Herman wrote:
> On 07/27/2007 09:54 AM, Mike Galbraith wrote:
> 
> > On Fri, 2007-07-27 at 08:00 +0200, Rene Herman wrote:
> > 
> >> The remaining issue of updatedb unnecessarily blowing away VFS caches is 
> >> being discussed (*) in a few thread-branches still running.
> > 
> > If you solve that, the swap thing dies too, they're one and the same
> > problem.
> 
> I still wonder what the "the swap thing" is though. People just kept saying 
> that swap-prefetch helped which would seem to indicate their problem didnt 
> have anything to do with updatedb.

I haven't rummaged around in the VM in quite a long while, so don't know
exactly where the balance lies any more, and have never looked at
swap-prefetch, but the mechanism of how swap-prefetch can help the
"morning after syndrome" seems simple enough:

Reclaim (swapout) a slew of application pages because there are
truckloads of utterly bored pages laying about when updatedb comes along
and introduces memory pressure in the middle of the night.  Updatedb
finishes, freeing some ram (doesn't matter how much) swap-prefetch
detects idle CPU, and begins faulting swapped out pages back in.  In the
process of doing so, memory pressure is generated, and now these freshly
accessed pages are a less lovely target than the now aging VFS caches
that updatedb bloated up, so they shrink back down enough that the
balance you had before updatedb ran is restored... with the notable
exception that cached data is now toast, so what you gained by faulting
god knows how frequently used pages back in isn't _necessarily_ going to
help you.  Heck, it could even step on what was left of your cached
working set after updatedb finished.

> Also, I know shit about the VFS so this may well be not very educated but to 
> me something like FADV_NOREUSE on a dirfd sounds like a much more promising 
> approach than the convoluted userspace schemes being discussed, if only 
> because it'll actually be implemented/used.

I like Andrew's mention of a future option... put that sucker and
everybody who looks like him in a resource limited container.

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/27/2007 11:26 AM, Mike Galbraith wrote:

> On Fri, 2007-07-27 at 10:28 +0200, Rene Herman wrote:

>> I still wonder what the "the swap thing" is though. People just kept
>> saying that swap-prefetch helped which would seem to indicate their
>> problem didnt have anything to do with updatedb.
> 
> I haven't rummaged around in the VM in quite a long while, so don't know 
> exactly where the balance lies any more, and have never looked at 
> swap-prefetch, but the mechanism of how swap-prefetch can help the 
> "morning after syndrome" seems simple enough:

As far as I've googled things together, the below scenario won't happen:

> Reclaim (swapout) a slew of application pages because there are 
> truckloads of utterly bored pages laying about when updatedb comes along 
> and introduces memory pressure in the middle of the night.

Ack (*)

> Updatedb finishes, freeing some ram (doesn't matter how much)

Will be very little and swap-prefetch at least in its current form needs 
more than very little to start doing anything:

http://ck.kolivas.org/patches/swap-prefetch/2.6.21-swap_prefetch-38.patch

| /*
|  * Set max number of entries to 2/3 the size of physical ram  as we
|  * only ever prefetch to consume 2/3 of the ram.
|  */

However, okay, let's just ignore that and pretend it kicks in even with the 
little free memory updatedb itself left behind when it finished:

> swap-prefetch detects idle CPU, and begins faulting swapped out pages 
> back in. In the process of doing so, memory pressure is generated, and
> now these freshly accessed pages are a less lovely target than the now
> aging VFS caches that updatedb bloated up, so they shrink back down
> enough that the balance you had before updatedb ran is restored...

The story now again breaks down here. Over at:

http://lkml.org/lkml/2007/2/9/112

we have swap-prefetch's author saying:

| swap prefetch stores the data at the tail end of the lru list which
| means that even if you do want to use that ram for something else,
| the prefetched pages will be immediately dropped.

So those aging VFS caches will never be replaced by anything prefetched it 
seems and any prefetching stops after the couple of pages that updatedb 
itself freed have been taken again.

A subsequent bit in that same message reads:

| It can't help the updatedb scenario. Updatedb leaves the ram full and
| swap prefetch wants to cost as little as possible so it will never
| move anything out of ram in preference for the pages it wants to swap
| back in.

which I'll take as confirmation.

> with the notable exception that cached data is now toast, so what you
> gained by faulting god knows how frequently used pages back in isn't
> _necessarily_ going to help you.

Also file-backed pages (such as the program binaries itself). However, _if_ 
prefetching were to take place, I'd be okay assuming it helps.

> Heck, it could even step on what was left of your cached working set
> after updatedb finished.

Given swap-prefetch's focus on being as free as possible I don't believe it 
should hurt any. And yes, it's always going to help _some_ workload shifts 
and as far as I'm concerned is a "makes sense" kind of tweak even if it's 
not the be all end all so again, not against swap-prefetch or its merger or 
anything...

>> Also, I know shit about the VFS so this may well be not very educated but to 
>> me something like FADV_NOREUSE on a dirfd sounds like a much more promising 
>> approach than the convoluted userspace schemes being discussed, if only 
>> because it'll actually be implemented/used.
> 
> I like Andrew's mention of a future option... put that sucker and
> everybody who looks like him in a resource limited container.

As to the (*) above -- now that I actually know about the "swappiness" 
thing, shouldn't setting that to 0 around updatedb runs really be quite 
effective and if it's not, is there anything to be fixed there? Bjoern 
Steinbrink (added to CC) earlier in this thread tried that and said stuff 
went weird.

Rene.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Mike Galbraith]

On Fri, 2007-07-27 at 13:09 +0200, Rene Herman wrote:
> On 07/27/2007 11:26 AM, Mike Galbraith wrote:
> 

> > Updatedb finishes, freeing some ram (doesn't matter how much)
> 
> Will be very little and swap-prefetch at least in its current form needs 
> more than very little to start doing anything:
> 
> http://ck.kolivas.org/patches/swap-prefetch/2.6.21-swap_prefetch-38.patch
> 
> | /*
> |  * Set max number of entries to 2/3 the size of physical ram  as we
> |  * only ever prefetch to consume 2/3 of the ram.
> |  */
> 
> However, okay, let's just ignore that and pretend it kicks in even with the 
> little free memory updatedb itself left behind when it finished:

Hm.  I didn't read the patch, so I'm only going on what you quoted.
>From that, all I see is a limit on how much will be used total, and 2/3
of physical ram is a bunch.  This quote doesn't say free ram, it says
physical ram.  If it really does use only free ram, that indeed sounds
pretty pointless.  I believe the users who say their apps really do get
paged back in though, so suspect that's not the case.

Anyway, I only offered a simple explanation of how swap-prefetching can
indeed help (and possibly hurt) with something like updatedb, not an
analysis of it's current implementation ;-)  I'd have to read it, and
test it myself to do that, but my world doesn't have a need for it,
so...

	-Mike

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Rene Herman]

On 07/27/2007 01:48 PM, Mike Galbraith wrote:

> physical ram.  If it really does use only free ram, that indeed sounds
> pretty pointless.

Con's quote from a bit below that seems to confirm the "only free" nicely.

> I believe the users who say their apps really do get paged back in
> though, so suspect that's not the case.

Stopping the bush-circumference beating, I do not. -ck (and gentoo) have 
this massive Calimero thing going among their users where people are much 
less interested in technology than in how the nasty big kernel meanies are 
keeping them down (*).

Nick Piggin has been unable to get anyone to substantiate anything it seems 
and even this thread alone (and I privately) received a few "oh, heh, sorry, 
I don't actually have a friggin' clue what I'm talking about" responses. As 
such, I believe it's fairly safe to dump the updatedb thing in the garbage 
as not a practical problem.

Leaves the issue of for example a midnight backup run that could very well 
itself grow large enough to leave massive amounts of free memory at exit 
which swap-prefetch _would_ help with. I haven't much opinion on how 
important such situations are but trying to do something to help those seems 
sensible in itself.

Rene.

(*) which isn't to say that you guys aren't in fact nasty big kernel meanies 
ofcourse.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: updatedb

[Tilman Schmidt]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

Rene Herman schrieb:
> On 07/27/2007 01:48 PM, Mike Galbraith wrote:
> 
>> I believe the users who say their apps really do get paged back in
>> though, so suspect that's not the case.
> 
> Stopping the bush-circumference beating, I do not. -ck (and gentoo) have 
> this massive Calimero thing going among their users where people are much 
> less interested in technology than in how the nasty big kernel meanies are 
> keeping them down (*).

I think the problem is elsewhere. Users don't say: "My apps get paged
back in." They say: "My system is more responsive". They really don't
care *why* the reaction to a mouse click that takes three seconds with
a mainline kernel is instantaneous with -ck. Nasty big kernel meanies,
OTOH, want to understand *why* a patch helps in order to decide whether
it is really a good idea to merge it. So you've got a bunch of patches
(aka -ck) which visibly improve the overall responsiveness of a desktop
system, but apparently no one can conclusively explain why or how they
achieve that, and therefore they cannot be merged into mainline.

I don't have a solution to that dilemma either.

-- 
Tilman Schmidt                    E-Mail: tilman@imap.cc
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]