why its dead now?

why its dead now?

[Nitin Gupta]

Hi,
    I've been working on 'compressed cache' feature
(http://linuxcompressed.sourceforge.net/) for some time now. I'm
basically porting it to 2.6 kernel series as it has already been
developed for 2.4.x kernels.
   I'm wondering why this project is dead even when it showed great
performance improvement when system is under memory pressure.

Are there any serious drawbacks to this?
Do you think it will be of any use if ported to 2.6 kernel?

Your feedback will be really helpful.

Thanks

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why its dead now?

[Rodrigo S de Castro]

Hi Nitin,

I didn't know you were working on a port of it to 2.6 version. 

The project has been dead because I didn't have time to work on it after I 
finished my Master's degree and also because nobody showed interest and had 
enough time to port it to 2.6. Although it was not completely stable, it has 
been used in many patchsets (ck, wolk) and I had really good feedbacks from 
its users, in particular desktop users about smoother system degradations 
when under memory pressure. It has never been officialy announced since there 
is still a good deal of work to make it work with high memory systems, to 
make it thread safe and probably some more testing and adjustments. With 2.6, 
we still have rmap implementation that may help us improving the dynamic 
adaptivity heuristic. 

Answering your questions, from my experience developing the 2.4 version, it 
doesn't seem to have any serious drawbacks, but it has not yet been 
extensively tested to prove to be a valid idea (although our benchmarks show 
to be) and it didn't reach an implementation level where it could be 
considered to be possibly a configuration option, at least. I think it may be 
really useful to port it to 2.6, for various reasons, such as:

- better change to prove this concept, 
- it may turn out to be a good option for embedded systems, 
- chance to improve the adaptivity heuristic (with rmap and maybe with other 
2.6 mm updates, besides some ideas I have)
- with a thread safe, see how well it works with SMP systems.

I am interested in porting it to 2.6 and it's possible, although not yet sure, 
that I get back to working on this project in the next weeks. Let's discuss 
the status of your port and how we could work together to make it happen (we 
could discuss further on the lc-devel list).

Best regards,

Rodrigo


On Sunday 13 November 2005 16:35, Nitin Gupta wrote:
> Hi,
>     I've been working on 'compressed cache' feature
> (http://linuxcompressed.sourceforge.net/) for some time now. I'm
> basically porting it to 2.6 kernel series as it has already been
> developed for 2.4.x kernels.
>    I'm wondering why this project is dead even when it showed great
> performance improvement when system is under memory pressure.
>
> Are there any serious drawbacks to this?
> Do you think it will be of any use if ported to 2.6 kernel?
>
> Your feedback will be really helpful.
>
> Thanks
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why its dead now?

[Rik van Riel]

On Mon, 14 Nov 2005, Nitin Gupta wrote:

> Are there any serious drawbacks to this?

Compressed caching may be hard to tune right for some workloads.
I cannot see any other drawbacks - certainly nothing that should
keep you from working on it...

> Do you think it will be of any use if ported to 2.6 kernel?

It has the potential to be very useful.

-- 
All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why its dead now?

[Scott F. H. Kaplan]

[-- Attachment #1: Type: text/plain, Size: 1629 bytes --]

On Mon, Nov 14, 2005 at 12:05:58AM +0530, Nitin Gupta wrote:

> I'm wondering why this project is dead even when it showed great
> performance improvement when system is under memory pressure.

As he stated himself, the primary reason is that its primary
maintainer (Rodrigo) can no longer dedicate time to it.

> Are there any serious drawbacks to this?

As Rik pointed out, the main complication is adapting the compressed
cache size which can be trickier for some workloads than others.  The
original, 2.4.x-line of linuxcompressed used a method of adaptivity
developed by those working on that project.  It seemed to work for
many cases, but also could suffer performance degredation for some
workloads.

There is also the possibility of the adaptive method about which I
wrote in my dissertation and in a USENIX 1999 paper (see the
linuxcompressed page -- I believe it has links for these).  This
adaptive method is much less likely to adapt badly for some workloads,
but it also requires more extensive changes to the way in which the
kernel stores referencing history.

For completely different purposes, we have a 2.4.x kernel that
maintains this history efficiently.  If you (or anyone else) are
interested at some point in porting this reference-pattern-gathering
code forward to the 2.6.x line, then you could easily apply this other
adaptive mechanism to compressed caching.

> Do you think it will be of any use if ported to 2.6 kernel?

Sure.  I think that the potential for compressed caching to ease the
performance degredation under memory pressure is only getting better
as hardware continues to evolve.

Scott

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: why its dead now?

[Rik van Riel]

On Tue, 15 Nov 2005, Scott F. H. Kaplan wrote:

> For completely different purposes, we have a 2.4.x kernel that
> maintains this history efficiently.  If you (or anyone else) are
> interested at some point in porting this reference-pattern-gathering
> code forward to the 2.6.x line,

Marcelo already did some work on that:

	http://linux-mm.org/PageTrace

-- 
All Rights Reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: why its dead now?

[Scott F. H. Kaplan]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Rik,

On Sat, Nov 19, 2005 at 06:30:01PM -0500, Rik van Riel wrote:
> On Tue, 15 Nov 2005, Scott F. H. Kaplan wrote:
> 
> > For completely different purposes, we have a 2.4.x kernel that
> > maintains this history efficiently.  If you (or anyone else) are
> > interested at some point in porting this reference-pattern-gathering
> > code forward to the 2.6.x line,
> 
> Marcelo already did some work on that:
> 
> 	http://linux-mm.org/PageTrace

Nope.  Marcelo's work on reference trace collection overlaps with
other work I've done (kVMTrace), but I'm referring to something
completely different.

Specifically, I'm talking about gathering LRU miss histograms
(A.K.A. ``miss rate curves'') online in the kernel.  A paper in ASPLOS
2004 presented this idea, as did we in an ISMM 2004 paper on
automatically resizing garbage collectors.  We have a kernel with much
lower overhead than the ASPLOS paper presents.

These histograms can be used to perform various kinds of cost/benefit
calculations for current reference patterns.  In this case, it could
be used to implement the method I presented in a USENIX 1999 paper for
dynamically adaptic compressed cache sizes.  It's a mechanism that
would be difficult to trick into maladaptivity.

Scott

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]