meminfo or Rephrased helping the Programmer's help themselves...

meminfo or Rephrased helping the Programmer's help themselves...

[John Carter]

The http://www.faqs.org/faqs/unix-faq/programmer/faq/ describes this 
nicely as a Frequently Unanswered Question...

I guess Linux has been evolving but the Un*x as broad standard seems to be 
stagnating...

So assuming the what the FUQ (appropriate abbreviation too) says is true, 
then my question becomes....

Given that I can fudge things in userland.

How would one compute an index of "Badness" from the 
information in /proc/meminfo.

ie. How would my daemon, on looking at /proc/meminfo decide...
  A) Hey, bad stuff is going down, time tell nice programs we're being 
     nibble to death.
  B) If I grant the friendly caring program this N megabyte chunk of 
     memory, bad things will happen.

As a refresher, this is what I see in /proc/meminfo right now...

        total:    used:    free:  shared: buffers:  cached:
Mem:  262033408 245161984 16871424        0  3399680 138022912
Swap: 534601728 117628928 416972800
MemTotal:       255892 kB
MemFree:         16476 kB
MemShared:           0 kB
Buffers:          3320 kB
Cached:         113780 kB
SwapCached:      21008 kB
Active:         137960 kB
Inact_dirty:     51408 kB
Inact_clean:     21688 kB
Inact_target:    42208 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       255892 kB
LowFree:         16476 kB
SwapTotal:      522072 kB
SwapFree:       407200 kB
Committed_AS:   329292 kB

-- 


John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : john.carter@tait.co.nz
New Zealand

Good Ideas:
Ruby                 - http://www.ruby-lang-org - The best of perl,python,scheme without the pain.
Valgrind             - http://developer.kde.org/~sewardj/ - memory debugger for x86-GNU/Linux
Free your books      - http://www.bookcrossing.com

---------- Forwarded message ----------
Date: Thu, 5 Sep 2002 09:58:40 +1200 (NZST)
From: John Carter <john.carter@tait.co.nz>
To: linux-mm@kvack.org
Subject: Helping the Programmer's help themselves...

One major problem with all the OOM handling ideas are they are hostile. 

They assume the situation is out of control and desperate.

What the programmers need is help to avoid getting the user into such dire 
straits. Consider two scenarios....

    * Nibbled to Death by Ducks. The program is malloc'ing many tiny
      chunks of memory. Eventually it starts thrashing. A quarter of
      an hour later it runs out of memory and malloc returns 0 or some
      friendly OOM killer hits it on the head. 

      Assume the OOM killer doesn't, what can the programmer do? Pop
      up a friendly dialog box, and shutdown neatly? Nah! Not enough
      memory to do that!

    * Grabbing a Large Chunk. I was using Ghost View. For various
      reasons it asked for a huge amount of memory. Malloc didn't
      return zero, there was enough swap. However, the system turned
      to sticky mud and stayed that way until I could kill the X
      server 15 minutes later...

Now assume that these two programs are written by responsible, caring 
programmers. What could they have done to stop entering this domain? 

Nothing. The OS hates users. ;-)

The first "Nibbled To Death by Ducks" scenario could be resolved by a
"memory low, system getting slow" signal.

Now if the OS had a "memory getting low, system getting slow" signal,
and could send that signal to all programs, _before_ things got
desperate. Then the programmer could start bailing out in a clean
and friendly manner. 

Most programs, such as the X server would just ignore it. However a
well behaved memory intensive non-critical program would respond by,
pausing, presenting a modal "Memory Low. Do you want me to die, or
continue?" dialog. 

On recieving the inverse signal, "memory fine, system go" signal the
dialog box would go away and things would continue. Some fuzzy
heuristics would be needed to tide the system over transient
fluctuations.

The "Grabbing a Large Chunk" scenario could be resolved by a
"friendly, caring malloc".

If the OS had a "malloc, but not at the cost of the system" malloc,
then as any programmer knows when he is going to be grabbing a really
large chunk, he can use that. 

So when he grabs a large chunk he uses the "friendly caring malloc"
and checks the return code. If the answer is bad, he pops up a
friendly message that tells the user that he really doesn't want to do
that and why. The user may at his own choice and peril say "do so
anyway".

I'm sure the friendly caring malloc could be written in userland, any 
hints on how?

The "memory getting low, system getting slow" signals would probably
need some OS support, but perhaps could be fudged by a userland daemon.

-- 


John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : john.carter@tait.co.nz
New Zealand

Good Ideas:
Ruby                 - http://www.ruby-lang-org - The best of perl,python,scheme without the pain.
Valgrind             - http://developer.kde.org/~sewardj/ - memory debugger for x86-GNU/Linux
Free your books      - http://www.bookcrossing.com


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: meminfo or Rephrased helping the Programmer's help themselves...

[M. Edward (Ed) Borasky]

On Fri, 6 Sep 2002, John Carter wrote:

> The http://www.faqs.org/faqs/unix-faq/programmer/faq/ describes this
> nicely as a Frequently Unanswered Question...
>
> I guess Linux has been evolving but the Un*x as broad standard seems to be
> stagnating...
>
> So assuming the what the FUQ (appropriate abbreviation too) says is true,
> then my question becomes....
>
> Given that I can fudge things in userland.
>
> How would one compute an index of "Badness" from the
> information in /proc/meminfo.
>
> ie. How would my daemon, on looking at /proc/meminfo decide...
>   A) Hey, bad stuff is going down, time tell nice programs we're being
>      nibble to death.
>   B) If I grant the friendly caring program this N megabyte chunk of
>      memory, bad things will happen.

I've been looking at this sort of thing for a year and a half now. I've
begged and pleaded on the main kernel list for documentation, meaningful
performance statistics, control knobs, etc. Linux is not
enterprise-ready until it has these. I've even made a proposal; see

	http://www.borasky-research.net/Cougar.htm

I *could* devote the rest of my life to building what *I* believe is
necessary, but given the chaotic nature of the Linux development process
relative to, say, SEI level 2, I'm reluctant to devote what little
sanity I have left to jousting this particular windmill. There are too
many other fun things in life, many of which you'll find on my main web
site.

As to what constitutes "badness", that depends entirely on what the
computer in question is doing to earn its keep. If it's a web server,
the real measures of goodness or badness are things you measure with a
web performance tool like Segue SilkPerformer. If it's an engineering
workstation, it has to be able to do big scientific/graphic calculations
while not impeding the flow of e-mail, documentation, etc. in the
organization where the engineer works.

So, getting back to your theme ... what does the computer you're
attempting to manage do for a living? What are the cost tradeoffs
between your programming time and buying more RAM? Do you have all of
the memory leaks out of your code?
-- 
M. Edward Borasky
znmeb@borasky-research.net

The COUGAR Project
http://www.borasky-research.com/Cougar.htm

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: meminfo or Rephrased helping the Programmer's help themselves...

[Rik van Riel]

On Fri, 6 Sep 2002, M. Edward (Ed) Borasky wrote:

> I've been looking at this sort of thing for a year and a half now. I've
> begged and pleaded on the main kernel list for documentation, meaningful
> performance statistics, control knobs, etc. Linux is not
> enterprise-ready until it has these. I've even made a proposal; see
>
> 	http://www.borasky-research.net/Cougar.htm

That's a _very_ high overview proposal.  A good start would be
defining what statistics are missing from the kernel and how
exactly you'd want to collect those.

> I *could* devote the rest of my life to building what *I* believe is
> necessary, but given the chaotic nature of the Linux development process
> relative to, say, SEI level 2, I'm reluctant to devote what little
> sanity I have left to jousting this particular windmill.

One step at a time.  It _is_ possible to put better statistics
into the Linux kernel, as long as they are sent in in manageable
chunks.

Speaking of which, I need to resubmit my iowait statistics patch.

regards,

Rik
-- 
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/		http://distro.conectiva.com/

Spamtraps of the month:  september@surriel.com trac@trac.org

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

RE: meminfo or Rephrased helping the Programmer's help themselves...

[M. Edward Borasky]

Yes, it is a high-level proposal - I adhere to the top-down philosophy of
software design, as well as the SEI standards for software engineering
process. One does not communicate about large software objects like the
Linux kernel in small manageable chunks of C code in that process. Perhaps
the fact that I insist on a design specification, requirements documents,
code reviews, etc., is the reason nobody has volunteered to join the
project.

I think a team of three could pull it off in six months; there isn't that
much kernel code that has to be done. All the hooks are there in the /proc
filesystem, they just need to be organized in a rational manner. The scheme
Windows has for PerfMon is much better than the haphazard results in the
/proc filesystem, which have been submitted over the years in "manageable
chunks". The rest of Cougar is R code - R is extremely well documented - and
database work, for which any ODBC-compliant RDB will work.

The first task that needs to be done is to develop a high-level model of the
Linux kernel. There are numerous modeling/simulation/analysis techniques
that can be used for such models. Generalized Stochastic Petri Nets (GSPNs)
are probably the best known, and I believe a related package, DSPNExpress,
is available for Linux in an academic settings. See Christoph Lindemann's
home page at

http://ls4-www.cs.uni-dortmund.de/~Lindemann/

for the details.

M. Edward (Ed) Borasky
mailto: znmeb@borasky-research.net
http://www.pdxneurosemantics.com
http://www.meta-trading-coach.com
http://www.borasky-research.net

Coaching: It's Not Just for Athletes and Executives Any More!


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

RE: meminfo or Rephrased helping the Programmer's help themselves...

[John Carter]

On Fri, 6 Sep 2002, M. Edward Borasky wrote:

> Yes, it is a high-level proposal - I adhere to the top-down philosophy of
> software design, as well as the SEI standards for software engineering
> process. One does not communicate about large software objects like the
> Linux kernel in small manageable chunks of C code in that process. Perhaps
> the fact that I insist on a design specification, requirements documents,
> code reviews, etc., is the reason nobody has volunteered to join the
> project.

Ah. Culture shock.

The kernel development I have seen that worked and got in was of the 
"Show me the code" variety. ie. What gets into the kernel and attracts 
developers is not requirement specs and design documents, it is nifty 
code that mostly works.

First some comments on your Cougar web page...

a) There is one graphical tool that does a lot. Procmeter.

b) The plethora of tools for the task is a feature not a bug. The Unix 
design philosphy is to produce tools that do one job well. The result is a 
flock of single purpose tools that you can glue together via a scripting 
language as needed.

c) R. Love it. I wrote a largish program in it that had nary a single loop 
in it. Boggled the minds of my co-workers but it was really neat clean 
good code.

d) ODBC. Hate it. It is a fundamentally broken spec. I looked at it in
detail a few years back, I can't remember the details, just the
conclusion. Very Microsoftish and broken and a pig to create a server for.  
Maybe it has changed some in the intervening years (Ah yes, it is all
coming back to me, each client needed a database specific backend. ie.
There wasn't (at that stage) a database neutral "over-the-wire" protocol,
so you had to create both a server and a client stub. It would have been
so easy to define a db neutral over the wire protocol and a single client
stub. Instead you have to load a ODBC client for every type and version of
database you want to talk to.)

Anyway, serving ODBC is perhaps not your intent. Do you intend serving
ODBC or merely using a client to stuff data into Postgres or MySql?

e) Cougar would use the infrastructure that I need. Perhaps here is common 
ground. What is needed is a good clean _simple_ API to get the data out of 
the kernel. A requirements and design exercise on that would not go amiss. 

It is _vital_ that the API be simple to use, cheap in CPU cycles, simple 
to implement. I would recommend it be phased, a _simple_ first pass and a 
second more complex version and the design be such that no change need be 
made to the initial api. (Why? Some of the stuff you want will probably be 
hairier than most need, and hairy to implement. Project could easily get 
stuck. Remember, "Show me the code", we would need to get something nifty 
and tasty to attract interest and get it in the kernel.

I'm game for making input into such a requirement & design doc of that 
limited scope.

Once we have that we can look to see how to implement.

> The first task that needs to be done is to develop a high-level model of the
> Linux kernel.

Perhaps for Cougar. But for my interest and involvement the first thing is 
a simple, well thought out API to get the data we both need out of the 
kernel.

I will invest some thought in that and come up with an RFC. Will you 
please send me a list of the sort of info that you need out of the kernel. 
I will try sort it into 
  * Can be done in userland.
  * Needed from kernel.
  * Needed from kernel, but too hairy for first phase.


-- 


John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : john.carter@tait.co.nz
New Zealand

Good Ideas:
Ruby                 - http://www.ruby-lang-org - The best of perl,python,scheme without the pain.
Valgrind             - http://developer.kde.org/~sewardj/ - memory debugger for x86-GNU/Linux
Free your books      - http://www.bookcrossing.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

RE: meminfo or Rephrased helping the Programmer's help themselves...

[M. Edward Borasky]

M. Edward (Ed) Borasky
mailto: znmeb@borasky-research.net
http://www.pdxneurosemantics.com
http://www.meta-trading-coach.com
http://www.borasky-research.net

Coaching: It's Not Just for Athletes and Executives Any More!

-----Original Message-----
From: owner-linux-mm@kvack.org [mailto:owner-linux-mm@kvack.org]On Behalf Of
John Carter
Sent: Sunday, September 08, 2002 3:21 PM
To: M. Edward Borasky
Cc: linux-mm@kvack.org
Subject: RE: meminfo or Rephrased helping the Programmer's help
themselves...
Importance: Low


Anyway, serving ODBC is perhaps not your intent. Do you intend serving
ODBC or merely using a client to stuff data into Postgres or MySql?
I picked ODBC because R has an ODBC interface, as does Minitab and nearly
every Microsoft tool. The data will be mostly stored sequentially and
accessed by medium-to-complex queries. There is a *lot* of data if you
sample a bunch o' counters every 15 seconds or thereabouts, but there is not
the need to stick records in the middle. I do a lot of this sort of thing by
hand using Access now. Access by Windows clients is a requirement - ODBC is
one way to get that but not the only way. The main purpose of this part of
Cougar is to accumulate high-frequency performance data efficiently in a
disk database and allow queries against it. I had not thought about whether
to use Postgres, MySql or something else. The database requirements are
quite simple: capture high-frequency data in real time in binary and serve
it up with standard queries and stored procedures.


e) Cougar would use the infrastructure that I need. Perhaps here is common
ground. What is needed is a good clean _simple_ API to get the data out of
the kernel. A requirements and design exercise on that would not go amiss.

It is _vital_ that the API be simple to use, cheap in CPU cycles, simple
to implement. I would recommend it be phased, a _simple_ first pass and a
second more complex version and the design be such that no change need be
made to the initial api. (Why? Some of the stuff you want will probably be
hairier than most need, and hairy to implement. Project could easily get
stuck. Remember, "Show me the code", we would need to get something nifty
and tasty to attract interest and get it in the kernel.

I'm game for making input into such a requirement & design doc of that
limited scope.

Once we have that we can look to see how to implement.
That sounds like a good idea. My thoughts were patterned around the Windows
"PerfMon" API. That is, a "filesystem" structure but memory resident. My
requirements:
The fundamental variable is a *counter*. Counters come in two kinds:
cumulative, like the number of I/Os performed by a disk since bootup, and
static, like the number of kilobytes in page cache. Each counter has a
name - usually something very much like the name it has in the kernel code,
a label - a brief English or other language title such as one might use on a
graph, e.g., "CPU System Jiffies", a text description - what the counter is
a count or size of (like the "Explain" field in Windows PerfMon), a unit -
milliseconds, operations, kilobytes, etc., and of course a value. Since the
destination for these counters and their differences and ratios is a table
in a database, one would need the facility to easily create, say,
comma-separated value (CSV) files from the data structure. For speed, this
"filesystem" should be in RAM and in binary. Actually, the "/dev/kmem" API
from a number of other Unices isn't all that bad; you have a name table and
offsets into a memory area. The rest of it could be quite easily added.


> The first task that needs to be done is to develop a high-level model of
the
> Linux kernel.

Perhaps for Cougar. But for my interest and involvement the first thing is
a simple, well thought out API to get the data we both need out of the
kernel.
Actually - given the counters and some thinking, it isn't all that hard to
build an *empirical* model of the kernel, which is what I'd want to have
anyhow.


I will invest some thought in that and come up with an RFC. Will you
please send me a list of the sort of info that you need out of the kernel.
I will try sort it into
  * Can be done in userland.
  * Needed from kernel.
  * Needed from kernel, but too hairy for first phase
Everything I want is in the kernel - I want to make it available on a
read-only basis to userland code. The bare minimum right now is everything
that is in /proc/stat, /proc/meminfo, the disk queuing information in
/proc/partitions, the network data (packets and bytes in and out for each
interface, error counts - the kind of thing that "netstat" produces).
.


--


John Carter                             Phone : (64)(3) 358 6639
Tait Electronics                        Fax   : (64)(3) 359 4632
PO Box 1645 Christchurch                Email : john.carter@tait.co.nz
New Zealand

Good Ideas:
Ruby                 - http://www.ruby-lang-org - The best of
perl,python,scheme without the pain.
Valgrind             - http://developer.kde.org/~sewardj/ - memory debugger
for x86-GNU/Linux
Free your books      - http://www.bookcrossing.com

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: meminfo or Rephrased helping the Programmer's help themselves...

[Eric W. Biederman]

"M. Edward Borasky" <znmeb@aracnet.com> writes:

> Yes, it is a high-level proposal - I adhere to the top-down philosophy of
> software design, as well as the SEI standards for software engineering
> process. One does not communicate about large software objects like the
> Linux kernel in small manageable chunks of C code in that process. 

No but you merge with people in small manageable chunks of C code.

> Perhaps
> the fact that I insist on a design specification, requirements documents,
> code reviews, etc., is the reason nobody has volunteered to join the
> project.

All you currently have is a slide show.  I see nothing that even clearly
states the problem you are trying to solve.

> I think a team of three could pull it off in six months; there isn't that
> much kernel code that has to be done. All the hooks are there in the /proc
> filesystem, they just need to be organized in a rational manner. The scheme
> Windows has for PerfMon is much better than the haphazard results in the
> /proc filesystem, which have been submitted over the years in "manageable
> chunks". The rest of Cougar is R code - R is extremely well documented - and
> database work, for which any ODBC-compliant RDB will work.

In Linux the emphasis has been a system that doesn't need tuning, for
most tasks.  So it is no surprise the tuning nobs and the monitoring
you need to apply them are underdeveloped.  They haven't been much
used or needed.  

As for your comments on an ascii text version of /proc being
inefficient, that is an assertion that needs backing up.  There are
some inefficiencies in /proc but I have not seen ascii text being the
primary problem.

And with that observation I place an extreme doubt you have the skills
to accomplish what you would like to accomplish.  

> The first task that needs to be done is to develop a high-level model of the
> Linux kernel.

You obviously are clear what you are thinking of here, but I am not.
You need a high-level model of the Linux kernel in what sense?


Eric
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/