I had worked on getting the sysstat utilities onto unRAID in the past, and am working on a plugin to do it now.

 

 

How the data is collected is a sticky wicket.  Higher the resolution, the bigger the data files.

 

Yes, I could make it all configurable as hell, but for starters, I want to get it working with something simple, as the data collection backend needs to go into cron.

 

What I am leaning towards is:

 

For HIGH resolution -- sample once every 5 seconds.  Will keep 1 hour of high-resolution data.

For LOW resolution  -- sample once every minute.  Will keep 1 month of low resolution data.

 

Comments?

 

 

In the past I used to change resolution based on usage patterns.

Higher resolution during business hours (5 to 15 minutes ) and hourly at other times.

Low resolution seems fine to me.

What kind of size of data files are we talking about here... that must be the key factor as mentioned in the OP

What kind of size of data files are we talking about here

 

Preliminary estimate is about 8.5MB/hr at high resolution

FWIW, this utility will require:

 

- php upgrade package

- lighttpd webserver package

- a number of libs necessary for php gd

 

 

 

Making the graphs requires gd, and a number of graphics support libs.

 

However, there are some flash and javascript tools that can use JSON-like data access, to do the graphics in the browser.

 

Anyone have comments one way or the other?

 

 

Making the graphs requires gd, and a number of graphics support libs.

 

However, there are some flash and javascript tools that can use JSON-like data access, to do the graphics in the browser.

 

Anyone have comments one way or the other?

 

I gave it some thought. I don't have a preference on how the graphics are generated.

Do what's easier and more globally supported.

Actually, I just found another possible solution...

 

http://ploticus.sourceforge.net/doc/welcome.html

Actually, I just found another possible solution...

 

http://ploticus.sourceforge.net/doc/welcome.html

That looks very powerful.

After playing around with Ploticus, I've decided to use it.  It handles the large datasets from sysstat fast, and it understands the date/time stamps natively, as time-series data.

 

And it only needed one library (libpng)!

 

 

Will it still require "php upgrade package" and "lighttpd webserver package"?

After playing around with Ploticus, I've decided to use it.  It handles the large datasets from sysstat fast, and it understands the date/time stamps natively, as time-series data.

 

And it only needed one library (libpng)!

 

 

 

This is an excellent find! Cant wait to see some of the datasets.

Will it still require "php upgrade package" and "lighttpd webserver package"?

 

Most likely not, as I'm using it with shell_exec, and not CGI.

Will it still require "php upgrade package" and "lighttpd webserver package"?

 

I confirmed that it works with the php CLI and emhttp in stock unRAID.

 

Ploticus has a few quirks I need to work through, but I think this will turn out OK.

 

 

Will it still require "php upgrade package" and "lighttpd webserver package"?

 

I confirmed that it works with the php CLI and emhttp in stock unRAID.

 

Ploticus has a few quirks I need to work through, but I think this will turn out OK.

 

What's a disk utilization graph look like?

At first I did not see that much need for this (personally).

But after some thought, it would be good to see how monthly parity check & daily locate db scan with --md5summing show up with CPU and Disk Utilization.

 

 

Will it still require "php upgrade package" and "lighttpd webserver package"?

 

I confirmed that it works with the php CLI and emhttp in stock unRAID.

 

Ploticus has a few quirks I need to work through, but I think this will turn out OK.

 

 

This is pretty damn cool 

This is pretty damn cool  Cool

 

It has already helped me discover a problem I didn't know I had.  I seems when I resume my test server from s3 suspend, disk i/o drops significantly, but not because of i/o wait.... curious.

 

I think I'm also going to use this for smarthistory, when I get around to rewriting it.

Hello,

 

Would rrdtool work?  Something like this http://oss.oetiker.ch/rrdtool-trac/wiki/SystemMonitorScripts

 

Although, pliticus looks nice.

 

Al

Would rrdtool work?

 

I looked into that first, but it was a real pain trying to get the sadc data into rrd format.

What's a disk utilization graph look like?

 

Here are the stats that are available in sysstat:

tps 
Indicate the number of transfers per second that were issued to the device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size.

rd_sec/s
Number of sectors read from the device. The size of a sector is 512 bytes.

wr_sec/s
Number of sectors written to the device. The size of a sector is 512 bytes.

avgrq-sz
The average size (in sectors) of the requests that were issued to the device.

avgqu-sz
The average queue length of the requests that were issued to the device.

await 
The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.

svctm 
The average service time (in milliseconds) for I/O requests that were issued to the device.

%util 
Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

 

Here a graph of what I have working so far.  The activity was copying a 4.5GB .iso file to disk 1.

 

Great Job BubbaQ!

Thanks... did you try it?

 

I have not at this time. I'm back to work and have been concentrating there.

And this weekend I'm making plans to go to PA.

I plan to try it next week along with retrofitting my plugins to beta-2.

Since the directory changed I have to alter all the slackbuild scripts.

Out of curiosity can we have long term data sets. i.e. stats since initial turn on.

 

Obviously we would not want super fine resolution because of the large log load but I am thinking generic trending over the entire life of the server?