Find average load

[OOagent137]

I have Plesk and it gives the average load over 1 minute, 5 minutes, and 15 mintutes. I'd like to know the average load over a day/week/month. How could I do this?

[awsolutions]

Hello,

Search freshmeat.net or http://sourceforge.net/ for a script that might log the results into a database and tell you about it.

I don't know if one does exists, but if it did it would be listed on thsoe two sites.

[Striker]

Actually, I'll be posting a script addon for MRTG soon so you can watch your loads like that.

[OOagent137]

Striker, what is MRTG?

Awsolutions, You seem to know a lot about servers (compared to me anyway). How do you keep tabs on your server?

[awsolutions]

Hey,

1.) Have a strict TOS and enforce it! I had a client install a proxy cgi script that all kids in a firewalled college got access to. I deleted it right away.

2.) I monitor TOP alot when I am logged into the box. I like to watch my server - it looks fun! I also watch pstree to make sure httpd doesn't have too many child proccesses.

3.) In /etc/httpd/conf/httpd.conf there is an option for server-status in the extended mode. I enabled it so I can (from a certain IP address) monitor what my apache is serving out. This is great for catching the warez kiddies!

4.) phpSysInfo (sourceforge.net I believe has it) let's you see your server specs real quick from the web.

5.) As for monitoring I use a utility called SIM that will test the basic servers (mysql, ssh, qmail, proftp, httpd, named) and will restart them and e-mail you a notification if one goes down. This is set up in cron to run every 5 minutes.

6.) Third party external monitoring service that will send me emails to notify me if the server is down. I am trying to have this hooked up to my cell phone but nextel doesn't (to my knowledge haven't had time to fool around) e-mail to phone service yet.

All in all using these techniques and a bandwidth monitor (too many to name) have allowed me to catch suspicious behavior before it became a harm to other clients. In the cases where it did cause harm, the box was promptly put back up and i simply watched or fed through error logs or access logs to try and see what the culprit was and then either renamed the file or set it's execution to none until the client was notified.

Thanks

Also, I typed up lengthy instructions for your mysql dump only to have IE crash with a DNS error for me;-( I will retype them soon

[amplify]

Run cron.php every hour by entering the below into your cron jobs (edit to suit your needs):

0 * * * * /usr/bin/php -q /path/to/cron.php

Cron.php:

<?php
$host = "localhost";
$user = "username";
$pass = "password";
$db = "database";

$dbh=mysql_connect ("$host", "$user", "$pass") or die ('I cannot connect to the database because: ' . mysql_error());
mysql_select_db ("$db");
$uptime = explode(": ", `uptime`);
$uptime = explode(", ", $uptime[1]);
mysql_query("INSERT INTO `uptimes` ( `id` , `loads` , `timed` ) VALUES ('', '$uptime[1]', '".time()."')");
?>


Uptime.php:

<?php
$host = "localhost";
$user = "username";
$pass = "password";
$db = "database";

$dbh=mysql_connect ("$host", "$user", "$pass") or die ('I cannot connect to the database because: ' . mysql_error());
mysql_select_db ("$db");
$uptime = explode(": ", `uptime`);
$uptime = explode(", ", $uptime[1]);
echo "One minute load average: $uptime[0]<br>";
echo "Five minute load average: $uptime[1]<br>";
echo "Ten minute load average: $uptime[2]<br>";
echo "-------------------------------------<br>";
$dailyUptime = mysql_fetch_array(mysql_query("select round(avg(loads),2) as up from uptimes where timed > ".(time()-86400)));
$weeklyUptime = mysql_fetch_array(mysql_query("select round(avg(loads),2) as up from uptimes where timed > ".(time()-604800)));
$monthlyUptime = mysql_fetch_array(mysql_query("select round(avg(loads),2) as up from uptimes where timed > ".(time()-2592000)));
echo "Daily load average: $dailyUptime[up]<br>";
echo "Weekly load average: $weeklyUptime[up]<br>";
echo "Monthly load average: $monthlyUptime[up]<br>";
?>


MySQL table structure:

CREATE TABLE uptimes (
id int(6) NOT NULL auto_increment,
loads decimal(3,2) NOT NULL default '0.00',
timed varchar(255) NOT NULL default '',
PRIMARY KEY (id)
) TYPE=MyISAM;


Example (http://3nom.com/~sratix/up/uptime.php) - Example of Uptime.php. Of course, you can pretty it up with jazzy fonts and HTML later. :D


This works, but I'm not the best programmer. I hope this is what you're looking for. :)

P.S: Send me your revisions of this so that I can learn from my mistakes!

[awsolutions]

How would that be accurate? It would only run once every hour recording only the past 10 minutes? You are loosing 50 mintues of data.

A better way, run as cron every 10 minutes recording only the last 10 minutes of data and then averaging that out.

[Striker]

MRTG stands for Multi Router Traffic Grapher and is normally used to track bandwidth usage; however, it can be easily reconfigured to track other data.

Here is cpu usage (http://www.myhotsprings.com/mrtg/cpu.html) for my server.
Here is memory usage (http://www.myhotsprings.com/mrtg/mem.html) for my server.

I'll be posting scripts and examples of how to implement this later. But before that, you'll need to read the howto I wrote on installing MRTG (http://www.serverbeach.com/forums/showthread.php?s=&threadid=144)

[OOagent137]

Thanks for all the suggestions thus far.

I have a lot of work cut out for me in this area.

Striker, I plan on installing MRTG later and will use your tutorial Striker.

Amplify, your script appears to be something I could use. I think I will go w/ a pre-written one though, uneless it's to inflexible. Then I will use yours.


Awsolutions:

1) To enforce your TOS, what techniques do you use? Just look at every single clients websites and log files?

2) I run top a lot as well. I think my problem is that I don't understand what the output means a lot of the time. Also, with pstree, how many child processes are too many? What do those processes represent?

3) This sounds interesting. I need to look into how to do that, I'm sure it's somewhere on Google.

4) I will check this out.

5) I will check this tool out as well.


Thanks for all your help guys. I appreciate this.

[awsolutions]

Hello,

Well, I use a bandwidth script to show all my clients usage: http://forum.plesk.com/showthread.php?s=&threadid=8597 PLESK ONLY (Sorry guys!) I am sure there are others for every CP.

Then any that are high I view there webstats directory and look specifically at the highest data transfer files. These will usually be either forums (viewtopic.php=) or exe, mp3, and such. Next is evaluating weather the data is illegal. Of course mp3's are and I do not host them unless it is a bands website OR a copyright letter has been faxed. I have had issues with the RIAA in the past and would prefer not to deal with them again!

Also, the next step I do about once a month (well more like once every 3 but I should every month) is a locate *.fileextension and then this will give me a list of all such files. I can track down alot of illegal files very often with this command. Do all the popular warez extensions:
zip
rar
wma
rm
ra
mp3

I had a big list on my old desk but I got a new one this summer and msiplaced it for now. There were about 10 -15 extensions I would look for.

Thats it as far as warez.

For scripts I have enables apache server status in my httpd.conf file. Here you can often catch some wildfire scripts as well as with pstree. this one shot screen shows everything apache is doing - it's also fun to watch by clicking reload every 10 seconds!
It's easy to set up and I would be more than happy to help you.

You can then see which websites are high load. I had a site getting 1 million hits a day, but not too bad on bandwidth. So it was causing alot of cpu load but not bw load. Boy it was about just the only site showing up in the apache server status. You can also catch cgi-scripts that will run away as they will show up for a long time on the screen. If you click reload and it's still there after a few reloads you may want to check the script out! This is how I caught a proxy script some college student placed on my server.

As for pstree and child proccesses here are the stats for my server with 752 domains:
init-+-atd
|-bdflush
|-4*[couriertcpd]
|-crond
|-eth0
|-httpd-+-httpd---XXXXX.cgi
| `-20*[httpd]
|-httpsd-+-9*[httpsd]
| `-3*[rotatelogs]
|-kapmd
|-keventd
|-2*[kjournald]
|-klogd
|-kscand
|-ksoftirqd_CPU0
|-kswapd
|-kupdated
|-4*[logger]
|-mdrecoveryd
|-6*[mingetty]
|-named---named---3*[named]
|-ntpd

|-qmail-send-+-qmail-clean
| |-qmail-lspawn
| |-qmail-rspawn---6*[qmail-remote]
| `-splogger
|-rcd
|-safe_mysqld---mysqld---mysqld---44*[mysqld]
|-sshd---sshd---sshd---bash---su---bash---pstree
|-sshd
|-stunnel
|-syslogd
`-xinetd---qmail-smtpd


Mysql is pretty accurate and is always at 40-60. Httpd is around 28-34 usually. Mail is about 6 unless the queue is going and then it will be 100 (server max setting I put in). I removed a few things from above to protect my specific firewall program and the cgi program running ;-)

On my other box with less than 100 domains (70ish I believe)
init-+-atd
|-bdflush
|-4*[couriertcpd]
|-crond
|-eth0
|-httpd---18*[httpd]
|-httpsd-+-8*[httpsd]
| `-3*[rotatelogs]
|-kapmd
|-keventd
|-2*[kjournald]
|-klogd
|-ksoftirqd_CPU0
|-kswapd
|-kupdated
|-4*[logger]
|-mdrecoveryd
|-6*[mingetty]
|-named---named---3*[named]
|-ntpd

|-qmail-send-+-qmail-clean
| |-qmail-lspawn
| |-qmail-rspawn
| `-splogger
|-rhnsd
|-safe_mysqld---mysqld---mysqld---2*[mysqld]
|-sshd---sshd---sshd---bash---su---bash---pstree
|-sshd
|-stunnel---stunnel
|-syslogd
`-xinetd-+-in.proftpd
`-qmail-smtpd

So I hope some of this helps!

[OOagent137]

It has helped, a bunch!

You mentioned a few things with apache, something about extended mode and putting some sort of enables on. How do I do that (I don't understand Apache that well, but I've configured it for PHP, SSI, and done some other minor tweaks with a home box)?

Oh, and btw Awsolutions, thanks for the TONS of help you've provided. I am finally starting to get some grasp on server administration. It's nice to be able to know what I'm doing a little bit w/ the server :)

[awsolutions]

OO,

Good to hear! I once got help to ;-) / Still do get help!

For extended mode:

login in as root:
# pico /etc/httpd/conf/httpd.conf

Search for (Ctrl + w) ExtendedStatus

Make sure it says:
ExtendedStatus On

Next:
Search for server-status

You will see:
#<Location /server-status>
# SetHandler server-status
# Order deny,allow
# Deny from all
# Allow from
#</Location>


Uncomment and put in the following:
<Location /server-status>
SetHandler server-status
# Order deny,allow
# Deny from all
Allow from xxx.xxx.xxx.xxx
</Location>

The x's should be your IP address or range so only you have access but not mandatory(you could keep this one commented). Then restart apache:
/etc/rc.d/init.d/httpd restart

Then go to:
http://www.yourdomain.com/server-status

Hope this helps!

[OOagent137]

Thanks for the tip brian.

I got it working, however, I wasn't able to restrict access. For instance, anyone (I tried from several different IPs and had friends trr) could access that page on any domain on our server.

Here is what I have:

<Location /server-status>
SetHandler server-status
# Order deny,allow
# Deny from all
Allow from 65.67.115.xxx
</Location>

*the xxx represents real numbers, just didn't want to post my IP on the web, not like it really matters that much anyway though.

What am I doing wrong? Thanks.

[nosleep]

Try uncommenting these two lines:

# Order deny,allow
# Deny from all


so you have:

<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from 65.67.115.xxx
</Location>

[OOagent137]

That worked like a champ (why didn't I think of that).

Ok, I set it up for my IP for right now, but it does occasionally change. I guess I need to call my ISP to see what range it will change to. Once I find that, how do I setup a range instead of a single IP?

Also, how can I specify multiple IPs?

Thanks.

[OOagent137]

I'm also interested in analyzing this info from the /server-status page.

What are the stats that matter? I'm thinking "CPU, SS, Conn, and Slot".

Now, what are normal values for these? I mean, I'm seeing around 4 seconds max for most scripts for the CPU. Is that normal? High? Low?

Also, the Conn, what is a normal value for that? What is a high value?

For the Slot, what is that a measure of? All Apache says is this "Total megabytes transferred this slot". What does that mean? What is a slot? How long of a time does that measure? Ect....

Also, what are the different mode types and do they matter?

Basically, I just need to know when I need to start paying attention to a site.

[OOagent137]

Awsolutions, I looked at PHP SysInfo on sourceforge.net. However, the only thing I could find said that the type was a source.gz (I assume this means source code). Where did you DL it at?

Also, on sites like sourceforge, how can I download the file directly to the server? I mean, a URL like www.sourceforge.net/file.tar.gaz isn't available (it is, but I have no way of knowing it). The download link automatically pops up.

Thanks.

[awsolutions]

Hello,

An Ip block I beleive can be added by :
XXX.XXX.XXX.*

Multiple ones can be seperated by a comma.

I think this is how*

Sorry about the commented out section before!

[Tino Didriksen]

Originally posted by OOagent137
...I looked at PHP SysInfo on sourceforge.net. However, the only thing I could find said that the type was a source.gz...It's PHP, so the only thing you will find is the source code. PHP isn't compiled to binaries; it's an interpreted language.

In other words, PHP Sysinfo's source.gz is the correct file to get.

[OOagent137]

The comma thing didn't work but on a hunch I did this:


123.123.123.123 && 321.321.321.321

That worked.

Also, for the range of IPs....it's this:

123.123.123. && 321.321.321.321

Thanks for the help with this. I'm glad I have this working.

[OOagent137]

Ok, I set up PHP SysInfo.

There is one problem though. I go to the file and I get this:

-------------------------------------------

System Information: N.A. (xxx.xxx.xxx.xxx)
System Vital
Canonical Hostname N.A.
Listening IP xxx.xxx.xxx.xxx
Kernel Version N.A.
Uptime 0 minutes
Current Users 2
Load Averages N.A. N.A. N.A.

Network Usage
Device Received Sent Err/Drop

Hardware Information
Processors N.A.
Model N.A.
Chip MHz N.A.
Cache Size N.A.
System Bogomips N.A.
PCI Devices none
IDE Devices none
SCSI Devices none

Memory Usage
Type Percent Capacity Free Used Size
Physical Memory % 0.00 KB 0.00 KB 0.00 KB
Disk Swap % 0.00 KB 0.00 KB 0.00 KB

Mounted Filesystems
Mount Type Partition Percent Capacity Free Used Size
/ /dev/hda2 13% 45.48 GB 7.04 GB 55.33 GB
/boot /dev/hda1 23% 27.11 MB 8.88 MB 37.95 MB
/dev/shm none 0% 247.39 MB 0.00 KB 247.39 MB
Totals : 13% 45.75 GB 7.05 GB 55.61 GB

-------------------------------------------

Thus, to me, it only appears to be picking up my hard drive partitions correctly (though every IP listed was correct). What's the deal?

[freqdafunk]

i get these same error (or lack of infos)

anyone have clues?

[awsolutions]

php open base probably?

Try that vhost.conf I was talking about ;-)

[OOagent137]

This is the post he was referring to:

http://www.serverbeach.com/forums/showthread.php?threadid=139&highlight=openbasedir

Ok, well, I made a vhosts.conf file in this directory:

/home/httpd/vhosts/mydomain.com/conf/vhosts.conf

And, w/ the -w switch, I fired up Pico and added these lines:

<Directory /home/httpd/vhosts/mydomain.com/httpdocs>

php_admin_value open_basedir /home/httpd/vhosts/mydomain.com/httpdocs:/tmp

</Directory>

Then I did this:

[root@s1 conf]# locate my_apci_rst
/usr/local/psa/admin/bin/my_apci_rst
/usr/local/psa/admin/sbin/my_apci_rst

I ran both b/c I wasn't sure which one to run. Still, the script works the same. However, the script is located in here:

http://www.mydomain.com/test/phpSysInfo-2.1/

Would it effect it if it's not in the root directory?

Any other ideas?

[OOagent137]

In case any are wondering, this is how I got it installed properly. In the vhost.conf file (/home/httpd/vhosts/domain.com/conf, you may have to create it), place the following text:

<Directory /home/httpd/vhosts/domain.com/httpdocs>
php_admin_value safe_mode 0
php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs/:/"
</Directory>

That did the trick for me.

[freqdafunk]

well, I did that and it didn't work. then went back and tried disabling open_basedir in my php.ini file and that didn't work either.

bummer, ah well - at least it shows me SOME stats :) guess i'll have to try some other script.

[knightfoo]

Originally posted by OOagent137
In case any are wondering, this is how I got it installed properly. In the vhost.conf file (/home/httpd/vhosts/domain.com/conf, you may have to create it), place the following text:

<Directory /home/httpd/vhosts/domain.com/httpdocs>
php_admin_value safe_mode 0
php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs/:/"
</Directory>

That did the trick for me.

Doing this completely defeats the purpose of having open_basedir :) The open_basedir is designed to prevent scripts from opening files outside of the domain they are hosted in. Setting this to / means that if someone were to exploit a script, they could read any file on your system, possibly making it easier to launch an attack or compromise your server. If you store any kind of confidential information on your server (credit card info, personal info) it is very dangerous to leave yourself open like this. You should be very cautious when changing this setting, especially if you set it to "/".

-knightfoo

[knightfoo]

There is a program named "sar" installed on RedHat servers which records data at certain periods (maybe hourly, but it can be adjusted) which you can review at a later date. You can log in as root and just run "sar" for some basic output, but you should check out the man pages for more information. sar records a lot of information, such as memory usage, swap page usage, cpu usage, number of processes, rate that processes are spawned, etc.

-knightfoo

[OOagent137]

Well, I could set it for the specific directory of phpSysInfo.

However, you really recommend never taking open_basedir off?

[awsolutions]

Well, you set it to the / directory. You should set it to:

php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs/:/tmp:/proc" for phpsysinfo.

[OOagent137]

Oh, btw knightfoo, thanks for the tip on the "sar" program. That looks like something I would be interested in using. I need to figure out how to get it to automatically store to a text file and give me averages per week/day, but that looks like a real useful tool.

[OOagent137]

I tried what you said Brian. It gives me the base directory error again:

This is what I have:

<Directory /home/httpd/vhosts/domain.com/httpdocs>
php_admin_value safe_mode 0
php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs/:/tmp:/phpSysInfo-2.1/"
</Directory>

Basically, I go to www.domain.com/phpSysInfo-2.1/ if I want to check the script. Any ideas why this doesn't work?

[awsolutions]

That's not what I said!
I said:

<Directory /home/httpd/vhosts/domain.com/httpdocs>
php_admin_value safe_mode 0
php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs/:/tmp:/proc"
</Directory>



With openbase it starts out with the directory you want to give permission to and then the areas you are giving it permission to read / write. You had phpSysInfo and php already has permission to read there since it is in a lower directory. PhpSysInfo reads /proc/ (load averages) and one other directory for mounted file systems I believe (someone needs to help both of us out there since I don't have that displaying on my sys info anymore after a server migration)

So a valid on would look like:

php_admin_value open_basedir "/home/httpd/vhosts/domain.com/httpdocs:/
proc:/tmp"

[OOagent137]

Ok, I'm beginning to understand what I'm putting into the vhost.conf file a bit better.

As far as the mounted directories, I'm not sure. You might try these:


/ or /dev/hda2 or /boot or /dev/hda1 or /dev/shm

I'm curious about this line though:

php_admin_value safe_mode 0

Now, I have safe mode off on my server. So why do I need it?

Also, knightfoo's warning has made me a bit nervous. Am I ok with allowing access to these directories? Should I be willing to take off the open_basedir warning for clients? Thanks for the help!