BigAdmins

Whether you’re unsatisfied with your Solaris system performance or just want to get the most out of what the machine is capable of, there is a cyclical process to improve it, which consists of determining where the processing slow-down is occuring (the bottleneck), fixing it, then repeating the process, until the most significant bottlenecks are reduced.

First, we should learn about some tools to help us monitor system performance. SAR (the system activity reporter) is the time-honored (and very cryptic) standard UNIX performance monitoring tool. How do we use SAR to see what the Sun server has been doing?

SAR collects data on the state of the system using a program called “sadc” (the system activity data collector), which is usually launched from two standardized shell scripts (called sa1 and sa2) that can be launched from “cron” as frequently as you want to collect performance data by adding a few lines to your crontab with:

	mysunserver# crontab -e

Now add these lines..

	# Starting at 8am collect system activity records
	# every 20 minutes for 12 hours
	# 20 minutes = 1200 seconds
	# 12 hours with 3 samples each hour = 36 loops
	0 8 * * 1-5 /usr/lib/sa/sa1 1200 36

	# After the 12 hour period,
	# collect a system activity report
	30 20 * * 1-5 /usr/lib/sa/sa2 -A

After a full day of systems monitoring, we can use “sar” to extract the collected data:

	root@mysunserver:~$ sar

	SunOS mysunserver 5.10 Generic_125100-06 sun4v    07/25/2007

	23:00:01    %usr    %sys    %wio   %idle
	23:20:00       0       0       0     100
	23:40:00       0       0       0     100
	00:00:00       0       0       0     100
	00:20:00       0       0       0     100
	00:40:02       0       1       0      99
	01:00:01       0       2       0      98
	01:20:05       0       2       0      98
	   .
	   .
	   .

In the default SAR display, we see, at every sample interval, how the computer was spending it’s time.

Each of the four columns shows how the CPU’s time was being divided:

• %usr: running UNIX processes (“user time”)
• %sys: processing kernel system calls on behalf of processes (“system time”)
• %wio: sitting around waiting to read/write data (to disks or network devices), as demanded by processes (“waiting for I/O”)
• %idle: just plain doing nothing as there were no processes that demanded the CPU’s time (“idle”).

NOTE: These statistics are not quite the same as the system “load average”, which, precisely defined, is the AVERAGE number of PROCESSES that are in the process table which are WAITING for the CPU to be able to service them. In theoretical terms, the “load average” resembles a percentage of processor utilization that can exceed 100% when the system is overutilized (load avg. > 1.00) or fall below 100% when the CPU is not fully utilized (load avg. < 1.00). Theoretically, a system that sustained a load average utilization of 0.50 would operate with the same apparent performance if the CPU ran at half the clock-speed.

If may be best to think of load average as CPU demand, and to think of the SAR statistics as actual CPU utilization.

If SAR reports the CPU is more than 10% idle, then your system is probably not being held back by a slow processor or slow storage devices. The culprit may just be inefficiently-written software, which fails to optimize for the particular platform which it runs on.

Time spent by the system waiting for I/O (%wio) is one of the most important to attack. This “wasted” time is sometimes called a “wait state”. As those I/O processes are necessary, it is only “wasteful” because the CPU just sits around doing nothing until the disk has had a chance to fetch the requested data, or upload the data, or do whatever other I/O activity which was requested by the UNIX process.