Recommended Tools: Sysstat Performance and Usage Activity Monitoring Tool for Linux

Recommended Tools:  Sysstat Performance and Usage Activity Monitoring Tool for Linux

-----------------------------------

Affected Roles:  Administrator, Owner

Related Digital Watchdog VMS Apps:  DW Spectrum® IPVMS

Last Edit:  January 21, 2022

-----------------------------------

Sysstat System Performance Activity Monitoring Tool

Sysstat is a system monitoring tool that can be used to track the performance and usage of system resources.  Additionally, Sysstat features a tool that can be scheduled to collect system performance and activity data by using cron.

This article will outline how to install and use the Sysstat System Performance Activity Monitoring Tool for Linux.

**NOTE:  This is an optional, supplemental tool that can be installed for Linux-based DW Spectrum® Servers. DW Blackjack® Servers shipping with Ubuntu 18.04 will come pre-loaded with this tool.

Supported/Affected Devices:

• DW Blackjack® Tower Server Series (Linux)
• DW Blackjack® Rack Server Series (Linux)
• DW Blackjack® E-Rack Series (Linux)
• DW Blackjack® P-Rack Series (Linux)
• DW Blackjack® Cube Series (Linux)
• DW Blackjack® Bolt Series

Enabling Repository Access

By default, DW Blackjack® Servers have OS updates disabled to prevent unexpected operating issues that may occur from auto-updates.

As so, Servers using Ubuntu have repository access disabled by default as well.

To install the Sysstat System Performance and Usage Activity Monitoring Tool, the repository will need to be made accessible to acquire the installation files.  If Sysstat is already installed and you are just seeking operating instructions, you can skip to the Sysstat Features and Commands section.

**DISCLAIMER:  Please be aware that DW Blackjack® Servers have OS updates and repository access disabled by default.  Altering a DW Blackjack® Server’s OS configuration from default settings are performed at the owner’s risk.

**NOTE:  When using the Linux command line, be aware that commands are case-sensitive. Press the ENTER key after each command, unless otherwise instructed.

Add Repository Access

1. Open the Terminal program on the Linux computer.  You can do this by using the Search function in the system Unity menu.

Alternatively, you can simultaneously press the Ctrl+Alt+T keys on the keyboard to launch the Terminal program.

2. The Terminal window will display.

To add the repository, use the command:

sudo add-apt-repository “deb http://us.archive.ubuntu.com/ubuntu/ xenial universe multiverse”

3. You will be prompted to enter the superuser password.  Use the root admin password (will not display):

Dw5pectrum

**NOTE:  For older DW Blackjack units purchased prior to June 18, 2021, the previously utilized default Linux OS login was “admin/admin”.

4. Follow adding the repository by adding the main repository as well:

sudo add-apt-repository “deb http://us.archive.ubuntu.com/ubuntu/ xenial main universe multiverse”

5. Next, to download updates, use the command:

sudo add-apt-repository “deb http://us.archive.ubuntu.com/ubuntu/ xenial-updates universe multiverse”

Follow with command:

sudo apt-get update

6. Allow the system to finish fetching the data packages.

You may enter the command again to double-check that the available updated data packages have been obtained by the computer.

sudo apt-get update

Repairing Broken Repository Dependencies

If an error occurs while attempting to add the repository, there is a chance that the atp-get command may be encountering issues when retrieving third-party dependencies.  For example, such an error may display as:

The following packages have unmet dependencies:

   E: Unable to correct problems, you have held broken packages.

If you encounter an issue with repository access or package corruption, refer to Repairing Broken Repository Dependencies for Ubuntu Servers for troubleshooting help.

Installing Sysstat with Linux

The Sysstat installation package can be found in the default repository as a package in all major Linux distributions or can be obtained manually if needed.

Otherwise, to acquire Sysstat version 12.5.4:

1. Open the Terminal program on the Linux computer.  You can do this by using the Search function in the system Unity menu.

Alternatively, you can simultaneously press the Ctrl+Alt+T keys on the keyboard to launch the Terminal program.

2. The Terminal window will display.

Download the latest version of the syssstat package by using the command:

wget http://pagesperso-orange.fr/sebastien.godard/sysstat-12.5.4.tar.gz

**NOTE:  If you must download a different version than Sysstat v12.5.4, change the version number at the end of the command.  For example, to download version 11.0.0 instead, the command would be “wget http://pagesperso-orange.fr/sebastien.godard/sysstat-11.0.0.tar.gz”.

3. After acquiring the Sysstat package, extract the downloaded package and access the associated directory to begin the compiling process.

Use the commands:

tar –xvf sysstat-12.5.4.tar.gz

cd sysstat-12.5.4/

4. [OPTIONAL] For compilation setup, you have two options:
   1. ./iconfig – provides flexibility for selecting and entering the customized values for each parameter

sysstat-12.5.4 ./iconfig

1. 2. ./configure --help – provides a list of different supported options for configuration

sysstat-12.5.4 ./configure --help

5. Moving ahead, use the following commands to compile the sysstat package:

./configure

make

make install

6. After the compilation process has completed, an output similar to below will display.

Verify the sysstat version by using the command:

mpstat -V

Updating Sysstat in Linux

By default, sysstat uses “/usr/local” as its prefix directory, while all binary/utilities are installed in the “/usr/local/bin” directory. 

However, if there is an existing sysstat package already installed, it will be stored in the “/usr/bin” directory.  This is due to the existing sysstat package preventing updated versions from being reflected as the “$PATH” variable does not have the “/usr/local/bin” directory set.

If there is an existing sysstat package installed, but the tool must be upgraded, remove the existing sysstat version and make sure that “/usr/local/bin” exists in the “$PATH” or set the –prefix option to “/usr” during the compilation process before updating.

1. Use the commands:

apt-get remove sysstat

./configure --prefix=usr

make

make install

2. Verify the updated version of sysstat with the command:

mpstat -V

Sysstat Features and Commands

The following are included tools included in the Sysstat packages:

mpstat – Processor Statistics

The mpstat feature can be used to view statistical data on the performance of a system’s CPUs, for both single and dual systems.

The following commands can be utilized for use with the mpstat feature.

• Using the mpstat command without specifying options will display the Global Average Activities by all CPUs.

mpstat

• Using the –P (Indicate Processor Number) flag and ‘ALL’ will display statistics about all CPUs, starting with zero (0) as the first CPU.

mpstat –P ALL

• You can display the statistics for the number of CPU iterations (N) after a set n seconds interval with the average performance of each CPU with the following command.

mpstat –P ALL <N> <n>

**NOTE:  The example below is set for two (2) iterations at five (5) second intervals.

• Using the –I flag will display the total number of interrupt statistics per processor.

mpstat –I

• Using the –A flag will display all of the above information with a single command.  This is similar to using the “-u –I ALL –P ALL” command.

mpstat –A

pidstat – Process and Kernel Threads Statistics

The pidstat feature can be used to monitor processing and currently running threads that are managed by the kernel.

The following commands can be utilized for use with the pidstat feature.

• Using the pidstat command without any argument will display all active tasks.

pidstat

• Using the –P (Indicate Processor Number) flag and ‘ALL’ will display all active and non-active tasks

pidstat –p ALL

• Using the –d flag will provide I/O statistics at (n) second intervals.  This can be useful if a system is experiencing heavy I/O activity and need information about the processes consuming the resources.

pidstat –d <n>

**NOTE:  The example below is set to display I/O statistics at two (2) second intervals.

• Using the –t and –p flags, specifying a Process ID (TGID) number (x), a time interval (n), and how frequently (z) will display CPU statistics along with threads about the specified PID.

pidstat –t –p <x> <n> <z>

**NOTE:  The example below is set to display CPU statistics for Process ID (4164) at a time interval of two (2) seconds, three (3) times.

• Using the –rh flag, specifying a time interval (n), and how frequently (z) will display the processes for memory utilization.

pidstat –rh <n> <z>

**NOTE:  The example below is set to display memory statistics at a time interval of two (2) seconds, three (3) times.

• Use the –G flag to display instances of a specified process (x).

pidstat –G <x>

**NOTE:  The example below is set to display processes of the VBox service.

• • Use the –t flag to display all threads

pidstat –t –G <x>

**NOTE:  The example below is set to display processes of the VBox service.

• Use the –R flag to display real-time priority and scheduling information.

pidstat -R

sar – System Activity Reporter

The sar feature can be used to obtain reports about the system’s overall performance.  It does this by calculating the rates and ratio of system request completion times and I/O counts that are tracked by the Linux Kernal by default.  This can be helpful when trying to identify system bottlenecks that may be causing performance issues.

**NOTE:  The sar feature reports all activities over a period of time.  This means that sar is only useful when used during period of normal operation, rather than during periods of downtime such as during weekends or holidays.

The following commands can be used for the sar feature:

• To invoke sar interactively, use the –u and –o flags to create a file tilted as “sarfile” in the sar feature’s directory.  This will collect a number of reports (N) at a specified interval (n) of CPU details.

sar –u –o sarfile <n> <N>

**NOTE:  The example below is set to a time interval of two (2) seconds, five (5) times.

• To invoke sar non-interactively, use the /usr/local/lib/sa/sa1 and /usr/local/lib/sa/sa2 cron scripts.
  • /usr/local/lib/sa1 – a shell script that can be used for scheduling cron to create a daily binary log file.
  • /usr/local/lib/sa2 – a shell script that can be used to change a binary log file into a human-readable form.

Use the following cron entries to make this non-interactive:

            In the above example, the sa1 script will call for the sadc (System Activity Data Collector) utility to fetch the data at a two (2) second interval.

            The sa2 script will call for the sar utility to change the binary log file into a human-readable form.

• Use the –q flag to check run queue length, the total number of processes running, and the load average of the processes.

sar –q <n> <N>

**NOTE:  The example below is set to a time interval of two (2) seconds, five (5) times.

• Use the –F flag to check statistics about the mounted file systems.

sar –F <n> <N>

**NOTE:  The example below is set to a time interval of two (2) seconds, four (4) times.

• Use the –n DEV command to view network statistics.

 sar –n DEV <n> <N> | egrep –v lo

• Use the –d flag to view block device statistics (like iostat).

sar –d <n> <N>

• Use the –r flag to display memory statistics.

sar –r <n> <N>

• Use the safd –d command to extract data in a format that can be processed by databases.

safd –d <log file directory> -- -n DEV | grep –v lo