Monitoring Sun X86 and X64 Systems with Hardware Sentry

How to monitor Sun X86 and X64 systems with Hardware Sentry KM.

Related Topics

This section covers the Sun Fire systems running on an AMD Opteron processor, including Sun Fire v20z, v40z, X2100, X4200 and all of their evolutions. Such systems are often referred as Sun X86 (i.e. Intel 32 bits) systems while they actually run on X64 processors (AMD 64 bits, followed by Intel’s EM64T). This platform is sometimes also referred as “i86pc” by Sun.

About Sun X64 systems

Sun X64 systems can be considered as the entry level server series from Sun. They are actually simple “PC” servers very similar to what is found from Dell, HP (ProLiant) and IBM (xSeries).

Sun X64 systems can run the following operating systems:

  • Sun Solaris 10
  • Microsoft Windows Server 2003
  • RedHat Enterprise Linux

Unlike their SPARC-based counterparts, Sun X64 servers feature an internal RAID controller (OEM’d from LSI).

Hardware Instrumentation

Solaris system utilities (prtdiag, psrinfo, kstat, etc.)

This section only applies to Sun X64 servers running the Solaris operating system.
Unlike in the classic sun4u architecture, there is no Solaris utility that provides information about the environment of the server (temperature sensors, voltage sensors, fans and power supplies): neither prtdiag nor prtpicl is able to report such information.

However, all of the other Solaris utilities used by Hardware Sentry in the sun4u architecture work the very same way in sun4v systems:

  • psrinfo for the processors
  • ifconfig, kstat and ndd (if required) for the network cards
  • iostat and dd for the internal disks

Some of these utilities require root privileges for proper execution: ndd and dd (in some cases).
Please read carefully the section about sun4u servers to learn more about how Hardware Sentry deals with these utilities.

Linux system utilities (ethtool, smartctl)

This section applies to Sun X64 machines running a Linux-based operating system only. As with any server running Linux, there are some system utilities that will provide information about the network cards and the disks (as seen by the operating system, i.e. non RAID disks).

These commands include:

  • ifconfig, mii-tool and ethtool for the network interfaces
  • smartctl and dd for the disks

In normal circumstances (a default Linux setup), mii-tool, ethtool, dd and smartctl will require root privileges.

WMI (Windows Management Instrumentation)

This section applies to Sun X64 machines running the Windows operating system only.
As with all servers running Windows, the built-in WMI stack provides information about non-RAID disks and network connections.
It needs to be noted that the WMI layer will only see hardware pieces as they are seen by the operating system. In some cases, virtual disks (from a SAN) are improperly exposed as real physical disks.

Sun Management Center (SMC)

The Sun Management Center agent (“SunMC” or “SMC” agent) does not support Sun X64 systems, running Solaris, Linux or Windows.
Hardware Sentry does not need and thus does not use the SMC agent to discover and monitor the hardware components of Sun X64 systems (i86pc).

IPMI (ipmitool)

IPMI is a new hardware instrumentation technology developed by Intel. IPMI’s goal is to standardize the way the manufacturers provide hardware health information on their servers. It is intended to replace the manufacturers’ specific technologies that used to be implemented to retrieve information like temperatures, voltages, fan speeds, etc.
IPMI’s implementation consists of a Baseboard Management Chip (BMC) installed in the server as well as a protocol to communicate with this chip. IPMI’s specification allows communicating with the BMC either through its dedicated LAN interface (out-of-band monitoring) or by using internal system buses through a driver (in-band monitoring).

Currently, Hardware Sentry only provides support for in-band monitoring with IPMI. This signifies that Hardware Sentry and the PATROL Agent need to be installed locally on the managed server and communicate with the local BMC.

The in-band IPMI communication protocol is being implemented as a driver by various vendors in different ways:

  • Dell provides its own IPMI driver for Windows and Linux
  • IBM provides the OSA implementation
  • OpenIPMI is an open-source IPMI driver for Linux
  • Sun provides a customized version of the OpenIPMI driver for Linux and Solaris 10
  • HP provides its own IPMI driver for Windows, Linux and HP-UX
  • Intel provides an IPMI driver for Windows 2000
  • Microsoft provides an IPMI driver with Windows Server 2003 R2

Unfortunately, these different implementations are most often not compatible with each other and each server manufacturer and OS vendor only supports one implementation. If you plan to monitor the hardware of an IPMI-enabled server with Hardware Sentry, you must carefully check which driver needs to be installed on your server model. Deciding which IPMI driver to install depends on the manufacturer and model of the server as well as the operating system and its version.

Installing an inappropriate or outdated IPMI driver can lead to severe problems, including server crashes.

Sun X64 systems are equipped with a BMC chip (Baseboard Management Controller) which reports information about the hardware status of the server and its environment (temperatures, fans, etc.).
Hardware Sentry does not support IPMI over the network and will only communicate with the BMC chip “in-band”, by using the ipmitool utility which communicates with the proper IPMI driver installed on the system.
Ipmitool is an open-source utility that ships with most recent Linux distributions as well as Solaris 10. It is not supported on Windows however. This means that Hardware Sentry will not be able to retrieve any environment information on Sun X64 servers running Windows.
On Linux systems, ipmitool is typically installed in the /usr/local/bin directory. On Sun Solaris systems, ipmitool is typically installed in the /usr/sfw/bin directory. If not already present on the managed system, ipmitool can be downloaded from sourceforge.net/projects/ipmitool/.

In order to work properly, ipmitool first requires the installation of the proper IPMI driver:

  • On Sun Solaris 9 systems: lipmi
  • On Sun Solaris 10 systems: bmc (the driver is pre-installed)
  • On Linux: OpenIPMI (pre-installed in most recent 2.4 and 2.6 kernel-based distributions)
  • On windows: IPMI is not supported (Microsoft’s own driver only partially works with Sun’s BMC chip)

Ipmitool also requires root privileges.

LSI utility (lsiutil)

Most Sun X64 servers include an LSI c1030-based RAID controller. LSI provides a command-line utility to manage these controllers on Linux and Solaris: lsiutil. The lsiutil for Sun servers can be downloaded on LSI’s Web site: docs.broadcom.com/docs/12351668.
Hardware Sentry uses the lsiutil command to discover and monitor the physical and RAIDs volumes in the system. The lsiutil file must be placed in a directory declared in the PATH environment variable (/usr/bin for example).
The lsiutil command requires root privileges.

SmartMon tools and dd on Linux systems (smartctl)

Most recent distributions of Linux include the SmartMon tools which report the S.M.A.R.T. status of the non-RAID disks (SCSI, IDE or ATA). S.M.A.R.T. is a technology embedded in hard drives which monitor their activity and analyze the trend of errors in order to predict an imminent failure.
Hardware Sentry is able to interrogate the /usr/sbin/smartctl utility to report whether a disk is about to fail.
For all disks reported by smartctl as an internal physical disk, Hardware Sentry will also make a read test to ensure that the disk is available.
The smartctl and dd utilities require root privileges.

Setting up Hardware Sentry on Sun X64 servers (i86pc) running Solaris

Pre-requisites

  • The server is running Sun Solaris 9 or 10
  • There is no need for the SMC agent to be installed
  • On Solaris 9, the ipmitool utility and the lipmi driver are installed
  • On Solaris 10, the Solaris 10 Update 3 has been installed
  • The lsiutil utility is installed if the LSI controller has been configured in RAID mode
    inline

Installation procedure

  • Install the PATROL Agent on the server (versions 3.5.00 and upward are supported, version 3.7.00 minimum is recommended) if it has not been already done.
  • Install Hardware Sentry KM for PATROL on the server (this can be done at the same time as the PATROL Agent). Please follow the instructions of the Installation Guide of Hardware Sentry.

Configuration

Some system utilities used by Hardware Sentry require root privileges. To ensure that Hardware Sentry can use these utilities to discover and monitor the hardware components of a Sun server (i86pc), you can either configure Hardware Sentry to execute all of its external commands as root or configure it to use the sudo utility for a specified list of commands.

To configure Hardware Sentry to impersonate as root for all of its external commands, [right-click] on the main “Hardware” icon › [KM Commands] › [This System’s Settings] › [Connection, Credentials and Connectors…] and enter the root login and password in the first step of the wizard.
To configure Hardware Sentry to use sudo, follow the same procedure but click on the “Sudo options” in the first step of the wizard. Then select which commands Hardware Sentry will use sudo for.

Below is the list of commands that will require root privileges:

  • ipmitool
  • lsiutil
  • /usr/bin/dd will be used on all Solaris systems but requires root (actually “sys”) privileges only on pre-Solaris 10 systems. Pre-Solaris 10 systems can be configured not to require such privileges to access /dev/rdsk/cXtYdZsN devices.
  • /usr/sbin/ndd will be used on some models of Solaris systems equipped with network cards whose driver is dmfe, bge or e1000g. But this list is not exhaustive and also depends on the version of Solaris. The best way to know for sure if the ndd will be used on a Solaris system is to check whether Hardware Sentry is able to collect the LinkStatus parameter without root privileges.

Please note that the sudo utility must have been installed on the system and configured to allow the PATROL Agent’s default account to execute the selected commands as root. This can be done in the /etc/sudoers file.

Discovered components and monitored parameters

Connectors

When configured properly, the following connectors should be automatically selected by Hardware Sentry in order to monitor a Sun X64 system running Solaris (i86pc):

  • IPMI - In-band (IpmiTool)
  • Sun Solaris - Processors (psrinfo)
  • Sun Solaris - Sun Disks
  • LSI Logic - LsiUtil - RAID
  • Sun Solaris - Network

Components and monitored parameters

In turn, the following components and parameters are discovered and monitored:

  • Server model
  • Temperature sensors
  • Fans, speed
  • Voltage sensors
  • Power supplies, status (when available)
  • Processors, type and frequency, status
  • Physical disks, vendor, size, serial number, error count and status
  • Network cards, vendor, model, connection speed, status, link status and error percentage

Troubleshooting

Check the previous section about troubleshooting sun4u systems regarding the processors, internal disks and network cards.

Hardware Sentry does not report any sensor information

It could mean that:

  • Hardware Sentry does not have sufficient privileges to execute the ipmitool command
  • Ipmitool has not been properly installed
  • The IPMI driver has not been properly installed

Hardware Sentry does not report any disk information

It could mean that:

  • Hardware Sentry does not have sufficient privileges to execute the dd and lsiutil commands
  • The lsiutil utility has not been properly installed
  • The lsiutil utility is not located in a directory declared in the PATH environment variable

Discovery process is too long

If the discovery takes very long to complete or if sensor objects are intermittently marked as missing and trigger alarms, it is probably due to the slowness of the IPMI driver. When launched by Hardware Sentry, the ipmitool command can take minutes to complete and report the value of all sensors in the system. If the command takes too long to complete, Hardware Sentry considers the command as stalled and kills the corresponding process. The update 3 of Solaris 10 fixes the slowness issue with the IPMI driver. To handle this issue, Hardware Sentry uses a timeout of 6 minutes of the ipmitool commands which should be largely sufficient.

Setting up Hardware Sentry on Sun X64 servers running Linux

Pre-requisites

  • The server is running RedHat or SuSe Linux
  • On older versions of Linux, the ipmitool utility and the OpenIPMI driver are installed
  • The lsiutil utility is installed if the LSI controller has been configured in RAID mode

Installation procedure

  • Install the PATROL Agent on the server (versions 3.5.00 and upward are supported, version 3.7.00 minimum is recommended) if it has not been already done.
  • Install Hardware Sentry KM for PATROL on the server (this can be done at the same time as the PATROL Agent). Please follow the instructions of the Installation Guide of Hardware Sentry.

Configuration

Some system utilities used by Hardware Sentry require root privileges. To ensure that Hardware Sentry can use these utilities to discover and monitor the hardware components of a Sun X64 server, you can either configure Hardware Sentry to execute all of its external commands as root or configure it to use the sudo utility for a specified list of commands.

To configure Hardware Sentry to impersonate as root for all of its external commands, right-click on the main “Hardware” icon › KM Commands › This System’s Settings › Connection, Credentials and Connectors… and enter the root login and password in the first step of the wizard.

To configure Hardware Sentry to use sudo, follow the same procedure but click on the “Sudo options” in the first step of the wizard. Then select which commands Hardware Sentry will use sudo for.

Below is the list of commands that will require root privileges:

  • ipmitool
  • lsiutil
  • /usr/sbin/smartctl
  • /bin/dd
  • ethtool
  • mii-tool

Please note that the sudo utility must have been installed on the system and configured to allow the PATROL Agent’s default account to execute the selected commands as root. This can be done in the /etc/sudoers file.

Discovered components and monitored parameters

Connectors

When configured properly, the following connectors should be automatically selected by Hardware Sentry in order to monitor a Sun X64 system running Linux:

  • IPMI - In-band (IpmiTool)
  • Linux - Network
  • SmartMon Tools
  • LSI Logic -LsiUtil - RAID

Components and monitored parameters

In turn, the following components and parameters are discovered and monitored:

  • Server model
  • Temperature sensors
  • Fans, speed
  • Voltage sensors
  • Power supplies, status (when available)
  • Physical disks, vendor, size, serial number, status and predicted failure
  • Network cards, vendor, model, connection speed, status, link status and error percentage

Troubleshooting

Hardware Sentry does not report any sensor information

It could mean that:

  • Hardware Sentry does not have sufficient privileges to execute the ipmitool command
  • Ipmitool has not been properly installed
  • The IPMI driver has not been properly installed

Hardware Sentry does not report any disk information

It could mean that:

  • Hardware Sentry does not have sufficient privileges to execute the smartctl, dd and lsiutil commands
  • The lsiutil utility has not been properly installed
  • The lsiutil utility is not located in a directory declared in the PATH environment variable

Discovery process is too long

If the discovery takes very long to complete or if sensor objects are intermittently marked as missing and trigger alarms, it is probably due to the slowness of the IPMI driver. When launched by Hardware Sentry, the ipmitool command can take minutes to complete and report the value of all sensors in the system. If the command takes too long to complete, Hardware Sentry considers the command as stalled and kills the corresponding process. To handle this issue, Hardware Sentry uses a timeout of 6 minutes of the ipmitool commands which should be largely sufficient.