EMC VNXe - File System Running Out of Space

  • kb1273
  • Oct. 27, 2020 - Last Update: Nov. 5, 2020
  • 2 min read

This article outlines a known issue with EMC VNXe systems which can be exacerbated by monitoring the platform with Hardware Sentry KM. It also provides a workaround for the problem.

Problem VNXe Numeric Value Extraction

Related Topics

Problem

For some EMC VNXe arrays (including VNXe3100, VNXe3150, and VNXe3300), the primary Storage Processor will reboot into service mode and the management features of the array will no longer function if the /EMC/backend/log_shared of the file system runs out of space.

This issue within the VNXe may occur earlier or more frequently when monitoring these systems with Hardware Sentry KM since the hardware monitoring relies on the EMC’s uemcli command line interface. Every time the VNXe receives a uemcli command, it logs the authentication in the /EMC/backend/log_shared/EMCSystemLogFile.log file. These entries cause the file to grow, which in turn fills the file system.

Resolution (Using Monitoring Studio)

The solution is to periodically check the usage of the file system by running the command df -h /EMC/backend/log_shared and if use percentage is high, run a command to truncate the logs: psql -U c4 c4LoggingDatabase -c 'truncate "CST_LogRecord" cascade' .

You can automate this process with Monitoring Studio X. You only have to create a Template, apply it to all your VNXe systems and provide the required credentials.

Procedure

Step 1: Create a Monitoring Template.

  1. Within the Monitoring Studio X UI, navigate to the Studio section.
  2. Click New Template.
  3. Name your Template and click Create. If the systems use the same credentials, they can be specified on this page as well.

Warning! You must use a service account to run these commands. Operator/Admin accounts will not work!

Step 2: Add a Command-Line Monitor for File System Usage

  1. On the right pane, click Add and select Command Line.

  1. Specify the command to check the filesystem usage: df -h /EMC/backend/log_shared.

  1. Click Create, and go to the Template configuration page.

Step 3: Add a Command-Line Monitor to Truncate the Logs

  1. On the right pane, click Add and select Command Line.

  1. Specify the command to truncate the logs: psql -U c4 c4LoggingDatabase -c 'truncate "CST_LogRecord" cascade'.

  1. Click Create.

Step 4: Add a Numeric Value Extraction.

  1. From the navigation tree on the left side, select the Command Line Monitor created in Step 2.
  2. On the right side, click Add, and select Numeric Value Extraction.

  1. Configure the extraction to check All Lines, and to look for the value before the string %.
  2. Under Parameters and Alerts, enable an alert for Value.
  3. Configure the alert to Alert when Value > and specify the value (File System usage in percent) above which you want the truncation to occur.
  4. Specify the Alert Action as Run Monitor, choosing the Monitor created in Step 3.

Step 5: Apply to Host

  1. From the navigation tree on the left, select Hosts, and choose the VNXe system to apply the template to. If it does not exist, select Hosts > New Host and add it.
  2. In the Templates section of the host screen, use the switch next to the Template to enable it for the host.

Problem VNXe Numeric Value Extraction