Purpose

Provide a troubleshooting and response procedure when a Filesystem Capacity Warning alert is triggered on a Head Node.

This KB helps engineers quickly identify the issue, perform basic checks, and escalate when required.

Alert Name

Filesystem Capacity Warning – Head Node

What the Alert Is

This alert indicates that the filesystem usage on the Head Node has crossed the warning threshold.

High filesystem utilization may lead to:

• Application logging failures

• Job scheduling issues

• System instability

• Potential service interruptions if capacity reaches critical levels

Summary

The monitoring system detects when the filesystem capacity approaches its configured limit.

The alert is generated to notify engineers so that corrective action can be taken before the filesystem becomes full.

Severity

Warning

Immediate investigation is recommended to prevent escalation to a critical filesystem full condition.

L1 Engineer Actions

Follow these steps when the alert is received.

Step 1 – Log the Alert

• Record the alert in the Alert Summary sheet

Step 2 – Create Ticket

• Open an incident ticket for tracking

Step 3 – Escalation (If Required)

If the issue appears significant:

• Create an internal ticket

• Inform L2 Engineer

Step 4 – Documentation

After confirmation with L2:

• Update the Issues Master Sheet
  

L2 Engineer Actions

L2 engineers should perform deeper troubleshooting.

Step 1 – Check Filesystem Status

Run the following commands:

df -h
lsblk
free -h

This will show:

• filesystem usage

• storage device mapping

• system memory usage

Step 2 – Identify High Usage Filesystem

Determine which filesystem is consuming high storage.

Example:

du -sh /* 2>/dev/null

or

du -sh /var/* 2>/dev/null

Step 3 – Clean Unnecessary Files

Free up space by removing unnecessary data such as:

• old log files

• temporary files

• unused data

• old core dumps

• old kernel files

Common cleanup locations:

/var/log
/tmp
/var/tmp

Step 4 – Verify Filesystem After Cleanup

Re-check usage:

df -h

Confirm that the filesystem utilization has reduced.

Step 5 – OEM Escalation

If the issue persists or cannot be resolved internally:

Escalate to OEM / platform support team with:

• node hostname

• filesystem details

• command outputs (df -h, lsblk)