Node Exporter Down - Troubleshooting Guide

Node Exporter Down - Troubleshooting Guide

Purpose

This document provides a standardized approach for handling and troubleshooting the Node Exporter Down alert, including L1 and L2 actions, escalation flow, and resolution criteria.

Alert Name

Node Exporter Down

Alert Description

This alert is triggered when the Node Exporter service is not responding.
Node Exporter is a system-level monitoring agent used with Prometheus to collect OS and hardware metrics from a Linux server (node).

Severity

P2 – High Impact

Possible Causes

  1. Node is down or unreachable (network issue)
  2. Node Exporter service crashed or stopped
  3. High system load or resource exhaustion
  4. Firewall or port blocking issues
  5. Prometheus scraping failure

L1 Engineer Actions

Step 1 – Log the Alert

  1. Review alert details:
    1. Timestamp
    2. Node name
    3. Alert source

Step 2 – Create Ticket

  1. Monitor the alert for 20 minutes to check auto-resolution
  2. If issue persists:
    1. Create an internal ticket
    2. Include all alert details

Step 3 – Escalation (If Required)

  1. Inform L2 Engineer after 20 minutes if unresolved
  2. Share all relevant details

Step 4 – Documentation

After confirmation with L2, update the Issues Master Sheet/ Issues Tracker Sheet.

L2 Engineer Actions

Step 1 – Check Node Reachability

ping <node_ip>
ssh <user>@<node_ip>
  1. Verify node is reachable over network.
  2. Confirm SSH access

Step 2 – Check Node Exporter Service Status

systemctl status node_exporter
  1. Verify if service is running

Step 3 – Restart Node Exporter Service

systemctl restart node_exporter

Step 4 – Verify Service Status

systemctl status node_exporter
  1. Ensure service is active

Step 5 – Check Port Accessibility (Default: 9100)

netstat -tulnp | grep 9100
  1. Confirm port is listening

Step 6 – Verify Monitoring Tools

  1. Check Prometheus: Verify target status (UP/DOWN)
  2. Check Grafana: Look for missing metrics

Step 7 – Verify System Health

top
df -h
free -m
  1. Check CPU, Memory, and Disk usage

Step 8 – Log Analysis & Escalation

journalctl -u node_exporter
  1. Investigate logs for errors.
  2. If issue persists:
    1. Inform customer / create external ticket

Step 9 – Reboot Node (If Required)

reboot
  1. Perform only if:
    1. Node is unresponsive
    2. Service repeatedly fails

Resolution

The issue is considered resolved when:
  1. Node is reachable
  2. Node Exporter service is running
  3. Metrics are visible in Prometheus/Grafana
  4. Alert is cleared

Escalation

  • L1 → L2: If not resolved within 20 minutes
  • L2 → OEM: If infrastructure issue persists

      • Related Articles

      • Network Down Extern (P1 Critical Alert) - Troubleshooting Guide

        Purpose This document provides a standardized approach for handling and troubleshooting the Network Down - Extern alert, including L1 and L2 actions, escalation procedures, and resolution criteria for critical incidents. Alert Name Network Down - ...

      • ZFS Pool Capacity Critical Alert - Troubleshooting Guide

        Purpose This document provides a standardized approach for handling and troubleshooting the ZFS Pool Capacity Critical alert, enabling rapid response to prevent storage exhaustion, performance degradation, and potential service outages. Alert Name ...

      • ZFS Pool Capacity Warning Alert - Troubleshooting Guide

        Purpose This document provides a standardized approach for handling and troubleshooting the ZFS Pool Capacity Warning alert, helping L1 and L2 engineers prevent storage exhaustion and maintain system stability. Alert Name ZFS Pool Capacity Warning ...

      • Fan Speed Critical - Troubleshooting Guide

        Purpose Provide a troubleshooting guide for engineers when a Fan Speed Critical alert is generated on a server node. This article helps engineers quickly identify cooling issues and perform corrective actions to prevent overheating and potential ...

      • Filesystem Capacity Critical in Head Node - Troubleshooting guide

        Purpose Provide a standard troubleshooting and resolution procedure when filesystem capacity on a Head Node reaches critical levels. When to Use Use this KB when the monitoring system triggers the alert: Filesystem Capacity Critical – Head Node This ...