Node Exporter Down - Troubleshooting Guide

KB 265063 - Node Exporter Down - Troubleshooting Guide

Purpose

This document provides a standardized approach for handling and troubleshooting the Node Exporter Down alert, including L1 and L2 actions, escalation flow, and resolution criteria.

Alert Name

Node Exporter Down

Alert Description

This alert is triggered when the Node Exporter service is not responding.
Node Exporter is a system-level monitoring agent used with Prometheus to collect OS and hardware metrics from a Linux server (node).

Severity

P2 – High Impact

Possible Causes

  1. Node is down or unreachable (network issue)
  2. Node Exporter service crashed or stopped
  3. High system load or resource exhaustion
  4. Firewall or port blocking issues
  5. Prometheus scraping failure

L1 Engineer Actions

Step 1 – Log the Alert

  1. Review alert details:
    1. Timestamp
    2. Node name
    3. Alert source

Step 2 – Create Ticket

  1. Monitor the alert for 20 minutes to check auto-resolution
  2. If issue persists:
    1. Create an internal ticket
    2. Include all alert details

Step 3 – Escalation (If Required)

  1. Inform L2 Engineer after 20 minutes if unresolved
  2. Share all relevant details

Step 4 – Documentation

After confirmation with L2, update the Issues Master Sheet/ Issues Tracker Sheet.

L2 Engineer Actions

Step 1 – Check Node Reachability

ping <node_ip>
ssh <user>@<node_ip>
  1. Verify node is reachable over network.
  2. Confirm SSH access

Step 2 – Check Node Exporter Service Status

systemctl status node_exporter
  1. Verify if service is running

Step 3 – Restart Node Exporter Service

systemctl restart node_exporter

Step 4 – Verify Service Status

systemctl status node_exporter
  1. Ensure service is active

Step 5 – Check Port Accessibility (Default: 9100)

netstat -tulnp | grep 9100
  1. Confirm port is listening

Step 6 – Verify Monitoring Tools

  1. Check Prometheus: Verify target status (UP/DOWN)
  2. Check Grafana: Look for missing metrics

Step 7 – Verify System Health

top
df -h
free -m
  1. Check CPU, Memory, and Disk usage

Step 8 – Log Analysis & Escalation

journalctl -u node_exporter
  1. Investigate logs for errors.
  2. If issue persists:
    1. Inform customer / create external ticket

Step 9 – Reboot Node (If Required)

reboot
  1. Perform only if:
    1. Node is unresponsive
    2. Service repeatedly fails

Resolution

The issue is considered resolved when:
  1. Node is reachable
  2. Node Exporter service is running
  3. Metrics are visible in Prometheus/Grafana
  4. Alert is cleared

Escalation

  • L1 → L2: If not resolved within 20 minutes
  • L2 → OEM: If infrastructure issue persists