Purpose
This article provides guidance for engineers to identify and respond to the GPU Temperature Critical alert in GPU compute nodes. It outlines the alert meaning, possible causes, and the required L1 and L2 troubleshooting steps.
Alert Name
GPU Temperature Critical
Alert Description
This alert is triggered when the GPU temperature exceeds 85°C.
High GPU temperature typically occurs due to heavy workloads, insufficient cooling, or hardware issues affecting airflow.
Severity
P3 – Medium Impact
The system may still be operational, but prolonged high temperature can lead to:
Possible Causes
-
High GPU workload or sustained compute jobs
-
Cooling system inefficiency
-
Fan malfunction or reduced airflow
-
Data center ambient temperature increase
-
Blocked air vents or dust accumulation
-
Thermal throttling conditions
L1 Engineer Actions
Follow these steps when the alert is received.
Step 1 – Log the Alert
Step 2 – Create Ticket
Step 3 – Escalation (If Required)
If the issue appears significant:
Step 4 – Documentation
After confirmation with L2:
L2 Engineer Actions
Perform deeper analysis and corrective actions:
Step 1 - Detailed GPU and process check
Use:
nvidia-smi
free -h
Identify processes consuming excessive GPU memory.
Step 2 - Check airflow and cooling
Verify:
Fan functionality
Airflow obstruction
Rack cooling conditions
Step 3 - Check thermal throttling
Confirm whether GPU is throttling due to high temperature using:
Step 4 - Analyze system workload
- Identify abnormal or stuck GPU-intensive jobs.
- Consider stopping or redistributing workloads if required.
Step 5 - Reboot the Server
Also, if possible, check with the Hard Reset.
Resolution
Depending on the root cause:
-
Reduce GPU workload temporarily
-
Improve airflow or cooling
-
Replace faulty fans if detected
-
Clean air vents or filters
-
Adjust environmental cooling in the data center
Server reboot / Hard reset
Escalation
Escalate to hardware support/vendor if:
-
GPU temperature remains critical despite normal workload
-
Cooling components are faulty
-
GPU shows repeated thermal throttling
Recent Articles
KB-630208 | How to use Find and Locate to search for files in Linux
Purpose To provide Linux administrators and Field Engineers with guidance on using the find and locate utilities to efficiently search for files and directories within a Linux filesystem based on criteria such as name, type, size, timestamps, ...
KB-630164 | How to use rsync to Synchronize Files
Purpose To provide a clear and practical procedure for using the rsync utility to synchronize files and directories between local systems and remote hosts, including push and pull operations, while preserving file attributes and optimizing data ...
KB-630107 | How to create a Linux swap file
Purpose To provide Field Engineers and Linux administrators with a standardized procedure for creating, configuring, and validating a Linux swap file. This procedure helps ensure sufficient virtual memory is available when physical RAM is exhausted ...
MSI All-in-One (AIO) PC – Physical Damage Policy for Warranty Claims
MSI All-in-One (AIO) PC – Physical Damage Policy for Warranty Claims MSI's standard warranty does not cover physical or accidental damage to an All-in-One (AIO) PC. If the damage is determined to be caused by external factors rather than a ...
KB 134833 - Troubleshooting NVSM Alert NV-CPU-XX – Unrecoverable CPU Internal Error
Purpose This document provides a general troubleshooting procedure for the NVIDIA System Management (NVSM) alert NV-CPU-XX, which indicates that a CPU has reported an internal error. The article outlines how to verify whether the alert represents an ...