Unauthenticated Ray Agent Enables Hidden GPU Crypto Mining via Rootkit
Written by AI from the linked story. Not reviewed by an editor. How this works
Attackers exploited an internet-exposed Ray job agent that required no authentication, allowing them to submit malicious remote computing jobs without any credentials. Once inside, they deployed the SANSAR toolkit — a sophisticated combination of a Rust-based controller, a kernel rootkit, and a tampered NVIDIA monitoring library — to mine cryptocurrency while making GPU utilization appear to be 0% in monitoring dashboards. This attack demonstrates that even robust monitoring infrastructure can be rendered blind when adversaries can manipulate the data sources themselves at the kernel or library level. The incident matters because GPU servers are high-value targets, and the ability to fake telemetry means organizations may remain unaware of ongoing resource theft and intrusion for extended periods.
Tactical Insight
Immediate actions
- Require authentication and authorization on all Ray job agent endpoints, treating them as privileged interfaces even in internal or research environments.
- Audit all GPU servers for unexpected processes, kernel modules, and replaced system libraries that could indicate a rootkit or tampering.
- Rotate credentials and revoke access tokens on any system exposed to the internet without authentication.
Long-term improvements
- Enforce network-level controls (firewall rules, private VPC placement) so that distributed computing frameworks like Ray are never directly reachable from the public internet.
- Implement integrity monitoring on critical system libraries (e.g., NVIDIA NVML) and kernel modules to detect unauthorized replacements or modifications.
- Apply the principle of least privilege to all compute cluster APIs, requiring mutual TLS or token-based authentication before any job submission is accepted.
Detection measures
- Collect GPU utilization metrics from multiple independent sources (hypervisor level, out-of-band IPMI, cloud provider metrics) so a tampered in-guest library cannot be the sole data source.
- Deploy behavioral anomaly detection on network egress and power/thermal telemetry to flag mining activity even when software-level metrics are spoofed.
- Use kernel integrity verification tools (e.g., IMA/EVM on Linux) to detect unauthorized kernel module loading consistent with rootkit installation.