Trainy logo

Member of Techincal Staff

Salary
$140K–$250K
Hiring from
United States
Work type
Remote
Posted

626,985 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

About the role**
Trainy operates the infrastructure behind some of the world's leading foundation model labs. Every day, our platform keeps more than 10,000 GPUs healthy, scheduled, and productive across multiple clouds and bare-metal clusters, where a single bad node can stall a training run that costs thousands of dollars an hour. We're looking for an early-career engineer to join our operations team and help keep that fleet running. You'll work on Konduktor, our Kubernetes-native orchestration platform, with a focus on the systems that detect failing hardware, surface problems before customers notice them, and automate the fixes. You'll also spend time onsite with our customers in the Bay Area, working directly with the teams training frontier models. This is a rare chance to work hands-on with frontier-scale AI infrastructure from day one.\
What you'll be doing**

  • Supporting day-to-day operations of large GPU clusters for foundation model labs, triaging node failures, job disruptions, and customer escalations alongside senior engineers.
  • Visiting customer sites in the Bay Area twice a week to work alongside their infrastructure and research teams, gather feedback, and resolve issues in person.
  • Building and extending Konduktor's health controller to detect GPU, network, and node faults, then automatically cordon, drain, and remediate affected machines.
  • Designing and shipping TypeScript front ends that give our team and our customers real-time visibility into cluster health, job status, and fleet utilization.
  • Turning recurring operational work into automation, replacing manual runbooks with tooling and controllers.
  • Writing clear incident reports and runbooks, and feeding what you learn from production back into the platform.

What we need to see

  • BS in Computer Science, Computer Engineering, or a related field, or equivalent practical experience (internships, research, open source, or a homelab count).
  • Working proficiency in TypeScript and at least one of Python or Go. You'll write front-end visualizations as well as backend tooling and automation.
  • Experience building web front ends with a modern framework (React or similar), and an instinct for turning raw telemetry into views that make problems obvious. Familiarity with Grafana or Prometheus is a plus.
  • Hands-on familiarity with Linux and the command line, including reading logs, inspecting processes, and diagnosing basic system and networking issues.
  • Foundational knowledge of Kubernetes concepts such as pods, nodes, deployments, taints, and labels, and comfort using kubectl across multiple clusters and contexts.
  • A methodical approach to troubleshooting: forming hypotheses, gathering evidence, and writing up clear findings and runbooks.
  • Strong written communication and the ability to work directly with customers' ML and infrastructure teams during incidents.
  • Ability to be onsite at customer locations in the San Francisco Bay Area twice a week.

Ways to stand out from the crowd

  • Fluency with agentic software development, such as using tools like Claude Code, Cursor, or Codex to plan, implement, test, and review changes, along with good judgment about when to trust, verify, or rewrite their output.
  • Familiarity with ML frameworks (PyTorch, JAX, Megatron/NeMo) and distributed HPC infrastructure (Slurm, Lustre, InfiniBand/RoCE), including common failure modes like NCCL hangs and straggler nodes.
  • Exposure to GPU infrastructure and NVIDIA tooling such as DCGM, nvidia-smi, and XID error interpretation.
  • Experience writing Kubernetes controllers or operators (controller-runtime, kubebuilder), especially for automated node health detection, cordoning, and remediation.
  • Experience with charting and data visualization libraries (D3, Recharts, ECharts, or similar) for real-time or time-series data.
  • Familiarity with job queueing and batch scheduling on Kubernetes (Kueue, MultiKueue, or Volcano).
  • Experience with GitOps workflows (FluxCD or Argo CD) and multi-cloud environments (GKE, OKE, EKS).
  • Contributions to open-source infrastructure projects or a portfolio of tools you've built to monitor or automate systems.

Visa: US citizen/visa only.

Similar jobs

Apply for this job100% free