Auxo Talent logo

Senior Operations Engineer - AI/HPC platform

Auxo TalentApplies on LinkedInData & Analytics
Hiring from
Romania
Work type
Remote
Posted

626,998 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

Senior Operations Engineer

AI/HPC platform and infrastructure operations

Contract: Fixed-term contract (FTC), 12 initial months, remote

Salary: Gross monthly, depending on location and experience


About the role

Join us as a Senior Operations Engineer on a large-scale AI/HPC platform spanning European and US sites. You will provide hands-on engineering and third-line support, owning complex faults from diagnosis through to a permanent fix and building the automation and platform improvements that keep the service reliable. The work covers bare-metal Linux, Kubernetes, GPU compute, storage and networking.

You report to the Technical Lead, mentor the Engineers handling first- and second-line support, and help research and AI teams run their workloads successfully. You own difficult platform work within agreed standards and escalate major risks and architecture decisions to the Technical Lead.


What you will do

  • Own third-line escalations: trace failures across the workload, container, scheduler, OS, GPU, network and storage layers, find the root cause and deliver fixes that stop it recurring.
  • Run the Kubernetes platform: build, operate and improve clusters and shared services (scheduling, registries, persistent storage, observability), and manage Run:ai project, quota and priority settings while keeping tenants isolated.
  • Automate: write and maintain tested, peer-reviewed production automation using Ansible, Python or Bash, Git and CI/CD or GitOps, with clear rollback paths.
  • Manage the server lifecycle: handle bare-metal imaging, patching, reprovisioning and return to service, validating host, GPU and fabric health, and plan upgrades, maintenance and recovery exercises including backup restoration and rollback tests.
  • Support workloads: diagnose failed or slow distributed training and inference jobs with users, using logs, GPU telemetry and benchmarks, and keep container and package delivery reliable across ARM64 and x86.
  • Stay compliant: apply access, secrets and security controls, and produce change records and validation evidence under GxP procedures and quality approvals.
  • Work with vendors: lead investigations with OEMs and specialist teams, direct remote hands for physical repairs, and work with Partner Operations on vendor processes.
  • Improve and coach: strengthen monitoring and diagnostics with Platform & AI Enablement, and turn incident findings into runbooks, automation and coaching for Engineers.
  • Cover on-call: take part in the rotating support rota across European and US hours, providing senior leadership during incidents.


What you will bring

  • Substantial hands-on production experience in platform, DevOps, Linux, HPC or GPU engineering, including ownership of complex incidents and changes in business-critical environments, and the ability to explain work you personally delivered.
  • Deep Linux administration: boot and service failures, kernel and driver compatibility, resource pressure, filesystems, networking and performance under load, and confidence investigating beyond runbooks.
  • Strong production Kubernetes skills: control-plane and worker lifecycle, container runtimes, Helm, operators, CNI and CSI-backed storage, and diagnosing cluster failures as well as deploying workloads.
  • Proven automation with Ansible or similar plus Python or Bash, using version-controlled, tested automation and CI/CD or GitOps workflows.
  • Hands-on GPU cluster operation (NVIDIA drivers, CUDA, GPU visibility in containers, multi-node workloads) and an understanding of distributed AI workloads (NCCL, GPU memory, data throughput), able to separate application problems from infrastructure faults.
  • Bare-metal provisioning at scale using a tool such as Base Command Manager or equivalent, including remote management, node isolation and validation before return to service.
  • Strong networking and storage troubleshooting (TCP/IP, DNS, routing, load balancing, NFS, persistent volumes), plus a practical grasp of RDMA over Ethernet or InfiniBand and fabric congestion.
  • Experience managing container images and dependencies, and operating multi-tenant platforms with role-based access, workload isolation, certificates and secrets.
  • Disciplined change execution with testing, rollback and auditable evidence, and willingness to complete GxP training and authorisation before independently doing regulated work.
  • Clear English, sound judgement under pressure, the ability to guide less experienced engineers, and willingness to complete security vetting and work rotating shifts, including evenings and out-of-hours on-call.


Nice to have

  • NVIDIA platform tooling such as Mission Control, Base Command Manager, Run:ai, GPU Operator, Network Operator, DCGM or NetQ.
  • Dense GPU systems (for example GB300 NVL72), ARM64 Linux, NVLink, GPUDirect RDMA, Spectrum-X, ConnectX or BlueField.
  • VAST or similar scale-out storage, ArgoCD, JFrog Artifactory, Dragonfly, Prometheus, Grafana and Alloy, plus stateful Kubernetes services such as PostgreSQL high availability with tested backups and failover.
  • PyTorch, JAX or TensorFlow environments, inference services such as NVIDIA NIM, experience in GxP or another validated environment, and awareness of power, thermal and liquid-cooling dependencies.


Qualifications and onboarding

Linux, Kubernetes or NVIDIA infrastructure certifications are a plus, but equivalent hands-on experience counts equally. GxP and local operating-procedure training and competency sign-off are completed during onboarding.


Similar jobs

Apply for this job100% free