Senior ML Inference / Systems Engineer

*hiring on behalf of our client

Title: Senior ML Inference / Systems Engineer

Time: Full-Time

Location: Remote (Germany)

Compensation: $250,000 - $300,000 USD / year + competitive equity

The Company

Our client is a high-growth research lab and inference provider focused on real-time AI for consumer-facing applications. They build first-party speech models, serve LLMs, and operate the infrastructure behind modular APIs designed for high-volume, real-time workloads. Their platform serves over 10 trillion LLM tokens monthly to hundreds of millions of users across companions, healthcare, education, and media. Backed by top-tier venture capital firms, they collaborate with leading technology partners to power sub-second multimodal experiences at global scale.

Role Overview

Our client is seeking an experienced Senior ML Inference / Systems Engineer to optimize, scale, and maintain high-throughput, low-latency machine learning serving systems. In this role, you will take ownership of the full model deployment lifecycle, from containerizing and profiling models to implementing cutting-edge acceleration techniques across distributed GPU clusters. You will work in a fast-paced, high-impact environment focused on solving latency, throughput, and distributed systems challenges in production.

Key Responsibilities

• Design, build, and optimize high-performance inference pipelines for large-scale production workloads

• Implement advanced model acceleration techniques including quantization, distillation, continuous batching, paged attention, and speculative decoding

• Profile and optimize low-level code to maximize performance and efficiency across NVIDIA GPUs

• Scale distributed inference infrastructure using modern serving frameworks, Kubernetes, Ray, and custom load-balancing solutions

• Containerize models and collaborate closely with research teams to deploy stable, high-throughput systems into production

• Troubleshoot and resolve complex system bottlenecks to ensure high availability and sub-second latency across multi-GPU and multi-node setups

• Write clean, high-performance systems code and contribute to core engineering architecture decisions

Qualifications

• 5+ years of experience as an ML inference engineer, systems engineer, or senior backend engineer working with high-performance production systems

• Deep technical expertise in modern inference serving frameworks such as vLLM or TRT-LLM

• Proficiency in systems programming and GPU acceleration using C++, CUDA, Rust, or highly optimized Python

• Hands-on experience deploying and managing distributed systems at scale using Kubernetes, Ray, and Docker

• Demonstrated background implementing model acceleration strategies such as caching, quantization, distillation, or speculative decoding

• Strong foundation in systems engineering, low-latency architectures, and multi-GPU/multi-node scaling

• Bachelor’s, Master’s, or PhD in Computer Science, Mathematics, Physics, or equivalent practical systems engineering experience

• Excellent problem-solving skills with a strong bias for shipping stable, high-impact solutions in ambiguous environments