Senior AI Inference Engineer (LLM & Speech Models)

Senior AI Inference Engineer (LLM & Speech Models)

仕事概要

In short

You'll design and build Recho's inference framework for speech models and LLMs, and you'll own its throughput and GPU cost. This is a hands-on senior role: you'll architect the system and write much of the code yourself.

• Team language: English (Japanese not required)
• Salary: ¥10M–¥20M per year, plus stock options
• Location: Tokyo (Hanzomon), hybrid. Candidates based outside Japan are welcome. We offer visa sponsorship and relocation support, and we're open to discussing fully remote arrangements.

You might be a fit if you are…

• An MLOps / ML platform engineer who has built and tuned LLM inference stacks (vLLM, SGLang, etc.)
• A speech recognition or speech synthesis engineer/researcher who wants to shift toward inference systems
• An engineer who has optimized throughput and cost for inference workloads on large GPU clusters

About the role

At Recho, we run training, evaluation, data generation and large-scale batch inference across multiple speech models and LLMs. Your core mission is to independently design and implement a high-efficiency inference framework that serves all of them.

This isn't about wrapping a model in an API or tweaking config files. You'll need to understand each model's architecture, inputs and outputs, sequence lengths, pre- and post-processing, and GPU characteristics. From there, you'll separate and integrate model execution, batching, data processing and scheduling into a reusable inference architecture.

On the LLM side, you'll build on mature engines such as vLLM and SGLang. You'll tune model adaptation, inference parameters, batching, prefill/decode behavior and request priority. Across both speech and LLMs, your north star is throughput and GPU cost.

What this role is not

• Not an on-call or SRE role. SLA management, incident response and infrastructure maintenance for online services are not the focus.
• GPU cluster operations and general DevOps are handled by a separate team.
• Writing CUDA kernels or building an inference engine from scratch is not the main job.
• RL training infrastructure belongs to our separate RL Infrastructure Engineer role. You'll partner with that team on the inference side.

What you'll do

1. Build a common inference framework for speech models

• Design a shared inference architecture for ASR, TTS, speech-to-speech and other audio generation models
• Draw clear boundaries between shared and model-specific modules, with adapter/plugin interfaces for onboarding new models
• Build the inference APIs, SDKs and tooling our model teams use every day

2. Optimize large-scale batch inference for audio

• Apply dynamic batching and bucketing for variable-length audio, and run data-parallel inference across GPUs and nodes
• Build asynchronous pipelines that overlap preprocessing, inference, postprocessing and storage I/O
• Measure and remove bottlenecks based on end-to-end processing time and cost

3. Adapt and optimize LLM inference

• Stand up LLM inference on mature engines such as vLLM and SGLang
• Tune tensor/pipeline parallelism, continuous batching, KV cache and prefix cache
• Design scheduling that accounts for prefill/decode characteristics and task priority

4. Plan GPU usage for inference workloads

• Handle GPU allocation and capacity planning for batch inference and evaluation
• Orchestrate GPU inference jobs with Kubernetes, Slurm and Ray
• Track and reduce GPU usage and inference cost per model and per task

We're happy to go into more detail during the intro call and interviews.

Why this role

• Set the playbook in an area that doesn't have one yet. Inference for variable-length audio, codecs, vocoders, speech-to-speech and multimodal models has no established best practices. What you design becomes the foundation for every future model and team.
• Own inference end to end. You'll cover both speech and LLMs, from framework design to batch inference and scheduling. That means you optimize the whole system, not just one piece of it.
• Directly move the business. Inference efficiency determines GPU cost per unit of generated speech and response latency. Those decide how competitive our product is for the major enterprises running it in production.

Tech stack

• Languages & frameworks: Python, PyTorch
• LLM inference: vLLM, SGLang, TensorRT-LLM, Triton Inference Server
• Speech models: ASR, TTS, speech-to-speech, audio codecs, vocoders
• Orchestration: Kubernetes, Slurm, Ray
• Low-level optimization (nice to have): C++, CUDA, Triton, NCCL

Our product

We build the Recho AI Voice Platform, which delivers voice conversations natural enough to be hard to tell apart from a human. We also deploy and operate Voice AI Agents for large enterprises.

Our edge is a platform for building and improving high-quality agents quickly. It captures industry know-how and turns user feedback into a fast, continuous improvement loop. Our agents now handle complex, large-scale contact center operations in finance, manufacturing and the public sector. In some deployments, they've resolved every call without handing off to a human agent.

Hiring process

Intro call → Case assignment & first interview → Paid work trial* → Final interview → Offer meeting

*The work trial is 1–2 consecutive days at our Tokyo office, working on real tasks alongside the team, and it's paid. It's a chance for both sides to check fit and get a feel for the work.

• If you're based outside Japan, we'll agree on the format together. We won't ask you to travel to Japan just for this step.
• If your schedule or your current employer's side-work policy makes the trial difficult, just let us know.
• We may also ask for references.

About Recho

Recho develops foundation models for customer-support AI agents and also deploys them for clients, all within one organization. Our core strength is voice AI technology we built entirely in-house, which resolves the industry's long-standing trade-off between quality and cost.

We've closed a Series A of approximately ¥700M and are rapidly rolling out to demanding production environments at major companies such as SBI Shinsei Bank and KDDI.

Learn more

• Corporate website
• Team interviews (note, in Japanese)
• Company deck
• Case study: Partnership with KDDI and Altius Link (Japanese)
• Case study: SBI Shinsei Bank (Japanese)
• 2-minute voice demo — give it a listen!

必須スキル

Must-haves

Language: Professional-level technical communication in English
Overseas applicants welcome: If you relocate to Japan, we provide work visa sponsorship and relocation support.

• Strong Python and PyTorch skills. You design systems and implement the modules yourself.
• Hands-on experience with inference infrastructure on GPU clusters, with a track record of performance analysis and cost reduction
• Experience with async/concurrent processing, distributed data processing, and batching and parallel inference for variable-length data
• Ability to coordinate technically with multiple research and model development teams
• Comfortable reading technical literature and discussing it in English

Plus deep expertise in at least one of the following. Having both is a plus; you can pick up the other after joining.

• LLM inference: Hands-on experience with vLLM, SGLang or similar, and a deep understanding of inference mechanics (prefill/decode, KV cache, etc.)
• Speech model inference: Hands-on experience with ASR, TTS or similar, and a deep understanding of speech inference pipelines from pre- to post-processing

歓迎スキル

Nice-to-haves

Advanced models & speech

• Inference experience with audio generation models (diffusion, flow matching) or audio codecs
• Hands-on experience across multiple model types, such as ASR, TTS and speech-to-speech
• Interest or experience in building and optimizing RL rollout inference environments

Performance & low-level optimization

• Designing dynamic batching for variable-length audio, or distributed/parallel inference across multiple GPUs and nodes
• Working knowledge of C++, CUDA, Triton and NCCL, and using profilers to pinpoint bottlenecks
• Managing and optimizing GPU cost and compute budgets

Platform design & technical leadership

• Designing internal AI frameworks, inference libraries or model SDKs
• Open-source contributions to vLLM, SGLang, PyTorch, Ray or similar
• Setting technical roadmaps, design standards or code review guidelines

Languages

• Japanese or Chinese. Our team works in English, but Japanese (the company's working language) or Chinese (spoken by many of our colleagues) makes cross-team collaboration even smoother.

求める人物像

What we look for

• Hands-on: You don't stop at architecture. You're on the front line writing the code and care about getting the implementation right.
• Data-driven: You tackle performance and cost problems with profilers and benchmarks, not gut feel, and you optimize for the whole system.
• Pragmatic: You don't reinvent the wheel. You pick the right open-source tools and frameworks and extend only what we actually need.
• Platform-minded: You enjoy building tools and libraries that make other teams, especially model developers, more productive. That's Team-First in practice.

Our values

These are the values our team relies on to deliver Recho's value as fast as possible.

• Exceed Expectations: We keep exceeding the expectations of customers and teammates by delivering both speed and quality.
• Team-First: Each of us takes professional ownership, and we maximize the team's results, not individual ones.
• Continuous Feedback: We engage seriously and speak candidly for the sake of results. The harder something is to say, the more we say it with honesty, respect and care.
• Earn Trust: Trust is the foundation of every collaboration and every challenge. We act ethically and keep personal and company matters separate. Trust takes time to build and a moment to lose.

応募概要

給与

Salary
Compensation is based on your current compensation, skills and experience.

• Annual salary: ¥10M–¥20M
• Salary review: Once a year, based on your contribution and company performance
• Stock option plan

勤務地

WeWork Hanzomon (Hanzomon PREX North, 2-3-2 Kojimachi, Chiyoda-ku, Tokyo)
Hybrid: mostly in the office, with some remote work. For candidates based outside Japan, remote arrangements are open to discussion.

雇用形態

Employment type: Full-time, permanent. Starting with a paid contract engagement first is also an option.

勤務体系

Working hours

• Discretionary work system (professional type under Japanese labor law)
• Deemed working hours: 9 hours/day
• Start and end times are up to you. As a general guideline, the team's core working window is 10:00–19:00, adjusted as work requires.

Scope of duties and work location
This disclosure is required under Japanese labor law.

• Duties: AI inference infrastructure development at hire; subject to change as determined by the company
• Work location: Hanzomon WeWork at hire (some remote work possible); subject to change as determined by the company

Holidays & leave
Weekends, Japanese public holidays, year-end/New Year holidays (Dec 29–Jan 3), special leave, and paid annual leave (10 days in the first year)

試用期間

Probation period: 3 months, with no change in terms

福利厚生

Benefits

Insurance

• Full Japanese social insurance (health, pension, employment)
• Commuting allowance (up to ¥25,000/month)

Office

• WeWork Hanzomon, 2 minutes from Hanzomon Station, with a dedicated floor, focus booths and meeting rooms
• Free drinks, including coffee, soft drinks and alcohol
• High-quality office chairs
• MacBook Air and external monitor provided

Productivity

• Dev tools such as Claude Code, Cursor and Devin
• AI tool stipend (up to ¥30,000/month)
• Amazon Business account for work-related books and equipment

Team

• Daily lunch meeting covering work updates and casual conversation
• Monthly team social on the first Friday evening, at the office with food

Smoking

• Indoor smoking prohibited; outdoor smoking area available