Principal Software Engineer

Make Your Mark:

You'll Get To:

We are looking for an accomplished, hands-on Principal Engineer with 18+ years of software engineering experience and a strong record of setting technical direction in Agentic AI, Generative AI systems and Software/Data engineering. This is the most senior individual contributor role on the Agentic AI platform. You will set the multi-year technical vision and own the architecture across products, while staying close enough to the code to make the critical calls. You will guide Senior Staff Engineers, Staff Engineers and Technical Leads across several teams.

The ideal candidate has led 5+ enterprise-scale AI programs from strategy and solution design through to production at scale, and has shaped platform architecture that several product teams build on. You can define an organization-wide AI architecture strategy and present it to executives, and you can also prototype and write production-grade Python when a problem needs it. You bring the vision to set direction across teams, the judgment to make high-stakes architecture and investment decisions, and the credibility to raise the engineering bar across the company.

The role spans strategy, architecture and execution. You will set the AI platform roadmap, define the governance and trust frameworks that make agents enterprise-ready on a secure, compliant financial platform, and establish the engineering standards that every AI product team follows.

Key Responsibilities

Technical Strategy & Vision

• Define and own the multi-year technical strategy and architecture roadmap for the Agentic AI platform across all product lines.
• Make build-vs-buy, technology selection and platform investment decisions, and present the trade-offs, risks and ROI to executive leadership.
• Track the agentic AI landscape: frameworks, protocols, models, regulation. Turn those shifts into strategic bets and platform direction.
• Identify new opportunities where AI can deliver business value across product areas, and shape them into fundable programs.

Architecture Leadership

• Own the end-to-end architecture of the Agentic AI system: agent registry, policy engine, metrics ingestion pipeline, anomaly detection and HITL orchestration.
• Define the reference architecture for multi-agent systems: trust tier boundaries (T0–T3), separation of concerns, fault isolation and full observability.
• Set organization-wide patterns for data flows, integrations (REST, gRPC, SSE, event-driven) and storage strategies for AI workloads in a multi-tenant FinTech environment.
• Own the design principles for durable DAG execution, pause and resume of long-running financial close workflows, event-sourced audit trails and SLA-tracked HITL approval gates.
• Run the architecture governance process: ADRs, cross-team design reviews and production readiness assessments for critical systems.

Technical Leadership & Organizational Influence

• Split the platform into programs and workstreams. Assign ownership to Senior Staff and Staff Engineers, with clear API contracts, interface definitions and success metrics.
• Act as the senior technical authority for Engineering, Product, Security, Compliance, Finance and Data Platform leaders, and align architecture decisions across organizations.
• Represent the AI Platform in executive and cross-functional technical forums, and externally at industry events, through publications and in partner discussions.
• Find and resolve cross-team technical dependencies and systemic risks early in planning.

Mentorship & Talent Development

• Grow Senior Staff Engineers, Staff Engineers and Technical Leads into the next generation of technical leaders through architecture working sessions, design clinics and 1:1 coaching.
• Review designs from senior engineers. Find gaps in governance, observability or compliance thinking and guide them to better solutions rather than overriding them.
• Shape technical hiring standards and help attract top senior talent to the organization.

Engineering Standards & Governance

• Define and drive adoption of engineering standards across all AI product teams: agent identity patterns, audit trail requirements, HITL design principles, Kafka topic design and eval harness setup.
• Set the quality bars: performance benchmarks, eval thresholds, observability standards and security posture.
• Own the agent governance and certification framework, including the T1 agent certification pipeline and its evaluation harnesses for model quality, factuality and safety at scale.
• Drive the production rollouts and release readiness for all releases
• Support the on-call and be the first point of contact for any incidents reported internally or from customers

Hands-on Technical Depth

• Step in directly on the hardest problems with code, prototypes, design sketches or paired debugging: novel system design, complex integrations, critical production incidents. This is not a review-only role.
• Guide the design and delivery of intelligent agents for complex, multi-step financial reasoning, using ReAct, Plan-and-Execute and other advanced reasoning strategies.
• Set the orchestration strategy across LangGraph, Temporal, Google Workflows or equivalent, with documented architectural reasoning.
• Define deployment architecture on GCP (Vertex AI Agent Builder, Cloud Run, GKE) or AWS/Azure equivalents, including Kubernetes namespace isolation for T1 certified partner agents.
• Shape MCP (Model Context Protocol) server and client patterns for external agent integration (T2/T3 trust tiers via SSE/HTTPS).
• Guide RAG architecture with vector stores (Pinecone, Weaviate, pgvector) for retrieval over domain-specific financial documents.
• Lead model strategy across foundation models (Gemini, GPT-4, Claude), covering fine-tuning, prompt engineering and evaluation for accounting tasks such as close commentary and variance explanation.
• Own the inference cost strategy: caching, batching, model routing and quantization. Keep token costs visible in Control Tower.

What You'll Bring:

Required

• 18+ years of software engineering experience, including significant time as a senior or principal-level technical leader.
• A record of setting technical strategy and architecture across multiple teams or product lines.
• Led 5+ enterprise-scale AI/ML or GenAI programs from strategy through to production at scale.
• Deep expertise in agentic AI architectures, multi-agent orchestration, LLM evaluation and AI governance.
• Strong hands-on proficiency in Python and distributed systems design.
• Extensive experience with cloud-native architecture on GCP, AWS or Azure, including Kubernetes and event-driven systems such as Kafka.
• Proven ability to influence executives and align technical direction across Engineering, Security, Compliance and Business stakeholders.
• Experience mentoring Senior Staff and Staff Engineers into senior technical leaders.

Preferred

• Experience building AI systems in regulated industries: FinTech, financial close, accounting or audit.
• Familiarity with SOX, audit trail and compliance-by-design patterns.
• Industry visibility through patents, publications, conference talks or open-source contributions.

Thrive at BlackLine Because You Are Joining:

• A technology-based company with a sense of adventure and a vision for the future. Every door at BlackLine is open. Just bring your brains, your problem-solving skills, and be part of a winning team at the world's most trusted name in Finance Automation!
• A culture that is kind, open, and accepting. It's a place where people can embrace what makes them unique, and the mix of cultural backgrounds and varying interests cultivates diverse thought and perspectives.
• A culture where BlackLiner's continued growth and learning is empowered. BlackLine offers a wide variety of professional development seminars and inclusive affinity groups to celebrate and support our diversity.

BlackLine is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity or expression, race, ethnicity, age, religious creed, national origin, physical or mental disability, ancestry, color, marital status, sexual orientation, military or veteran status, status as a victim of domestic violence, sexual assault or stalking, medical condition, genetic information, or any other protected class or category recognized by applicable equal employment opportunity or other similar laws.

BlackLine recognizes that the ways we work and the workplace itself has shifted. We innovate in a workplace that optimizes a combination of virtual and in-person interactions to maximize collaboration and nurture our culture. Candidates who live within a reasonable commute to one of our offices will work in the office at least 3 days a week.

Accommodations:

BlackLine is committed to creating an inclusive and accessible experience for all candidates. If you require a reasonable accommodation that would better enable your success during the application or interview process, please complete this form.

Picture Yourself at Blackline

Wondering what it’s like to work at BlackLine? Don’t take our word for it. We’ve been voted Best Place to Work by Inc. Magazine four years in a row. Built in LA ranks us in their top 100 Best Places to Work in LA. Reviewers on Glassdoor say we have “smart and kind people,” “phenomenal culture,” and “good work/life balance.” Sound good? Check out our open positions and see if there’s a fit for you.

Beware of Recruitment Scams

At BlackLine, we follow a comprehensive recruitment process which is undertaken by our talent acquisition team. We conduct personal interviews of all potential job seekers and do not follow practice of making employment offers based solely on a candidate’s resume. We do not solicit any payment under the name of security deposit and/or visa/work-permit processing fee, etc., either refundable or non-refundable, at any stage of the recruitment process. Openings at BlackLine are listed on our career webpage in addition to being advertised in job portals.

Job seekers are requested to be vigilant of any fraudulent or suspicious activities in protection of their interest. BlackLine shall not be held liable for any loss or damage, direct or indirect, incurred as a result of this act.