Technical Program Manager, AI Benchmarking

Title: Technical Program Manager, AI Benchmarking Location: New Delhi, India Work Authorization: Current and ongoing work authorization in India Position: Full-time Application deadline: September 18, 2026, 6 pm IST (applications may be reviewed as they are submitted) Role Overview: PxD is looking for a Technical Program Manager to lead delivery of a national initiative benchmarking Indian AI models for agricultural advisory. Government advisory platforms are increasingly powered by large language models, and a growing set of Indian foundational models, fine-tuned variants, and voice-based systems have emerged as potential alternatives — but little rigorous evidence exists on how they perform inside live advisory workflows. You will run a multi-partner benchmarking program that produces decision-ready evidence for the Ministry of Agriculture and Farmers’ Welfare on which models can be safely and effectively integrated into existing advisory systems. PxD will co-design the benchmark framework and act as consortium facilitator and government coordination partner for this work, convening government agencies, academic technical evaluators, benchmark partners, and model providers. You will be the person who holds that consortium together day to day: setting the benchmarking architecture with government, contracting and managing academic and technical partners, overseeing the evidence pipeline from technical testing through farmer-facing user research, and turning the results into recommendations a ministry can act on. This position is based in New Delhi, with travel to partner institutions and project states. About PxD: PxD is a rapidly scaling global non-profit that harnesses the power of technology, data science, and behavioral economics to provide targeted information to farmers in low- and middle-income countries. Providing the right information to the right people, in the right way and at the right time, increases incomes, protects the environment, and improves well-being. We are evidence-based: we test our interventions with randomized controlled trials to prove our impact and cost-effectiveness, to simultaneously learn from and improve our service, and to demonstrate our value to funders and partners. Responsibilities: Government & Stakeholder Management Serve as PxD's day-to-day point of contact for the benchmarking program, working through PxD's embedded PMU team to align the Ministry of Agriculture and Farmers' Welfare on priority use cases, target languages and geographies, candidate models, benchmarking baselines, and approval pathways. Establish the governance framework for the consortium — partner roles, decision rights, steering cadence, and escalation routes — and keep it functioning as the program moves through its stages. Negotiate system and data access, including access to government advisory environments for substitution testing, and maintain a permissions tracker across government, academic, and advisory data sources. Facilitate workshops and consultations with government counterparts, model providers, and domain experts; manage the funder relationship and reporting. Benchmarking Design & Technical Delivery Own the benchmarking framework end to end: advisory use cases, benchmark tasks, datasets, human-rating rubrics, safety and language criteria, tool-use and source-grounding assessments, cost comparison templates, and decision thresholds. Work with existing benchmark initiatives to adapt and localise established assets rather than building a parallel benchmark. Coordinate technical benchmarking across three layers — standalone model evaluation, substitution testing within the live government advisory stack, and a neutral benchmarking harness where direct system access is unavailable — alongside assessment of speech recognition, translation, text-to-speech, and multilingual performance. Manage latency, reliability, and total-cost-of-ownership analysis, and oversee the cloud compute, GPU, and API resources that represent the single largest cost line in the program. Safeguard the independence and neutrality of the benchmarking by maintaining clear separation between benchmark development, model evaluation, and model development roles of partner organisations, with transparent disclosure of potential conflicts of interest. Evidence Generation & Synthesis Oversee pre-deployment user evidence generation with agronomists, extension personnel, and compensated farmer participant testers, assessing whether advisory outputs are comprehensible, actionable, trustworthy, and usable in local languages. Manage IRB submission and third-party field vendor selection and oversight for user testing. Consolidate findings across technical performance, user evidence, safety, operational readiness, and cost into a single decision framework, and translate them into recommendations a non-technical government audience can act on. Author the final evidence synthesis and decision report identifying models ready for pilot deployment, models requiring further optimization, and key implementation risks. Grants, Contracts & Partner Management Contract and manage academic and technical partners, sandbox and compute hosts, domain-expert consultants, and field vendors, monitoring performance against agreed deliverables. Ensure contractual and donor compliance throughout implementation, including data-processing agreements and dataset licenses. Coordinate with finance and operations teams on budgets, procurement, and financial reporting against grant milestones. Team & Knowledge Management Work alongside research and field staff supporting the program, providing technical direction and day-to-day coordination. Document benchmarking tools, protocols, and methodology so future evaluations across agricultural and other public-sector advisory systems can build on this work. Contribute to PxD's institutional knowledge management and cross-country learning on AI advisory. What You'll Bring: Required Master's degree (or equivalent experience) in Computer Science, Data Science, Engineering, Public Policy, Economics, Agriculture, Development Studies, Management, or another relevant field. 5–8 years of progressively responsible experience managing complex technical programs or projects. Working fluency with modern AI systems — LLMs, retrieval-augmented generation, prompting, model evaluation, and the practical trade-offs between accuracy, latency, safety, and cost. Demonstrated experience managing multi-stakeholder projects involving government agencies, academic institutions, and external technical partners. Strong project management skills with the ability to run parallel workstreams against fixed donor milestones. Ability to translate technical evidence into clear, decision-oriented recommendations for non-technical senior audiences. Strong stakeholder management and relationship-building abilities. Excellent organizational skills and exceptional attention to detail. Preferred Demonstrated experience with AI model evaluation (academic/ personal/ work projects). Direct experience designing or running model evaluations, benchmarks, or structured A/B testing of AI systems. Experience with Indian-language NLP, speech technology (ASR/TTS), or multilingual model performance. Experience working with state or national government programs, or with digital public infrastructure. Background in agriculture, climate, technology, or international development. Familiarity with causal inference and experimental design. Comfort operating with significant ambiguity and limited existing infrastructure, having built or launched a new program from scratch. Proficiency in one or more Indian languages. How to apply: To apply, please visit the PxD website and use the online application form to submit a CV, cover letter, and brief responses to the screening questions. Applications will be reviewed on a rolling basis until the application deadline of Sep 18, 2026, 6 pm IST. PxD is committed to building an inclusive and diverse organisation. We don’t make decisions based on gender, gender identity, class, ethnicity, sexual orientation, or any other personal characteristics that are unrelated to the role. We understand that everyone has different lifestyles and we accommodate flexible working to make sure you are as productive and happy as you can be. We will use any personal data you provide for recruitment purposes only.