Description

In this role, you will focus on designing evaluation pipelines for large language models, collaborating with experts to enhance model performance, and championing best practices in AI evaluation to ensure reliability and ethical standards.

Requirements

• •PhD in Computer Science, Artificial Intelligence, Machine Learning, or related field
• •Research or hands-on experience with large language models, NLP evaluation, or agent-based AI systems
• •Strong understanding of LLM performance measurement, prompt evaluation, and reliability testing
• •Proficiency in Python and familiarity with ML libraries such as PyTorch, Transformers, and LangChain
• •Comfort with experimental design, data analysis, and communicating technical findings clearly
• •Experience with LLM evaluation frameworks (e.g., OpenAI Evals, HELM, LM Harness, or custom auto-eval tools)
• •Familiarity with retrieval-augmented generation (RAG), tool-using agents, or agentic evaluation methodologies
• •Experience in cloud-based ML development (AWS, Azure, or GCP)
• •Record of publications or preprints in top-tier venues (e.g., NeurIPS, ACL, EMNLP, ICLR) or equivalent research contributions
• •Interest in Responsible AI, fairness, and interpretability research

Tasks

• •Design and execute evaluation pipelines for LLMs and agentic systems
• •Assess reasoning, factual accuracy, and alignment
• •Build tools for automatic evaluation and synthetic dataset creation
• •Implement LLM-as-a-judge workflows and continuous benchmarking systems
• •Collaborate with applied scientists, ML engineers, and product managers
• •Translate evaluation results into model improvements and product insights
• •Prototype new evaluation metrics and contribute to internal reports
• •Support publications and presentations on evaluation methods
• •Promote reproducibility, transparency, and ethical AI evaluation

Tools & Technologies

Benefits