Description

In this role, you will lead the evaluation of generative AI systems, ensuring their performance and safety. Your work will involve designing frameworks, refining metrics, and mentoring junior team members while staying at the forefront of NLP advancements.

Requirements

• •Ability to think from first principles
• •Leadership by example
• •Open and respectful communication
• •Ability to thrive in a fast-paced environment
• •Willingness to navigate uncertainty
• •Extensive experience in deploying LLMs
• •Strong experience in evaluating LLMs
• •Proven track record with large language datasets
• •Excellent written and verbal communication skills
• •Solid programming skills in Python
• •Passion for applied research in AI
• •Published research in generative AI (nice to have)
• •Hands-on experience with model explainability tools (nice to have)
• •Familiarity with cloud-based platforms (nice to have)

Tasks

• •Lead the design and implementation of evaluation frameworks for generative AI systems
• •Establish and refine metrics for model quality, including fidelity and bias detection
• •Perform technical evaluations and benchmarking on large language models
• •Design custom evaluation approaches based on client needs and scientific research
• •Develop technical AI evaluation frameworks and tools with the product management team
• •Define metrics for evaluating model performance, such as accuracy and coherence
• •Curate and manage high-quality datasets for evaluating large language models
• •Mentor junior data scientists on best practices for LLM evaluation
• •Stay updated on advancements in NLP and LLM evaluation methods

Tools & Technologies