Senior Research Data Engineer(m/w/x)

Description

You will drive frontier research by building massive data pipelines for foundation models. Your day-to-day involves scaling complex Python solutions to manage petabytes of multimodal data.

Requirements

• •Degree in scientific or technical field
• •Experience as Data Engineer or similar
• •Extensive experience with Python data ecosystem
• •Skills in EDA and ML feature engineering
• •Developing and deploying data pipelines
• •End-to-end ownership of data solutions
• •Experience with distributed computing and IaC
• •Excellent communication and collaboration skills
• •LLM training data preparation skills
• •Knowledge of NLP and GPU workflows
• •Experience with dynamic workflow orchestration
• •Linguistics expertise or multilingual skills
• •Fluency in C++, Go or Rust

Tasks

• •Collaborate on ambitious frontier research projects
• •Architect and build scalable data pipelines
• •Download and prepare multimodal unstructured data
• •Utilize Kubernetes, Dask, and Ray stacks
• •Debug and fix low-level open-source issues
• •Deploy complex Python solutions to cloud infrastructure
• •Operate data processing at a massive scale
• •Engineer solutions for text, code, and audio
• •Partner with scientists and platform teams
• •Champion data quality and availability standards
• •Ensure mission-critical reliability of pipeline jobs
• •Maintain high-quality code and documentation

Tools & Technologies

Benefits