Description

As a Data Engineer, you will focus on building scalable infrastructure and managing data transfers. You will also implement advanced ML models and ensure high-quality datasets for training generative models.

Requirements

• •Proficiency in Python
• •Familiarity with cloud computing platforms (AWS, GCP, or Azure)
• •Experience with image and video processing libraries (e.g., OpenCV, FFmpeg)
• •Ability to optimize and parallelize data processing workflows
• •Familiarity with data annotation and captioning processes
• •Knowledge of machine learning techniques for data cleaning
• •Interest in developing large-scale data acquisition systems
• •Experience with natural language processing for image/video captioning
• •Experience with data deduplication techniques at scale
• •Experience with big data processing frameworks (e.g., Apache Spark, Hadoop)
• •Understanding of ethical considerations in data collection

Tasks

• •Develop and maintain scalable infrastructure for image and video data acquisition
• •Manage and coordinate data transfers from licensing partners
• •Implement and deploy ML models for data cleaning and processing
• •Create tools to visualize and cluster data effectively
• •Optimize data processing workflows for billion-scale datasets
• •Ensure data quality, diversity, and proper annotation for training
• •Transform user preferences into trainable data formats
• •Collaborate in the model development loop to update data as needed

Tools & Technologies