Senior Data Engineer(m/w/x)

Anforderungen

• 7+ years building/operating production data pipelines
• Strong experience with streaming systems (Kafka or AWS Kinesis) and event-driven architectures
• Hands-on experience with processing frameworks (Flink, Spark, Beam, or equivalent)
• Proficiency in Python and/or Java/Scala (Go a plus)
• Strong SQL skills and data modeling for analytics/aggregates
• Strong experience with AWS and cloud-native data patterns
• Experience with data warehouses/OLAP (Snowflake/Redshift/BigQuery) and/or real-time analytics stores (ClickHouse/Druid)
• Familiarity with SQL + NoSQL ecosystems (Postgres/MySQL + DynamoDB/Cassandra/Redis)
• Experience with orchestration and CI/CD for data pipelines (Airflow/Argo/Step Functions or equivalents)
• Clear communication and collaboration skills
• Programmatic advertising domain knowledge
• Experience building feature stores or ML feature parity
• Lakehouse experience (Delta/Iceberg/Hudi), incremental processing, backfill strategies
• Strong data governance practices (lineage, access controls, PII handling, privacy-by-design)
• Experience operating at extreme scale and optimizing cost/performance

Aufgaben

• Build streaming data pipelines
• Ingest and process high-volume event data
• Handle schema evolution and late data
• Implement data deduplication
• Create canonical data aggregates
• Produce durable, well-defined data rollups
• Ensure consistent semantics and SLAs for rollups
• Deliver feature-ready datasets
• Provide near-real-time signals
• Support model training and scoring
• Enable data retrieval for agents
• Facilitate agent decision loops
• Publish governed datasets to analytics systems
• Deliver datasets to data warehouses
• Support dashboards and ad-hoc queries
• Enable operational reporting
• Implement data reconciliation and backfills
• Establish audit trails and quality checks
• Ensure data correctness for reporting
• Ensure data correctness for measurement
• Optimize pipeline performance and cost
• Tune pipeline throughput and latency
• Optimize storage formats and partitioning
• Manage compute spend for workloads
• Apply optimizations to streaming and batch workloads
• Ensure operational excellence and observability
• Instrument pipelines with metrics, logs, and traces
• Define Service Level Indicators (SLIs)
• Define Service Level Objectives (SLOs)
• Drive fast issue detection
• Conduct root-cause analysis
• Collaborate with cross-functional teams
• Partner with Backend, ML/DS, and Analytics
• Work with Platform and SRE teams
• Define data contracts and schemas
• Develop robust data products

Berufserfahrung

AusbildungWeitere Informationen zur Ausbildungsanforderung

Sprachen

Tools & Technologien