Azure Data Engineer/Scientist

Project context

We are building a research and execution platform for Systematic Equity on Databricks. At its core is a bi-temporal feature store based on the medallion architecture (Bronze/Silver/Gold) serving as a “Golden Source” for signals, reporting and applications, the consolidation of legacy systems, high-performance onboarding of historical market data (45+ years, daily frequency) and a compliant SDLC.

Your responsibilities

  • Design & implementation of a bi-temporal EAV feature store (versioning/vintage to avoid look-ahead bias).
  • Development of incremental/streaming pipelines on Databricks Delta Lake for large time series.
  • Development of a security master for cross-source asset harmonisation.
  • Implementation of the medallion architecture: ingestion, transformation, feature persistence.
  • Development of a Python library for research/PM (panel queries, integration into investment workflows).
  • Compliance by design: change control, environment separation, auditability.
  • Performance optimisation, observability/monitoring, documentation & knowledge transfer.

Must-haves

  • 4+ years of data engineering with Databricks/Spark (PySpark), Delta Lake, SQL.
  • Proven experience with time series/versioning (bi-temporal, valid-from/to, vintage).
  • Hands-on Azure experience (Unity Catalog, ADLS, ADF, identity/secrets) or a comparable cloud.
  • Clean engineering: tests, CI/CD, packaging, code reviews.
  • Fluent German & English, written and spoken.

Nice-to-haves

  • Finance/market data (e.g. LSEG/Refinitiv, Worldscope) & security master concepts.
  • Power BI/analytics for stakeholder demos.
  • Experience with feature stores/MLOps and data governance.

Tech stack (in the project)

Azure, Databricks (Unity Catalog, Delta Lake, Streaming, Spark), Python/PySpark, SQL, Azure Data Factory; data sources: LSEG, Refinitiv, Worldscope/DAL.

Deliverables (first 3–6 months)

  • Feature store v1 including vintage-capable panels and a documented query layer.
  • Production-grade ingestion/transformation (Bronze→Silver→Gold) with monitoring.
  • Security master MVP and mappings across data sources.
  • Python package for research/PM including tests & documentation.

General conditions

  • Location: Remote, occasionally Frankfurt a. M.
  • Start: at short notice
  • Model: permanent employment or freelance (full-time preferred)

This is a remote position. Both permanent employment and freelance work are possible.