Azure Data Engineer/Scientist
Project context
We are building a research and execution platform for Systematic Equity on Databricks. At its core is a bi-temporal feature store based on the medallion architecture (Bronze/Silver/Gold) serving as a “Golden Source” for signals, reporting and applications, the consolidation of legacy systems, high-performance onboarding of historical market data (45+ years, daily frequency) and a compliant SDLC.
Your responsibilities
- Design & implementation of a bi-temporal EAV feature store (versioning/vintage to avoid look-ahead bias).
- Development of incremental/streaming pipelines on Databricks Delta Lake for large time series.
- Development of a security master for cross-source asset harmonisation.
- Implementation of the medallion architecture: ingestion, transformation, feature persistence.
- Development of a Python library for research/PM (panel queries, integration into investment workflows).
- Compliance by design: change control, environment separation, auditability.
- Performance optimisation, observability/monitoring, documentation & knowledge transfer.
Must-haves
- 4+ years of data engineering with Databricks/Spark (PySpark), Delta Lake, SQL.
- Proven experience with time series/versioning (bi-temporal, valid-from/to, vintage).
- Hands-on Azure experience (Unity Catalog, ADLS, ADF, identity/secrets) or a comparable cloud.
- Clean engineering: tests, CI/CD, packaging, code reviews.
- Fluent German & English, written and spoken.
Nice-to-haves
- Finance/market data (e.g. LSEG/Refinitiv, Worldscope) & security master concepts.
- Power BI/analytics for stakeholder demos.
- Experience with feature stores/MLOps and data governance.
Tech stack (in the project)
Azure, Databricks (Unity Catalog, Delta Lake, Streaming, Spark), Python/PySpark, SQL, Azure Data Factory; data sources: LSEG, Refinitiv, Worldscope/DAL.
Deliverables (first 3–6 months)
- Feature store v1 including vintage-capable panels and a documented query layer.
- Production-grade ingestion/transformation (Bronze→Silver→Gold) with monitoring.
- Security master MVP and mappings across data sources.
- Python package for research/PM including tests & documentation.
General conditions
- Location: Remote, occasionally Frankfurt a. M.
- Start: at short notice
- Model: permanent employment or freelance (full-time preferred)
This is a remote position. Both permanent employment and freelance work are possible.