Scalable data architecture & reporting pipelines with Snowflake, Airflow & Snowpark
Case study
Company
dmTECH GmbH
Industry
Drugstore
Duration
9 months (11/2024 – 07/2025)
Introduction
For dmTECH (the IT unit of dm-drogerie markt), Innovaforge developed a robust end-to-end data pipeline for Planisware reporting. The result: faster delivery of critical reporting functions, smooth historisation & migration in Snowflake, and reusable framework components for SCD/CDC. The project created the foundation for data-driven decisions in more than 500 stores.
Challenge
dmTECH wanted to integrate Planisware and ERP data into a central Data Warehouse daily, reliably and auditably in order to manage project & resource planning across the company – with high data quality, clear governance and short release cycles right through to reporting (MicroStrategy).
- Heterogeneous source systems (Planisware, ERP) behind a virtualisation layer (Denodo)
- Complex business logic & historisation (SCD types, CDC)
- Requirement for stable, reusable orchestration with monitoring & fast troubleshooting
- Unclear infrastructure: The existing infrastructure and the Azure cloud services were confusing and complex, making management and optimisation difficult.
Solution
- Medallion architecture in Snowflake (on GCP) with a clear separation of Bronze/Silver/Gold and a dimensional model (facts & dimensions)
- Denodo as a virtualisation layer for high-performance, controlled data delivery
- Orchestration: Apache Airflow (daily workflows, SLA checks, alerting)
- Transformations: Snowpark Python – robust, testable pipelines with clear business rules
- CI/CD & infrastructure: GitLab + Terraform for automated deployments on Kubernetes
- Monitoring & quality: runtime optimisation & data quality checks in Airflow and Snowflake dashboards
Results
- Speed & reliability: critical reporting functions (MicroStrategy) delivered significantly faster – thanks to automated, robust Airflow workflows.
- Data quality & governance: clean migration of historical Planisware data to Snowflake, auditable historisation (SCD) & CDC processes.
- Scalability & reuse: framework components allow rapid onboarding of additional data sources and reduce time-to-value for new use cases.
- Business impact: well-founded, data-driven decisions in >500 stores thanks to reliable, up-to-date management information.
- daily orchestration of all relevant data flows (Airflow)
- end-to-end CI/CD from code to deployment (GitLab, Terraform, Kubernetes)
- Historised data basis (SCD), change-driven processing (CDC) for reporting in MicroStrategy
Conclusion
Innovaforge took on architecture, implementation, performance engineering & documentation – in close consultation with the business departments, IT, the MicroStrategy teams and stakeholders. Particular strengths: rapid implementation of demanding requirements, excellent troubleshooting of performance bottlenecks and structured knowledge transfer to the internal team.
Who we are
Innovaforge IT Consulting & Softwareentwicklung GmbH offers comprehensive services in the fields of data architecture, data engineering and data analytics. We help companies to leverage their data in order to optimise their business processes, develop new products and services, and strengthen their competitiveness.







