Cloud migration of the existing reporting system

Company

Carl Zeiss AG

Industry

Production

Customer since

2024

Introduction

Migration of Carl Zeiss's existing reporting to a scalable, cloud-native solution. The focus is on overcoming the legacy system's challenges and transforming it into a future-proof architecture that meets the growing demands of data processing and analysis.

Challenge

Carl Zeiss's previous reporting, based on an on-premise monolith, was no longer able to meet the demands of the modern production environment. The following challenges were identified.

  • Data explosion: The rapidly growing volume of manufacturing data required a new architecture and technical solutions to guarantee scalability.
  • Changed business requirements: The migration had to take account of and integrate new business requirements from the business side.
  • Slow data processing: Processing the data from Kafka events was time-consuming and inefficient.
  • Poor code quality: The existing code was unclear, poorly structured and poorly modularised, which made maintenance and further development difficult.
  • Data quality: The data mart contained numerous inaccuracies and errors that compromised the reliability of the reports.
  • Unclear infrastructure: The existing infrastructure and the Azure cloud services were confusing and complex, making management and optimisation difficult.

Solution

  • Architecture decomposition: The overall architecture of the old project was decomposed and simplified in order to achieve greater security, modularity and maintainability.
  • Pipeline implementation: Pipelines for receiving data in various formats (REST, blob storage, MongoDB) were implemented to ensure efficient and flexible data ingestion.
  • Data model development: New data models were developed to transform the data according to the defined business logic. Databricks, PySpark, Delta Tables, Azure Data Factory and SQL queries were used for this purpose.
  • Data Warehouse: A scalable and high-performance Data Warehouse based on Azure SQL and Databricks was built to make the data available for analyses and reports.
  • Real-time ETL: The ETL process was optimised to enable near real-time processing of the data.
  • Cloud configuration: Various services and DWH systems in the Azure cloud environment were configured and optimised.
  • Team leadership: The project team was led effectively, tasks were planned and prioritised, responsibilities delegated and daily team meetings held.
  • Data quality: Data quality was continuously improved and aligned with business requirements.
  • Python programming: Clean and efficient Python code was developed to guarantee the functionality and performance of the system.
  • Monitoring system: A monitoring system for ETL and data science services in Azure was built based on Log Analytics Workspace.
  • Architecture optimisation: The entire IT and data architecture was improved and simplified to increase efficiency and scalability.

Results

  • Scalability: The new system is able to process and analyse the growing volumes of data efficiently. (Scalability increased by 400 %).
  • Flexibility: The cloud-native architecture allows flexible adaptation to changing business requirements. (Simplified integration of new data sources and business logic).
  • Performance: Data processing is now significantly faster and more efficient. (Data processing time reduced by 80 %).
  • Maintainability: The code is clean, modular and well documented, which simplifies maintenance and further development. (Downtime reduced to below 5 %).
  • Data quality: The quality of the data has been significantly improved, increasing the reliability of the reports. (Data quality improved by 15 %).
  • Transparency: The infrastructure and cloud services are clear and transparent, making administration easier.
  • Real-time analyses: Near real-time processing of the data enables timely analyses and decisions.
  • Cost optimisation: The cloud infrastructure allows flexible scaling of resources and thus optimisation of costs. (Operating costs reduced by 60 %).

Conclusion

Migrating Carl Zeiss's reporting to the cloud was a complete success. By overcoming the existing challenges and implementing a modern, scalable architecture, the foundation for a future-proof and efficient reporting environment was created. The company benefits from improved data quality, faster data processing, greater flexibility and an optimised cost structure.

Who we are

Innovaforge IT Consulting & Softwareentwicklung GmbH offers comprehensive services in the fields of data architecture, data engineering and data analytics. We help companies to leverage their data in order to optimise their business processes, develop new products and services, and strengthen their competitiveness.

Your success in safe hands: with certified expertise and experience