Azure Data Factory

Project goals:

• Setup ETL process based on Azure Data Factory, to move data between OLTP.
• MS SQL Server DB (on-premises environment) and Data Mart based on Azure Database for MySQL.
• Implement several aggregations and transformations into ETL

Accomplishments:

• The ADF pipeline has been designed to move extract data from SQL Server, perform all required transformations and load the data into the Data Mart.
• The service has been integrated with an on-premises monitoring system (Zabbix), to ensure robust, production-ready management and monitoring capabilities.
• Azure AD roles and users have been set up to fit the enterprise security/access policies.

Technologies:

• Azure Cloud
• Azure Data Factory
• Azure Database for MySQL
• Microsoft SQL Server
• Zabbix

Workload:

• 60 hours

Project team:

• Senior DevOps Engineer
• Senior Data Engineer