In this blog post, I will introduce two configuration-driven Azure Data Factory pipeline patterns I have used in my previous projects, including the Source-Sink pattern and the Key-Value pattern. The Source-Sink pattern is primarily used for parameterising and configuring the data movement activities, with the source location and sink location of the data movement configured in a … Continue reading Configuration-Driven Azure Data Factory Pipelines
Tag: SQL
SSIS in Azure #1 – Periodically Ingesting Data from SQL Database into Azure Data Lake using SSIS
*The source code created for this blog post is located here. The low cost, schema-less and large column attributes of Azure Data Lake Store along with the large number of supported analytic engines (e.g., Azure Data Lake Analytics, Hive and Spark) makes it a prefect store-everything repository for enterprise data. We can offline the copies of business … Continue reading SSIS in Azure #1 – Periodically Ingesting Data from SQL Database into Azure Data Lake using SSIS
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 4)
This is the last part of the blog series demonstrating how to build an end-to-end ADF pipeline for data warehouse ELT. Introduction & Preparation Build ADF pipeline for dimension tables ELT Build ADLA U-SQL job for incremental extraction of machine cycle data Build ADF pipeline for fact table ELT In the previous part we created … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 4)
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 3)
This is the third part of the blog series to demonstrate how to build an end-to-end ADF pipeline for data warehouse ELT. The part will describe how to build an ADLA U-SQL job for incremental extraction of machine cycle data from Azure Data Lake store and go through the steps for scheduling and triggering the … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 3)
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 2)
This is the second part of the blog series to demonstrate how to build an end-to-end ADF pipeline for extracting data from Azure SQL DB/Azure Data Lake Store and loading to a star-schema data warehouse database with considerations on SCD (slow changing dimensions) and incremental loading. Introduction & Preparation Build ADF pipeline for dimensional tables … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 2)
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 1)
This blog series demonstrates how to build an end-to-end ADF pipeline for extracting data from Azure SQL DB/Azure Data Lake Store and load to a star-schema data warehouse database with considerations of SCD (slow changing dimensions) and incremental loading. The final pipeline will look as: The machine cycle records will be load from the csv … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 1)
Workaround for Building Azure Data Warehouse using Visual Studio
When creating Azure Data Warehouse, I have found the limitation that the Visual Studio SSDT SQL projects does not support Azure Data Warehouse. It causes much pain to the data warehouse development without support on source control and nice code organisation as those offered by SSDT SQL Projects. Fortunately, I have found a trick to … Continue reading Workaround for Building Azure Data Warehouse using Visual Studio
Tableau #2 – Visualising Eurovision Voting Path
In the last blog post, I've created a box-and-whisker plot to compare UK's performance on Eurovision contest with others. That would be good to have a nice Viz to show where the votes from for each country. This can be achieved using Tableau Map and Path. It is straightforward to create this Viz using the Path … Continue reading Tableau #2 – Visualising Eurovision Voting Path
Creating Custom SSRS Spatial Map Reports for Dynamics CRM 2011
A CRM system in an organisation is often the central place to store business data related to customers, sales, and markets, which are highly valuable for any business. The BI feature in Dynamics CRM 2011 is very powerful which allows developers or even end-users to create report tables or charts very easily. However, it is … Continue reading Creating Custom SSRS Spatial Map Reports for Dynamics CRM 2011
Using SSIS and Excel Services to Build a Lightweight Reporting Solution for SharePoint-Based Applications (Part 1)
Part 1 – Introduction & Building the Back-End Reporting Database Part 2 – Developing SSIS Package to Import Data from SharePoint Lists to Reporting Database Part 3 – Setup SQL Server Agent Job to periodically execute the SSIS Package Part 4 – Building and Publishing Excel Reports on SharePoint Sharepoint is a rapid application development platform … Continue reading Using SSIS and Excel Services to Build a Lightweight Reporting Solution for SharePoint-Based Applications (Part 1)
Data Ninjago (Finsight-Tech Blogs) 
You must be logged in to post a comment.