Microsoft claims Azure Storage providing both high availability and strong consistency. It sounds good but obviously violates the CAP theorem as the 'P' (network partitioning) is not avoidable in the real world. In theory, you can only achieve either high availability or strong consistency in a distributed storage system. I have done a bit of … Continue reading How Azure Storage Cheats Over the CAP Theorem
Tag: Azure Data Lake Store
Power Query – Parameterised Files Loading from a Year-Month-Day folder Hierarchy
In one of my previous blog post, I described an approach to load text files from Azure Data Lake Store to Power BI filtered by the date period specified by users through setting the period start and period end parameters. That approach is capable to load text files located at the same folder. However, there … Continue reading Power Query – Parameterised Files Loading from a Year-Month-Day folder Hierarchy
Power Query – Parameterised Files Loading from Azure Data Lake Store within a Given Date Range
Power BI now supports data load from Azure Data Lake Store. We can connect to a folder in the Azure Data Lake Store and load all files from that folder. However, we often don't want to or aren't able to load all the files in the Azure Data Lake Store folder into Power BI due to … Continue reading Power Query – Parameterised Files Loading from Azure Data Lake Store within a Given Date Range
SSIS in Azure #1 – Periodically Ingesting Data from SQL Database into Azure Data Lake using SSIS
*The source code created for this blog post is located here. The low cost, schema-less and large column attributes of Azure Data Lake Store along with the large number of supported analytic engines (e.g., Azure Data Lake Analytics, Hive and Spark) makes it a prefect store-everything repository for enterprise data. We can offline the copies of business … Continue reading SSIS in Azure #1 – Periodically Ingesting Data from SQL Database into Azure Data Lake using SSIS
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 4)
This is the last part of the blog series demonstrating how to build an end-to-end ADF pipeline for data warehouse ELT. Introduction & Preparation Build ADF pipeline for dimension tables ELT Build ADLA U-SQL job for incremental extraction of machine cycle data Build ADF pipeline for fact table ELT In the previous part we created … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 4)
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 3)
This is the third part of the blog series to demonstrate how to build an end-to-end ADF pipeline for data warehouse ELT. The part will describe how to build an ADLA U-SQL job for incremental extraction of machine cycle data from Azure Data Lake store and go through the steps for scheduling and triggering the … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 3)
End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 1)
This blog series demonstrates how to build an end-to-end ADF pipeline for extracting data from Azure SQL DB/Azure Data Lake Store and load to a star-schema data warehouse database with considerations of SCD (slow changing dimensions) and incremental loading. The final pipeline will look as: The machine cycle records will be load from the csv … Continue reading End-to-End Azure Data Factory Pipeline for Star Schema ETL (Part 1)
Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL
When analysing daily usage of a machine that can run across days, we need to split the time of a single machine running cycle into right days. As the example below shows, the second machine (D00002) run across two days and the third machine (D00003) run across three days. To analyse the daily usage of … Continue reading Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL
Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics
The raw telemetry data collected from IoT sensor is normally event-based, e.g., a "Device On" message when the device starts to run, and a "Device Off" message when the device stops. One common data preprocessing task is to transform the raw "On/Off" telemetry data into device cycle records with the start time, end time and … Continue reading Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics
Data Ninjago (Finsight-Tech Blogs) 
You must be logged in to post a comment.