Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL

When analysing daily usage of a machine that can run across days, we need to split the time of a single machine running cycle into right days. As the example below shows, the second machine (D00002) run across two days and the third machine (D00003) run across three days. To analyse the daily usage of … Continue reading Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL

Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics

The raw telemetry data collected from IoT sensor is normally event-based, e.g., a "Device On" message when the device starts to run, and a "Device Off" message when the device stops. One common data preprocessing task is to transform the raw "On/Off" telemetry data into device cycle records with the start time, end time and … Continue reading Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics

Workaround for Building Azure Data Warehouse using Visual Studio

When creating Azure Data Warehouse, I have found the limitation that the Visual Studio SSDT SQL projects does not support Azure Data Warehouse. It causes much pain to the data warehouse development without support on source control and nice code organisation as those offered by SSDT SQL Projects. Fortunately, I have found a trick to … Continue reading Workaround for Building Azure Data Warehouse using Visual Studio

Extracting Features from IoT Sensor Data using R

In my previous blog I introduced the common patterns to extract features from IoT sensor data using Python. Although R is not my primary machine learning language it is becoming ubiquitous in Microsoft's data analytics ecosystem after they acquired Revolution Analytics, the major commercial distributor of R. Considering the increasing popularity of R on Microsoft … Continue reading Extracting Features from IoT Sensor Data using R

Extracting Features from IoT Sensor Data using Python

The previous blog post discusses three common patterns for extracting feature from IoT sensor data: Window-based descriptive statistics Seasonal pattern Trend pattern This blog post introduces how to implement those three patterns in Python. Window-based descriptive statistics There are three main types of descriptive statistics based on what they describe: distribution (e.g., skewness and kurtosis), … Continue reading Extracting Features from IoT Sensor Data using Python

Feature Extraction of IoT Sensor Data

Feature extraction is an important step in IoT-related machine learning process that transforms the temporal data of machine component state into a format supported by machine learning algorithms. The extracted features need to be informative, i.e. need to carry the information that can contribute to the prediction. Due to the temporal characteristic of IoT sensor … Continue reading Feature Extraction of IoT Sensor Data

Building SPFx Client-Side Web Parts using Chart.js

Building SPFx Client-Side Web Parts using Chart.js

Chart.js is a very handy Javascript charting library, lightweight, flexible, sleek UI and free of charge. In this blog post, I will walk through the steps to build SharePoint Framework client-side web parts using Chart.js library. I have created a demo client-side chart web part with the source code on github. This web part allows … Continue reading Building SPFx Client-Side Web Parts using Chart.js

Setup an Azure Dev VM for Testing Power BI + SQL Server 2016 Integration

Setup an Azure Dev VM for Testing Power BI + SQL Server 2016 Integration

This blog post walks through the steps to setup an Azure dev vm for testing Power BI + SQL Server 2016 integration: Provision the Azure VM using SQL Server 2016 developer image Configure the SSAS/SSRS services Install development software Install sample databases Integrate SSRS and Power BI Service Install and configure Personal Power BI Gateway (Optional) … Continue reading Setup an Azure Dev VM for Testing Power BI + SQL Server 2016 Integration

ABC Classification Analysis with Tableau

In one of my previous posts, I have conducted an ABC Classification analysis using MDX against a SSAS Cube. In this post, I will conduct an ABC Classification analysis using Tableau calculations. In this example, we want to classify the products based on their sales amount into three categories: ‘A’ (make up the top 5% of total sales), … Continue reading ABC Classification Analysis with Tableau

Inspecting SharePoint Metadata using Power Query

Inspecting SharePoint Metadata using Power Query

Power Query is a self-service ETL tool from Microsoft which is able to extract data from a broad range of data sources, transform the data, and finally load the data to PowerPivot. However, I have recently found that Power Query could also be a very handy SharePoint Metadata viewer to inspect the schema of SharePoint sites, lists, content … Continue reading Inspecting SharePoint Metadata using Power Query