One of the major use cases of industrial IoT is predictive maintenance that continuously monitors the condition and performance of equipment during normal operation and predict future equipment failure based on previous equipment failure and maintenance history. With an accurate equipment failure prediction organisations can reduce cost from unplanned breakdown and unnecessary preventive maintenance. Driven … Continue reading Questions to Ask when Starting a Predictive Maintenance Project
Evaluate Feature Importance using Tree-based Model
Tree-based model can be used to evaluate the importance of features. In this blog post I go through the steps of evaluating feature importance using the GBDT model in LightGBM. LightGBM is the gradient boosting framework released by Microsoft with high accuracy and speed (some test shows LightGBM can produce as accurate prediction as XGBoost … Continue reading Evaluate Feature Importance using Tree-based Model
Tuning Hyper-Parameters using Grid Search
Hyper-parameters tuning is one common but time-consuming task that aims to select the hyper-parameter values that maximise the accuracy of the model. Normally, cross validation is used to support hyper-parameters tuning that splits the data set to training set for learner training and the validation set to test the model. Python scikit-learn package provides the GridSearchCV class … Continue reading Tuning Hyper-Parameters using Grid Search
Trigger Azure Analysis Service Processing in Azure Data Factory
There is one important feature missing from Azure Data Factory. In SSIS, at the end of the ETL process when the new data has been transformed and load into data warehouse, the SSAS processing task can be run to process the cube immediately after the new data has flow into the data warehouse. However, Azure … Continue reading Trigger Azure Analysis Service Processing in Azure Data Factory
Build Custom Component Status Diagram for Power BI using Synoptic
One common visualisation requirements of IoT analytics is to display the current status of machine components using status diagram of the machine in an interactive reporting tool such as Power BI. The task to build custom machines diagram can be very time-consuming for report authors. Fortunately, I found we can actually build machine diagrams for … Continue reading Build Custom Component Status Diagram for Power BI using Synoptic
Issues with Azure Streaming Analytics + Power BI Real-Time Streaming for IoT Hot-Path Analytics
Event Hub+Azure Streaming Analytics+Power BI Real-Time Streaming is the recommended approach from Microsoft for IoT hot-path analytics. The combination of those techniques provides a simple and efficient way to implement streaming analytics. However, I did meet some issues with this approach when designing hot-path analytics solutions for IoT projects. Azure Streaming Analytics does not support … Continue reading Issues with Azure Streaming Analytics + Power BI Real-Time Streaming for IoT Hot-Path Analytics
Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL
When analysing daily usage of a machine that can run across days, we need to split the time of a single machine running cycle into right days. As the example below shows, the second machine (D00002) run across two days and the third machine (D00003) run across three days. To analyse the daily usage of … Continue reading Handling Across-Day Cycle Issue in Daily Usage Analysis using U-SQL
Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics
The raw telemetry data collected from IoT sensor is normally event-based, e.g., a "Device On" message when the device starts to run, and a "Device Off" message when the device stops. One common data preprocessing task is to transform the raw "On/Off" telemetry data into device cycle records with the start time, end time and … Continue reading Generate Device Cycle Records from Raw Telemetry Message using Azure Data Lake Analytics
Workaround for Building Azure Data Warehouse using Visual Studio
When creating Azure Data Warehouse, I have found the limitation that the Visual Studio SSDT SQL projects does not support Azure Data Warehouse. It causes much pain to the data warehouse development without support on source control and nice code organisation as those offered by SSDT SQL Projects. Fortunately, I have found a trick to … Continue reading Workaround for Building Azure Data Warehouse using Visual Studio
Extracting Features from IoT Sensor Data using R
In my previous blog I introduced the common patterns to extract features from IoT sensor data using Python. Although R is not my primary machine learning language it is becoming ubiquitous in Microsoft's data analytics ecosystem after they acquired Revolution Analytics, the major commercial distributor of R. Considering the increasing popularity of R on Microsoft … Continue reading Extracting Features from IoT Sensor Data using R
Data Ninjago (Finsight-Tech Blogs) 
You must be logged in to post a comment.