Tag: Machine Learning

QuantFlow: Brute-Force Grid Search for Stock Behaviour Patterns

QuantFlow: Brute-Force Grid Search for Stock Behaviour Patterns

Every stock behaviour pattern has parameters. A compression breakout depends on the rolling window. A VWAP reclaim depends on the volume threshold. A failed breakout depends on the forecast horizon. Change one number and a "profitable" pattern becomes noise. The typical workflow: tweak a parameter → rerun → query results → squint at a table … Continue reading QuantFlow: Brute-Force Grid Search for Stock Behaviour Patterns →

QuantFlow: Detecting Trading Opportunities Through Market Lead-Lag Profiling

QuantFlow: Detecting Trading Opportunities Through Market Lead-Lag Profiling

Context: Some trading ideas often pop into my mind unexpectedly, but most of the time I am too lazy to investigate them further. The root cause of this "laziness" is not the effort required to explore the idea itself, but rather the amount of pre-work needed before I can actually start working on it. Collecting … Continue reading QuantFlow: Detecting Trading Opportunities Through Market Lead-Lag Profiling →

dqops Data Quality Rules (Part 2) – CFD, Machine Learning

dqops Data Quality Rules (Part 2) – CFD, Machine Learning

The previous blog post introduces a list of basic data quality rules that have been developed for my R&D data quality improvement initiative. Those rules are fundamental and essential for detecting data quality problems. However, those rules have existed since a long, long time ago and they are neither innovative nor exciting. More importantly, those … Continue reading dqops Data Quality Rules (Part 2) – CFD, Machine Learning →

Data Quality Improvement – Rule-Based Data Quality Assessment

Data Quality Improvement – Rule-Based Data Quality Assessment

As discussed in the previous blog posts in my Data Quality Improvement series, the key for successful data quality management is the continuous awareness and insights of how fit your data is being used for your business. Data quality assessment is the core and possibly the most challenging activity in the data quality management process. … Continue reading Data Quality Improvement – Rule-Based Data Quality Assessment →

Data Quality Improvement – Conditional Functional Dependency (CFD)

Data Quality Improvement – Conditional Functional Dependency (CFD)

To fulfil the promise I made before, I dedicate this blog post to cover the topic of Conditional Functional Dependency (CFD). The reason that I dedicate a whole blog post to this topic is that CFD is one of the most promising constraints to detect and repair inconsistencies in a dataset. The use of CFD … Continue reading Data Quality Improvement – Conditional Functional Dependency (CFD) →

Execute R Scripts from Azure Data Factory (V2) through Azure Batch Service

Introduction One requirement I have been recently working with is to run R scripts for some complex calculations in an ADF (V2) data processing pipeline. My first attempt is to run the R scripts using Azure Data Lake Analytics (ADLA) with R extension. However, two limitations of ADLA R extension stopped me from adopting this … Continue reading Execute R Scripts from Azure Data Factory (V2) through Azure Batch Service →

The Tip for Installing R packages on Azure Batch

Problem In one project I have been recently working with, I need to execute R scripts in Azure Batch. The computer nodes of the Azure Batch pool were provisioned with Data Science Virtual Machines which already include common R packages. However, some packages required for the R scripts, such as tidyr and rAzureBatch, are missing … Continue reading The Tip for Installing R packages on Azure Batch →

Scaffolding Azure Machine Learning Experiments

*please download the source code here Microsoft has released the public preview of their newest data science service, Azure Machine Learning, that contains a collection of components to support the end-to-end machine learning solution. The Azure Machine Learning Workbench and the Azure Machine Learning Experimentation service are the two main components offered to machine learning practitioners … Continue reading Scaffolding Azure Machine Learning Experiments →

Questions to Ask when Starting a Predictive Maintenance Project

One of the major use cases of industrial IoT is predictive maintenance that continuously monitors the condition and performance of equipment during normal operation and predict future equipment failure based on previous equipment failure and maintenance history. With an accurate equipment failure prediction organisations can reduce cost from unplanned breakdown and unnecessary preventive maintenance. Driven … Continue reading Questions to Ask when Starting a Predictive Maintenance Project →