Exploring the future of financial data architecture: AI-native financial data foundation, large-scale financial data processing, and enterprise data quality & governance.
Category: Machine Learning / Data Mining
Every stock behaviour pattern has parameters. A compression breakout depends on the rolling window. A VWAP reclaim depends on the volume threshold. A failed breakout depends on the forecast horizon. Change one number and a "profitable" pattern becomes noise. The typical workflow: tweak a parameter → rerun → query results → squint at a table … Continue reading QuantFlow: Brute-Force Grid Search for Stock Behaviour Patterns
Context: Some trading ideas often pop into my mind unexpectedly, but most of the time I am too lazy to investigate them further. The root cause of this "laziness" is not the effort required to explore the idea itself, but rather the amount of pre-work needed before I can actually start working on it. Collecting … Continue reading QuantFlow: Detecting Trading Opportunities Through Market Lead-Lag Profiling
One of the reasons — actually, the core reason — I chose DolphinDB as the built-in streaming engine for QuantFlow's streaming execution layer is that it's really fast, even for the kind of complicated computation that requires chained steps. Thanks to that speed, and with QuantFlow's MarketState engine and FeatureDAG compiler on top, we can … Continue reading QuantFlow Fun – Build a Low-Latency Feature Monitor Dashboard
The previous blog post introduces a list of basic data quality rules that have been developed for my R&D data quality improvement initiative. Those rules are fundamental and essential for detecting data quality problems. However, those rules have existed since a long, long time ago and they are neither innovative nor exciting. More importantly, those … Continue reading dqops Data Quality Rules (Part 2) – CFD, Machine Learning
To fulfil the promise I made before, I dedicate this blog post to cover the topic of Conditional Functional Dependency (CFD). The reason that I dedicate a whole blog post to this topic is that CFD is one of the most promising constraints to detect and repair inconsistencies in a dataset. The use of CFD … Continue reading Data Quality Improvement – Conditional Functional Dependency (CFD)
Problem In one project I have been recently working with, I need to execute R scripts in Azure Batch. The computer nodes of the Azure Batch pool were provisioned with Data Science Virtual Machines which already include common R packages. However, some packages required for the R scripts, such as tidyr and rAzureBatch, are missing … Continue reading The Tip for Installing R packages on Azure Batch
When we process data using Pandas library in Python, we normally convert the string type of categorical variables to the Categorical data type offered by the Pandas library. Why do we bother to do that, considering there is actually no difference with the output results no matter you are using the Pandas Categorical type or … Continue reading Why Bother to Use Pandas “Categorical” Type in Python
*please download the source code here Microsoft has released the public preview of their newest data science service, Azure Machine Learning, that contains a collection of components to support the end-to-end machine learning solution. The Azure Machine Learning Workbench and the Azure Machine Learning Experimentation service are the two main components offered to machine learning practitioners … Continue reading Scaffolding Azure Machine Learning Experiments
I have written a Jupyter notebook describing the Exploratory Data Analysis using Python as shown below:
One of the major use cases of industrial IoT is predictive maintenance that continuously monitors the condition and performance of equipment during normal operation and predict future equipment failure based on previous equipment failure and maintenance history. With an accurate equipment failure prediction organisations can reduce cost from unplanned breakdown and unnecessary preventive maintenance. Driven … Continue reading Questions to Ask when Starting a Predictive Maintenance Project
Data Ninjago (Financial Data Architecture) 
You must be logged in to post a comment.