Author: Linxiao Ma

Robust DolphinDB – How does DolphinDB Achieve Scalability, Reliability, Resilience, Consistency, and Monitorability

Robust DolphinDB – How does DolphinDB Achieve Scalability, Reliability, Resilience, Consistency, and Monitorability

What makes me buy into DolphinDB: Friendly DolphinDB –  Cross-Exchange Arbitraging Case Speedy DolphinDB – Why is DolphinDB so fast? Robust DolphinDB – Reliable, Scalable, Resilient, Consistent, and Monitorable Cost Effective DolphinDB – Worth the Money DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database As a high-performance database built for business-critical financial applications … Continue reading Robust DolphinDB – How does DolphinDB Achieve Scalability, Reliability, Resilience, Consistency, and Monitorability →

Speedy DolphinDB – Why is DolphinDB So Fast?

Speedy DolphinDB – Why is DolphinDB So Fast?

What makes me buy into DolphinDB: Friendly DolphinDB –  Cross-Exchange Arbitraging Case Speedy DolphinDB – Why is DolphinDB so fast? Robust DolphinDB – Reliable, Scalable, Resilient, Consistent, and Monitorable Cost Effective DolphinDB – Worth the Money DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database In my last blog post, I showcased a cross-exchange … Continue reading Speedy DolphinDB – Why is DolphinDB So Fast? →

Friendly DolphinDB – Cross-Exchange Arbitraging Case

Friendly DolphinDB – Cross-Exchange Arbitraging Case

What makes me buy into DolphinDB: Friendly DolphinDB –  Cross-Exchange Arbitraging Case Speedy DolphinDB – Why is DolphinDB so fast? Robust DolphinDB – Reliable, Scalable, Resilient, Consistent, and Monitorable Cost Effective DolphinDB – Worth the Money DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database I reckon there is no doubt that kdb+ has … Continue reading Friendly DolphinDB – Cross-Exchange Arbitraging Case →

Buy-Side Financial Data Engineering (3) – Market Data Management

Buy-Side Financial Data Engineering (3) – Market Data Management

Buy-Side Financial Data Engineering (1) - Overview Buy-Side Financial Data Engineering (2) - Financial Instruments Buy-Side Financial Data Engineering (3) - Market Data Management As a data guy, two thoughts immediately come to my mind when I hear the term "Finance Market Data", 1) They are bloody expensive; 2) What a chore to handle all … Continue reading Buy-Side Financial Data Engineering (3) – Market Data Management →

Buy-Side Financial Data Models (2) – Financial Instruments

Buy-Side Financial Data Models (2) – Financial Instruments

Buy-Side Financial Data Engineering (1) - Overview Buy-Side Financial Data Engineering (2) - Financial Instruments Buy-Side Financial Data Engineering (3) – Market Data Management The second article of my "Buy-Side Financial Data Models" focuses on the "Financial Instruments" data domain. Financial instruments data is complex and difficult to manage. In the meantime, it is crucial to … Continue reading Buy-Side Financial Data Models (2) – Financial Instruments →

Buy-Side Financial Data Models (1) – Overview

Buy-Side Financial Data Models (1) – Overview

Buy-Side Financial Data Engineering (1) - Overview Buy-Side Financial Data Engineering (2) - Financial Instruments Buy-Side Financial Data Engineering (3) – Market Data Management This is the first blog post of the "Buy-Side Financial Data Models" series I am planning to write. To kick off this blog series, this post provides a high-level overview of the … Continue reading Buy-Side Financial Data Models (1) – Overview →

Spark Structured Streaming Deep Dive (8) – Session Window

Spark Structured Streaming Deep Dive (8) – Session Window

From Spark v3.2, session window is natively supported by Spark Structured Streaming. Session window based aggregation is a common requirement of streaming data processing, especially in the use cases such as user behaviour analytics. In this blog post, I will discuss how session window works under the hood in Spark Structured Streaming. Compared to the … Continue reading Spark Structured Streaming Deep Dive (8) – Session Window →

Spark Structured Streaming Deep Dive (7) – Stream-Stream Join

Spark Structured Streaming Deep Dive (7) – Stream-Stream Join

This blog post discusses another stateful operation supported by Spark Structured Streaming, Stream-Stream Join, which joins two streaming datasets. Unlike static datasets join, for the rows reaching to one side of the input streams in a micro-batch, the matching rows would highly likely be not received in the other side of the input streams at … Continue reading Spark Structured Streaming Deep Dive (7) – Stream-Stream Join →

Spark Structured Streaming Deep Dive (6) – Stateful Operations

Spark Structured Streaming Deep Dive (6) – Stateful Operations

There are two types of streaming processing modes, Stateless and Stateful. Stateless is easy to understand that each message is processed independently without the needs to maintain the states across multiple messages. The challenge and fun one is the Stateful streaming processing where the processing of a message depends on the result of the processing … Continue reading Spark Structured Streaming Deep Dive (6) – Stateful Operations →

Spark Structured Streaming Deep Dive (5) – IncrementalExecution

Spark Structured Streaming Deep Dive (5) – IncrementalExecution

Spark Structured Streaming reuses the Spark SQL execution engine, including the analyser, optimiser, planner, and runtime code generator. QueryExecution is the core component of the Spark SQL execution engine, which manages the primary workflow of a relational query execution using Spark. IncrementalExecution is a variant of QueryExecution that supports the execution of a logical plan … Continue reading Spark Structured Streaming Deep Dive (5) – IncrementalExecution →