Exploring the future of financial data architecture: AI-native financial data foundation, large-scale financial data processing, and enterprise data quality & governance.
Tag: Stream Processing
For many years, a large portion of systematic strategies relied on relatively low-frequency signals. These approaches worked well when they were under-explored, but over time they have been widely researched, increasingly arbitraged, and structurally compressed in edge. As a result, a growing share of remaining opportunity has shifted toward market microstructure — order flow dynamics, … Continue reading How QuantFlow Handles Large-Scale Market Data
One of the reasons — actually, the core reason — I chose DolphinDB as the built-in streaming engine for QuantFlow's streaming execution layer is that it's really fast, even for the kind of complicated computation that requires chained steps. Thanks to that speed, and with QuantFlow's MarketState engine and FeatureDAG compiler on top, we can … Continue reading QuantFlow Fun – Build a Low-Latency Feature Monitor Dashboard
In the previous blog post, we explored option Greeks calculations using the BSM model. In this post, I’ll have some coding fun by implementing real-time Greeks calculations with the formulas from the last post, but this time using the DolphinDB stream processing framework. Here, I plan to mimic a portfolio consisting of option contracts with … Continue reading Real-Time Option Greeks Calculation with DolphinDB
What makes me buy into DolphinDB: Friendly DolphinDB – Cross-Exchange Arbitraging Case Speedy DolphinDB – Why is DolphinDB so fast? Robust DolphinDB – Reliable, Scalable, Resilient, Consistent, and Monitorable Cost Effective DolphinDB – Worth the Money DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database When I chat about DolphinDB with someone, it’s common … Continue reading DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database
From Spark v3.2, session window is natively supported by Spark Structured Streaming. Session window based aggregation is a common requirement of streaming data processing, especially in the use cases such as user behaviour analytics. In this blog post, I will discuss how session window works under the hood in Spark Structured Streaming. Compared to the … Continue reading Spark Structured Streaming Deep Dive (8) – Session Window
This blog post discusses another stateful operation supported by Spark Structured Streaming, Stream-Stream Join, which joins two streaming datasets. Unlike static datasets join, for the rows reaching to one side of the input streams in a micro-batch, the matching rows would highly likely be not received in the other side of the input streams at … Continue reading Spark Structured Streaming Deep Dive (7) – Stream-Stream Join
There are two types of streaming processing modes, Stateless and Stateful. Stateless is easy to understand that each message is processed independently without the needs to maintain the states across multiple messages. The challenge and fun one is the Stateful streaming processing where the processing of a message depends on the result of the processing … Continue reading Spark Structured Streaming Deep Dive (6) – Stateful Operations
Spark Structured Streaming reuses the Spark SQL execution engine, including the analyser, optimiser, planner, and runtime code generator. QueryExecution is the core component of the Spark SQL execution engine, which manages the primary workflow of a relational query execution using Spark. IncrementalExecution is a variant of QueryExecution that supports the execution of a logical plan … Continue reading Spark Structured Streaming Deep Dive (5) – IncrementalExecution
This blog post deep dive into the Azure Event Hubs Connector for Apache Spark, the open-source streaming data source connector for integrating Azure Event Hubs with Spark Structured Streaming. The Azure Event Hubs Connector implements the Source and Sink traits with the EventHubSource and the EventHubSink for receiving streaming data from or writing streaming data … Continue reading Spark Structured Streaming Deep Dive (4) – Azure Event Hub Integration
This blog post discusses another main component in the Spark Structured Streaming framework, Sink. As the KafkaSink will be covered when discussing the Spark-Kafka integration, this blog post will focus on ForeachBatchSink, ForeachWriteTable, FileStreamSink and DeltaSink. Spark Structured Streaming defines the Sink trait representing the interface for external storage systems which can collect the results … Continue reading Spark Structured Streaming Deep Dive (3) – Sink
Data Ninjago (Financial Data Architecture) 
You must be logged in to post a comment.