DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database

DolphinDB – An Integrated Financial Data Platform, Not Just a Time-Series Database

What makes me buy into DolphinDB: 

When I chat about DolphinDB with someone, it’s common for them to bring up those open-source time-series databases for comparison. That’s when I have to step in and say, ‘Hold up, mate, you’re comparing apples to oranges!’ Sure, TSDB is one of DolphinDB’s core storage engines, but it’s just one ingredient in the recipe; it is not the primary value that DolphinDB offers.

To be honest, I have to admit that I can’t see the point in comparing DolphinDB with those general-purpose time-series databases. They are simply designed for different purposes. While those general-purpose time-series databases offer great value in helping people efficiently manage their time-series data, DolphinDB, on the other hand, is a platform specifically designed with features to help professionals address their business needs in the financial services space. The TSDB engine is just one of the components that support those features.

Design Hammer for Nails

The founders of DolphinDB have provided a fitting metaphor for their journey in developing DolphinDB: ‘Design a Hammer for Nails.’ I believe this metaphor captures the essence of DolphinDB: it was born and grew by addressing real-world financial needs. In other words, the designs and features of DolphinDB are driven by the quest for optimal solutions to real-world business requirements.

To handle complex analytical queries of massive financial data volumes, DolphinDB introduced the OLAP engine. To maximise the performance of time-series financial data under extreme conditions like HFT, DolphinDB added the TSDB engine starting from version 2.0. To preserve the primary key for supporting the streaming of line-of-business OLTP data into DolphinDB through CDC, DolphinDB created the PKEY engine. To enable real-time AI generation, the VectorDB engine was introduced to support RAG systems in efficiently searching information from large knowledge bases. To process text-based alternative financial data, DolphinDB added the TextDB engine. To support use cases like financial trading systems with workloads from high-frequency, high-concurrency updates and queries on relational tables requiring extremely high data consistency, DolphinDB introduced IMOLTP, an in-memory OLTP engine.

For the most commonly required time-series window aggregations in financial market data, DolphinDB embedded the Time-Series Streaming Engines. To address the interpretive overhead in use cases like options implied volatility (IV), where vectorisation is not applicable, DolphinDB added support for Just-In-Time (JIT) compilation. To maximise computational power and extract alpha, DolphinDB developed Swordfish, pushing real-time computation latency to the microsecond level, and introduced GPU support to fully leverage the potential of parallel computing.

To support financial professionals in authoring efficient complex calculations and modelling with high productivity, DolphinDB offers over 2,000 built-in functions, carefully optimised for high performance. To cater to FICC (Fixed Income, Currencies, and Commodities) business scenarios, DolphinDB developed the Curve Fit Engine and the Pricing Engine. Additionally, to enable users to conduct end-to-end investment research activities, DolphinDB provides a wide range of modules and plug-ins.

To reduce the learning curve for users, DolphinDB introduced the Python Parser, allowing financial professionals to reuse their existing skills. To reduce IT dependency and accelerate results delivery, DolphinDB supports Unified Batch-Streaming, reducing the involvement of IT. To streamline users’ workflows with industry best practices, DolphinDB offers a suite of integrated solutions, such as the Starfish platform (for supporting factor investing) and cryptocurrency trading solutions.

From DolphinDB’s journey, it’s clear that DolphinDB is a different species compared to general-purpose time-series databases. It is not just a database engine, but rather an integrated platform designed to help financial professionals solve a wide range of financial requirements.

A Integrated Financial Data Platform

Before discussing DolphinDB from the perspective of a data platform, I think it’s important to first consider what businesses actually need from a data platform.

As an integrated platform, it needs to offer comprehensive support across the business process, rather than just focusing on a single task, which is the role of a tool, not a platform. Taking investment research as an example, the platform should support business activities at various stages, such as data collection, pre-trading analysis, factor mining, strategy development, backtesting, live trading, post-trading analysis, and performance attribution.

A financial institution typically spans a wide range of business lines, from research and trading to finance, risk management, and compliance. It can participate in various markets, including equities, forex, fixed income, and commodities, and may also employ a variety of trading strategies. It is unrealistic for a single system to offer solutions that meet all these requirements. Instead, the platform must be flexible, providing a suite of frameworks, components, and services that enable the business to build solutions tailored to diverse business scenarios.

In addition, the platform needs to support a clear separation between IT workloads and business workloads. While the IT teams are responsible for keeping the platform reliable and sustainable, as well as developing domain-agnostic frameworks, the business teams can focus entirely on implementing business logic.

After discussing the requirements of a financial data platform, let’s explore how the DolphinDB data platform meets those needs.

*Please note that the following views describing the DolphinDB platform are not from DolphinDB. Instead, they are just my personal interpretation based on insights from DolphinDB summits and other DolphinDB documentation.

First, based on the information from DolphinDB summit and documentation, I learnt DolphinDB comes with a layered design, including the infrastructure layer, business middleware layer, and the business interface layer.

The infrastructure layer serves as the foundation of the platform, consisting of the core distributed system for data storage and computing, along with components that ensure the platform’s robustness. DolphinDB’s infrastructure layer provides a solid, powerful, and flexible foundation, featuring a variety of built-in storage engines that enable flexible data modelling to meet specific business use cases. The wide range of computing engines offered by DolphinDB is designed and optimised specifically for financial applications. The infrastructure layer is domain-agnostic and is the responsibility of platform engineers to maintain.

The business middleware layer serves as the “Lego Blocks” for building financial applications on DolphinDB data platform. DolphinDB comes with a repository of domain-specific engines, functions, modules, and plug-ins, which is continuously expanding to support more business scenarios. The business middleware is essential for the data platform’s flexibility, customisability, and extensibility. This layer is designed to be loosely coupled with the infrastructure layer, meaning that as long as the infrastructure maintains a stable structure, new middleware components can be added without disruption. DolphinDB now offers a marketplace for modules and plug-ins, and more third-party “Lego Blocks” are expected to created in the future.

The business interface layer is where business teams interact with the platform’s features. This layer serves to separate business workloads from IT workloads. Business users work with the languages, frameworks, and tools provided by this layer to meet their needs, without needing to understand the platform’s internal complexities. DolphinDB offers a suite of development languages and interfaces, such as DolphinScript, Python Parser, and a wide range of APIs, enabling business users to create business-specific solutions. Additionally, DolphinDB provides a set of business solution frameworks to streamline and standardise the entire process for specific business scenarios, such as the complete lifecycle of a factor investing process. These solution frameworks are built with industry best practices, which not only shorten time-to-market but also reduce the effort spent on trial and error.

Based on a solid infrastructure foundation, the modular and reusable business middleware provides the flexibility required to support diverse business scenarios and processes. The business-facing development interfaces and solution frameworks allow business teams to focus on their business logic, while the underlying platform management can be handled by the IT teams.

Recommended Reading:

  1. Advancing Towards an Enterprise-Level Real-Time Computing Platform
  2. DolphinDB Cryptocurrency Solutions White Paper
  3. Quick Start Guide of Python Parser in Quantitative Analysis Scenarios
  4. Crossing Data Boundaries: The Vision for an Enterprise-Level Real-Time Computing Platform – 2024 DolphinDB Annual Summit (in Chinese)
  5. Gravity: 2023 DolphinDB Annual Summit Keynote – Leading the Innovation of Infrastructure Software in the Digital Finance Era (in Chinese)
  6. DolphinDB Starfish Factor Development and Management Platform White Paper (in Chinese)

Leave a comment