I have been planning the UX components and research engine for QuantFlow. I thought both would become key pillars of the platform. The UX layer would simplify how engineers build data pipelines and how quants interact with data. The research engine would automate large parts of the quant research workflow, allowing users to focus on ideas rather than infrastructure and implementation details.

However, the more I worked on QuantFlow, the more confused I became about its role. The reason is simple. Many of the operations that I originally designed QuantFlow to perform for engineers and quants can now be performed remarkably well by AI agents.

The more capable the AI becomes, the more I find myself questioning some of the assumptions behind the product. Do users really need a dedicated UX? Do they need prebuilt research functions? Do they even need to interact with QuantFlow directly?

AI Was Pretty Much an Idiot Initially

A few months ago, I was convinced that AI could automate certain tasks, but only when those tasks were clearly defined and structured by humans. Humans were still firmly in the driver’s seat.

This belief seemed correct during the early development of the latest version of QuantFlow. The metadata design was naive. The data mapping and transformation logic were mismatching. The multi-task execution framework was nowhere near what I had envisioned. I still had to code the most of the core framework myself.

Something Changed

As QuantFlow evolved, I noticed the AI became smarter and smarter. Initially, creating a new market data feed required manual work. Field mappings had to be defined. Data transformations had to be written. Data quality rules had to be specified. However, for the recent a couple of weeks, AI can generate an entire feed provider with detailed mappings, validation rules, and transformation logic in a single attempt. In many cases, it not only meets my expectations but also considers edge cases that I had overlooked.

Also, I rarely run Dagster pipelines myself anymore. Instead, I describe what I want in plain English. The AI identifies the correct CLI command and fills in the suitable parameters, and executes it, review result, rerun if any error occured.

The same thing happened elsewhere. One good example is the Behaviour Profiler. My original expectation was straightforward. I would build a profiling framework, then continuously refine YAML configurations, adjust parameters, rerun experiments, analyze results, and iterate manually. Once the framework was in place, I found myself spending less and less time editing configurations. The AI would generate configurations, run experiments, analyze outputs, compare results, and suggest the next direction. The role of the human shifted from operator to reviewer.

This was the moment I started questioning the entire product strategy. I thought I was building tools to automate quant research. Instead, I was watching AI automate quant research using the tools I had built.

The Wrong Bet

Originally, I thought the Feature Library, FeatureDAG, and Research Engine would become the most valuable components of the platform. After all, these were the “smart” parts. I assumed they would become the core differentiators of QuantFlow. Ironically, they turned out to be exactly the areas where AI performs remarkably well. Instead of the pre-defined libraries, AI is good at generating those on demand.

This forced me to rethink a fundamental assumption. Perhaps these components do not need to be products. Perhaps they are simply capabilities that AI can generate when needed, use once, and discard. The things I thought would become the most valuable parts of QuantFlow may not need to exist as permanent components at all.

In addition, looking ahead, I suspect AI will increasingly handle:

• ETL implementation
• Data mapping
• Routine data quality rules
• Configuration generation
• Standard research workflows
• Routine experiments
• Basic data analysis
• Boilerplate feature engineering

These activities are highly structured. They follow patterns.

What AI Still Needs from QuantFlow

Interestingly, the things that remain important are not necessarily the things I originally considered exciting.

Common Data Models

AI can generate mappings, transformations, features, and workflows, but it still needs a stable representation of financial concepts. Without that anchor, different agents may interpret the same data differently. A CDM provides a common language that keeps AI aligned with business reality.

Now, the main challenge for QuantFlow is to evolve the CDM from being human- and machine-readable to AI-readable, especially for the FICC data model. I am currently doing some literature review in this area and will write a few blog posts if I find anything interesting.

High Performance Data Infrastructure

AI still needs access to data. The larger the datasets become, the more important infrastructure becomes. AI does not eliminate the need for data platforms. If anything, it increases their importance.

Trust, Governance, and Observability

Humans need confidence that AI-generated outputs are correct. That means: Monitoring, Auditability, Reproducibility, Lineage, Governance. AI can do all those job for sure, however, do you really want AI to govern themself? The more autonomous the AI becomes, the more critical these capabilities become for human to have.

Novel Problems

AI is excellent at pattern recognition. It is much less reliable when facing entirely new situations. The problems that still require deep investigation, domain expertise, and original thinking remain valuable.

• *DolphinDB