AI-Native Financial Data Foundation (30): Exploring a Retrieval Pattern for the Financial Semantic Layer

AI-Native Financial Data Foundation (30): Exploring a Retrieval Pattern for the Financial Semantic Layer

In the previous article, I raised a practical question: what should happen when a user does not use the exact field name, semantic concept, or use-case name stored in the graph?

A user may ask: “Do we have the fixed coupon needed for valuation?”. However, the graph may use governed names such as “FixedRate“. The system therefore needs to translate business language into the appropriate graph entities and retrieve the evidence needed to answer the question.

A useful retrieval flow is:

user question
→ classify intent and extract context
→ search for candidate entities
→ resolve the intended entity
→ execute a constrained graph traversal
→ return governed evidence
→ generate an explanation

Search helps bridge user language and graph vocabulary. Entity resolution determines what the user most likely means. Traversal retrieves the relevant evidence path. Governance metadata controls how confidently the result can be explained.

The rest of this article uses a broker IRS filled-trade onboarding scenario to examine how this could work in practice.

The problem we need to solve

Assume a broker sends a daily filled-trades CSV for interest rate swaps containing fields such as:

trade_id, product_type, notional, ccy, fixed_rate, float_index,
effective_date, maturity_date, pay_freq, reset_freq, counterparty,
trade_status

At the physical level, this is simply a broker trade feed. For an AI-native financial data foundation, however, the important question is not only whether these columns exist. It is whether the dataset provides the governed semantic concepts required by a use case such as “IRS Valuation Readiness“.

The user may say “fixed coupon”, while the graph stores FixedRate. They may say “day count”, while the graph stores DayCountConvention. They may ask whether the feed can “support IRS valuation”, while the graph represents this through multiple required concepts, mappings, validation rules, and review statuses.

The retrieval layer therefore needs to connect:

natural business language
→ governed semantic entities
→ verified graph evidence

This is more than a text-search problem. It requires intent interpretation, contextual entity resolution, controlled graph traversal, and evidence governance.

The retrieval pattern

Step 1 – Classifying intent and extracting context

Intent classification and context extraction should be considered together. When a user asks a question, the retrieval layer needs to interpret the question in two ways. First, it needs to understand the user’s business meaning. Second, it needs to decide what type of graph evidence should be retrieved. This second part is important because different questions require different graph traversal patterns.

For example:

Can this broker feed support IRS valuation?

This is not a single-field lookup. It is asking whether a dataset can support a use case. The expected graph evidence is a readiness assessment: required concepts, provided concepts, missing concepts, and mappings that still require review.

A structured interpretation may look like this:

{
"intent": "readiness_assessment",
"confidence": 0.93,
"context": {
"dataset_hint": "broker feed",
"product_type_hint": "IRS",
"use_case_hint": "valuation",
"term_to_resolve": null
}
}

Another question may be:

Do we have the fixed coupon needed for valuation?

This is more specific. The system needs to resolve “fixed coupon” to a semantic concept, then check whether the dataset provides that concept for the valuation use case. The expected graph evidence is no longer a full dataset readiness assessment. It is a concept-level evidence path.

fixed coupon → FixedRate → source field → dataset → valuation use case → validation rule

A structured interpretation may look like this:

{
"intent": "concept_lookup",
"confidence": 0.89,
"context": {
"dataset_hint": "broker feed",
"product_type_hint": "IRS",
"use_case_hint": "valuation",
"term_to_resolve": "fixed coupon"
}
}

A practical intent model can be organised around a small number of semantic task families, such as:

IntentTypical questionRetrieval pattern
concept_lookupWhat does X mean? Why is X required? Do we have X?field / term → concept → use case → rule
readiness_assessmentIs this feed ready? What is missing? What blocks readiness?dataset × use case → present / missing / review
alternative_sourceWhere can I find X? Is there another source?concept → fields → datasets → systems
mapping_blocker_analysisWhich mappings block readiness? Which fields need review?dataset × use case → mappings requiring review / low confidence

The goal is not to create a large taxonomy of intents. The goal is to map user questions to reusable graph retrieval patterns.

In other words, intent classification answers the question:

Which retrieval pattern should be used for this user question?

Context extraction answers the question:

Which dataset, product type, use case, field, or business term should be used as search hints?

Step 2 – Scoped hybrid candidate search

Hybrid candidate search means finding possible graph entry points using multiple retrieval signals.

It can combine:

exact lookup
alias / search-term lookup
full-text search
vector search
graph-context filtering

The purpose is not to answer the question immediately. The purpose is to find candidate graph nodes from which traversal may begin. This matters because user language and graph vocabulary are often different.

Hybrid search finds possible matches. But in a large graph, search should not blindly scan the whole graph and trust the top result. Many nodes may have similar names. A phrase such as “rate” may match FixedRate, FloatingRateIndex, Spread, DiscountRate, RepoRate, and many other concepts.

So search needs to be scoped using graph context such as:

dataset
product type
use case
asset class
source system
concept category
relationship proximity
governance status

For example:

User phrase: fixed coupon
Context: IRS valuation readiness, Broker A IRS Filled Trades

A candidate result may look like this:

{
"term": "fixed coupon",
"candidates": [
{
"node_id": "concept:FixedRate",
"label": "SemanticConcept",
"name": "FixedRate",
"score": 0.91,
"matched_by": ["alias", "fulltext", "use_case_context"],
"context_match": true
},
{
"node_id": "concept:CouponRate",
"label": "SemanticConcept",
"name": "CouponRate",
"score": 0.78,
"matched_by": ["vector"],
"context_match": false
}
]
}

Search finds candidates. It does not decide the answer.

Step 3 – Entity resolution

Entity resolution is the step that chooses the exact graph node the user likely means. Hybrid search returns candidates. Entity resolution selects one.

For example:

User says: fixed coupon
Search finds: FixedRate, FixedCouponRate, CouponRate, FixedLegRate
Entity resolution selects: FixedRate

The selected node then becomes the entry point for graph traversal. Entity resolution should consider:

search score
score gap
context match
relationship proximity
use-case relevance
dataset relevance
product-type relevance
governance status

A deterministic scoring policy is useful because it makes the behaviour easier to test.

For example:

If top_score - second_score > 0.15, select the top candidate.
If a candidate is connected to the relevant use case, boost it.
If candidates remain close after boosting, return an ambiguity response.

A resolved result may look like this:

{
"resolution_status": "resolved",
"selected_entity": {
"node_id": "concept:FixedRate",
"label": "SemanticConcept",
"name": "FixedRate"
},
"confidence": 0.89,
"reason": "The phrase fixed coupon matches FixedRate, and FixedRate is required by IRS Valuation Readiness."
}

An ambiguous result may look like this:

{
"resolution_status": "ambiguous",
"term": "day count",
"candidates": [
"DayCountConvention",
"DayCountBasis",
"DayCountFraction"
],
"answer_policy": {
"allowed_claim": "ambiguous",
"must_ask_clarifying_question": true,
"allow_speculation": false
}
}

This is better than silently traversing from the wrong node.

Step 4 – Constrained graph traversal

After entity resolution, the retrieval layer should not freely expand the graph. It should use a constrained traversal pattern.

For example:

IntentTraversal pattern
concept_lookupfield / term → concept → use case → validation rule
readiness_assessmentdataset × use case → present / missing / requires review
alternative_sourceconcept → fields → datasets → source systems
general_explorationcontrolled Text-to-Cypher or fallback retrieval

This is safer than open-ended graph traversal.

If the user asks: “Why is fixed_rate required?”, the system should not return every node connected to fixed_rate. It should retrieve the evidence path that explains requirement:

fixed_rate → FixedRate → IRS Valuation Readiness → FixedRateMustBePresent

This keeps the answer focused. It also makes the retrieval behaviour testable. Each traversal pattern need also have a known input shape and expected output shape.

Step 5 – Governed evidence

The graph should return more than facts. It should return governed evidence. For example, the result should not only say:

fixed_rate maps to FixedRate

It should also say:

mapping confidence = 0.88
review_status = requires_sme_review
mapping_status = proposed
validation_rule = FixedRateMustBePresent
evidence_path = fixed_rate → FixedRate → IRS Valuation Readiness → FixedRateMustBePresent

This metadata should control the LLM’s response. If a mapping requires SME review, the LLM should not say it is confirmed. If a required concept is missing, the LLM should not say the feed is ready. If confidence is low, the LLM should use cautious language.

A governed output may include an answer_policy:

{
"answer_policy": {
"allowed_claim": "not_ready",
"must_flag_review_items": true,
"must_list_missing_concepts": true,
"must_suggest_alternatives": true,
"allow_speculation": false,
"confidence_floor": 0.70
}
}

This is important because the policy is not just decoration. It tells the LLM how the evidence may be explained.

A governed retrieval layer should also define what happens when retrieval fails. The system should not hide failure behind a fluent LLM answer.

For example:

{
"status": "no_candidates",
"searched_terms": ["haircut"],
"searched_indices": ["exact", "alias", "fulltext", "vector"],
"answer_policy": {
"allowed_claim": "unknown",
"must_acknowledge_gap": true,
"allow_speculation": false,
"suggest_human_review": true
}
}

Expected answer:

I could not find a governed concept matching "haircut" in the current semantic graph.
This may mean the concept has not been onboarded yet,
rather than that it does not exist in the business domain.

A trustworthy retrieval system should explain not only what it found, but also what it failed to find.

Step 6 – LLM Explanation

The final step is LLM explanation. The LLM should not decide the facts or invent missing relationships. Its role is to turn the retrieved evidence into a clear business answer while preserving confidence levels, review requirements, missing information, and other boundaries defined by the governed result.

Why this is less work than it looks

At this point, the retrieval pattern may look like it introduces a lot of moving parts. This can easily create the impression that a huge amount of manual definition is required. But the actual work is smaller than it first appears if the pattern is designed around reusable graph retrieval primitives.

The system does not need a new intent, a new graph query, a new prompt, and a new response format for every user question. Instead, many user-facing questions can be decomposed into a smaller set of reusable retrieval patterns.

For example, these questions look different:

  • Can this broker feed support IRS valuation?
  • What is missing for valuation?
  • Which fields block valuation readiness?
  • Which mappings still require review?

But they can reuse the same underlying comparison pattern:

dataset
→ provided concepts
→ compare with required concepts for a use case
→ classify as approved, missing, or requiring review

So the important distinction is this:

User-facing questions can grow quickly.
Reusable graph retrieval primitives should grow much more slowly.

This does not mean there is no modelling work. Semantic concepts, mappings, aliases, use-case requirements, validation rules, confidence scores, and review status still need to be captured. But the work becomes cumulative rather than repetitive.

That is why this pattern is more manageable than it may first appear. The system does not need to define a new route for every question. It needs a compact set of reusable graph retrieval capabilities that can be composed across many questions.

Conclusion

This retrieval pattern gives the Agentic Semantic Layer a clearer shape.

The key idea is to separate responsibilities:

Search finds candidates.
Entity resolution selects the intended node.
Traversal retrieves governed evidence.
The LLM explains the evidence.
Governance metadata controls the caveats.

This is different from a generic chatbot retrieval pattern. For governed financial semantic questions, the important output is not only a relevant text chunk. The important output is a controlled evidence path with confidence, review status, missing concepts, alternative sources, and answer boundaries.

Leave a comment