This is intentionally a technical, list-style article. Its purpose is to provide a reasonably exhaustive catalogue of candidate techniques for reducing superficial language variation without changing the user’s intended business meaning.

Rather than proposing one preferred normalisation approach, it is intended to serve as a reference for designing the query-planning service: which techniques are available, what each is useful for, where each can fail, and when each might be selected or combined in an enterprise semantic-query pipeline.

Why Query Normalisation Matters

Business users rarely express the same concept in exactly the same way as the semantic layer.

A user may write:

floatingRateIndex
floating_rate_index
floating rate index
float rate idx
floating rate indx

These expressions may refer to the same concept, but exact or lexical retrieval may treat them differently.

Queries may also contain:

• spelling mistakes;
• abbreviations;
• camelCase and snake_case identifiers;
• source-system terminology;
• historical names;
• inconsistent punctuation;
• singular and plural forms;
• incomplete business phrases.

Query normalisation tries to reduce these differences before intent classification, mention extraction, and entity resolution.

One thing to note, query normalisation must also be conservative. The following expressions are related, but they are not necessarily equivalent:

FixedRate
fixed rate
fixed-rate payout
fixed leg

A system that normalises too aggressively may improve retrieval recall while silently changing the user’s meaning.

The objective should therefore be:

Create useful alternative representations of the query while preserving the original language, technical identifiers, and transformation history.

This blog article organises the candidate methods into five categoreis:

1. Surface and Structural Normalisation

This category standardises the basic representation of the query without intentionally changing its meaning.

It includes:

• Unicode normalisation;
• whitespace normalisation;
• case normalisation;
• punctuation normalisation;
• tokenisation;
• language detection.

These are normally the lowest-risk methods.

1.1 Unicode Normalisation

Characters that look identical may have different Unicode representations.

Unicode normalisation can standardise:

• composed and decomposed characters;
• full-width and standard-width characters;
• different apostrophe forms;
• different dash characters.

This is useful when questions are copied from emails, documents, PDFs, spreadsheets, or different keyboard layouts.

1.2 Whitespace Normalisation

Typical operations include:

• trimming leading and trailing spaces;
• replacing repeated spaces with one space;
• standardising tabs and line breaks.

1.3 Case Normalisation

Ordinary language can be converted to lowercase for lexical processing. However, the original casing should remain available because it may reveal technical structure. Case normalisation should therefore create an additional lexical representation rather than overwrite the original form.

1.4 Punctuation Normalisation

Different punctuation forms can be aligned, however, they should not be removed indiscriminately. It may carry meaning in:

• qualified names;
• versions;
• paths;
• URIs;
• namespaces;
• code identifiers.

For example:

cdm.product.asset.InterestRatePayout
CDM 6.23
FpML 5-12

should not be flattened without preserving their original structures.

1.5 Tokenisation

Tokenisation determines the units used by later processing.

A general tokenizer might process:

Where does floatingRateIndex map to?

as:

Where, does, floatingRateIndex, map, to

A technical tokenizer may additionally produce:

floating, rate, index, floatingRateIndex

Word tokenisation

Word tokenisation separates ordinary text into word-level tokens.

It is useful for:

• BM25;
• intent patterns;
• phrase matching;
• general natural-language processing.

Whitespace tokenisation

Whitespace tokenisation simply splits text on spaces. It is fast, but it does not understand camelCase, qualified names, or punctuation-heavy identifiers.

Pattern-based tokenisation

Regular expressions or specialised parsers can identify:

• camelCase expressions;
• snake_case expressions;
• qualified paths;
• versions;
• function names;
• source-system identifiers.

Subword tokenisation

LLMs and embedding models use internal subword tokenisation methods, commonly based on WordPiece, Byte Pair Encoding (BPE), or SentencePiece. These methods allow models to encode unfamiliar words and technical identifiers by breaking them into smaller units.

2. Technical and Lexical Normalisation

This category creates searchable lexical representations while preserving the technical expressions found in enterprise queries.

The methods can be divided into three groups:

1. Identifier Decomposition
2. Morphological and Lexical Normalisation

This category is particularly important for enterprise semantic query because user questions often mix natural language with model names, source fields, functions, paths, and formal identifiers.

2.1 Identifier Normalisation

Queries may contain:

floatingRateIndex
floating_rate_index
InterestRatePayout
MapSwapPayout
trade.tradeDate
cdm.product.asset.InterestRatePayout

These expressions should be searchable both as complete identifiers and as decomposed lexical forms.

camelCase decomposition

floatingRateIndex → floating rate index

PascalCase decomposition

InterestRatePayout → interest rate payout

snake_case decomposition

floating_rate_index → floating rate index

kebab-case decomposition

floating-rate-index → floating rate index

Qualified-name decomposition

cdm.product.asset.InterestRatePayout
→ [cdm, product, asset, InterestRatePayout, interest, rate, payout]

The complete qualified name should remain available for exact matching.

Acronym-aware splitting

Simple splitting rules can damage acronyms.

For example:

FpMLTrade

should ideally become:

[FpML, Trade]

rather than:

[Fp, M, L, Trade]

This often requires an acronym dictionary or domain-specific tokenisation rules.

2.2 Morphological Normalisation

Morphological normalisation reduces grammatical variation in ordinary-language tokens.

Stemming

Stemming removes endings using heuristic rules:

[mapping, mapped, maps] → map

It is fast and useful for some lexical-retrieval tasks.

However, stemming may:

• produce unnatural roots;
• merge unrelated words;
• damage technical terminology;
• reduce explainability.

For enterprise semantic queries, stemming should be used cautiously and should not be applied to protected identifiers.

Lemmatisation

Lemmatisation maps words to their dictionary forms:

[am, is, are, was, were] → be

It is generally more linguistically accurate than stemming but requires more processing.

Singular and plural normalisation

attributes → attribute
mappings   → mapping
payouts    → payout

This is useful for natural-language phrases, but exact identifiers should remain untouched.

Stop-word handling

Words such as:

[the, is, of, to]

may be removed or downweighted in lexical retrieval.

However, they should not automatically disappear from Query Understanding because phrases such as:

[maps to, part of, different from]

carry operational meaning.

3. Terminology Normalisation

Terminology normalisation connects the words used in a query with other approved expressions that carry the same, or nearly the same, business meaning. Its purpose is to reduce variation in how a concept is expressed.

It should remain a lexical process:

User expression → Approved alternative expressions

It should not decide which semantic entity the expression represents.

For example:

IRS → interest rate swap

is terminology normalisation.

But:

interest rate swap → a specific semantic entity ID

belongs to Entity Resolution.

Terminology normalisation can be organised into four groups:

1. Abbreviation Expansion
2. Synonym Normalisation
3. Canonical Terminology Alignment
4. Context-Scoped Terminology

3.1 Abbreviation Expansion

Enterprise queries frequently contain abbreviations and acronyms:

[IRS, OIS, FRN, CDS, FpML, CDM, idx, qty, amt]

A governed abbreviation dictionary can provide approved expanded forms:

abbreviation: IRS
expanded_form: interest rate swap
domain: interest_rate_derivatives
status: APPROVED

The original abbreviation should always be preserved. A normalised representation may contain both forms:

original_form: IRS
alternative_forms:
  - interest rate swap
relationship: ABBREVIATION_OF

Ambiguous abbreviations

Some abbreviations have more than one valid expansion.

For example, IR may mean interest rate, informaiton retrieval, investor relations.

Terminology normalisation should retain the valid lexical alternatives and use available context to rank them.

original_form: IR
alternative_forms:
  - value: interest rate
    domain: financial_markets
    confidence: 0.91
  - value: information retrieval
    domain: technology
    confidence: 0.14

This is still lexical interpretation. It does not resolve the expression to a formal semantic entity.

Common implementation methods

Abbreviation expansion may use:

• governed dictionaries;
• domain-specific dictionaries;
• phrase lookup;
• surrounding-word analysis;
• business-domain context;
• model-assisted suggestions for unknown terms.

Model-generated expansions should remain hypotheses until validated against governed terminology.

3.2 Synonym Normalisation

Synonym normalisation connects approved alternative expressions.

Examples include:

IRS         ↔ interest rate swap
float index ↔ floating index
rate leg    ↔ interest rate leg

Not all related terms are true synonyms. For example:

[fixed leg, fixed rate, fixed rate specification]

are related expressions, but they should not automatically be treated as interchangeable. The terminology model should distinguish relationship strengths. Useful relationship types include:

EXACT_EQUIVALENT
APPROVED_SYNONYM
NEAR_EQUIVALENT
RELATED_TERM

For example:

term: rate leg
alternative: interest rate leg
relationship: APPROVED_SYNONYM
domain: interest_rate_derivatives

A weaker relationship may be represented as:

term: fixed leg
alternative: fixed-rate leg
relationship: NEAR_EQUIVALENT

The relationship type should influence how strongly the alternative is used by downstream retrieval.

3.3 Canonical Terminology Alignment

Canonical terminology alignment maps multiple accepted expressions to one preferred business term.

For example:

[float index, floating index, floating-rate benchmark]
        ↓
floating rate index

A governed rule might look like:

canonical_term: floating rate index
accepted_forms:
  - term: float index
    relationship: CANONICAL_FORM_OF
  - term: floating index
    relationship: CANONICAL_FORM_OF
  - term: floating-rate benchmark
    relationship: NEAR_EQUIVALENT

The canonical term provides a consistent lexical representation for:

• intent classification;
• mention extraction;
• BM25 retrieval;
• terminology matching;
• logging and analysis.

3.4 Context-Scoped Terminology

Some expressions are only valid within a particular context. This may include:

• historical terminology;
• source-system terminology;
• organisation-specific terminology;
• product-specific terminology;
• version-specific terminology.

Historical terminology

A term may have been replaced by a newer preferred expression.

original_term: legacy payout term
preferred_term: current payout term
relationship: HISTORICAL_TERM_FOR
valid_until: CDM_5

The system should preserve the historical form when the query concerns an older semantic version. It should not automatically rewrite every historical term into the current terminology.

Source-system terminology

A source platform may use its own lexical conventions.

original_term: RT_INDEX
source_system: MUREX
preferred_term: floating rate index
relationship: SOURCE_SYSTEM_TERM_FOR

This is a terminology relationship, not a source-to-target data mapping. It states that two expressions carry related business meaning within a defined source-system context. It does not state that one physical field populates a particular target attribute.

Organisation-specific terminology

An institution may use internal terms for common market concepts.

Phrase-Aware Processing

Terminology often consists of multi-word expressions:

interest rate swap
floating rate index
business day convention
fixed interest rate leg

These phrases should be processed as units rather than as unrelated tokens.

Common methods include:

• longest-match-first lookup;
• n-gram phrase detection;
• trie-based phrase matching;
• phrase dictionaries;
• synonym graphs;
• model-assisted phrase detection.

For example:

fixed interest rate leg

may contain several candidate phrases:

interest rate
interest rate leg
fixed interest rate
fixed interest rate leg

Terminology normalisation should preserve plausible phrase interpretations when the correct boundary is uncertain.

It should not use phrase matching to decide the final semantic entity.

4. Error Recovery

This category handles queries that may contain accidental errors.

It includes:

• edit-distance correction;
• character n-gram matching;
• dictionary-constrained correction;
• frequency-based correction;
• contextual spelling correction;
• LLM-assisted correction.

A spelling error is different from an abbreviation or synonym. An abbreviation is intentional; a spelling error is accidental.

4.1 Edit-Distance Correction

Edit distance measures how many character-level changes are required to transform one term into another. It is useful for detecting likely spelling mistakes when the incorrect term is still close to a known word or identifier.

Common methods include:

• Levenshtein distance, which counts insertions, deletions, and substitutions;
• Damerau-Levenshtein distance, which also counts adjacent character transpositions;
• Jaro-Winkler similarity, which is particularly effective for short strings and terms that share the same beginning.

For example:

raet → rate

Damerau-Levenshtein treats this as a single adjacent-character transposition rather than two separate edits.

Edit-distance methods work well for small spelling errors, but they can produce misleading matches when terms are short or when several technical identifiers have similar names. They should therefore be constrained by the governed vocabulary and combined with context.

4.2 Character n-grams

Character n-grams break a word into overlapping sequences of characters.

For example, using two-character n-grams:

rate → [ra, at, te]

A misspelled term can still share many of these fragments with the correct term, even when exact matching fails.

Character n-grams are useful for:

• partial matching;
• spelling variations;
• errors in the middle of technical identifiers;
• comparing terms with similar character structure.

They are often more tolerant than whole-word matching, but they can also retrieve unrelated terms that share common fragments. For this reason, n-gram similarity should normally be used for candidate generation rather than final correction.

4.3 Dictionary-constrained Correction

Correction candidates can be restricted to governed vocabularies such as:

• entity names;
• aliases;
• operator names;
• standards;
• business terminology;
• known source-system fields.

This is safer than correcting against a general-language dictionary.

4.4 Frequency-based Correction

The system can rank correction candidates based on how frequently they occur in:

• entity catalogues;
• historical queries;
• domain corpora;
• approved aliases.

Frequency is useful evidence, but it should not override stronger domain or contextual signals.

4.5 Contextual Correction

The surrounding query can help determine the intended correction.

For example, fixed rat payout strongly suggests fixed rate payout.

The words fixed and payout make rate much more plausible than another edit-distance candidate.

4.6 LLM-assisted Correction

An LLM can often infer the intended term from context.

However, it may also rewrite a valid technical name into a more common form. LLM-generated corrections should therefore be treated as hypotheses and validated against governed vocabulary.

5. Query Enrichment and Transformation

Query enrichment and transformation go beyond strict normalisation.

Normalisation converts an expression into an equivalent or near-equivalent form. Enrichment and transformation may introduce related concepts, infer context, reformulate the query, or divide it into multiple subqueries.

These techniques can improve retrieval and interpretation, but they also create a greater risk of changing the user’s intended meaning.

This category contains four distinct capabilities:

1. query expansion;
2. query rewriting;
3. query decomposition;

These capabilities should be implemented and governed separately.

5.1 Query Expansion

Query expansion introduces additional terms or concepts to improve retrieval recall.

For example, fixed leg may be expanded into:

interest rate leg
fixed rate payout
fixed rate specification

This differs from strict normalisation:

Normalisation
= equivalent or near-equivalent representations
Expansion
= related terms introduced to improve retrieval recall

An expanded expression is not necessarily another name for the original expression. It may represent a related concept, component, parent, child, or alternative level of abstraction.

For this reason, every expansion should preserve the relationship between the original term and the expanded term.

Dictionary-based Expansion

Dictionary-based expansion uses governed sources such as:

• approved business terminology;
• related-term dictionaries;
• domain-specific glossaries;
• curated aliases;
• controlled concept mappings.

It is generally more controllable and explainable than model-generated expansion.

For example:

fixed leg → interest rate leg

The expansion record should identify whether the added term is an approved synonym, a related business term, or merely a retrieval aid.

Ontology-based Expansion

Ontology-based expansion follows explicit semantic relationships.

Possible relationships include:

• broader concept;
• narrower concept;
• parent;
• child;
• component;
• containing object;
• related concept.

For example:

fixed leg
→ InterestRatePayout
→ FixedRateSpecification

These terms are related, but they are not equivalent. The relationship type must therefore be preserved:

expanded_forms:
  - value: InterestRatePayout
    relationship: POSSIBLE_BUSINESS_CONCEPT
  - value: FixedRateSpecification
    relationship: POSSIBLE_COMPONENT

Ontology expansion should normally be constrained by the current intent and expected entity type. Otherwise, traversing the ontology may introduce too many loosely related candidates.

Embedding-based Expansion

Embedding-based expansion uses vector similarity to identify related terms, entities, or descriptions. It can discover expressions that are not present in governed terminology dictionaries.

For example, a phrase such as:

the side paying the constant rate

may be close to:

fixed leg
fixed rate payout
fixed rate specification

This method is flexible, but it is more difficult to govern and explain.

Semantically close results may:

• represent different entity types;
• appear at different levels of abstraction;
• belong to different business contexts;
• be related without being interchangeable.

Embedding neighbours should therefore be treated as retrieval candidates, not resolved meanings.

Pseudo-relevance Feedback

Pseudo-relevance feedback performs an initial retrieval, extracts common terms or concepts from the highest-ranked results, and uses them to create an expanded query.

A simplified process is:

Original query
    ↓
Initial retrieval
    ↓
Extract terms from top results
    ↓
Expanded query
    ↓
Second retrieval

This may improve recall when the initial results are relevant. However, if the initial results are wrong, the system may reinforce the wrong interpretation and cause query drift. Pseudo-relevance feedback should therefore be applied cautiously in enterprise semantic query systems. It should normally require:

• a sufficiently reliable initial result set;
• entity-type consistency;
• domain consistency;
• limits on the number of added terms;
• traceability to the source results.

LLM-generated Expansion

An LLM can generate:

• alternative business expressions;
• related search phrases;
• likely technical terminology;
• candidate entity names;
• alternative formulations of the same information need.

This can be useful when users express technical concepts in informal business language.

However, generated terms should carry less evidential weight than:

1. the original query;
2. exact identifiers;
3. approved synonyms;
4. governed aliases;
5. explicit ontology relationships.

LLM-generated expansions should be labelled as generated hypotheses rather than accepted terminology.

5.2 Query Rewriting

Query rewriting creates a reformulated version of the user’s query. Its purpose is usually to make the query easier for downstream components to parse, classify, or execute.

Unlike query expansion, rewriting produces a new query structure rather than simply adding related terms.

Grammar and Spelling Rewriting

A grammatically incomplete or misspelled query may be rewritten into a clearer form:

where floating index maps

becomes:

Where does the floating rate index map?

This type of rewrite should preserve technical identifiers whenever possible.

For example, the system should not replace:

floatingRateIndex

with a more generic phrase if the identifier may refer to an exact model element.

Conversational Rewriting

Conversational queries often depend on earlier turns.

For example:

What about the fixed side?

may be rewritten as:

What is the fixed side of the previously discussed fixed-float interest rate swap?

The rewrite should record which conversational context was used. It should not silently assume context from an unrelated or uncertain previous turn.

Intent-preserving Rewriting

Intent-preserving rewriting converts a natural-language request into a clearer operational form without changing its intended task.

For example:

show me changes to InterestRatePayout

may become:

Compare InterestRatePayout across the selected CDM versions.

The rewrite makes the comparison intent explicit, but it should not invent which versions are being compared unless they are already known from the conversation or application context.

5.3 Query Decomposition

Query decomposition divides a complex question into smaller subqueries.

For example:

Explain where floatingRateIndex maps and how it is transformed.

may be decomposed into:

1. Find the mapping for floatingRateIndex.
2. Retrieve the associated transformation logic.

Decomposition is useful when a query contains:

• multiple intents;
• multiple entities;
• sequential dependencies;
• retrieval and explanation tasks;
• comparison and transformation questions;
• requests requiring different data sources.

The decomposition should preserve the relationship between the subqueries.

For example:

original_query: Explain where floatingRateIndex maps and how it is transformed.
subqueries:
  - id: q1
    intent: MAPPING_LOOKUP
    text: Find the target mapping for floatingRateIndex.
  - id: q2
    intent: MAPPING_LOGIC_QUESTION
    text: Explain the transformation logic associated with the mapping.
    depends_on:
      - q1

This is preferable to treating the two subqueries as unrelated searches.

Conclusion

Query normalisation is not only cleaning language. It is creating controlled and traceable representations that support exact search, lexical retrieval, fuzzy matching, vector retrieval, intent classification, mention extraction, and entity resolution.

Some methods merely standardise the query’s surface form. Others expose technical structure, apply governed terminology, recover likely errors, or introduce related meaning and context.

These methods should not be treated as equally safe or interchangeable.

Low-risk transformations can usually be applied automatically. Medium-risk terminology and correction methods should be governed and retained as hypotheses. High-risk enrichment and rewriting should remain optional, traceable, and clearly separated from the original query.

The objective is not to produce one perfect rewritten query. It is to preserve the original question while generating useful alternatives with clear provenance and controlled semantic risk.

The central principle is:

Enterprise query normalisation should reduce linguistic variation without collapsing distinct business meanings.

• *AI-Native Financial Data Foundation