Entity Screening Algorithms

1. What We Solve

The platform screens people, companies, banks, vessels and aircraft against sanctions and screening sources.

Input parameters:

• name or legal name;
• country;
• date of birth;
• registration number, IMO, BIC/SWIFT or another identifier;
• entity type;
• source list selection;
• historical as-of screening date;
• minimum score threshold.

Output:

• candidate matches;
• score from 0 to 100;
• explanation of why the score was assigned;
• source, regulation, annex and grounds;
• for the VoP profile: MTCH, CMTC, NMTC, NOAP.

---

2. Two Algorithm Profiles

Short formula:

Standard = find sanctions candidates and estimate match risk.
VoP = take the retrieved candidates and classify the name using EPC close-match rules.

---

3. Standard Screening Pipeline

---

4. Step 1: Normalization

Goal: remove noise before comparison. Both the submitted query and every stored name variant pass through the same pipeline — normalization runs once at import time (stored as pre-computed columns) and again at query time, so formatting differences, script variations and legal suffixes never prevent a match on their own. Six steps, run in this order:

1. Lowercase. Script-neutral — applies the same way to Latin, Cyrillic, Arabic and every other script.

2. Arabic normalization. Diacritical vowel marks (harakat) are stripped, the four alef variants (آ أ إ ٱ) collapse to bare alef (ا), four further letter variants normalize (ى→ي, ة→ه, ؤ→و, ئ→ي), and the definite article ال (al-) is removed from the front of a word.

3. Arabic-to-Latin transliteration. What remains is mapped to approximate Latin equivalents, so it indexes alongside the Latin-script transliterations already present in the source data:

The remaining 22 base Arabic letters map one-to-one (ب→b, ت→t, ج→j, and so on). Verified live end to end: خالد normalizes to خالد (no diacritics or alef variants to strip in this word) and then transliterates to khald.

4. Legal-form suffix stripping. A fixed list of suffixes is stripped by regex — jsc, oao, ooo, ao, pao, ojsc, pjsc, npo, npk, llc, ltd, co, corp, inc, plc, ag, gmbh, sa, sas, srl, bv, nv, ab, as, oy, pub among them. Worth knowing: this list is narrower than the one VoP normalization uses — SIA and UAB are not in it, so general screening leaves a name like SIA Baltija as sia baltija, not baltija. VoP's separate, broader suffix table does strip them; general sanctions screening does not need to, because the token and trigram paths already tolerate the extra word.

5. Character whitelist. Only lowercase Latin, lowercase Cyrillic, ASCII digits and whitespace survive; everything else becomes a space.

6. Whitespace collapse. Multiple consecutive spaces become one; leading and trailing spaces are trimmed.

Examples, verified live end to end:

Why it matters:

Without normalization, "Sirius JSC", "JSC Sirius" and "Sirius" look like different strings.
After all six steps, they become comparable.

---

5. Step 3: Trigram Similarity

Name: Trigram Similarity Technology: PostgreSQL pg_trgm, similarity() function Index: GIN on name_normalized

How it works:

A string is split into overlapping three-character fragments. The score is the Jaccard coefficient of the query's trigram set and the stored name's trigram set:

similarity = |trigrams(query) ∩ trigrams(name)| / |trigrams(query) ∪ trigrams(name)|

Worked example, run live against pg_trgm.similarity():

Query:        "ivanov ivan"
Trigrams (8): "  i", " iv", "iva", "van", "an ", "nov", "ov ", "ano"

Stored:       "ivanov ivan petrovich"
Trigrams (18): the 8 above, plus "  p", " pe", "pet", "etr", "tro", "rov",
               "ovi", "vic", "ich", "ch "

Intersection: 8   (every query trigram appears in the stored set)
Union:        18
similarity:   8 / 18 ≈ 0.444  ->  raw score ≈ 44

This is a candidate, but not a strong hit on its own — it needs the token-set or phonetic paths below to confirm it, or a country/DOB bonus to clear the review threshold.

Meaning:

• good for similar names;
• robust to small differences;
• weak when word order changes unless token-sort is also used.

---

6. Step 4: Token-Sort Trigram

Name: Token-Sort Trigram Similarity Technology: pg_trgm.similarity() on sorted words Index: GIN on name_tokens_sorted

How it works:

Words are sorted alphabetically and then compared with trigram similarity.

Example, run live:

Query:   "Sirius Trading LLC"
After normalization: "sirius trading"
Token sort:          "sirius trading"

Stored:  "Trading House Sirius LLC"
After normalization: "trading house sirius"
Token sort:          "house sirius trading"

similarity("sirius trading", "house sirius trading") ≈ 0.714  ->  raw score ≈ 71

Meaning:

• solves First Last vs Last First;
• useful for companies where legal-name word order often varies;
• complements ordinary trigram matching rather than replacing it.

---

7. Step 5: Double Metaphone

Name: Double Metaphone Phonetic Matching Technology: PostgreSQL fuzzystrmatch, dmetaphone() Goal: find names that are written differently but sound similar.

Examples, run live against dmetaphone() — the codes below are the actual output, not illustrative:

Step-by-step example:

Query:  "Mikhail Ivanov"
Codes:  "MKL AFNF"

Stored: "Michael Ivanoff"
Codes:  "MKL AFNF"

Result: identical phonetic codes -> phonetic path gives a strong match.

For Cyrillic, transliteration runs first:

"Михаил Иванов" -> cyrillic_to_latin -> "Mikhail Ivanov" -> Double Metaphone

This is also the only bridge between the Cyrillic and Latin alphabets: the two scripts share no trigrams, so a Cyrillic query and a Latin stored name score zero on trigram similarity no matter how alike they read. Double Metaphone is what lets «Орлан» find a Latin-registered ORLAN at all. See the script_corroboration field in §14 for how a single-token phonetic-only match is distinguished from a genuine spelling match once both sides are compared in one script.

---

8. Step 6: Levenshtein Edit Distance

Name: Levenshtein Edit Distance Technology: levenshtein_less_equal() Goal: short names and typos.

Levenshtein counts the minimum number of edits:

• insert;
• delete;
• substitute.

similarity = 1 - (levenshtein_distance / max(len(query), len(name)))

Example, run live:

Query:  "putin"
Stored: "putyin"

Distance: 1
max length: 6
similarity: 1 - 1/6 = 0.833  ->  raw score 83

Meaning:

• useful for short queries;
• important where trigram has too little information;
• not used as identity confirmation without other signals.

---

9. Step 7: Exact Token Matching

Name: Exact Token Matching Technology: token table + btree index Goal: fast deterministic lookup by words, aliases and transliterations.

token_set_score = count(query_tokens ∩ entity_tokens) / count(query_tokens)

Example:

Query: "Northbridge Trading"

Stored entity tokens:
  "northbridge"
  "trading"
  "northbridge trading"
  "nb trading"

Intersection: "northbridge"
Token-set score: 100

Meaning:

• strong signal for aliases;
• especially useful for transliteration;
• exact legal/vessel aliases can avoid the single-token fuzzy cap;
• for natural persons, this is the path a short single-token query still has available once fuzzy matching is gated off — see §12.

---

10. Step 8: Identifier Matching

Name: Identifier Matching Technology: normalized identifiers / document tables Goal: find an entity by exact official identifier.

Supported types, with the confidence level each one is assigned in score_breakdown.identifier_confidence when it forces a match — and why:

Example:

Input:  "4000 3012 345"
Stored: "40003012345"

normalize_identifier() makes both strings identical.
Result: document_match = true, final_score = 100.

Meaning:

Identifier match is stronger than name match.
If the official identifier matches, the score is forced to 100.

---

11. How Score Is Assigned

Base formula:

raw_name_score = max(
  trigram_normalized,
  token_sort_trigram,
  phonetic_double_metaphone,
  levenshtein,
  exact_token_score
) * 100

Then:

pre_cap_score = raw_name_score + dob_bonus + country_bonus
final_score = apply_caps_and_identifier_override(pre_cap_score)

Bonuses:

Override:

Worked example — "Ivanov Ivan", country RU, no date of birth supplied:

Trigram similarity:      72   (Algorithm §5)
Token-sort similarity:   75   (Algorithm §6)
Phonetic similarity:     90   (Algorithm §7)  <- maximum
Token-set score:         67   (Algorithm §9)

raw_name_score:  max(72, 75, 90, 67) = 90
country bonus:   entity.country = "RU"  ->  +5
dob bonus:       not provided  ->  +0

pre_cap_score:   90 + 5 = 95
single-token check: query has 2 tokens  ->  cap does not apply
identifier check:   not provided  ->  no override

final_score:     95  ->  "hit" tier, human review mandatory

---

12. Single-Token Safety

Problem:

Query: "Ivanov"

One token may be a surname, part of a name, an alias or a common word. Even when the fuzzy score is high, it does not always confirm identity — and for a short token, phonetic matching makes it worse: a four-character query such as "teit" collapses to the same phonetic skeleton as several unrelated names (Tito, Tata, Daud, ...). The rule differs by entity type.

Legal person, vessel, aircraft, bank. The original rule still applies unchanged: a fuzzy-only or phonetic-only single-token match is capped at score 84 — placed in the review tier, not confirmed — but it is still returned. An exact-token alias match or an identifier match is not capped and can reach 100.

Natural person. Short single-token queries do not reach the fuzzy or phonetic path at all. Below a configurable normalized length (single_token_min_fuzzy_length, default 5 characters), the query only runs through exact-token and exact-identifier matching. Fuzzy matching reopens for one specific candidate only when a date of birth was supplied with the query and it matches that candidate's date of birth — merely supplying some DOB is not enough, it has to match the candidate.

Verification of Payee is exempt from the natural-person gate — see §17. It has its own regulated close-match rules and needs to evaluate short names the same way regardless of length.

The threshold is configurable per organization; a lower value screens more short names through the fuzzy path, a higher value keeps the gate stricter.

---

12a. Multi-Token Phonetic-Only Cap

The single-token cap above does not cover every collision. Double Metaphone truncates its output to 4 characters, and for multi-syllable Slavic surnames that is short enough for genuinely unrelated names to land on the same code by coincidence:

"Aleksandra Lukachenko"  →  ALKS LKXN
"Laktionov Aleksandr"    →  ALKS LKXN   ← identical, unrelated surname
"Lokshin Aleksandr"      →  ALKS LKXN   ← identical, unrelated surname

Because the phonetic path compares the whole query string against the whole candidate string, one coincidental code match was enough to reach a perfect phonetic score — and with a two-word query, that one word can be half the string. Before this rule, all three names above scored 100% against the same query.

The cap applies only when every one of these holds:

• the match came from the fuzzy/phonetic path (not an exact token or an identifier);
• the query has two or more words — a one-word query is left alone, because that is the only way to bridge two scripts for a one-word name (see the EuroLine LLC / «Орлан» example below) and capping it would create real missed matches;
• no word in the query matched a word in the candidate exactly;
• script_corroboration is below 30%.

script_corroboration is a per-word minimum, not a whole-string comparison: transliterate both names to Latin, split into words, find each query word's single best-matching candidate word, and report the minimum of those per-word scores. Requiring every word to individually clear the bar matters: a first fix of this rule compared the whole strings at once and wrongly capped a genuine match where the surname agreed but the given name used a different transliteration (see the third example below) — measuring word-by-word and taking the minimum fixed that without reopening the original hole.

When all four hold, the raw score is capped at 60 before the date-of-birth and country bonuses are added — score_breakdown.phonetic_only_capped = true. A matching date of birth (+10) plus a matching country (+5) can still lift it to exactly 60 + 10 + 5 = 75, the platform's default review threshold, so real corroborating evidence is never thrown away — only a bare coincidence of sound is kept from reaching the same confidence as an actual match.

Query:      "Aleksandra Lukachenko"
Candidate:  "Laktionov Aleksandr" (Ukraine sanctions register)

phonetic similarity:    1.00  (4-char code collision)
token overlap:          0
script_corroboration:   5%    (worst-matching word: "lukachenko" vs its best
                                candidate word ≈ 5%, even though "aleksandra"
                                matches "aleksandr" at ≈ 75%)
→ below 30% → capped at 60, below the default 75 threshold

Candidate:  "Aleksandr LUKASHENKO" (Japan MOF) — same query
phonetic similarity:    1.00  (same code, but here it IS the real surname)
script_corroboration:   57%   (both words individually agree)
→ not capped, stays at 100

Candidate:  "Lukashenko Oleksandr Hryhorovych", alias "Lukashenka Aliaksandr"
            (Ukraine's own register — the SAME real person, Belarusian
            transliteration of the given name)
script_corroboration:   38%   (surname "lukachenko" vs "lukashenka" ≈ 37.5%,
                                which is what clears the bar — the given name
                                "aleksandra" vs "aliaksandr" is weaker but
                                irrelevant since the minimum is taken over
                                the WORST word, and the worst word here is
                                still well above 30%)
→ not capped, stays at 100

---

13. Our Rating System

Default threshold:

75

The threshold can be configured for each organization.

---

14. Score Breakdown

Each match contains an explainability payload:

script_corroboration exists because a phonetic-only match and a genuine cross-script match can otherwise carry the identical score. Screening "EuroLine LLC" against a Russian source can return "OOO ORLAN" at 89%: normalization strips the legal form to the single tokens euroline and orlan, Double Metaphone reduces both to the same consonant skeleton, and the single-token cap brings it to 84 plus a country bonus. Nothing about the spelling actually agrees. Screening «Орлан» against the same OOO ORLAN scores the identical 89% — but here the two names agree once both are read in one script. For a one-word query the score genuinely stays the same either way (see §12a for why); for a two-or-more-word query, low script_corroboration with zero token overlap now caps the score itself (phonetic_only_capped), because there the risk of silencing a real cross-script match is gone — a genuine multi-word match already agrees in spelling once transliterated.

Reading An Explainability Payload

Two real response shapes, and what each one actually means for a reviewer.

A fuzzy-name match:

{
  "match_source": "fuzzy_name",
  "raw_name_similarity": 90,
  "name_similarity": 90,
  "token_set_score": 67,
  "query_token_count": 2,
  "single_token_score_cap": false,
  "document_match": false,
  "identifier_confidence": null,
  "dob_match": false,
  "country_match": true,
  "as_of_date": "2026-05-11"
}

Found via the fuzzy-name path — one of trigram, token-sort or phonetic. Name similarity before any caps was 90. The query had two tokens, so no single-token cap applied. No identifier or date of birth was supplied, but the country matched, contributing the +5 bonus. The resulting 95 is name similarity plus a country bonus — evidence for review, not a confirmed identity by itself.

An identifier match:

{
  "match_source": "identifier",
  "raw_name_similarity": 100,
  "name_similarity": 100,
  "token_set_score": 100,
  "query_token_count": 2,
  "single_token_score_cap": false,
  "document_match": true,
  "identifier_confidence": "high",
  "dob_match": false,
  "country_match": false,
  "as_of_date": "2026-05-11"
}

Score 100 was forced by an exact registration-number match. identifier_confidence: "high" means the identifier is state-issued and globally unique — a near-certain identity confirmation, though local policy may still require a human sign-off before acting on it.

---

15. Documenting A Match For Audit Or Dispute

When a screening result needs to be defended later — to an auditor, a regulator, or a customer disputing a block — record these fields from the match, not just the headline score:

1. the submitted query text and every filter parameter used (country, date_of_birth, registration_number, entity_type, list_name, as_of_date);
2. match_source from the score breakdown;
3. whether document_match is true;
4. whether single_token_score_cap or short_token_fuzzy_suppressed is true;
5. as_of_date from the score breakdown;
6. the list name, annex, regulation and grounds text of the matched entity.

A result with document_match = false and single_token_score_cap = true is a review signal, never a confirmed identity — treat and record it as such, not as a positive identification.

---

16. What VoP Is

VoP = Verification of Payee.

Classic banking meaning:

Check whether the payee name corresponds to the IBAN/account identifier.

Regulatory context:

• Regulation (EU) 2024/886;
• EPC288-23 close-match guide (character encoding per EPC217-08);
• used by PSPs/banks before authorizing credit transfers.

Result codes:

---

17. Our VoP Wrapper

Our VoP is not a separate banking IBAN service.

It is a wrapper on top of standard sanctions screening:

screen_entity_vop()
  -> calls screen_entities()
  -> receives sanctions candidates
  -> applies vop_match() to each candidate
  -> adds vop_result / vop_scenario

It explicitly opts out of the natural-person short-token gate from §12 when it retrieves its candidate set: EPC288-23 close-match rules apply the same way to a short name as to a long one, so the sanctions-side suppression would be the wrong behavior here.

What it adds to the normal result:

---

18. VoP Normalization

VoP normalization runs after standard candidate retrieval, and it is a separate pipeline from §4 — not a reuse of it.

Order:

standard candidates -> vop_normalize_name() -> vop_match scenarios

VoP normalization steps:

Why Nordic expansion runs before unaccent. unaccent alone maps ø to a single letter o, but EPC217-08 requires the two-letter expansion ø -> oe. Running the explicit Nordic step first avoids the wrong substitution — order matters here specifically because both steps could otherwise touch the same character.

A broader suffix list than general screening. VoP's suffix table (vop_legal_form_aliases) is a data table — extendable with a single INSERT, no code deploy — and it's deliberately wider than the regex used in §4: it includes SIA, UAB, ZAO, PAO and similar Baltic/CIS forms that general screening's normalizer does not strip. Verified live: vop_normalize_name('SIA Baltija') returns baltija, while normalize_entity_name('SIA Baltija') — the general screening path — leaves it as sia baltija.

Important:

VoP normalization does not search candidates in the database.
It classifies candidates that were already retrieved.

---

19. VoP Close-Match Scenarios

Two precision notes worth knowing before relying on a CMTC result:

• s2b_transposition is a strict single adjacent swap, not general edit distance. It requires both names to be the same length and differ by exactly one pair of adjacent transposed characters. A name that differs by an insertion or deletion does not qualify here — it may still qualify under s2a_levenshtein if the edit distance is 2 or less, but the two scenarios are evaluated independently and reported separately.
• s2d_phonetic requires every token to match phonetically, not just one. Double Metaphone runs per word, and both the primary and alternate codes are compared for each token pair — a two-word name only qualifies as a phonetic match if both words carry a matching phonetic code.

For legal_person:

initial and phonetic scenarios do not apply.
Only exact, Levenshtein and transposition remain.

---

20. Classic VoP vs Our VoP

Client wording:

Classic VoP checks "name <-> account".
Our VoP checks "name <-> retrieved sanctions candidate" under VoP close-match rules.

---

21. VoP Examples

Both surfaces reach the same underlying check. Over MCP, call screen_entity_vop. Over plain REST, POST /v1/payee_verifications:

curl -X POST "https://api.compliance-mcp.com/v1/payee_verifications" \
  -H "Authorization: Bearer tk_..." \
  -H "Content-Type: application/json" \
  -d '{"query":"Jan Kowalski","date_of_birth":"1980-03-15","algorithm":"vop_epc288"}'

---

22. Organization Settings

An organization can configure its default screening preset.

Table:

organization_screening_presets

Main fields:

Example:

{
  "preset_name": "default",
  "list_names": ["EU_FSF", "OFAC_SDN", "UN_SC"],
  "algorithm_preset": "standard",
  "score_threshold": 85,
  "single_token_min_fuzzy_length": 5
}

---

23. How Preset Affects MCP Forms

The /entity form receives organization settings from the server:

If the user explicitly passes a request parameter, request override is stronger than organization default.

Example:

Organization default threshold = 85
User opens /entity without threshold -> form shows 85

User opens /entity threshold=70 -> form shows 70

---

24. Choosing Data Sources

The form has two modes:

Example selected sources:

EU_FSF
OFAC_SDN
UN_SC
UK_UKSL

If several sources are selected, the form passes them as one array to the bulk RPC:

screen_entities(list_names=["EU_FSF", "OFAC_SDN", "UN_SC"])

Source filtering and the global limit run in one SQL plan; the widget receives one bounded result.

---

25. Examples

A fuzzy person query. Screen "Ivan Ivanov" — two tokens, so the single-token gate never engages. Expect a high score, source-list evidence, and a score_breakdown that shows which of the fuzzy, exact and token-set paths actually won.

Single-token safety. Screen "Ivanov" alone as a natural person. Below the configured length floor, no fuzzy or phonetic candidate surfaces at all — only an exact-token or identifier hit would appear. Supply a date of birth that matches a specific candidate, and that candidate's fuzzy path reopens for just that one row.

An identifier match. Screen a registration number, IMO number, or BIC. Expect document_match = true, final_score = 100, and identifier_confidence of high or medium depending on the identifier type.

A VoP close match. Screen "J Kowalski" against a registered "Jan Kowalski". Expect vop_result = CMTC with vop_scenario = s2c_initial — the initial-plus-surname scenario, not a full name match.

An organization preset in effect. With a default preset of sources = EU_FSF + OFAC_SDN and threshold = 85, opening /entity shows those sources already selected and that threshold already filled in — nothing to configure before the first search.

---

26. One-Slide Summary

Our screening engine is multi-path:
normalization (including a dedicated Arabic pipeline) + trigram + token-sort +
Double Metaphone + Levenshtein + exact token + identifier matching.

Score is explainable:
max algorithm signal + DOB/country bonuses + safety caps + identifier override
+ script corroboration for cross-script phonetic matches.

Short single-token natural-person queries skip fuzzy/phonetic matching
entirely unless a supplied DOB confirms the candidate.

VoP is a wrapper with its own, broader normalization pipeline:
standard sanctions candidates + EPC288-23 close-match classification,
exempt from the short-token gate.

Organizations can configure:
default sources, algorithm preset, threshold, and the single-token length floor.