In the financial industry, businesses must ensure that every interaction with customers complies with strict regulations. Traditionally, this involves people manually reviewing conversations, which is slow and can lead to mistakes. In addition, as regulations change, this can be hard to maintain.
Rule-based automations can help automate this, but most customer interaction data is unstructured, such as call transcripts. Furthermore, data formats like PDFs or images can be difficult to extract information from. This makes it difficult to build simple rules over the data we have for automation.
In this article, we’ll use Lantern and Ecliptor to build an application that efficiently searches for relevant compliance context for customer interactions, and uses the context and LLMs to automate compliance checks.
We’ll use Ecliptor to parse and process unstructured documents into structured formats. Ecliptor helps financial institutions process messy data so that they can use the data to build applications. We’ll store this data in Lantern Cloud — Lantern enables vector search and text search in Postgres.
Step 1: Ingest compliance policy documents
Financial services organizations have compliance policies across a multitude of documents to adhere to. To make use of these documents, we’ll transform the PDFs to Markdown using Ecliptor's Document Ingest API. This endpoint preserves table formatting and document structure.
Dataset
You can download sample documents detailing compliance acts and regulations from here: Banking Compliance Regulations and Acts.
Make a call to Ecliptor's ingest endpoint
In your application, make a request to Ecliptor's API a the link to the PDF:
import requests
# Sample PDF URL
pdf_url = 'https://www.congress.gov/116/plaws/publ283/PLAW-116publ283.pdf'
# Ecliptor's PDF ingest endpoint
api_url = "https://api.ecliptor.com/ingest/pdf"
# Request payload
payload = {
"url": pdf_url,
}
# Make the POST request
response = requests.post(api_url, json=payload)
# Check if the request was successful
if response.status_code == 200:
# Process the response
result = response.json()
markdown_url = result.markdown_urlThe resulting markdown file contains the information in the PDF — we can now process this text into chunks for vector search.
In this article, we’ll use the "EQUAL CREDIT OPPORTUNITY ACT", accessible here. Download a sample of the generated markdown from Github.
Step 2: Create chunks for analysis
Simply converting the documents into text isn’t enough for effective searching and comparison. These documents are often lengthy and cover multiple topics, making it difficult to extract the relevant subset of information.
To address this, we break the text into smaller, meaningful sections — also referred to as chunks. One naive way to do this is to simply split text based on character count or sentence boundaries. However, this can leave out relevant context.
Ecliptor’s Smart Chunking API generates semantically meaningful chunks by analyzing the structure of the document, and injecting additional relevant information from elsewhere in the text if necessary. This approach allows us to get the most relevant and sufficient information to answer questions.
Make a call to Ecliptor's chunking endpoint
Pass the generated markdown file to Ecliptor’s chunking endpoint to receive a list of chunks to embed.
# Ecliptor's chunking endpoint, which accepts markdown files
api_url = "https://api.ecliptor.com/chunk"
# Request payload
payload = {
"url": markdown_url
}
# Make the POST request
response = requests.post(api_url, json=payload)
chunks = []
# Check if the request was successful
if response.status_code == 200:
# Process the response
result = response.json()
chunks = response.chunksOnce the API call is completed, you will have a list of roughly uniformly sized chunks which can be embedded using any embedding model.
Step 3: Store the chunks and generate embeddings
Next, we’ll use Lantern to store the chunks and index them for fast retrieval. You can sign up for a free database at Lantern Cloud.
Connect to the database
import psycopg2
conn = psycopg2.connect("postgresql://postgres:postgres@localhost:5432/postgres")
cur = conn.cursor()
create_table_query = f"CREATE TABLE compliance_documents (id INTEGER, chunk TEXT, vector REAL[1536]);"
cur.execute(create_table_query)
conn.commit()Generate embeddings using Open AI's embeddings model
Lantern can automatically generate embeddings of our data. To do this, you can simply enable an embedding generation job.
This can be done in the Lantern Cloud dashboard, or with SQL inside your database. We use the Python client below to set the OpenAI token and add an embedding generation job
cur.execute("""
ALTER SYSTEM SET lantern_extras.enable_daemon=true;
SELECT pg_reload_conf();
""")
cur.execute("""
ALTER DATABASE postgres SET lantern_extras.openai_token='OPENAI_KEY';
""")
cur.execute("""
SELECT add_embedding_job(
'compliance_documents', -- Name of the table
'chunk', -- Source column for embeddings
'vector', -- Destination column for embeddings
'openai/text-embedding-3-small' -- Embedding model to use
);
""")
conn.commit()More information about the embedding job service can be found here.
To see what embeddings were generated on your data, you can run the SQL query below.
SELECT vector FROM compliance_documents;Create Indexes for efficient search
We now have the contexts of our compliance documents and the corresponding generated embeddings stored in the compliance_documents table. The next step is to create indexes over the data we want to search, to enable faster search over a large number of documents.
We’ll create an HNSW index over our vectors with the L2 distance function.
cursor.execute(f"CREATE INDEX ON {TABLE_NAME} USING lantern_hnsw (vector dist_l2sq_ops);")
conn.commit()We are now ready to implement our compliance check application.
Step 4: Build an application to check customer interactions for compliance risks
Finally, we’ll build an application to check customer chat logs for compliance with regulations.
We’ll follow the following steps:
- Embedding: We will generate vectors for each customer support chat message.
- Search: We will use Lantern’s vector search to find the most relevant compliance chunks for each chat message.
- LLM: We will input the chat message and the relevant compliance text into an LLM to determine compliance, flagging potential violations.
Chat interactions data set
We’ll use a synthetically generated dataset of customer support chats.
In these chats, clients are asking the customer support agents questions about the bank’s credit assessment process.
The downloadable csv can be found here: Bank Customer Support
The dataset has the columns id (int), speaker_role (string), text (string), compliant (bool). We will use entries in the text column as input queries and use the `compliance`` column as ground truth for evlauating our model.
import pandas as pd
# Download the data set
df = pd.read_csv("bank_compliance_qa.csv")
# Grab all the queries
query_list = df['text'].tolist()Create queries from each of the texts into the index
First calculate the most similar documents for all of the queries. We will use the same embedding model used to embed the entire corpus.
cursor = conn.cursor()
similar_chunks = {}
for query in query_list:
cursor.execute("""
SELECT
id,
vector
FROM
compliance_documents
ORDER BY
vector <-> openai_embedding('openai/text-embedding-3-small', %s)
LIMIT 3;
""", (query,))
results = cursor.fetchall()
similar_chunks[query] = results
cursor.close()Flag non-compliant responses
Once we have found all the similar chunks, we have the information we need to be able to judge whether each chunk was adhering to compliance principles. We will use an LLM as a judge to flag possible non-compliance and return those responses.
from openai import OpenAI
from typing import List, Dict, Any
client = OpenAI(api_key='OPENAI_KEY')
def check_compliance(query: str, chunks: List[Dict[Any, Any]]) -> bool:
# Prepare the prompt
prompt = f"""
Question: {query}
Relevant texts:
1. {chunks[0]['id']}
2. {chunks[1]['id']}
3. {chunks[2]['id']}
Based on the question and the relevant texts, determine if the response is compliant with ethical and legal standards and company policy.
Return only a boolean value: True if compliant, False if not compliant. Do not return any other descriptive text, only the one word.
"""
# Make the API call
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a compliance checker. Respond with only 'True' or 'False'."},
{"role": "user", "content": prompt}
],
)
# Extract and return the boolean result
result = response.choices[0].message.content.strip().lower() == 'true'
return result
# Check compliance for each query and its similar chunks
compliance_results = {}
for query, chunks in similar_chunks.items():
is_compliant = check_compliance(query, chunks)
compliance_results[query] = is_compliant
# Print out non-compliant queries
print("Non-compliant queries:")
for query, is_compliant in compliance_results.items():
if not is_compliant:
print(f"- {query}")Summary
In this post, we demonstrated a system for ensuring compliance in banking customer support using a custom dataset. We leveraged document understanding APIs from Ecliptor, data storage and search in Postgres with Lantern Cloud, and LLMs to automatically reason about compliance.
Interested in learning more?
Lantern is building Postgres for AI applications. Learn more about how Lantern supports vector search at scale, or sign up for a free database at Lantern Cloud.
Ecliptor is currently in private beta for financial services companies. If you have complex documents and want to extract valuable insights for downstream applications like in this post, reach out to us at [andre@ecliptor.ai](mailto:andre@ecliptor.ai) or visit ecliptor.ai.