Back to Feed
Agents / Benchmarks & Evals

Automated Compliance Checking Using LLMs

Original: CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs
By Muhammad Roman, Karen Rafferty, Barry Devereux Published 2026-08-03

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • CTRAG achieved an F1-score of 78% and a recall of 85% in its final configuration.
  • The framework processes PDF documents through chunking and vector storage to enable context-aware regulatory questioning.
  • It utilizes a hybrid retrieval system to feed relevant documentation into LLMs for structured compliance assessment.
  • The system remains limited by challenges in evidence attribution, model hallucination, and high preprocessing costs associated with meta-chunking.

Summary & Methodology Analysis

The CTRAG framework functions by ingesting regulatory and company-provided PDF documents. The system employs chunking strategies, including a compute-intensive technique called meta-chunking, to break text into segments. These segments are converted into embeddings (numerical vector representations of data used to capture semantic meaning) and stored in a vector store for fast retrieval. When a regulatory control query is submitted, a hybrid retrieval system fetches the top-k document chunks relevant to the query to inform the subsequent generation process.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary goal of CTRAG?

It aims to automate manual compliance checking, which is typically time-intensive and inconsistent, by using LLMs to evaluate regulatory requirements.

Q2. Can this system handle complex regulatory scenarios?

Yes, it is designed to address complex and ambiguous requirements, although it initially faced challenges with indirect compliance scenarios involving third-party services.

Q3. What is the output of the system?

The system outputs a binary compliance status of either Pass or Fail along with the reasoning behind the decision.

Q4. Which models does the framework support?

The framework utilizes several models, including GPT-4o, GPT-4, GPT-3.5 Turbo, Gemini-Pro, and Gemini Flash.

Q5. How does the retrieval system operate?

It uses a hybrid retrieval system to fetch relevant chunks and incorporates in-context learning (a technique where a model learns from provided examples in its input prompt) to guide the LLM's final assessment.

Q6. What are the computational drawbacks of this approach?

Meta-chunking significantly increases preprocessing times due to its high computational complexity.

Q7. What are the known limitations of the framework?

The system struggles with evidence attribution, model hallucinations (where an LLM generates false or nonsensical information), and the need for improved explainability.

Q8. What libraries and tools are integrated into the pipeline?

The stack includes LangChain, LangChain RetrievalQA, RecursiveCharacterTextSplitter, and FAISS for vector storage.

Q9. Does the paper disclose the dataset scale?

The paper notes that the dataset used for testing the framework is small in scale.