Safety & Alignment / Benchmarks & Evals

Securing Retrieval Augmented Generation Systems

Original: Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems

By Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson Published 2026-08-21

Listen to the summary

Uses a voice available on your device

Speed Voice

On this page 5 sections

Related concepts 7 concepts

Key Takeaways

The evaluation agent reaches 91% accuracy and 100% precision on the TruthfulQA benchmark using Llama 3.3 70B.
The system successfully blocks unsafe advice in a secure coding use case with an F1 score of 92%.
Across three different large language models, the system demonstrates consistent discriminative power with a ROC-AUC between 0.73 and 0.81.
Per-model threshold calibration is essential to maintain performance because generation style significantly influences trust scoring.

Summary & Methodology Analysis

The proposed architecture functions as defensive middleware placed between a retrieval system and an Large Language Model. The system performs two primary stages of verification. Stage one screens retrieved documents for poison signals including linguistic anomalies and semantic outliers. Stage two utilizes an Natural Language Inference (NLI) verifier based on facebook/bart-large-mnli to compare retrieved premises against generated hypotheses. This NLI verifier runs on a central processing unit to compute a factuality score, which is fused into a final Trust Index alongside consistency metrics. A non-linear dampener is applied to this score when contamination levels are high.

Interactive System Flowchart

Click diagram to expand and zoom

Illustrative Implementation

A short sketch of the paper's core idea, not the authors' own code.

# Illustrative sketch (not from the paper)
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

def retrieve(q): return [{"text":"doc1","rel":0.9},{"text":"doc2","rel":0.7}]
def poison_signals(txt): return [0.1,0.2,0.05,0.0,0.15]  # 5 signals
def agg_poison(sig, rel): return sum(s*rel for s in sig)/len(sig)

tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
nli = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")

def nli_score(prem, hyp):
    inp = tokenizer(prem, hyp, return_tensors="pt", truncation=True)
    logits = nli(**inp).logits
    return torch.softmax(logits, 1)[0, 2].item()  # entailment probability

def trust_index(q, gen):
    docs = retrieve(q)
    trusts = []
    for d in docs:
        poison = agg_poison(poison_signals(d["text"]), d["rel"])
        nli_f = nli_score(d["text"], gen)
        raw = 0.4 * nli_f + 0.4 * (1 - poison) + 0.2 * d["rel"]
        trusts.append(torch.tanh(raw * 5).item())  # non‑linear dampener
    return max(trusts)

// Illustrative sketch (not from the paper)
const { pipeline } = require('@xenova/transformers'); // hypothetical inference API

async function retrieve(q) { return [{ text: "doc1", rel: 0.9 }, { text: "doc2", rel: 0.7 }]; }
function poisonSignals(txt) { return [0.1, 0.2, 0.05, 0.0, 0.15]; } // 5 signals
function aggPoison(sig, rel) { return sig.reduce((s, v) => s + v * rel, 0) / sig.length; }

async function nliScore(premise, hypothesis, nli) {
  const out = await nli([premise, hypothesis]); // returns logits
  const entail = softmax(out.logits)[2];
  return entail; // entailment probability
}
function softmax(arr) {
  const max = Math.max(...arr);
  const exp = arr.map(v => Math.exp(v - max));
  const sum = exp.reduce((a, b) => a + b, 0);
  return exp.map(v => v / sum);
}

async function trustIndex(q, gen) {
  const docs = await retrieve(q);
  const nli = await pipeline('text-classification', 'facebook/bart-large-mnli');
  const trusts = [];
  for (const d of docs) {
    const poison = aggPoison(poisonSignals(d.text), d.rel);
    const nliF = await nliScore(d.text, gen, nli);
    const raw = 0.4 * nliF + 0.4 * (1 - poison) + 0.2 * d.rel;
    trusts.push(Math.tanh(raw * 5)); // non‑linear dampener
  }
  return Math.max(...trusts);
}

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary purpose of this research?

The research aims to prevent knowledge poisoning and misinformation in retrieval-augmented generation systems by detecting malicious context before the model generates a response.

Q2. How does the agent detect poisoned information?

It uses a two-stage pipeline that combines pre-generation screening for linguistic or structural anomalies with post-generation factual verification using NLI.

Q3. Can this system be used in production environments?

The agent demonstrates high performance in secure coding assistant use cases, showing it is capable of blocking unsafe advice with an F1 score of 92%.

Q4. What is the impact of model choice on performance?

Generation style impacts results more than model size, meaning that per-LLM threshold calibration is necessary to maintain competitive accuracy.

Q5. Does the system verify if the LLM actually adopted the misinformation?

No, the agent only measures the detection of poisoned context before generation occurs.

Q6. What are the current limitations of this detection approach?

The system struggles to detect in-place edits like entity swaps and subtle semantic weakening.

Q7. Does the system work across different datasets?

Cross-dataset generalization is limited as shown by weaker FEVER results, indicating that domain-specific calibration is required.

Q8. What hardware is required for the NLI verifier?

The NLI verifier runs on a Central Processing Unit.

Q9. Which models were used for testing the generation?

The primary model used was llama3.3:70b, with additional testing performed using qwen3.5:35b.

Flag an issue