Back to Feed
Benchmarks & Evals / Agents

Automated Schema Guided Extraction for HTA

Original: Schema-Guided Hierarchical Information Extraction and Semantic Evaluation Using Generative AI
By Modhurita Mitra, Jan-Willem Versteeg, Maarten D. Schermer, Shiva Nadi Najafabadi, Marie L. De Bruin, Lourens T. Bloem Published 2026-08-06

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • Developed a JSON Schema based framework to encode hierarchical data structures for automated attribute extraction.
  • Utilized a custom path based semantic matching algorithm to automate the validation of nested attributes against a gold standard.
  • Achieved an F1 score of 90 percent for 12 out of 14 attributes when using Claude Opus 3 on NICE documents.
  • The evaluation process incorporates an automated rubric to classify results into categories like exact, semantic, useful, or non-match.

Summary & Methodology Analysis

The methodology centers on a schema-driven pipeline designed to solve the complexity of unstructured HTA documents. The process begins by defining domain knowledge through a unified JSON Schema, which encodes hierarchical relationships and attribute cardinality. This schema is passed alongside the raw PDF text in a single zero-shot prompt, which is an inference technique where the model performs tasks without prior task-specific examples, to the generative AI model to extract required attributes.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the core problem this paper addresses?

It addresses the difficulty of extracting unstructured data from complex, hierarchical, and specialized HTA documents, which currently requires tedious manual effort by domain experts.

Q2. How does the proposed framework extract data?

It uses a single zero-shot prompt that includes the predefined JSON schema and the PDF text to guide the generative AI model.

Q3. What performance metrics were achieved?

The framework reached an F1 score of 90 percent for 12 out of 14 attributes using Claude Opus 3.

Q4. How are the extracted results validated?

The paper uses a custom path-based semantic matching algorithm to compare extracted nested attributes against a gold standard dataset.

Q5. What rubric is used for classification?

Results are classified as exact, semantic, useful, or non-match, with generative AI used to assist in evaluating non-exact lexical matches.

Q6. What are the limitations regarding the gold standard?

The evaluation requires a gold standard created by a domain expert, making it suitable only for test or development sets, and the reliance on a single expert may introduce subjectivity or errors.

Q7. Which models were utilized in this study?

The paper references Claude Opus 3, Claude Opus 4.6, Gemini 3.1 Pro, GPT OSS 120B, GPT 3.5, and GPT 4.

Q8. What are the risks associated with the model implementation?

Reliance on proprietary model APIs introduces risks concerning reproducibility, cost, and long-term functional availability.

Q9. What datasets were mentioned in the paper?

The paper worked with NICE, ZIN, and HAS HTA documents, as well as the Oxford Dictionary of National Biography.