Structuring Agentic Research with Evidence Graphs
Listen to the summary
Uses a voice available on your device
Audio options
On this page 5 sections
Related concepts 3 concepts
Key Takeaways
- Crase significantly outperforms proprietary agents, achieving a recall at 50 of 0.3659 compared to 0.1220 on the ICLR dataset.
- The system uses human expert judgment for validation, with edge pruning decisions agreeing with experts in 84.0% of cases.
- The architecture operates on a controlled corpus of 500K arXiv papers, ensuring retrieval is constrained to relevant citation neighborhoods.
- The method relies on a claim grounding formulation supported by specific entailment datasets to verify the relevance of research edges.
Summary & Methodology Analysis
Crase improves research retrieval by replacing open-ended exploration with a structured evidence graph approach. In the initial phase, a Qwen2.5-32B-Instruct model performs query decomposition to generate focused sub-queries. Each sub-query triggers a single Semantic Scholar API call to retrieve a set of seed papers. These papers form the basis of a citation neighborhood, which acts as a bounded exploration space for the agent.
Interactive System Flowchart
Illustrative Implementation
A short sketch of the paper's core idea, not the authors' own code.
# Illustrative sketch (not from the paper)
import torch
from collections import defaultdict
def decompose_and_fetch(q):
subqs=["sub1","sub2"]; seeds=[]
for _ in subqs: seeds+=["paperA","paperB"]
return seeds
def citation_neighbors(p): return ["c1","c2"]
def entailment(c,a): return torch.rand(1).item()
def build_graph(seeds):
g=defaultdict(dict)
for s in seeds:
for n in citation_neighbors(s):
sc=entailment("claim","abs")
if sc>0.5: g[s][n]=sc
return g
def random_walk(g,steps=10,alpha=0.85):
scores={n:0 for n in g}; cur=next(iter(g))
for _ in range(steps):
scores[cur]+=1
if torch.rand(1).item()<alpha and g[cur]: cur=next(iter(g[cur]))
else: cur=next(iter(g))
return sorted(scores,key=scores.get,reverse=True)[:5]
def crase(q):
return random_walk(build_graph(decompose_and_fetch(q)))
print(crase("example query"))// Illustrative sketch (not from the paper)
const torch=require('torch'); // placeholder
function decomposeAndFetch(q){
const subqs=["sub1","sub2"], seeds=[];
subqs.forEach(()=>seeds.push("paperA","paperB"));
return seeds;
}
function citationNeighbors(p){return["c1","c2"];}
function entailment(c,a){return Math.random();}
function buildGraph(seeds){
const g={};
seeds.forEach(s=>citationNeighbors(s).forEach(n=>{const sc=entailment("claim","abs");if(sc>0.5){if(!g[s])g[s]={};g[s][n]=sc;}}));
return g;
}
function randomWalk(g,steps=10,alpha=0.85){
const scores={};Object.keys(g).forEach(n=>scores[n]=0);
let cur=Object.keys(g)[0];
for(let i=0;i<steps;i++){
scores[cur]++;
if(Math.random()<alpha && Object.keys(g[cur]).length){cur=Object.keys(g[cur])[0];}else{cur=Object.keys(g)[0];}
}
return Object.entries(scores).sort((a,b)=>b[1]-a[1]).slice(0,5).map(e=>e[0]);
}
function crase(q){return randomWalk(buildGraph(decomposeAndFetch(q)));}
console.log(crase("example query"));
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem Crase solves?
It addresses the lack of control and transparency in open-ended agentic search systems by implementing a structurally bounded research process.
Q2. How does Crase retrieve information?
It decomposes queries into sub-queries, retrieves seed papers via Semantic Scholar, and builds a citation graph to explore relevant evidence.
Q3. Is the system more accurate than existing models?
Yes, it achieves a recall at 50 of 0.3659, which is substantially higher than the 0.1220 achieved by proprietary-agent baselines.
Q4. What happens if a relevant paper is outside the initial citation neighborhood?
The paper states that relevant papers outside the citation neighborhood induced by the initial seeds cannot be recovered.
Q5. What is the role of MsciNLI and SciNLI?
These datasets provide premise-hypothesis pairs with entailment labels used to adapt the claim-grounding formulation of the system.
Q6. How large is the corpus used for testing?
The system was evaluated on a controlled corpus of roughly 500K arXiv papers.
Q7. How was the quality of the citation graph pruning evaluated?
The authors used expert human evaluation and found that Crase agreed with majority human judgment in 84.0% of cases for 50 citation edges.
Q8. Does the system guarantee perfect accuracy in evidence grounding?
No, the claim extractor and entailment model are learned components and can make mistakes.
Q9. What specific model is used for query decomposition?
The system uses Qwen2.5-32B-Instruct to decompose each query into sub-queries.