GRID Framework Extracts Security Threat Graphs from CTI Text at 68% F1
Training a 4B-parameter model to turn raw cyber threat intelligence reports into structured knowledge graphs — without an LLM judge in the reward loop — now beats the judge-based approach on recall and costs less to run.
Explanation
Security teams drown in unstructured threat reports. Knowledge graphs (think: a machine-readable map of "malware X exploits vulnerability Y via technique Z") would let AI agents reason over that data — but building those graphs automatically has been a mess. Large language models hallucinate domain-specific entities, and training them end-to-end on graph outputs is expensive and unstable because you need another LLM to score every output.
GRID sidesteps both problems. First, it generates its own training supervision by extracting graphs from CTI (Cyber Threat Intelligence) articles and then revising the source text to align tightly with those graphs — creating traceable article-graph pairs without human annotation. Second, instead of asking an LLM judge to score full graph outputs during training, it converts the learning task into a bank of multiple-choice questions and regex-matchable triple targets. Cheap, deterministic rewards, reusable across training runs.
Two models were trained on this pipeline, both based on Qwen3-4B-Instruct: a Task-bank Reward model and an End2End Reward model. Tested across 249 CTI articles from five public datasets (GRID, CASIE, CTINexus, MalKG, SecureNLP), the Task-bank model hit 84.62% precision, 64.91% recall, and 68.53% F1 — the best recall in the benchmark and near-top F1, at lower token cost than the judge-based alternative.
The practical upshot: a 4B model with structured rewards outperforms the more expensive LLM-as-judge setup on the metric that matters most for threat intelligence (recall — missing an attack technique is worse than a false alarm). The task bank is built once and reused, which matters for teams that need to retrain as the threat landscape shifts.
What to watch: whether this pipeline generalizes beyond English-language CTI and how it holds up against proprietary threat intel formats that don't resemble public benchmark articles.
The core contribution is a two-stage supervision pipeline that decouples graph-quality signal from LLM-judge latency and variance. Stage one uses an extraction pass to produce candidate knowledge graphs from CTI articles, then applies KG-conditioned text revision to create article-graph alignments — effectively a self-supervised grounding step that anchors entity and relation labels to source spans. Stage two reframes document-to-graph learning as a scripted task bank: four-option multi-select questions probe entity/relation classification, while triple-level regex targets provide token-exact matching rewards. Both reward types are deterministic and offline-computable, eliminating the per-step LLM judge call that makes End2End RL expensive and reward-noisy.
Both extractors are fine-tuned from Qwen3-4B-Instruct-2507 via RL. The Task-bank Reward model achieves 84.62% source-averaged precision, 64.91% recall, and 68.53% Avg F1 across 249 held-out CTI articles spanning five datasets. The End2End Reward model — which does use LLM-as-judge precision/recall signals — scores 76.91% / 53.85% / 58.06%, a meaningful gap that validates the task-bank design. Ablations confirm that Choice-only Reward (questions without triple matching) and End2End SFT without RL both underperform, isolating the RL + structured reward combination as the key driver.
The ontology-guided extraction pipeline paired with the Task-bank model is the recommended deployment path: best recall (critical in threat intel, where false negatives carry operational risk), lower token usage, and a reusable reward bank that survives across post-training iterations as new CTI data arrives.
Open questions worth tracking: (1) The benchmark is 249 articles — respectable for this niche, but small enough that dataset-specific ontology drift could inflate cross-dataset numbers. (2) All five source datasets are English; CTI is increasingly multilingual. (3) The paper doesn't report inference latency or graph size distributions, which matter for real-time SOC (Security Operations Center) integration. (4) The KG-conditioned text revision step is the least-specified component — its quality directly gates supervision quality, and failure modes there aren't characterized. A falsifier: if the task-bank rewards overfit to the ontology used during supervision, performance on novel threat actor TTPs (Tactics, Techniques, and Procedures) not represented in training graphs should degrade sharply.
Reality meter
Why this score?
Trust Layer A 4B-parameter model trained with deterministic, offline task-bank rewards can construct security knowledge graphs from CTI text more accurately and cheaply than LLM-as-judge end-to-end reward training.
A 4B-parameter model trained with deterministic, offline task-bank rewards can construct security knowledge graphs from CTI text more accurately and cheaply than LLM-as-judge end-to-end reward training.
- Task-bank Reward model achieves 84.62% source-averaged precision, 64.91% recall, and 68.53% Avg F1 across 249 CTI articles from five datasets.
- End2End Reward model (LLM-as-judge) scores 76.91% precision, 53.85% recall, and 58.06% Avg F1 — consistently below the task-bank variant.
- Task-bank rewards are built once offline and reused across post-training runs, outperforming online LLM-as-judge reward and weaker ablations (Choice-only Reward, End2End SFT without RL).
- Both models are based on Qwen3-4B-Instruct-2507; the pipeline uses KG-conditioned text revision to create traceable article-graph alignments for supervision.
- Evaluation spans five public CTI datasets: GRID, CASIE, CTINexus, MalKG, and SecureNLP.
- 249 articles is a small benchmark; cross-dataset generalization claims rest on a narrow empirical base.
- The KG-conditioned text revision step — which gates all downstream supervision quality — is not fully specified or ablated in the excerpt.
- No inference latency, deployment cost figures, or graph size distributions are reported, making real-world SOC integration claims hard to verify.
Results are concrete, multi-dataset, and include ablations isolating the reward design — the core claim is well-supported within the scope of the experiment.
The paper is measured; it reports F1 gaps honestly and does not claim production readiness, though 'end-to-end' framing slightly overstates automation given the unsupervised revision step.
Better recall on CTI extraction at lower cost is operationally meaningful for security teams, but the 249-article benchmark and English-only scope limit how broadly the impact claim can be extended today.
- 1 source on file
- Avg trust 90/100
- Trust 90/100
Time horizon
Community read
Glossary
- Knowledge Graph (KG)
- A structured representation of information organized as entities (nodes) and their relationships (edges), used here to extract and organize threat intelligence from articles.
- CTI (Cyber Threat Intelligence)
- Information about cybersecurity threats, attacks, and threat actors, typically documented in articles and reports that security teams analyze.
- Reinforcement Learning (RL)
- A machine learning approach where a model learns by receiving rewards or penalties for its actions, optimizing behavior over time without explicit labeled examples.
- LLM-as-judge
- Using a large language model to evaluate the quality or correctness of outputs, rather than using fixed rules or metrics.
- TTPs (Tactics, Techniques, and Procedures)
- The methods and patterns used by threat actors to conduct cyberattacks, ranging from high-level strategies (tactics) to specific technical actions (techniques and procedures).
- SOC (Security Operations Center)
- A centralized team and facility that monitors, detects, and responds to cybersecurity incidents in an organization.
- Ontology
- A formal framework that defines the types of entities, relationships, and concepts relevant to a specific domain, used here to structure threat intelligence extraction.
What's your read?
Your read shapes future topic weighting.
Your vote feeds topic weights, community direction and future prioritisation. Open community direction
Sources
Optional Submit a prediction Optional: add your prediction on the core question if you like.
Prediction
Will GRID's task-bank reward approach be adopted or replicated in at least one published security knowledge graph system within 12 months?