AI engineering case study / v0.1 release
DevSignal AI
DevSignal analyzes historical GitHub issues to help engineering teams find similar problems, identify potential duplicates, recommend labels, and understand how previous issues were fixed.
Measured v0.1 Signals
- Historical GitHub issues
- 15K+
- BM25 Recall@5
- 0.4839
- Dense Recall@5
- 0.5806
- Hybrid Recall@5
- 0.6129
- Heuristic label micro-F1
- 0.1256
- TF-IDF plus Logistic Regression label micro-F1
- 0.6129
- Tests at v0.1 release
- 71
Airflow and VS Code public issue corpus
How to read this: Higher is better. Recall@5 measures how often a known relevant issue appears in the first five results; micro-F1 summarizes precision and recall across label predictions. Bars use a 0–1 scale.
Retrieval comparison
- BM25
- 0.4839 Recall at five
- Dense
- 0.5806 Recall at five
- Hybrid
- 0.6129 Recall at fiveSelected
Hybrid retrieval improved Recall@5 from 0.484 to 0.613 on the evaluation set, surfacing more known relevant issues within the first five results.
Label comparison
- Heuristic
- 0.1256 micro F1
- TF-IDF plus Logistic Regression
- 0.6129 micro F1Selected
The learned classifier improved micro-F1 from 0.126 to 0.613, substantially outperforming the heuristic baseline on the evaluation set.
Problem -> Attempt -> Evidence
Same problem. Different language.
Keyword retrieval gave an explainable baseline, but related issues often described the same failure with different wording. Dense retrieval helped recover those cases; hybrid retrieval kept exact technical identifiers in the final path.
Architecture
A local evaluation-first system.
The architecture keeps ingestion, normalization, retrieval, service behavior and evaluation visible enough to test. Duplicate classification remains assistive because explicit ground truth is sparse.
- GitHub history
- Raw ingestion
- Normalization
- Historical corpus
- Lexical + Semantic retrieval
- Hybrid retrieval
- Intelligence services
- Evaluation
- FastAPI / CLI
01 / Context
What problem are engineering teams facing?
Engineering teams often answer new issues by remembering older, similar failures.
That memory is fragile when issue history spans thousands of tickets and multiple repositories.
02 / Product
What DevSignal actually does.
DevSignal searches historical issues for similar problems, potential duplicates, labels, related pull requests, commits, and previous-fix context.
The v0.1 product is intentionally local and evaluation-first.
03 / Data
Real GitHub issue corpus.
The current corpus includes 15K+ historical public GitHub issues across Airflow and VS Code.
Data is normalized into local JSONL artifacts so retrieval and evaluation are reproducible.
04 / First Approach
Lexical issue retrieval.
BM25 gives a simple, explainable baseline for issue search.
It remains useful because issue text often contains precise error messages, file names, and component language.
05 / What Didn't Work
Why keyword-only retrieval misses differently worded issues.
Similar issues can describe the same failure with different vocabulary.
Keyword-only retrieval becomes brittle when the signal is conceptual rather than lexical.
06 / Semantic Retrieval
Why dense retrieval was introduced.
Dense retrieval was added to catch semantic overlap that BM25 can miss.
It improved Recall@5 from 0.4839 to 0.5806 in the measured setup.
07 / Hybrid Search
Why BM25 plus dense performed better together.
The hybrid run combined lexical precision with semantic coverage.
Hybrid Recall@5 reached 0.6129, the best measured retrieval baseline in v0.1.
08 / Label Prediction
Weak heuristic baseline followed by trained classical ML baseline.
A heuristic label baseline reached 0.1256 micro-F1.
TF-IDF plus Logistic Regression reached 0.6129 micro-F1 and became the stronger explainable baseline.
09 / Duplicate Experiment
What was attempted and why reliable ground truth became the limitation.
Duplicate-pair extraction used explicit GitHub timeline events where available.
Duplicate classification remains data-limited because reliable labeled pairs are sparse.
10 / Previous Fixes
Cited resolution briefs.
DevSignal can surface previous-fix briefs that cite related issues, pull requests, and commits.
The current milestone does not claim deterministic resolution-brief evaluation metrics.
11 / Architecture
Actual v0.1 system.
The system uses local ingestion, normalized datasets, retrieval baselines, evaluation scripts, a CLI, and thin FastAPI routes over processed data.
It intentionally avoids production databases and distributed infrastructure for this milestone.
12 / Evaluation
Temporal split, leakage prevention, real metrics.
Evaluation uses temporal splitting to reduce future leakage.
Metrics are reported as measured baselines rather than business impact claims.
13 / Trade-Offs
What would change if this became a larger production product.
A larger product would need stronger data governance, drift monitoring, richer duplicate ground truth, and production retrieval infrastructure.
Those choices are deliberately outside the current local v0.1 scope.