Project — noul

A local, non-generative scorer solves single-file code search and loses multi-file answers at the shortlist

Shel Burkes, PhD·2026–present·Updated

Benchmarked 24 Sep·Held-out test scored 26 Sep 2026

AbstractA coding agent answers “where is the thing that does X” by reading files into its context, turn by turn5,6,8, and every file it reads is re-sent with every later turn. TypeSafe's Jev points at an alternative — typed, calibrated numbers instead of prose, in one forward pass — but is API-only, with its architecture unpublished. We asked how much of that shape can be rebuilt from open models running locally, with no text generation and no code sent anywhere. Here we show that a two-stage pipeline — BM25 and a 33M-parameter embedding model shortlisting 20 chunks for a 568M-parameter cross-encoder — puts a right file in the top three for 33 of 35 queries across two codebases, against 28 for keyword search, at 2.2–2.3 s per query, a tenth of the brute-force reranking time; its top-1 lead of two to three queries is too small to call. Under a strict rule counting a query only when every correct file reaches the top five, all 26 single-file queries pass, but only 4 of 9 multi-file queries do, fewer than keyword search's 5; a pre-registered fix spreading the shortlist across files made no difference on a held-out codebase (16 of 17 complete either way). The shortlist, not the reranker, is where multi-file answers are lost.

Describing code in natural language and retrieving it is the semantic code search task, benchmarked by CodeSearchNet1, CoSQA2 and CoIR3, where learned code encoders such as UniXcoder4 compete with lexical baselines. Repository-level localization for SWE-bench5 is usually done by LLM agents that read files turn by turn, such as SWE-agent6 and LocAgent8, or by the simpler staged pipeline of Agentless7, which already argued that an agent is not required; LLMs can also serve as the reranker itself17. Every file an agent reads is re-sent with every later turn. TypeSafe's Jev points at an alternative: a model that returns typed, calibrated numbers instead of prose, in one forward pass. Jev is API-only and its architecture is unpublished. We asked a narrower question: how far a local, non-generative scorer gets on that task, with no text generation and no code sent anywhere, measured by a strict every-correct-file rule and a pre-registered held-out test (Box 1, Methods). None of the architecture is new — the first stage combines BM259 with a small BGE embedding model10 by reciprocal rank fusion11, and the second follows the retrieve-then-rerank design of cross-encoder rerankers13; the question is what that stack delivers locally.

At top-1 the lead over keyword search is too small to call; the replicated gain is the shortlist

We benchmarked three retrievers on 41 queries over two small codebases — a JavaScript web application and a Python-and-notebook research codebase with scientific vocabulary — 35 of them answerable (Methods). On the labels as originally drafted, the two-stage pipeline led BM25 keyword search at top-1 by two queries; rerun on the current labels, after three disclosed widenings, by three (Tables 1 and 2). Out of 35, both leads are noise at this sample size, and an earlier lead for the big reranker on the web application did not replicate on the research codebase, where a 22M-parameter model did best. The gain that did replicate is recall into a shortlist: on both codebases, the reranker puts a right file in the top three for 33 of 35 queries against 28 for BM25, and 34 against 29 on the current labels. That points to its role — a reranker over a shortlist, not a top-1 oracle. The two-stage default was chosen on these same queries, so its edge over its near neighbors is not a finding.

Table 1 | Retrieval on the labels as originally drafted. 35 answerable queries over the two development codebases; P@1 and R@3 count any correct file (Box 1). Timing ranges span the two codebases.

PipelineP@1R@3Seconds per query
BM25 alone0.63 (22/35)0.80 (28/35)< 0.01
Reranker over every chunk0.66 (23/35)0.94 (33/35)6.7–25.6
Two-stage0.69 (24/35)0.94 (33/35)2.2–2.3

Table 2 | The same benchmark on the current labels. Rerun after the three label widenings (Methods), code and corpus unchanged.

PipelineP@1R@3Seconds per query
BM25 alone0.69 (24/35)0.83 (29/35)< 0.01
Two-stage0.77 (27/35)0.97 (34/35)2.1

The two-stage pipeline keeps the gain at a fraction of the cost

Reranking every chunk costs about 0.1 s per chunk, 25.6 s a query on the larger codebase. Every shortlist tested kept recall at three unchanged, and the default brings a query to 2.3 s: a tenth of the brute-force time on the larger codebase, a third on the smaller. The reranker is the only model that sees query and code together, which is what lets it match meaning rather than words, and it is also the only expensive step; the shortlist exists to keep it off every chunk in the repository, the standard retrieve-then-rerank design13.

Multi-file answers fail the strict rule, and the pre-registered fix did not replicate

The measures above count a query as right if any correct file appears. Counting it only when every correct file is in the top five, every single-file query passes (26 of 26), but only 4 of the 9 queries with several correct files do, fewer than keyword search (5) and brute force (6) (Fig. 1a). The shortlist keeps the best 20 chunks, which tend to come from one dominant file, so secondary files never reach the reranker. A pre-registered15 fix spread the shortlist across files, in the spirit of diversity-based reranking14: on these queries it lifted multi-file completeness from 4 to 6 of 9 (Fig. 1b), but on a fresh held-out codebase, labeled before noul ever ran on it with every label backed by a cited line, it made no difference — 16 of 17 complete either way (Fig. 1c). The default stays as it was. The one held-out miss was a source file pushed out of the top five by test files that mention the same functions, which is what gets measured next. Per-query retrievals behind these counts are in Extended Data Fig. 1.

Three panels showing the share of queries with every correct file in the top five, split into single-file and multi-file answers. a: on 35 development queries, BM25 22 of 26 single and 5 of 9 multi; reranking every chunk 26 of 26 and 6 of 9; two-stage 26 of 26 and 4 of 9. b: shortlist variants on the same queries: top 20 chunks 4 of 9 multi-file, one or two chunks per file 6 of 9, top 50 chunks 6 of 9 at 5.2 seconds a query against 2.1. c: on 17 held-out queries, top 20 chunks and one chunk per file both 10 of 10 single and 6 of 7 multi.
Fig. 1 | Single-file answers are solved; multi-file answers are where the shortlist loses files, and the fix did not replicate. Share of queries whose every correct file is in the top five, by whether the answer spans one file (squares) or several (circles). a, Three retrievers on the 35 development queries (two codebases). b, Shortlist variants for the two-stage pipeline on the same queries, with seconds per query on the larger codebase. The spread-across-files variants were designed after these queries exposed the problem, so their gain here is partly tuning. c, The pre-registered test on a third codebase, scored once: no difference. Development labels are Claude-drafted and unverified; held-out labels each carry a cited line checked by script.

Raw scores cannot say “not here”

On each codebase, two or three answerable queries score below the strongest query for a feature that does not exist, and the cut-off sits at a different score in each. Raw reranker scores are not probabilities. Turning them into probabilities is calibration16, the Jev-like part still to build; for reranker scores there is no established recipe to borrow.

Discussion

On these benchmarks, a local, non-generative two-stage scorer put every single-file answer in its top five in about two seconds, and its replicated gain over keyword search was recall into the shortlist rather than precision at top-1. Its weakness is equally specific: under the every-correct-file rule, multi-file answers come back complete less often than with plain keyword search, because the shortlist concentrates on one dominant file, and the pre-registered spread-across-files fix that helped on the development queries made no difference held-out.

The claim is bounded by the benchmark. Both development codebases are small, 60 and 9 files; the shortlist of 20 chunks is 8% of one and 31% of the other, so it has not had to prove itself on a repository where it discards almost everything. 35 development queries is a small sample, their labels are Claude's reading of the code and not independently verified, and the two-stage default was chosen on them. The held-out test is the stronger evidence, and it is one codebase of 17 queries.

The next tests are set: keep test files from crowding out source files; more held-out codebases with cited labels; then a large open-source repository, the scale localization benchmarks use5,8; calibration so “not here” becomes a probability; and the per-file yes/no ask mode. A Claude Code skill already fronts the pipeline with a small-model agent, which the local scorer is meant to replace for all but ambiguous cases; that agent will be scored on the same labels, so the comparison becomes a measurement rather than an anecdote.

Methods

Pipeline. find ranks files by how well they match a plain-language description, including descriptions whose words never appear in the code, the task semantic code search benchmarks measure1,2,3. Files are cut into overlapping 40-line chunks; notebooks are read as code and markdown, without outputs. A file scores as its best chunk. Retrieval runs in two stages: BM259 and a 33M-parameter embedding model (bge-small10), fused by reciprocal rank11, shortlist 20 chunks, and a 568M-parameter cross-encoder (bge-reranker-v2-m3, built on the M3 backbone12) then reads the query against only those 20.

Benchmark. Two small codebases: a JavaScript web application (60 files) and iridis, a Python and notebook research codebase with scientific vocabulary. 41 queries in all. 35 are answerable, and 12 of those are “gap” queries whose key words do not appear in the code. The other six describe features that do not exist, to test whether a scorer can say “not here”. Each answerable query lists every file that would be a fair top answer. The measures are precision at 1 and recall at 3, as in code retrieval benchmarks1,4, and the strict complete-in-top-5 rule (Box 1).

Labels. Who wrote the labels matters. Claude drafted them against the source. Three were widened after the first run, when the scorers found defensible answers the labels had missed; one of those turned up a real duplication in iridis, two separate implementations of the same skin-color extraction. None of the 41 development labels has been independently verified: they are Claude's reading of the code. The held-out codebase was labeled before noul ever ran on it, with every label backed by a cited line checked by script.

Extended Data — every query, every retrieval

The counts above aggregate over queries; Extended Data Fig. 1 shows each of the 52 answerable benchmark queries individually, with the files it asked for and the five that came back, under each shortlist variant.

A dot per benchmark query, 52 answerable queries across three repos, showing the share of that query's correct files in the top five. Almost every dot sits at 100%; a handful of multi-file queries sit at 50% or below. Buttons switch between four shortlist variants.
Extended Data Fig. 1 | Every benchmark query, one row each: which files were asked for, and what came back. The 52 answerable queries across the three benchmark repos, under each shortlist variant (buttons). Squares are single-file answers, circles multi-file; the position is the share of that query's correct files in the top five. Hovering a dot shows the query, its gold files, and the five files actually retrieved with their scores — the fastest way to see which file a multi-file answer loses.

Data availability

Both label sets — the development labels with their disclosed widenings, and the held-out labels with their cited lines — the benchmark runners and every per-query result are committed with the project. All scoring ran locally; no code left the machine.

Code availability

The noul repository is private. The pipeline, benchmark construction, label provenance and decision rules are summarized here; the two-stage pipeline described in Methods is the committed default, with results for find only.

Pre-registration statement

Pre-registered means committed to the project's own repository before scoring, not lodged with an external registry15. The held-out test of the spread-across-files shortlist was pre-registered: the third codebase was labeled before noul ever ran on it, every label backed by a cited line checked by script, and the test was scored once. The development results were not held out — the pipeline and its default were chosen on those same queries, and label widenings after the first run are disclosed above.

Competing interests

None.

References

  1. Husain H, Wu H-H, Gazit T, et al. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. arXiv 1909.09436 (2019). arXiv:1909.09436.
  2. Huang J, Tang D, Shou L, et al. CoSQA: 20,000+ Web Queries for Code Search and Question Answering. Proceedings of ACL-IJCNLP, 5690–5700 (2021). doi:10.18653/v1/2021.acl-long.442.
  3. Li X, Dong K, Lee YQ, et al. CoIR: A Comprehensive Benchmark for Code Information Retrieval Models. Proceedings of ACL, 22074–22091 (2025). doi:10.18653/v1/2025.acl-long.1072.
  4. Guo D, Lu S, Duan N, et al. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. Proceedings of ACL, 7212–7225 (2022). doi:10.18653/v1/2022.acl-long.499.
  5. Jimenez CE, Yang J, Wettig A, et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR (2024). arXiv:2310.06770.
  6. Yang J, Jimenez CE, Wettig A, et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS (2024). arXiv:2405.15793.
  7. Xia CS, Deng Y, Dunn S, Zhang L. Demystifying LLM-Based Software Engineering Agents (Agentless). Proceedings of the ACM on Software Engineering 2 (FSE), 801–824 (2025). doi:10.1145/3715754.
  8. Chen Z, Tang X, Deng G, et al. LocAgent: Graph-Guided LLM Agents for Code Localization. Proceedings of ACL, 8697–8727 (2025). doi:10.18653/v1/2025.acl-long.426.
  9. Robertson S, Zaragoza H. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval 3, 333–389 (2009). doi:10.1561/1500000019.
  10. Xiao S, Liu Z, Zhang P, et al. C-Pack: Packed Resources For General Chinese Embeddings. Proceedings of SIGIR, 641–649 (2024). doi:10.1145/3626772.3657878.
  11. Cormack GV, Clarke CLA, Buettcher S. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of SIGIR, 758–759 (2009). doi:10.1145/1571941.1572114.
  12. Chen J, Xiao S, Zhang P, et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. Findings of ACL, 2318–2335 (2024). doi:10.18653/v1/2024.findings-acl.137.
  13. Nogueira R, Cho K. Passage Re-ranking with BERT. arXiv 1901.04085 (2019). arXiv:1901.04085.
  14. Carbonell J, Goldstein J. The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. Proceedings of SIGIR, 335–336 (1998). doi:10.1145/290941.291025.
  15. Nosek BA, Ebersole CR, DeHaven AC, et al. The preregistration revolution. Proceedings of the National Academy of Sciences 115, 2600–2606 (2018). doi:10.1073/pnas.1708274114.
  16. Guo C, Pleiss G, Sun Y, Weinberger KQ. On Calibration of Modern Neural Networks. ICML (2017). arXiv:1706.04599.
  17. Sun W, Yan L, Ma X, et al. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. Proceedings of EMNLP, 14918–14937 (2023). doi:10.18653/v1/2023.emnlp-main.923.