Vault search bench
github.com/NaluKicks-808/vault-search-bench
A search test for any folder of linked markdown notes, Obsidian included. It needs no labelling, because the links you already wrote are the answer key: what you call a page inside a link is the question, and the page it points to is the right answer.
Why I built it
I asked why my second brain had so few search test cases when it holds hundreds of pages, and the answer key turned out to be sitting in the links the whole time. The bench showed my own search was far weaker than I thought, that a small plain fix closed most of the gap, and that a model reranker on top helps only when the words in the question are not the words on the page.
Start here
- Download the repository as a ZIP, or clone it.
- Point it at your notes folder and run it. It builds the test sets from your links and scores the plain rankers that come with it.
- Plug in your own search with one small function and compare it on the same questions.
- The model reranker is optional and off by default. Nothing leaves your machine unless you turn it on and add a key.
What it does not do
It measures finding a page by its name, by its description or by a sentence that cites it. It does not measure open questions with no single right page, and a result on one vault is not a promise about yours.