Kestrel: an operated AI system

Kestrel is a production AI job-market platform I designed, shipped, and operate solo: FastAPI + React, ten AI providers behind one interface, 4,600+ tests, published on PyPI under AGPL. The scoring filter is measured, not vibed: 93.6% recall / 74.6% precision on a 277-item human-judged benchmark, with the evals running in CI. AI cost per scoring run went from $6.51 to $0.84.

The problem

Job boards lie in small ways that compound. The location string says "Berlin, Germany; Munich" while the company's only office is in Paris. A "Remote" posting turns out to be US-remote three paragraphs into the description. Duplicates, dead links, agency cross-posts. Reading it all by hand eats hours a day; paying an LLM to read all of it uncritically just converts hours into dollars.

I was job-searching (still am). I couldn't out-muscle the market, so I built the platform to win it.

What got built

A pipeline: scrapers pull postings from multiple boards and ATS APIs, a geo-eligibility gate classifies each role from authoritative per-office data (never the free-text location string -- that's the part that lies), an AI scoring filter ranks fit, and survivors land in a review queue backed by SQLite with a proper application state machine. FastAPI backend, React frontend, a CLI for daily operation.

The interesting half is the scraping and multi-strategy scoring across providers, not the CV generation. Ten AI providers sit behind one interface, so a scoring run can route to whichever is cheapest or healthiest that day, and a provider outage stays an inconvenience.

The pipeline. The dashed box is the part that keeps the rest honest.

How it's operated

This is the part most side projects skip, and the part I'd want to be asked about.

• Evals run in CI. The scoring filter is regression-tested against a 277-item human-judged blind set on every change. Recall and precision have floors; a change that degrades them fails the build.
• Provider preflight and fallback. Before a paid run, the pipeline checks that providers are alive and funded. Mid-run failures distinguish "out of credit" from "rate-limited" (a 402 is not a 429, and retrying them the same way wastes either money or time) and fall through a provider chain instead of dying.
• Cost is a tracked metric. $6.51 per scoring run at the start, $0.84 now: batching, caching, and routing to cheaper providers. The playbook is public in awesome-llm-token-optimization.
• Privacy is the floor. Job-search data is personal by definition, so a privacy boundary blocks personal-data features from any provider without a zero-data-retention guarantee, each provider's retention tier is documented, and the whole thing self-hosts.

What broke, and what caught it

The original geo gate trusted the listing's location string and an isRemote boolean. Both lie. One "clean" batch let five ineligible roles through before the gate was rewritten around authoritative per-office data.

The failure I keep retelling happened later, during a port of the improved geo classifier. One rule got skipped: a bare "Remote" location has to consult the job description before defaulting to eligible (a posting that says "Remote" and then names US cities is a US-remote role). Ten junk roles got in and precision dropped from 74.6% to 65.2%. The blind-set regression test caught it; the rule went back in.

Nobody reviews my PRs here -- solo project. The benchmark is the reviewer. That's the actual argument for evals in CI, and it's why I trust this system enough to run my own job search through it.

Annotated repo tour

The repo is github.com/pleasedodisturb/kestrel. One honesty note before you click: the public repo is the OSS core. The 277-item blind-set benchmark gates my operated deployment's CI; the public repo carries the same eval architecture as a golden-set agreement harness (weighted kappa + NDCG@5) gating its nightly builds. If you have ten minutes, read these in order:

1. tools/job_scorer.py -- the scoring filter and the geo-eligibility gate. The gate is the hard-won part: it classifies from per-office data and treats the location string as untrusted input.
2. src/career_os/ai/fallback.py and factory.py -- ten providers behind one interface, plus the fallback chain.
3. src/career_os/services/scoring_eval.py -- the eval harness that gates scoring changes.
4. tools/tests/test_geo_gate.py -- unit cases for the geo classifier, including the "Remote that isn't" family of failures.
5. tests/test_provider_fallback.py -- what happens when a provider dies mid-run.
6. src/career_os/ai/privacy.py -- the privacy boundary: personal-data features raise instead of sending to a provider without a zero-data-retention guarantee.

One more honest number: the repo has 5 GitHub stars. Why a production-grade system has 5 stars while a parallel invention of the same idea has 62,000+ is its own page.