Why I stopped manually picking lead sources and built a bandit instead

LeadForge scrapes fifteen-plus sources: the YC dataset, Greenhouse and Lever and Ashby job boards, SEC EDGAR filings, a handful of community lists. Early on I picked which ones to run by feel — "the ATS boards were good last week, run those more." That's a bandit problem being solved by vibes, and the whole point of the machine was to remove me from loops like this.

The trap with lead sources is that you can't know which are good until you've already spent budget on the bad ones. Some source produces 10x better leads than another, but "better" only reveals itself three stages downstream — after filtering, after enrichment, after you know whether the contact was even real. Classic explore/exploit.

The reward function matters more than the algorithm

The bandit itself is unexciting — score per source, sample mostly-greedy, keep 20–35% of pulls for exploration so a cold source can redeem itself. The design decision that actually mattered is what counts as reward. Raw lead count is a vanity metric; a source that dumps 500 unqualified companies is worse than one that produces 20 real buyers, because every junk lead costs enrichment work and pollutes scoring.

# reward per scraped batch — quality only, volume irrelevant
reward = (
    n_qualified            # passed ICP scoring
  + n_contactable          # SMTP-verified email exists
  + 2 * n_decision_makers  # named founder/exec found
)

Decision-makers count double because a verified founder email is the single strongest predictor of a reply in my funnel data. Volume appears nowhere in the formula. A source can 10x its output and earn nothing if the output is noise.

What broke first

The first version rewarded per-run instead of per-lead-outcome, and it had a feedback lag bug: enrichment finishes hours after scraping, so a source got its reward before anyone knew whether its emails verified. The bandit spent two weeks confidently over-allocating to a job board whose leads looked plentiful at scrape time and evaporated at verification. The fix was deferring reward attribution until the enrichment phase closes — the source gets paid when the lead proves out, not when it arrives. Obvious in retrospect; most reward-lag bugs are.

The uncomfortable part

After four months, the allocation converged: the YC dataset and the ATS boards dominate, the community lists starve. Which is… exactly what I already believed intuitively. So was the bandit pointless? No — and this is the part I keep coming back to. My intuition was right this quarter. When a source's quality drifts — a board changes format, a dataset goes stale — the bandit notices within days and rebalances without me. Intuition doesn't re-run itself nightly. The model's job isn't to be smarter than me on day one; it's to keep being right after I've stopped paying attention.