I Built an AI Pipeline That Reads 20+ Tech Sources So I Don't Have To

I was drowning in tabs every morning. Hacker News, GitHub Trending, ArXiv, TechCrunch, AI lab blogs — all open, half-read, mostly redundant. The same announcement on 8 different sites. The same trending repo summarized in 3 newsletters.

Sound familiar?

So I built ScanBrief — an AI-powered intelligence pipeline that does my morning reading in 2 minutes.

What It Does

ScanBrief) ingests from 20+ high-signal sources on a daily schedule:

• Hacker News — Firebase API, top 30 stories with points/comments
• GitHub Trending — daily trending repos with star counts
• ArXiv — cs.AI, cs.LG, cs.SE papers
• AI Lab Blogs — OpenAI, Google AI, Anthropic
• Tech Press — TechCrunch, Ars Technica, The Verge
• Product Hunt, Reddit, Dev.to — coming in Sprint 1

Then it:

1. Deduplicates — URL hash + fuzzy title matching. Same story on HN and Reddit? You see it once.
2. Scores — Source authority x novelty bonus. Not just popularity — actual signal quality (0-100 composite).
3. Summarizes — Claude AI generates 2-sentence summaries for the top 15 items.
4. Delivers — Morning brief to Discord, email, Slack, or API.

Total read time: ~5 minutes. Covers what would take 2+ hours of manual scanning.

Why Not Just Use a Newsletter?

Newsletters are humans picking links. That's fine for casual reading. But for builders who need to stay current:

• Newsletters miss things — human bandwidth is finite. ScanBrief scans 92+ items per run.
• No deduplication — TLDR, The Rundown, and Morning Brew all cover the same story separately.
• No personalization — they can't score by what affects YOUR projects.
• Fixed schedule — ScanBrief delivers when you need it, how you need it.

The Stack

Deliberately simple:

• Python — ingesters, dedup engine, scorer, brief formatter
• Claude AI (Haiku) — summarization via CLI
• SQLite — item storage, dedup history, brief archive
• Discord webhooks — delivery (email + API coming)
• Cron — 6:30 AM daily, runs in ~2 minutes end-to-end

Total infrastructure cost: ~$2/month. All source APIs are free.

What Makes Dedup Hard (and Valuable)

Deduplication is the killer feature. Stage 1 catches exact URL matches. Stage 2 uses fuzzy title matching (85%+ similarity) to merge "Claude 5 Released" with "Anthropic Launches Claude 5." Future sprints add semantic embeddings for complete rephrases.

Getting this right is the difference between a useful brief and another noise source.

Coming Next: Project Relevance

The feature I'm most excited about: ScanBrief will scan your actual codebase and tech stack, then flag when a trending signal affects YOUR projects. "New Claude version released" automatically flags every project using the Claude API.

No newsletter can do this. It requires knowing what you're building.

Pricing

Early subscribers get Pro free for 3 months.

Try It

Head to scanbrief.dev) and sign up. You'll get your first brief tomorrow morning.

Signal compounds. Noise compounds too. Choose which one you're stacking.

Built by Prime Automation Solutions). Follow me @primex001).