50+ Event Sources, One Digest, and a 226s to 44s Speedup

Published on May 17, 2026 • 2 min read

Python Async Scraping Architecture

Written up in October 2026. Dates are when I actually built it.

Finding out what’s happening in Seattle means checking about fifty websites. I got tired of that and built a thing to do it for me: Seattle Events Collector. It’s over 4,700 lines of Python before you count the scrapers themselves, and most of the interesting part is figuring out that four websites listing the same show are talking about one show.

Every source returns the same object

Scrapers read HTML, JSON APIs, JavaScript-rendered pages (Playwright) and iCal feeds, and they all return a list of the same Event model. Static, API and Playwright scrapers run as concurrent phases, and results are gathered with return_exceptions=True, so one scraper blowing up can’t take the batch down. Every run, per source, gets logged to a scrape_runs table so I can see exactly who’s flaky.

Dedupe is three passes

Event IDs are sha256(source:source_id:date), which dedupes the same source across runs for free. Cross-source is where it gets fun:

  1. Exact ID match: merge.
  2. Fuzzy key: Pacific date + normalized venue + normalized title. Venue normalization strips accents (Unicode NFD), drops generic words like “theatre”, “club”, “hall”, drops a leading “the” and truncates to 20 chars. Title normalization turns & into and and w/ into with, strips “The Crocodile Presents:” prefixes and [SOLD OUT!] tags, drops a trailing year, and normalizes comma-separated artist lists so “A, B” matches “B, A”. Using the Pacific date (not the timestamp) absorbs timezone disagreements between sources.
  3. Title-prefix pass per (date, venue). One site calls it “Hot Mess”, another “Hot Mess - Playing bangers by Kesha…”. If one title’s slug is a prefix of another’s, they merge.

Merging isn’t “pick one”

Sources have a priority ranking (individual venue sites beat Ticketmaster beats secondary-market listings). The winner supplies the record, empty fields get filled in from the loser, tags are unioned, and the earliest first_seen_at survives so “just announced” stays correct.

The 226s to 43.7s story

The first full run took 226 seconds. After one round of fixes it took 43.7. Nothing clever, all failure handling: one aggregator sits behind a Cloudflare human-verification page, so I detect it and bail instead of waiting. Two sources were gated on that aggregator succeeding even though they didn’t need it, so I unhooked them. A source with a 404 URL and a date parser that choked on US-style dates got fixed. Total unique events went from 777 to 792 on the same run, because fixing broken scrapers beats scraping faster.