Seattle Events Collector

September 28, 2026 • Built with: Python, FastAPI, SQLite, Playwright

Python Web Scraping FastAPI SQLite Async

A pipeline that pulls events from 50+ Seattle venues, ticketing APIs and calendar feeds, merges the duplicates, and serves one digest as a web UI, RSS and iCal feeds, and personalized email. 63 commits from May to September 2026, over 4,700 lines of Python outside the scrapers.

Architecture

  • Scraping layer. Static HTML (BeautifulSoup, 35+ sites), JSON APIs (Ticketmaster, Bandsintown, ESPN, Luma, Eventbrite), JavaScript-heavy sites via Playwright, and iCal feeds. Every scraper returns the same Event model. The three phases run concurrently, results are gathered with return_exceptions=True, and each run per source is logged to a scrape_runs table (12,000+ recorded runs).
  • Dedupe and merge. Three passes: exact ID (sha256(source:source_id:date)), a cross-source fuzzy key, and a title-prefix pass within each date and venue. Merges follow a source-priority ranking, fill empty fields from the lower-ranked record, union tags, and keep the earliest first_seen_at.
  • Storage. SQLite in WAL mode with aiosqlite, an in-memory TTL cache for hot lookups, incremental ALTER TABLE migrations.
  • Delivery. FastAPI and Jinja2, with caching headers (5 minutes for the digest, 1 hour for feeds), gzip, keyboard navigation, and email digests via Resend where subscribers pick categories and frequency.

Interesting problems

Cross-source identity. The fuzzy key combines the Pacific-time date (to absorb timezone disagreements), a normalized venue (Unicode NFD, generic words like “theatre” and “hall” stripped, leading “the” dropped), and a normalized title (&/w/ expanded, “X Presents:” prefixes and [SOLD OUT!] tags removed, trailing years dropped, artist lists order-normalized).

Performance by failure handling. A full collection went from 226 s to 43.7 s by failing fast on a source behind a Cloudflare challenge, decoupling independent sources from it, and fixing broken URLs and date parsing.

Security review of my own code. An admin endpoint that sends the digest skipped authentication when credentials weren’t configured. I changed it to fail closed: no configured credentials means 401. The retry logic got the same treatment, failing immediately on 403, 404 and 410 and using exponential backoff for everything else.

Knowing when to stop. Three venues sit behind bot challenges (Cloudflare Turnstile, SiteGround Robot Challenge). I documented why in the code and chose not to chase them.

Tests

Unit tests cover the deduplicator, models, and scraper utilities. Built with AI coding assistants in the loop, with architecture decisions and review by me.