The problem
I kept missing price drops on items I wanted because checking manually was tedious, and most existing trackers were bloated browser extensions I didn't trust with my data. I wanted a small self-hosted service that just watched a URL and pinged me.
Role and impact
- Individual project, backend-focused.
- Tracks 40+ product URLs across 5 different retailers.
- Detected a 35% price drop within 10 minutes of it happening.
- Queue processes ~200 scrape jobs/hour with automatic retry and backoff, zero manual restarts in 3 months of uptime.
Technical decisions and challenges
Each retailer renders prices differently, and some require JavaScript execution. I compared plain HTTP requests with cheerio versus a headless browser for every job.
I ended up using Playwright only for sites that require it, falling back to a lightweight HTTP fetch otherwise, because running a full browser for every job would have made the worker queue far too slow and resource-hungry.
A flaky scrape could report a false price change and spam the user with alerts. I evaluated notifying on any change versus requiring confirmation across two consecutive scrapes.
I chose requiring two consecutive matching reads before firing a notification, since it eliminated nearly all false positives caused by transient page-render issues.
Detailed stack
| Layer | Technology | Why |
|---|---|---|
| Scraping | Playwright, cheerio | Handles both JS-heavy and static product pages |
| Queue | BullMQ (Redis) | Reliable retry/backoff for scraping jobs at scale |
| Notify | Resend | Simple transactional email for price-drop alerts |
Gallery
What I would do differently
I would add proxy rotation from the start instead of retrofitting it later. A few retailers started rate-limiting the scraper by IP once tracked URLs passed a certain volume, which single-IP scraping can't work around.