rip26-scraper

Polls a Kleinanzeigen search for Rock im Park ticket offers, classifies each one with an LLM, alerts you about full weekend passes via ntfy, and serves a small read-only web viewer with a price-history chart.

It is built to be a quiet, polite scraper: it polls infrequently with jitter, backs off on errors, looks like a real returning browser, and can rotate its exit IP through a Mullvad VPN when it gets rate-limited.

---

What it does

• Scrapes the search page on an interval (default ~60s, ±33% jitter) and stores new offers in SQLite.
• Classifies each offer with an OpenAI-compatible LLM into a category (weekend, day, addon, merch, wanted, unknown), plus ticket_count and price_per_ticket. For real ticket offers it fetches the full detail-page description so per-ticket pricing isn't fooled by truncated listing snippets.
• Notifies (ntfy + macOS desktop) only for weekend / 3-day passes — the ones worth acting on. Edited ads (title/price changes) are synced and re-alert.
• Serves a web UI: category badges, €/ticket, a price-history graph with a rolling 7-offer average, a "weekend only" toggle, and the time of the last successful scrape.

The two processes (scraper.ts, server.ts) share one SQLite database. The scraper is the only writer; the web server opens it read-only (WAL mode).

Requirements

• Node 24+ (the .ts files run directly via Node's built-in type stripping — no build step). Locally this repo uses fnm.
• Native module better-sqlite3 (compiled on install) and cheerio.
• Optional: an OpenAI-compatible API key, an ntfy topic, a Mullvad account.

Configuration

All settings are environment variables, loaded from .env. Copy the template and edit:

cp .env.example .env

Variable	Default	Purpose
INTERVAL_SECONDS	60	Base poll interval (jittered ±33%).
NTFY_TOPIC	(unset)	ntfy topic to publish to. Empty = ntfy disabled.
NTFY_SERVER	https://ntfy.sh	ntfy server.
NTFY_TOKEN	(unset)	Bearer token for protected topics.
OPENAI_API_KEY	(unset)	Key for the classification endpoint.
OPENAI_BASE_URL	https://api.openai.com/v1	OpenAI-compatible endpoint (Ollama, LM Studio, …).
OPENAI_MODEL	gpt-4o-mini	Model used for classification.
MULLVAD_ROTATE	false	Rotate the Mullvad exit IP when rate-limited.
MULLVAD_ACCOUNT	(unset)	Mullvad account number (required when rotating in Docker).
MULLVAD_LOCATIONS	de,nl,at,ch,…	Country codes cycled on each rotation.
MULLVAD_LOCKDOWN	true	Killswitch: block non-tunnel traffic (Docker).
PORT	3000	Web viewer port (host port in Docker).
DB_PATH	./offers.db	SQLite path; set to a volume path in Docker.
TZ	Europe/Berlin	Timezone for displayed timestamps.

LLM classification is enabled when OPENAI_API_KEY (or a custom OPENAI_BASE_URL) is set; otherwise offers are stored without categories.

Running locally

npm install
npm start      # the scraper (writer)
npm run serve  # the web viewer at http://localhost:3000

The first run seeds the database silently; you only get alerts for offers seen afterwards.

Running with Docker (recommended)

docker-compose.yml defines two services that share a db volume:

• scraper — runs the Mullvad daemon inside the container for exit-IP rotation, so it needs NET_ADMIN and /dev/net/tun (already declared).
• web — the read-only viewer, published via caddy-docker-proxy on the external caddy network (auto-TLS, no host port).

cp .env.example .env   # fill in MULLVAD_ACCOUNT, NTFY_*, OPENAI_API_KEY, …
docker compose up -d --build
docker compose logs -f scraper

Notes:

• The image bundles the amd64 Mullvad package. On arm64, adjust the download in the Dockerfile.
• The Mullvad device is persisted in a named volume (mullvad_state) so restarts reuse one device instead of registering a new one (accounts cap at 5 devices). Manage devices with docker compose exec scraper mullvad account list-devices -a <account>.
• On a cold start the web container may restart a few times until the scraper creates the database; it self-heals within seconds.
• The web entrypoint expects the caddy network to already exist (provided by your caddy-docker-proxy stack). Adjust the web service if you don't use it.

Exposing the web UI

The web service carries these labels:

labels:
  caddy: scraper.elbnerds.de
  caddy.reverse_proxy: "{{upstreams 3000}}"

Point them at your own hostname. Without caddy, publish a port instead (ports: ["3000:3000"]).

Data model

A single offers table keyed by adid, with scraped fields (title, price, location, url, description), LLM fields (category, ticket_count, price_per_ticket, normalized_at), and first_seen. A meta key/value table holds the last successful scrape time.

Notes

• .env is gitignored — keep your real secrets out of version control.
• Be respectful of the source site: keep INTERVAL_SECONDS at 60+ and don't run many instances against the same IP.