Not every site offers an API. Sometimes the data you need is public, scattered across HTML, and still worth collecting carefully.

Respect the source

Before writing a scraper, ask:

  • Is this allowed by the site's terms?
  • Can you rate-limit and cache politely?
  • Is there a simpler export, feed, or partnership path?

Good scraping is boring on purpose. It should not look like an attack.

Build for change

Layouts shift. Selectors break. The durable approach logs failures, alerts early, and keeps extraction logic small and testable.

When scraping is the right tool, treat it like production software — not a one-off script in a forgotten folder.