Building a scraper for

Published 2018-06-20 on Farid Zakaria's Blog

The start of a new project

A friend has recently asked if I can look into building a tool / site to scrape -- with a end-goal of building a system to automatically reserve for a desired permit.

This piqued my interest and lets take a look at what I can do! At a high level I imagined building:
register desired site -> continously scrape -> reserve -> notify via text

Looks like a good chance to put together some interesting technologies: web-framework (django?) & twilio to send notifications


Before beginning any project, I take a look at the current space and see if there are any current open source alternatives or even a paid platform to leverage.

I found the following:

I could not find a paid service and the OSS options seemed very difficult for non technical people to use.

Can I haz API?

Browsing online -- I was ecstatic when I came across which is a REST API for the website -- unfortunately it doesn't let you perform reservations and I couldn't decipher yet how to link them to the reservation portion. Perhaps it might be leveraged in the future!

Time to use our favorite reverse engineering tools: wireshark & Charles -- I ended up using Charles specifically because I find it easier to setup as a man-in-the-middle HTTPS proxy.

You can follow the simple guide on how to setup Charles as a HTTPS proxy here

Here is the raw request from Charles when searching locations matching whitney at
(unimportant parts stripped out)

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-www-form-urlencoded
Content-Length: 275
Connection: keep-alive
Upgrade-Insecure-Requests: 1


The important part is that it is x-www-form-urlencoded with locationCriteria=whitney.

The response is HTML however we can use various tools to strip out the desired list.