The start of a new project
A friend has recently asked if I can look into building a tool / site to scrape https://recreation.gov -- with a end-goal of building a system to automatically reserve for a desired permit.
This piqued my interest and lets take a look at what I can do! At a high level I imagined building:
register desired site -> continously scrape -> reserve -> notify via text
Looks like a good chance to put together some interesting technologies: web-framework (django?) & twilio to send notifications
Before beginning any project, I take a look at the current space and see if there are any current open source alternatives or even a paid platform to leverage.
I found the following:
I could not find a paid service and the OSS options seemed very difficult for non technical people to use.
Can I haz API?
Browsing online -- I was ecstatic when I came across ridb.recreation.gov which is a REST API for the recreation.gov website -- unfortunately it doesn't let you perform reservations and I couldn't decipher yet how to link them to the reservation portion. Perhaps it might be leveraged in the future!
You can follow the simple guide on how to setup Charles as a HTTPS proxy here
Here is the raw request from Charles when searching locations matching whitney at https://www.recreation.gov/unifSearch.do
(unimportant parts stripped out)
POST /unifSearch.do HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Firefox/60.0
Accept-Encoding: gzip, deflate, br
The important part is that it is x-www-form-urlencoded with locationCriteria=whitney.
The response is HTML however we can use various tools to strip out the desired list.