A constant rat race
Honestly, I don't even use hypem.com anymore since my music taste has changed, however I am constantly trying to keep my hypem chrome extension and python download script up to date with the changes to the site. People have been overall extremely generous in their compliments which is why I keep updating the script.
Although I may not use the site anymore, I am finding their constant attempts to thwart similar extensions interesting and have enjoyed the constant rate race. I've documented on my blog already several of their numerous attempts to thwart similar projects and looks like they released their latest incarnation. I hope this post will serve those who are attempting to enjoy their music beyond the confines of the site helpful.
So, What's Changed?
Previously, the link to the mp3 file was very easy to generate and scrape from their HTML page. Every song use to contain it's unique identifiers right in the 'div' element of that song. One simply had to parse out the 'key' and 'id' and construct a URL to http://hypem.com/serve/play/id/key.mp3 and your content was served on a platter. They've now added a few more tricks to their sleeve but it's still trivial getting access to the mp3.
- Firstly, they've moved where the 'key' is stored for each song. They've embedded the 'keys' of the current page in a 'script' object on the site. The interior of the script object is simply JSON.
- Secondly, they've changed the whole request mechanism for serving songs. Before as I mentioned, the /serve/play url immediately served up the Mp3. This is no longer the case. There is now an intermediate request which returns some JSON data and then finally the URL to the Mp3 is retrieved and served!
- Thirdly, this hasn't changed but don't forget about setting the cookie! Hypemachine checks for a AUTH cookie and without it will not authorize your get request for the song data! Simply store the cookie when you retrieve the HTML and use the same one when you make further requests. Voila!
Example
import urllib2
import urllib
from time import time
data = {'ax':1, 'ts':time()}
data_encoded = urllib.urlencode(data)
complete_url = "http://hypem.com/track/1q7nf?{}".format(data_encoded)
request = urllib2.Request(complete_url)
response = urllib2.urlopen(request)
html = response.read()
html_file = open("hype.html", "w")
html_file.write(html)
html_file.close()
After the page loads, they read the JSON data into a javascript variable and remove the script tag from the HTML body. So you won't be able to inspect a current page and find the code
Relevant Code from hype_functions_min.js
var req_url = '/serve/source/' + window.playList['tracks'][track].id + '/' + window.playList['tracks'][track].key;
var source_data;
var r = $.ajax({url: req_url,data: req_data,type: 'get',async: false,cache: false,dataType: 'json',error: function() {
log('playTrack /source/ request FAILED');
return false;
}});
try {
response = jQuery.parseJSON(r.responseText);
} catch (err) {
log("FAILED to parse JSON data");
return false;
}
15 archived comments
Ben Lart · 2012-10-05
Really cool! I hope you update the JS.
Farid Zakaria · 2012-10-05
thanks for the kind words! :)
Jon · 2012-10-05
Thanks for the investigation! hopefully this lets others or myself fix this up
Farid Zakaria · 2012-10-05
Hey Jon! Thanks, I’ll be updating my scripts as well and lookout for the new site soon! :)
Andrew · 2012-10-08
you’re the best for creating this extension, thanks for not giving up on it!
Farid Zakaria · 2012-10-09
thanks Andrew. I haven’t forgotten about the extension or the script and the people who use it either! I’ve just been focused on this new site to let you download.
All the goodies coming soon
Python Download Script Updated – Small Thoughts · 2012-10-30
[…] the python script. I’ve already documented many of the changes that needed to occur in my previous blog post but now here is the full source for those who are curious! You can download the script […]
Restap · 2012-11-24
“There is now an intermediate request which returns some JSON data and then finally the URL to the Mp3 is retrieved and served!”
This is not the case in my experience. I see no JSON response, just Status: 302[Moved Temporarily] redirects.
12:16:25.468[666ms][total 666ms] Status: 302[Moved Temporarily] GET http://hypem.com/serve/play/1rbkb/9097b60b6f3d3ad9a4212a66321d2314.mp3 Load Flags[LOAD_DOCUMENT_URI LOAD_INITIAL_DOCUMENT_URI ] Content Size[-1] Mime Type[text/html] Request Headers: Host[hypem.com] User-Agent[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0] Accept[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Language[nl,en-us;q=0.7,en;q=0.3] Accept-Encoding[gzip, deflate] Proxy-Connection[keep-alive] Cookie[__utma=1717032.1548772717.1353445981.1353445981.1353755681.2; __utmz=1717032.1353445981.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __qca=P0-1026700643-1353445981753; hblid=HTLhI2PQKnTXcuaVZqqhWBG786428581; olfsk=olfsk8030978083672631; AUTH=03%3A74e587adb3047850a3f48ba2986d2c82%3A1353445973%3A1448472171%3A09-NL; __utmb=1717032.1.10.1353755681; __utmc=1717032] Response Headers: Content-Type[text/html; charset=UTF-8] Content-Encoding[gzip] Transfer-Encoding[chunked] Location[http://api.soundcloud.com/tracks/68232982/stream?consumer_key=nH8p0jYOkoVEZgJukRlG6w] Server[nginx/1.2.3] Vary[Accept-Encoding] Date[Sat, 24 Nov 2012 11:15:59 GMT] Keep-Alive[timeout=5] Connection[Keep-Alive] Proxy-Connection[Keep-Alive]
12:16:26.135[97ms][total 97ms] Status: 302[Moved Temporarily] GET http://api.soundcloud.com/tracks/68232982/stream?consumer_key=nH8p0jYOkoVEZgJukRlG6w Load Flags[LOAD_DOCUMENT_URI LOAD_REPLACE LOAD_INITIAL_DOCUMENT_URI ] Content Size[28] Mime Type[application/xml] Request Headers: Host[api.soundcloud.com] User-Agent[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0] Accept[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Language[nl,en-us;q=0.7,en;q=0.3] Accept-Encoding[gzip, deflate] Proxy-Connection[keep-alive] Response Headers: Content-Type[application/xml; charset=utf-8] Content-Length[28] Via[1.1 varnish] Access-Control-Allow-Origin[*] X-Cacheable[NO:Cache-Control=no-cache] Location[http://ec-media.soundcloud.com/4wBtAmevZJkU.128.mp3?ff61182e3c2ecefa438cd02102d0e385713f0c1faf3b0339595666f90800ed15a826ef5293b6ad9fcb33ffb24facb26dcae86e95f821856747d1c31426457d321cb2e487e1&AWSAccessKeyId=AKIAJ4IAZE5EOI7PA7VQ&Expires=1353756170&Signature=Q8ZhVh2S7ZvRNrnPeKMCY%2FhCl2E%3D] X-Varnish[420384860] X-Cache[MISS] Date[Sat, 24 Nov 2012 11:16:00 GMT] Cache-Control[no-cache] Age[0] Access-Control-Allow-Methods[GET, PUT, POST, DELETE] X-Runtime[14] Access-Control-Expose-Headers[Date] Access-Control-Allow-Headers[Accept, Authorization, Content-Type, Origin] Server[nginx] Keep-Alive[timeout=5] Connection[Keep-Alive] Proxy-Connection[Keep-Alive]
12:16:26.233[430ms][total 4681ms] Status: 200[OK] GET http://ec-media.soundcloud.com/4wBtAmevZJkU.128.mp3?ff61182e3c2ecefa438cd02102d0e385713f0c1faf3b0339595666f90800ed15a826ef5293b6ad9fcb33ffb24facb26dcae86e95f821856747d1c31426457d321cb2e487e1&AWSAccessKeyId=AKIAJ4IAZE5EOI7PA7VQ&Expires=1353756170&Signature=Q8ZhVh2S7ZvRNrnPeKMCY%2FhCl2E%3D Load Flags[LOAD_DOCUMENT_URI LOAD_REPLACE LOAD_INITIAL_DOCUMENT_URI ] Content Size[4642271] Mime Type[audio/mpeg] Request Headers: Host[ec-media.soundcloud.com] User-Agent[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0] Accept[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Language[nl,en-us;q=0.7,en;q=0.3] Accept-Encoding[gzip, deflate] Proxy-Connection[keep-alive] Response Headers: Content-Type[audio/mpeg] Content-Length[4642271] X-Amz-Meta-Duration[290169] X-Amz-Meta-Job[4wBtAmevZJkU] X-Amz-Id-2[71CW4Y3nVq6TJrqKNXsdgPMvWUS1QmTch1z9x9u3V4voVm1OQJ1edA7jngs/Q4yr] Date[Sat, 24 Nov 2012 11:16:00 GMT] Cache-Control[max-age=252460800] X-Cache[HIT] X-Amz-Request-Id[3FCFC7577DAA11EF] X-Amz-Version-Id[Pf7cBj1iUxYbXm7T4Ra7GDRazH3.GV4o] Etag[“3738a8f86fe8628bb9b3467d6a59a2c0”] X-Amz-Meta-Bitrate[128] Last-Modified[Wed, 21 Nov 2012 01:59:12 GMT] Accept-Ranges[bytes] Server[ECAcc (ams/D18A)] Keep-Alive[timeout=5] Connection[Keep-Alive] Proxy-Connection[Keep-Alive]
Farid Zakaria · 2012-11-26
They moved the URL from where they serve media to ““http://hypem.com/serve/source/” rather than “http://hypem.com/serve/play/”. If you use the new URL you’ll see the new JSON data! :)
Update – Google Chrome HypeMachine Extension Alive and Well! – Small Thoughts · 2013-03-19
[…] changing to HypeMachine’s new private API (which I’ve detailed earlier) […]
mozilla firefox · 2013-06-27
I feel your angst. I never had any luck whatsoever with this kind of thing, either. So happy to know I’m not all by my lonesome!
Meronako · 2014-07-05
Danke, das hat gerade 10€ von mir gertetet. Bis vor kurzem konnte man die Musik auch noch direkt mit Banshee kaufen. Geht nun nicht mehr. Ich denke ich war das letzte mal bei Amazon Musik kaufen.
Azlizam · 2014-07-05
Freut mich, dass es dir helfen kotnne :)So lange es eine Mf6glichkeit gibt, ist ja alles gut. Aber so entgeht Amazon eine inzwischen immer grf6dfer werdende Kundenzahl
mplewis · 2014-09-29
Thanks for taking a look at the Hypem API! I’m trying to build an iOS app for Hype Machine, and this is coming in super handy.
Farid Zakaria · 2014-10-07
glad I could help!