The game resumes
Over the course of this weekend, my latest pet project MetaHypem has been shut down but by HypeMachine and not through us. HypeMachine seems to have added either our domain or instance running our server onto a black-list.
Here is a nice screen capture of what current data HypeMachine is returning my scraper. The MP3 Artist/Title was a friendly sting I guess.
I haven't done much publicizing of the server and daily hits to the site was only ~40 visitors (most returning). However clearly there was a need/love for having access to the music (~50k downloads). All the blogs that HypeMachine scrapes allow users to download the music. The purpose of a music blog is to help promote bands and new music. Keeping the music locked onto their platform seems going against their goal.
Where to go from here
I'm currently investigating (alongside my partner) different ways to resolve this issue. I'd love feedback from people expressing perhaps which they would prefer or even have experience with!
- Move the server to EC2. Perhaps they only block the IP at which I can just tear down the server to get a new IP.
- Change the scraping of HypeMachine to be distributed and have other instances perform the scraping.
- Release the source code to the project and let others run local instances.
- Figure out other inventive ways!
10 archived comments
Tyler B · 2013-01-28
Thank you for your work! I think changing the scraping of HypeMachine to be distributed so that your server is not the one bringing in traffic may be a smart idea.
Farid Zakaria · 2013-01-29
Yea.. I just need to figure out what level have they blocked me.
Gregory Allan · 2013-01-31
I’m really shocked hype machine would be this cruel. I think it was a really low blow.. their words sting. Sorry about this Farid. I come on your blog from time to time seeing what updates you’ve accomplished :)
Anthony V · 2013-01-31
Just figured it’d be funny, since I saw someone (your partner?) link to this on HN.
While you are right that blogs exist to promote music, they aren’t just file repositories. So by stripping away all the context (like metahypem.com does) that Hype Machine carefully shows and leaves intact, you just get a bunch of random, meaningless mp3 files. What’s the point of that? It’s also certainly not what blogs intend.
Ultimately, if this is a service you believe should exist, you should just build all the infrastructure we’ve built and then, if it truly matters more, and adds more value, you will be successful, and I have not problem with that.
Farid Zakaria · 2013-01-31
Hey Anthony,
Firstly, it was myself who wrote that comment. To be honest, it’s just a short term solution to get the media files for offline use. No one browses metahypem.com alone. They visit it once finding music they enjoy on HypeMachine. It’s just helping people get to that last step. I think it just shows it’s a feature HypeMachine should add to their infrastructure (I’m in in no way looking to replace it). If a blog allows the download of the song, HypeMachine should be itself by that logic exposing that functionality as well
Anyways, I use it just as a tool to learn new technologies. With http://metahypem.com I got to learn NodeJS and other new technologies. So tough to build something when you have no overall goal. Allowing users to download is a simple adequately scoped project.
It’s a shame a little on the wording used in the title/artist. I’ve been very respectable towards my projects.
Farid Zakaria · 2013-01-31
To be honest, my music taste has shifted from that on HypeMachine and I’ve been primarily following other genres via Soundcloud. I just like the idea of writing apps for it. The API is simple and clean.
Farid Zakaria · 2013-01-31
who could blame them. I’d probably do the same if someone was blasting my site with a script to rip all the songs. They should just expose to me a real API ;)
James · 2013-02-04
I’d love for the source code to be published ! Really loved the project, it’s a shame to see it taken off-line !
Farid Zakaria · 2013-02-04
hey James,
I’ll talk to my partner if he’d like to release the source. For now you can find my older version which was done in Django here: https://github.com/fzakaria/HypemFinderSite
The current site is NodeJS and much uberer ;)
Farid Zakaria · 2013-02-04
I fixed the site so it’s live just removed download button for now.