www.audio-circuit.dk download site bandwidth spam - Help needed

[Michael Chua]

Hi Jan

Sorry to hear about your problems. Check out eserviceinfo. If you can get something like theirs running, it'll prevent someone from downloading your files en masse.

Regards
Mike

[Jan Dupont]

I have come across some sites offering the exact same schematics and service manuals as on my site. The funny part is if I change the name of a schematic (removing a letter or correct a spell error), shortly after the other sites have the renamed schematic (and the old one as well). So they just continiusly download everything on my site and publish it on their own.....

[Jan Dupont]

Thanks Michael

[Jason]

Hi Jan,

Now that is very interesting. Probably being downloaded from their server IP, or perhaps they think they are smartypants for setting up a distrubuted downloading bot network that downloads manuals from all over the web (probably not just your website) and assembles them on their own websites in order to make money from google traffic and advertising.

Depending on how they do it, you could use random and constantly changing urls to confuse their bots (takes some programming) or also construct your urls using javascript to piece them together so a "stupid" bot can't just scrape your website for plain text and download the new files. This approach is done commonly to protect email addresses. As the bots get smarter they have learned to actually run the javascript on a page though...

Another angle might be to say something like this:

To download the schematic, take this url: http://www.audio-circuit.dk/XXXX/schematic1.pdf and replace the XXXX with "apple".

That way, there are no urls to scrape from your website, it requires a small amount of intelligence to "create" the url required to download but it's simple and obvious enough for a real person to work it out. Only real downside I can see - Google won't find or archive the manuals so less traffic for you.

If these guys are determined to "mine" your website in particular, there's not much you can do. What amount of bandwidth are we talking about? Bandwidth is cheap these days if you are with the right host. I'm a fan of dreamhost, and they usually have unlimited bandwidth offers every now and again..

[Jan Dupont]

Thanks Jason,

Many solutions are in play, and sakis (and some of his employed) are testing some to find the most friendly solution for both users and me. Thanks sakis

Yes they are gathering schematics from all over the net, to present on their own. Each time I deny a group of IP addresses from that area where they downloads from, they manage to alter the IP to a new area address

From the stats I can see that they use multiple downloads at the same time, using all bandwidth 24 hours a day = tons of GB each hour...........

[Jason]

Perhaps use mod_evasive in that directory, and when you get more than a few hits in a minute from any IP address it will go into a ban list, and instead of serving an error page, serve a lovely PDF containing an advertisement for your website.

[Jan Dupont]

serve a lovely PDF containing an advertisement for your website

Thanks. That could be fun;

"I'm just a F?=)(* A$$hole ripping other peoples websites!
Please go to Welcome to Audio Circuit Denmark to download the schematic"

[chris661]

Brilliant. Make sure you put the word original in there - then people will stop going to those sites when they realise what they are.

[Jan Dupont]

A solution is on it's way

Thanks to my Greek friend diyaudio member sakis, I expect to have my download site up and running again tommorow
Wednesday with Captchas

Thanks to all for your kind support
I hope I will be able to provide free Schematics and Service Manuals to the DIY community for many years to come......

criss661;
Yes! That could have been a real laught

[QSerraTico_Tico]

I have seen CDs full with collected schematics from the Internet offered on Ebay for .......money.