Why did BYOND block the Wayback Machine (Archive.org)

ID:183444
 
Jul 30 2007, 5:42 pm
Was the spider eating your bandwidth or something? Otherwise I can't see a reason personally.

Just wondering what the reasoning was behind it :)

Wish I could still see some of my old HUB entries that were deleted without warning several years ago.

~Kujila
Jul 30 2007, 7:00 pm
I think it had something to do with the Wayback mahine indexing and archiving every hub page BYOND has. Which is a lot of pages. And I believe it also does it with forum posts. And that's even more pages it has to download.

I think it was a bandwidth issue associated with the aforementioned. It was brought up a while back, anyway.
Jul 31 2007, 9:02 am
BYOND blocked ALL spider bots, this includes Google and whatnot too I believe. Mostly for bandwidth reasons, partially because it was indexing things it shouldn't have.
Jul 31 2007, 3:20 pm
In response to Nadrew
Can't robots.txt be set up to only block specific directories?