From: Cameron Kaiser (spectre_at_floodgap.com)
Date: 2004-02-06 17:16:13
> > > > - Is there an easy way to download the whole archive?
> > >
> > > If you have curl, and don't need the attachments... otherwise use your
> > > favourite browser's archive making facilities.
> >
> > May I say that as someone who pays for my bandwidth, I hate people who do
> > this, and I will block them indefinitely from my site if I see activity
> > of this sort. This drives up load average, diminishes outgoing traffic, and
> > is just plain rude.
>
> err isnt the purpose of websites that they are beeing downloaded? especially
> if they are archived? but maybe thats just me ... :=P
Don't be deliberately obtuse. I'm referring to people with abominations like
Offline Explorer or WebZIP that can, when parallelized, bring servers to their
knees. If someone wants to view the site, fine. Rapidly downloading it to disk
prevents others from accessing the site during that period, and on the size
that services like the HyTelnet gateway or Commodore Knowledge Base and
Secret Weapons are, locks my 384Kbps uplink for unacceptably long periods.
Having had this happen multiple times, if I notice a particular host
exhibiting spider-like activity that I don't allow (well-behaved bots likeo
Googlebot are specifically excepted), I ban IP blocks first and ask questions
later.
Incidentally, to robots.txt suggesters, the whole selling point of many of
those tools is that they ignore robots.txt. Good bots obey <meta> tags and
stay out of where they're not wanted. It's the badly behaved bots that I'm
banning.
Bluntly, I'm not buying extra bandwidth for degenerates who believe that
having a local copy of the entire Internet is their right over all other
regular requests, including from whom I bought the line for in the first
place -- me.
--
---------------------------------- personal: http://www.armory.com/~spectre/ --
Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- The things which hurt, instruct. -- Benjamin Franklin ----------------------
Message was sent through the cbm-hackers mailing list
Archive generated by hypermail pre-2.1.8.