Re: <nettime> fast-changing propaganda website archiving tools?

[John Young <jya@pipeline.com>]

Running an open site, not requiring infernal registration with vile
log-in and password, and worst of all, email address for confirmation,
is illuminating.

Visitors are far more diverse than the regulated kind which has almost
become the norm what with the exaggerated claims of hackery generated
by the cybersecurity cartel.

This has been our bot experience since 1996, nut-shelled:

The earliest bots were governmental, NSA in fact, before we knew what
a bot was until a wizard described the varmint. It was open in that
its origination was exactly an NSA server domain: 144.51. (It stopped
after a year or so or went behind a spoofed address) Most of the TLAs
followed suit openly, then went under cover.

This was before the truly world-class villainous search engines
(excluding the one and only Archive.org) began their drive to
commercial bot hegemon.  Site traffic increased as the search engines
spread their infectious file mongering.

As search engine technology spun of hundreds, maybe thousands, of bot
programs empowered individuals, institutions, governments,
competitors, thieves, good hearts and idiot savants to rake in files
without restraint.  They came to average 25-30% of bandwidth usage
despite use of robot.txt and htaccess.

The more powerful bots would take over the site until it was completly
drained. When it hosted a few hundred files, that was brief, but
50,000 files many of them hundreds of KB images, take a while.

We are determined to keep the site unfettered to readers, note
readers, not gobblers, and have a warning that downloading more than
100 files a day will lead to blockage of the orginating IP address.
None of the bots, being inhuman, abide this restriction. So we pick
off users one by one for sending to hell.

If anybody complains we point out the reason for the block. And
suggest they use an anonymizer to get files until a block is lifted.

However there are few complaints about blocks, so this seems to
indicate it doesn't matter to the bot user, there are plenty of other
free lunches elsewhere.

The worst bot of all -- Google. We block all search engines except
Google. The voracious son of a bitch comes several times a day,
meticulously tracked. It lies about what it is doing. We will make it
sorry it ever did. More on that when the bell tolls.

For debate: Search engines and bots are spam, killers are needed to
keep the Internet from succumbing to their onslaught. Don't believe
the crap about their advancing access knowledge -- that's a variation
on free cancer sticks. They discourage reading in favor of gathering
and hoarding.

A mind slightly open.


#  distributed via <nettime>: no commercial use without permission
#  <nettime>  is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: http://mail.kein.org/mailman/listinfo/nettime-l
#  archive: http://www.nettime.org contact: nettime@kein.org