Now I am using this technology for perfectly genuine reasons, but it strikes me that it would be very easy to abuse.  Also, for your inexperienced programmer, could easily get out of control.  What also strikes me, is that these two scenarios almost overlap if you aren’t careful.

Say for instance, you write a basic spider to read a site, pick up any links, read them too, and record all the information in a text file.  Say also, you forget to put in a checker for duplicate links.  Already you have a spider that will never stop.  In other words, you’ve put it into an infinite loop.  ”Oops” you might say, but consider the website you just spidered.  What you have essentially just done is a DoS attack.  Sure, it may not have done any damage, but it has the potential.

However, chuck in a few random commands, and your basic terrible script can suddenly emulate a real web user, give it a user-agent, and voila, spider away, steal what you want, the possibilities are endless and frankly, scary…