cURL, or "a client that groks the URLs", is a handy command line tool that is available for most platforms. It makes basically all url-related functions, such as up- & downloading, easy to go about. If you have a *nix system, it's probably installed. Otherwise, you can obtain it at http://curl.haxx.se/.

The basic way of usage is simply typing curl "http://www.everything2.com/" for instance. This will grab whatever comes out of the url http://www.everything2.com/ and spew it to standard output (if you're using a terminal, it will get printed there).

However, the real power of curl comes when you take into account the plethora of switches and options available to you. Some you should know are:

  • -f makes curl fail silently, not generating output on errors.
  • -O tells curl to give the file the same name as on the server, ie curl -O "http://www.everything2.com/index.pl" would save to a file called index.pl
  • -o (note the lower-case) tells curl to name the file as you specify, ie curl "http://www.everything2.com/index.pl" -o monkeysoy.html would save to a file called monkeysoy.html
  • -e "fakes" the referer (sic) field of the request, to make it appear as though coming from the server you specify. This is practical if a server blocks requests that come from outside.

Now, this is all fairly simple and not very devious. Let's get down and dirty, shall we?

One of the really sneaky parts is the way it can interpret the urls you give it. Say there's a website with a series of images on it, named sequentially. You've got your image1.jpg, image2.jpg, all the way up to image200.jpg. This would certainly be a hassle to type in by yourself. Sure, you could whip up some perl to do it for you, but still, there has to be an easier way!

Of course there is! Enter the [brackets]. Instead of typing out every url, simply replace the number in the aforementioned imagenames with a bracket containing a range, like so: curl -O "http://www.tehpixx0rs.com/stash/image[1-200].jpg" (I bet you're thinking up uses already, huh?). You can even pad with zeroes if that's the naming convention ([001-200]).

And as if that wasn't enough, there are the {curly brackets}... These can be used with a comma-delimited list of whatever you want, to make the urls even more complex! To wit: curl -O "http://www.monkeys.org/species/{rhesus,gorilla,spider}/closeup.jpg".

Oops. Did you spot the error up there? It's overwriting the previous monkey as it gets the next one! Now we need to use the sneaky parts of the -o switch. You can use placeholders for each of the variables in your url, with the first one being #1, the next #2 and so on up to #9. This makes the correct use of the previous line: curl "http://www.monkeys.org/species/{rhesus,gorilla,spider}/closeup.jpg" -o "#1-closeup.jpg" - or something to that effect.

So now, I'll leave you with a couple of examples of uses (all authentic, albeit I've changed the urls to protect the innocent). Your homework will be to figure out how and why they work, then find even better ways to abuse the inter-net (remember, though, that moderation is key!).

  • curl "http://www.mp3zforfree.org/" | perl -pe '/a href="([^"]+\.mp3)"/ig;if($1){$_="-f\n-O\nurl=\"$1\"\n"}else{$_=""};' | curl -K -
    I admit this could be done easier and prettier with wget, but if you don't have it handy, there is more than one way to do it.
  • curl "http://everything2.com/index.pl?node_id=762826&op=login&user=YOURNAME&passwd=YOURPASS" | perl -pe '/node_id="(\d+)"/i;if($1){$_="-f\n-o \"$1.xml\"\nurl=\"http://www.everything2.com/?node_id=$1&displaytype=xml\"\n"}else{$_=""};' | curl -K -
    Pretty useful if you need to back up your stuff, just remember that this will leave your username and password hanging in the terminal history, so you might want to find a better way.
  • curl -f "http://fakecomic.keenspace.com//comics/fake[1999-2002][01-12][01-31].{gif,jpg,png}" -o "fakecomic_#1#2#3.#4"
    This is very evil, and you shouldn't do it

But don't take my word for it, go out and try leeching entire websites yourself. Eat up that bandwidth, son!

Sources: curls man page, curl.haxx.se. Also sorry to baffo for being so slow with writing this. And I swear, this is for educational purposes only! Remember to abide by your local laws, y'hear? Any input or extra hints appreciated.