cURL

If you are not already using it, I suggest you start using cURL. cURL, or Client for URLs, or see URL comes in two flavours, a command line tool for getting and sending files using URL syntax and a library, libcurl, for use by other programs. It supports more than a dozen protocols (FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, FILE, LDAP), cookies, proxy tunneling, transfer resume, authentication (Basic, Digest, NTLM, Negotiate, kerberos…), SSL certificates, HTTP uploads, progress meter, speed limit, you name it. Here’s some examples:

Upload a file as multipart/form-data plus extra params to a URL:

curl -F upload=@localfilename -F press=OK [URL]

Use an agent of your choice:

curl -A "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL]

Get the last 500 bytes of a document:

curl -r -500 http://www.get.this/

ftp upload through a proxy:

curl --proxytunnel -x proxy:port -T localfile ftp.upload.com

Read and write cookies from a netscape cookie file:

curl -b cookies.txt -c cookies.txt www.example.com

Download resume:

curl -C - -o file http://www.server.com/

cURL is in Flash Player 9, Mac OS X, F-Secure and IBM’s BOINC among others. Interfaces exist for most major as well as other more obscure languages. Unfortunately, the javacurl interface supports a small subset of cURL’s features and is not well tested.