Automate FTP with macros

Has it been a while since you used FTP from the command line? While there are decent GUI-based FTP clients (such as gFTP), you can automate operations with the command-line version and handle file transfers with no user interaction at all.

When you try connecting to an FTP server and specify a username, FTP won’t do anything unless you type in your password, or specify it in the connection command, as in ftp bainm:myftppassword@192.168.1.2. You can take advantage of this syntax to automate your FTP transfers using a pipe:

echo get testfile.txt | ftp://bainm:myftppassword@192.168.1.2

Unfortunately, this doesn’t work for every distro. For example, it’s fine on SUSE but not on Red Hat or Debian. Therefore, you need a solution that works for all distros.

Create a file that contains the IP address or name of the machine that you want to access, your username, and your password, and save it as ~/.netrc:

machine 192.168.1.2
login bainm
password myftppassword

With ~/.netrc in place, you can send a command to FTP without having to input a password manually:

echo get testfile.txt | ftp 192.168.1.2
Connected to 192.168.1.2.
220 acamas.ilium FTP server (Version 6.4/OpenBSD/Linux-ftpd-0.17) ready.
331 Password required for bainm.
230 User bainm logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
local: testfile.txt remote: testfile.txt
227 Entering Passive Mode (192,168,1,2,9,203)
150 Opening BINARY mode data connection for 'testfile.txt' (0 bytes).
226 Transfer complete.
221 Goodbye.

At this point, you may want to think about creating a shell function that can carry out data transfers for you — for example:

function compileFTP {
cat <<STOP
prompt off
lcd $1
#Don't worry if the target directory already exists - FTP won't complain:
mkdir $2 
cd $2
mput *
STOP
}

You can then call the function with a command like:

compileFTP ~/webfiles webdirectory | ftp 192.168.1.2

FTP runs the commands in the function without you having to input a password. This means that you can automate any FTP operations that you need to carry out. For example, you might want to save the code from above into a file, make it executable with a command like chmod +x ~/FTP/updateWeb, then run that file daily at midnight using cron:

0 0 * * * ~/FTP/updateWeb

You’re not limited to writing functions in shell scripts — you can also write them for FTP. However, in that case, they’re called macros.

If you want to write an FTP macro, you’ll need to go back to the .netrc file, because that’s where the macros are stored. The rules for writing a macro are simple:

1. Start a new macro definition by using the macdef keyword.
2. Write your FTP commands.
3. End the macro definition by adding a blank line.

For example, you could add the following code to .netrc:

macdef updateWeb
prompt off
lcd /home/bainm/webfiles
mkdir webdirectory
cd webdirectory
mput *

Don’t forget that blank line!

You can call a macro from the Linux command line, but in a slightly different manner than the way you call a shell function:

echo "$ updateWeb" | ftp 192.168.1.2

You have to include a $ in the command to be sent to FTP to tell FTP that the command is a macro. Now you can change the crontab entry accordingly:

0 0 * * * echo "$ updateWeb" | ftp 192.168.1.2

At this point, you’re probably wondering about variables — you don’t want to have to create a new macro for every upload that you want to carry out. Fortunately, macros use variables the same way shell functions do. The macro, therefore, becomes:

macdef updateWeb
prompt off
lcd $1
mkdir $2
cd $2
mput *

You can now send the names of the directories to your macro rather than hard-coding them:

echo "$ updateWeb /home/bainm/webfiles webdirectory" | ftp 192.168.1.2

You can also create another macro that supplies the inputs to your generic one (remembering to use the $ to tell FTP that you’re calling a macro):

macdef myWebUpdate
$ updateWeb /home/bainm/webfiles webdirectory

What we’re doing here is defining the new macro by using the macdef keyword, then calling our original macro (updateWeb) and supplying it with the directories that we want to copy from and to (i.e. from /home/bainm/webfiles on the local machine to webdirectory on the server. You can now run the macro from the command line:

echo "$ myWebUpdate" | ftp 192.168.1.2

Of course, you’re not going to throw away your favorite FTP GUI. However, with some command line magic, you can automate FTP and run it with no human intervention at all. You can go home each evening knowing that all your Web servers will be updated or all your files backed up — all through the power of FTP macros and the .netrc file.