Backing up to the cloud

My previous post was about using the Jets3t synchronize tool on an iSeries server.
While synchronize works quite well, it is not flexible enough for what I had in mind. To be able to integrate Amazon S3 with a real life backup and DRP policy, I want a set of atomic functions that enable me to simply transfer particular data objects between S3 and the server.
Using the Jets3t rich set of S3 objects and the samples provided by the author, I found it quite easy to write additional proof of concept tools.

I now have Create Bucket, Upload Single file, and DownLoad Single File programs. These are all barebones tools, use them at your own risk..

All tools use the iUtils helper tool that provides for central management of parameters and credentials. In fact, iUtils uses exactly the same configuration file Synchronize does.

I timed the Upload and Download tools with a 500MB savefile, and on my network it took less than 30 minutes to go either way.

The usage is quite simple:

• CreateBucket requires a single parameter, the new bucket name.
• uploadSingleFile requires two parameters: the target bucket name, and the file name to upload
• DownLoadSingleFile requires two parameters: the source bucket name, and the file name to download.

Get the extended toolset files here. As I said before, these are barebones tools and there is a lot to improve. There is practically no graceful error handling, no compression and no encryption.

Using Amazon S3 in an iSeries backup/recovery scenario
S3 can be used as an offsite backup for some of your data, maybe even for all of your data. A library can easily be saved to a save file, and the save file uploaded to S3 storage until needed, at which time it can be downloaded and restored to your iSeries.

For example, look at the following set of commands that saves changed objects to a save file, zips it and sends it to S3. The extra zipping is required because the data compression built into the iSeries save commands is not very efficient, and because I have not implemented it yet as an integral part of the upload tool (although the functionality exists in the jets3t package).

/* create save file */
crtsavf s3wk200901

/* backup objects changed since December 31 to save file */
savchgobj obj(*all) lib(mylib) dev(*savf) savf(s3wk200901)
 refdate('31/12/2008') clear(*all) dtacpr(*high)

/* copy save file to IFS */
cpytostmf frommbr('/qsys.lib/mylib.lib/s3wk200901.file')
 tostmf('/backups/s3wk200901.savf') stmfopt(*replace)

/* further compress save file */
qsh cmd('jar  cMf  /backups/s3wk200901.zip /backups/s3wk200901.savf')

/* upload compressed save file to S3 */
qsh cmd('cpytos3.sh mybackup /backups/s3wk200901.zip')

Synchronize iSeries files with Amazon S3 virtual storage

Amazon Web Services, or AWS, is one of the new and hot promises to revolutionize IT.
I subscribed to AWS almost two years ago, but started to really use it only on last October.

AWS provides huge benefits to IT, developers and web site owners.
For IT, other than the ability to dynamically allocate servers and pay for the actual server utilization, AWS provides the ability to have a reliable, secure remote storage.

I went out to see whether Amazon S3 lives up to my expectations and can play a significant part in a disaster recovery plan. Some smart folks already did some math a while ago, and the economic drivers for using S3 are now higher than ever.

It turns out that there are quite a few S3 tools. For S3 management I ended up using the S3 organizer, a free FireFox extension, and the S3 Cockpit tool, which is actually part of a larger package, Jets3t.

There is a major difference between personal tools and tools appropriate to use in a datacenter. The biggest challenge for me was to find those tools that support multiple platforms, and that once configured, work in unattended mode with no hassle.

I settled for Jets3t - a set of libraries and tools written in Java.
On a windows server, installation is straightforward: just unzip the distribution archive, set up some environment variables and you are ready to start. Of course, Java is a prerequisite.

Jets3t consists of a set of libraries that wrap the S3 API with Java objects, and 4 working tools:

• Cockpit, a GUI application/applet for viewing and managing the contents of a S3 account.

• Synchronize, a console application for synchronizing directories on a computer with an Amazon S3 account.

• CockpitLite, a GUI application/applet for viewing and managing the contents of an S3 account, where the S3 account is not owned by the application's user directly but is made available via the Gatekeeper servlet.

• Uploader, a wizard-based GUI application/applet that S3 account holders (Service Providers) may provide to clients to allow them to upload files to S3 without requiring access to the Service Provider's S3 credentials.

The most promising tool, as far as I am concerned, is Synchronize.
Working in a manner similar to rsynch, this tool can keep an S3 bucket and a local folder in synch with each other, while providing both compression and encryption in the cloud.

After making it work on my PC, I set out to make it work on an iSeries server.

Setting up on iSeries
The first thing to do is to install Jets3t on your server, in any IFS directory you like.
I placed it in the /QIBM/UserData/aws/jets3t folder.

Jets3t relies on some environment variables to be defined, specifically JETS3T_HOME and JAVA_HOME.
Make sure both are defined before executing the synchronize.sh script.
I set up both as global variables by running these iSeries commands:

addenvvar JETS3T_HOME   '/QIBM/UserData/aws/jets3t'  LEVEL(*JOB) REPLACE(*YES)
addenvvar JAVA_HOME   '/QIBM/ProdData/Java400/jdk15'  LEVEL(*JOB) REPLACE(*YES)

Additionally, the unix shell scripts that came with the package assume that the shell is in /bin/sh
This probably is not true on your iSeries server, so either modify the shebang lines or run this command to solve the problem

ADDLNK OBJ('/usr/bin') NEWLNK('/bin') LNKTYPE(*SYMBOLIC)

Synchronize supports file compression and encryption, as well as various modes of operation that make synching a breeze. For example, to synch the /home/shalom/personal folder with a S3 bucket called shalom-personal, you will execute

synchronize.sh UP shalom-personal /home/shalom/personal/

You do not have to create the bucket prior to using synchronize, it will create the bucket if it does not exist. ( tip: use the --noaction switch to verify your actions before commiting to any changes. )

To learn how to actually use synchronize, read the manual.

To make it easier on the iSeries folks, here is a CL program and CMD that wrap the synchronize tool in a familiar interface.

In the next post I will explain how to extend Jets3t to a real on-demand storage device with some Java programming.