Synchronize iSeries files with Amazon S3 virtual storage

Amazon Web Services, or AWS, is one of the new and hot promises to revolutionize IT.
I subscribed to AWS almost two years ago, but started to really use it only on last October.

AWS provides huge benefits to IT, developers and web site owners.
For IT, other than the ability to dynamically allocate servers and pay for the actual server utilization, AWS provides the ability to have a reliable, secure remote storage.

I went out to see whether Amazon S3 lives up to my expectations and can play a significant part in a disaster recovery plan. Some smart folks already did some math a while ago, and the economic drivers for using S3 are now higher than ever.

It turns out that there are quite a few S3 tools. For S3 management I ended up using the S3 organizer, a free FireFox extension, and the S3 Cockpit tool, which is actually part of a larger package, Jets3t.

There is a major difference between personal tools and tools appropriate to use in a datacenter. The biggest challenge for me was to find those tools that support multiple platforms, and that once configured, work in unattended mode with no hassle.

I settled for Jets3t - a set of libraries and tools written in Java.
On a windows server, installation is straightforward: just unzip the distribution archive, set up some environment variables and you are ready to start. Of course, Java is a prerequisite.

Jets3t consists of a set of libraries that wrap the S3 API with Java objects, and 4 working tools:

• Cockpit, a GUI application/applet for viewing and managing the contents of a S3 account.

• Synchronize, a console application for synchronizing directories on a computer with an Amazon S3 account.

• CockpitLite, a GUI application/applet for viewing and managing the contents of an S3 account, where the S3 account is not owned by the application's user directly but is made available via the Gatekeeper servlet.

• Uploader, a wizard-based GUI application/applet that S3 account holders (Service Providers) may provide to clients to allow them to upload files to S3 without requiring access to the Service Provider's S3 credentials.

The most promising tool, as far as I am concerned, is Synchronize.
Working in a manner similar to rsynch, this tool can keep an S3 bucket and a local folder in synch with each other, while providing both compression and encryption in the cloud.

After making it work on my PC, I set out to make it work on an iSeries server.

Setting up on iSeries
The first thing to do is to install Jets3t on your server, in any IFS directory you like.
I placed it in the /QIBM/UserData/aws/jets3t folder.

Jets3t relies on some environment variables to be defined, specifically JETS3T_HOME and JAVA_HOME.
Make sure both are defined before executing the synchronize.sh script.
I set up both as global variables by running these iSeries commands:

addenvvar JETS3T_HOME   '/QIBM/UserData/aws/jets3t'  LEVEL(*JOB) REPLACE(*YES)
addenvvar JAVA_HOME   '/QIBM/ProdData/Java400/jdk15'  LEVEL(*JOB) REPLACE(*YES)

Additionally, the unix shell scripts that came with the package assume that the shell is in /bin/sh
This probably is not true on your iSeries server, so either modify the shebang lines or run this command to solve the problem

ADDLNK OBJ('/usr/bin') NEWLNK('/bin') LNKTYPE(*SYMBOLIC)

Synchronize supports file compression and encryption, as well as various modes of operation that make synching a breeze. For example, to synch the /home/shalom/personal folder with a S3 bucket called shalom-personal, you will execute

synchronize.sh UP shalom-personal /home/shalom/personal/

You do not have to create the bucket prior to using synchronize, it will create the bucket if it does not exist. ( tip: use the --noaction switch to verify your actions before commiting to any changes. )

To learn how to actually use synchronize, read the manual.

To make it easier on the iSeries folks, here is a CL program and CMD that wrap the synchronize tool in a familiar interface.

In the next post I will explain how to extend Jets3t to a real on-demand storage device with some Java programming.