tag:blogger.com,1999:blog-31191485380926009752024-03-05T22:49:16.201+02:00my bleeding noseI often find myself running in the darkness of accelerating technologies and changing society with only sporadic bursts of illumination. I guess my nose bleeds a lot.Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-3119148538092600975.post-71861112181946328722012-03-14T12:14:00.000+02:002012-03-14T12:14:11.211+02:00Experts ExchangeIt is hard keeping up with new stuff. Especially so when you are a veteran whose head is already packed with old stuff that's not willing to go away. However, new stuff, especially in dynamic fields like IT and software, has its own way of sneaking up on you. New stuff that you are forced to deal with on a moment's notice can make you hazy and discombobulated (just learned this word the other day..)<br />
<br />
My way of coping with this problem is to constantly keep in touch with new tech, new approaches, and new problems, so I do not face an insurmountable wall to climb whenever the reality of change hits me.<br />
<br />
One of the best tools I have is my membership in <a href="http://www.experts-exchange.com/" target="_blank">Experts Exchange</a>, and specifically the fact that I try to be on the giving side, an expert giving free advice. Each and every answer I provide hones my skills and teaches me something new. I learn even from questions that I have nothing to help with.<br />
<br />
Give it a try. <br />
<br />
<a href="http://www.experts-exchange.com/" target="_blank"><img alt="The New Experts Exchange is Here! Experience EE v.10!" height="155" src="https://redsourcesales.com/facebook/iPadLandingPage/imgs/EE-Port_Badge_150_155.jpg" title="Go to Experts Exchange" width="150" /><br />
</a>Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-52442465785241611282011-06-04T19:23:00.027+03:002011-06-06T11:45:23.945+03:00Things your IT can do in the cloud - #1: anonymous proxy<h3>Why anonymize your business? </h3><br />
In many people's minds browsing via an anonymizing proxy is associated with either hard-core anarchists or with opponents to oppressive regimes. <br />
<br />
However, there are several very good reasons to use anonymizing proxies in your business.<br />
You may want to hide from your competitors the fact that you visit their web site and look for specific information to acquire competitive intelligence. You also want to hide these visits from third party web sites, which may cooperate with your competition more closely than they do with you.<br />
<br />
When researching a new product, a firm searches various databases, vendor sites and academic sources for information on the relevant materials, processes, possible suppliers, and market information. When using subscription based information sources, the firm's confidentiality is protected contractually. That is not so in public sources. In many industries, knowing in advance that a certain firm is looking for information about certain processes and materials, is worth a lot of money.<br />
<br />
<br />
Prior to all mergers or acquisitions, a team of analysts will research the firm to be merged with or acquired. Just like in regular R&D, an exposure of intentions to the researched subject or to a third party can be very costly.<br />
<br />
Of course, there is no need to anonymize the entire traffic going out of your firm's network. The problem can be usually pinpointed to a few individuals who really need this protection, and even they do not need it all of the time.<br />
<br />
<h3>Using EC2 to anonymize your users</h3><br />
The idea to use An Amazon EC2 server as an anonymous proxy is not a new one. There are a number of articles already on the web explaining the technicalities of setting up proxy servers and tunneling in EC2. However, the existing articles aim at highly technical individuals and usually the setup is much more complex than what I am about to show. <br />
<br />
Technically speaking, there are several ways to achieve our goal. You can tunnel the traffic via SSH, setup a SOCKS server, or use anonymizing networks like TOR and I2P. I chose to showcase a simple http proxy using standard Apache, because it is so easy to set up and yet so effective.<a href="http://aws.amazon.com/cloudformation/">Using AWS CloudFormation</a>, you can have a 1-Click proxy up and running in no time!!<br />
<br />
I assume that the readers of this blog post already have an AWS EC2 account, along with their credentials, certificates and keys. You should also have some familiarity with the AWS self service portal. I will focus on what you need to specifically do to quickly and easily deploy an anonymizing apache proxy server. Scroll down to see the CloudFormation template used to automate the deployment. <br />
<br />
<br />
<b>Recipe materials:</b><br />
<ul><li>1 EC2 security group</li>
<li>1 Elastic IP<br />
</li>
<li>1 EC2 Micro Linux server<br />
</li>
<li>1 Apache httpd<br />
</li>
<li>A dash of configuration changes<br />
</li>
</ul><h3>Create an EC2 security group</h3><br />
We will use the EC2 security group to limit access to the proxy server. After all, we do not want free riders to use our proxy, especially since we will be paying for all of the traffic. <br />
First, you have to know the outbound addresses of your network. In this example, I used a bogus address of 79.181.46.194. Create a new Security group, and add a rule like the one in this screenshot:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWiTp6DRO56ZSF9oavHmByje3lPfFhZzEdTp_XnQjaHqnqa9popx2jiq-Ruf9S-8lII4tk_AoUkx1Ov3hqkRUgE-UC9xBdvMfBne20dUqcbVzbwc2-wntTjPv2mkROu2fI0seep4ZMhrdj/s1600/EC2-security-group-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="237" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWiTp6DRO56ZSF9oavHmByje3lPfFhZzEdTp_XnQjaHqnqa9popx2jiq-Ruf9S-8lII4tk_AoUkx1Ov3hqkRUgE-UC9xBdvMfBne20dUqcbVzbwc2-wntTjPv2mkROu2fI0seep4ZMhrdj/s320/EC2-security-group-2.png" width="320" /></a></div><br />
<h3>Start a new EC2 instance</h3><br />
For a small number of users, a Linux micro instance is more than enough. Start a new instance and select the basic 32bit Amazon Linux AMI, and then chose to start a micro size server. <br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKw21UfY5LpGLaLJ21Ks-CQNzN_xtAN221REir5a73hGyW6HT1F1Ki2tY6_fsRj0uQ2DZF31ObxEwnCGdIpaUFsz5ikUYMoMVXuifX9um0zoPUhh8sAKSqzOOhD5m_XLqg5dYegydbZH7D/s1600/EC2-instance-ami.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKw21UfY5LpGLaLJ21Ks-CQNzN_xtAN221REir5a73hGyW6HT1F1Ki2tY6_fsRj0uQ2DZF31ObxEwnCGdIpaUFsz5ikUYMoMVXuifX9um0zoPUhh8sAKSqzOOhD5m_XLqg5dYegydbZH7D/s320/EC2-instance-ami.png" width="320" /></a></div><br />
The Amazon Linux servers have almost nothing preinstalled on them, but they do have the <a href="https://help.ubuntu.com/community/CloudInit">cloud-init service</a>. The cloud-init service, developed by Canonical to be used on Ubuntu, allows you to pass to the server bootstrap configuration data, parameters and commands. Our instance will read the user data passed to it during initialization, and use it as the input for cloud-init. <br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7FdkEId0U9ecdoDDZfqXYucUKoFpbDY6GPsnT1H02N-zfP1Q7MoSbgZx_dQHgQpCimJadfl8Tazh1P3j7anCEtcwckhqQPUlhORivnLSCkDUE09-qA3brWDUwU_RuJmzT8wgX3OI0XfuP/s1600/EC2-instance-userdata.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7FdkEId0U9ecdoDDZfqXYucUKoFpbDY6GPsnT1H02N-zfP1Q7MoSbgZx_dQHgQpCimJadfl8Tazh1P3j7anCEtcwckhqQPUlhORivnLSCkDUE09-qA3brWDUwU_RuJmzT8wgX3OI0XfuP/s320/EC2-instance-userdata.png" width="320" /></a></div><br />
Here are the actual contents to be used for the user data. <br />
The "packages" section installs the latest Apache httpd service from Amazon's yum repository. <br />
<br />
The "runcmd" section appends the minimal set of Apache configuration directives that are required for the Apache proxy, and restart apache. Port 443 is added to support browser configurations that use it for SSL proxying.<br />
<br />
Before you copy and paste, take care to modify the IP address of your network. Although the EC2 Security group should take care of unwanted network access, I think that it is good practice to include some access control here as well. <br />
<br />
<code>#cloud-config<br />
<br />
packages:<br />
- httpd<br />
<br />
runcmd:<br />
- echo listen 443 >> /etc/httpd/conf/httpd.conf<br />
- echo ProxyRequests On >> /etc/httpd/conf/httpd.conf<br />
- echo ProxyVia Block >> /etc/httpd/conf/httpd.conf<br />
- echo \<proxy \*\> >> /etc/httpd/conf/httpd.conf<br />
- echo Order deny,allow >> /etc/httpd/conf/httpd.conf<br />
- echo Deny from all >> /etc/httpd/conf/httpd.conf<br />
- echo Allow from 79.181.46.194 >> /etc/httpd/conf/httpd.conf<br />
- echo \<\/Proxy\> >> /etc/httpd/conf/httpd.conf<br />
- service httpd restart<proxy \*\=""><br />
</proxy></code><br />
<br />
Last thing to do before you launch the instance, is to assign the previously created EC2 security group to our new micro instance. <br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZgwd3aYeQvqLB0y2F_95uRQdrunq9PWcDQ2HRqkfHoQw-BANMkLIb_oxRfWlk6cqbTNlQjxRLUtv-HtCJfoQJQFTaSzVW9oV4S3iJ4h_TwLpJwDs8RBXZE0vIRrVCj3hRwnWLX45J6A-F/s1600/EC2-instance-security-group.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZgwd3aYeQvqLB0y2F_95uRQdrunq9PWcDQ2HRqkfHoQw-BANMkLIb_oxRfWlk6cqbTNlQjxRLUtv-HtCJfoQJQFTaSzVW9oV4S3iJ4h_TwLpJwDs8RBXZE0vIRrVCj3hRwnWLX45J6A-F/s320/EC2-instance-security-group.png" width="284" /></a></div><br />
Before we turn to configure the end users, we may want to associate an Elastic IP with our new instance. We want a predictable environment, where the configuration changes to end users and our IT infrastructure are minimal. A new server instance in EC2 gets unpredictable IP address, and when we use an Elastic IP we can keep a known IP address and even use a DNS record to point at our proxy. The allocation of an EIP to a running instance happens after the instance is up and running.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx08fyDQmdW2wr0w_mJsOgLeOZ8sS3dUor8CPzNkdI_AbzV2VA0b5RhR_QXvuGzKpOvYflRd1fU3LpO2cXxBWopN3_Ivv1VCNDziRHkOsPtR7jCajS1MoNtrtOYznHhSlGAr5kXTJ8zzrE/s1600/EC2-instance-eip.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="158" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx08fyDQmdW2wr0w_mJsOgLeOZ8sS3dUor8CPzNkdI_AbzV2VA0b5RhR_QXvuGzKpOvYflRd1fU3LpO2cXxBWopN3_Ivv1VCNDziRHkOsPtR7jCajS1MoNtrtOYznHhSlGAr5kXTJ8zzrE/s320/EC2-instance-eip.png" width="320" /></a></div><div class="separator" style="clear: both; text-align: center;"></div><br />
That's it – start the server, wait 3 minutes, and you have a private yet anonymous proxy.<br />
<br />
<h3>How much does it cost?</h3>If you are a new customer, you are entitled to a period of 12 months of free tier discount, reducing the costs significantly.<br />
Assuming that your business needs a proxy 50% of the time, and a monthly bandwidth usage of 50GB, the setup used in this post costs<b> $26</b> a month, or<b> $15</b> after discount.<br />
If you plan to keep the proxy online 100% of the time, it will cost you a total of <b>$30</b> a month, or <b>$12</b> after discount.<br />
The reason for this seemingly strange pricing is the cost of Elastic IP. It does not cost you to use it, but you pay when you keep it reserved without usage. <br />
<br />
<a href="http://www.blogger.com/post-edit.g?blogID=3119148538092600975&postID=5244246578524161128" name="cloudformation"><br />
</a><br />
<h3>AWS CloudFormation Template</h3>CloudFormation allows you to bundle all of the resources needed for an application launch into a single, automatic job. In our case, we have 3 resources: a security group, an elastic IP, and a server instance. The following template starts everything in one step.<br />
<br />
<code>{<br />
"AWSTemplateFormatVersion" : "2010-09-09",<br />
<br />
"Description" : "Instant Anonymizing proxy",<br />
<br />
"Parameters" : {<br />
"KeyName" : {<br />
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",<br />
"Type" : "String",<br />
"Default" : "MySSHKeypair"<br />
},<br />
"MyNetwork" : {<br />
"Description" : "Outbound IP Address of your corporate network",<br />
"Type" : "String",<br />
"Default" : "100.101.102.103"<br />
},<br />
"MyEIP" : {<br />
"Description" : "Existing Elastic IP",<br />
"Type" : "String",<br />
"Default" : "50.51.52.53"<br />
},<br />
"InstanceType" : {<br />
"Description" : "Type of EC2 instance to launch",<br />
"Type" : "String",<br />
"Default" : "t1.micro"<br />
}<br />
<br />
},<br />
<br />
"Mappings" : {<br />
"RegionMap" : {<br />
"us-east-1" : {"AMI" : "ami-8c1fece5"},<br />
"us-west-1" : {"AMI" : "ami-3bc9997e"},<br />
"eu-west-1" : {"AMI" : "ami-47cefa33"},<br />
"ap-southeast-1" : {"AMI" : "ami-6af08e38"},<br />
"ap-northeast-1" : {"AMI" : "ami-300ca731"}<br />
}<br />
},<br />
<br />
<br />
"Resources" : {<br />
"Ec2Instance" : {<br />
"Type" : "AWS::EC2::Instance",<br />
"Properties" : {<br />
"KeyName" : { "Ref" : "KeyName" },<br />
"InstanceType" : { "Ref" : "InstanceType" },<br />
"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],<br />
"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},<br />
"Tags" : [ <br />
{<br />
"Key" : "Name",<br />
"Value" : "MyProxy"<br />
}<br />
], <br />
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[<br />
"#cloud-config","\n",<br />
"\n",<br />
"packages:","\n",<br />
"- httpd","\n",<br />
"\n",<br />
"runcmd:","\n",<br />
"- echo listen 443 >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo ProxyRequests On >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo ProxyVia Block >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo \"<proxy *>\" >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo Order deny,allow >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo Deny from all >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo Allow from " , { "Ref" : "MyNetwork" } , " >> /etc/httpd/conf/httpd.conf","\n",<br />
"- echo \"</proxy>\" >> /etc/httpd/conf/httpd.conf","\n",<br />
"- service httpd restart","\n" ]]}}<br />
}<br />
},<br />
<br />
"InstanceSecurityGroup" : {<br />
"Type" : "AWS::EC2::SecurityGroup",<br />
"Properties" : {<br />
"GroupDescription" : "All ports access from my corporate network",<br />
"SecurityGroupIngress" : [ {<br />
"IpProtocol" : "tcp",<br />
"FromPort" : "0",<br />
"ToPort" : "65535",<br />
"CidrIp" : { "Fn::Join" : [ "/" , [ { "Ref" : "MyNetwork" } ,"32" ] ] }<br />
} ]<br />
}<br />
},<br />
"IPAssoc" : {<br />
"Type" : "AWS::EC2::EIPAssociation",<br />
"Properties" : {<br />
"InstanceId" : { "Ref" : "Ec2Instance" } ,<br />
"EIP" : { "Ref" : "MyEIP" }<br />
} <br />
}<br />
},<br />
<br />
"Outputs" : {<br />
"ProxyIP" : {<br />
"Description" : "The IP address for the newly created Proxy server",<br />
"Value" : { "Ref" : "MyEIP" }<br />
}<br />
}<br />
}<br />
</code><br />
<br />
<br />
<br />
<br />
<h3>Advanced proxy setup</h3><br />
You may consider a more customized Apache configuration. <br />
<br />
Optimizing your Apache installation for performance and security is a good idea. It means stripping out all unnecessary apache modules, keeping only the bare minimum for our required functionality, and modifying some apache directives for increased security. <br />
<br />
Adding better access control with some kind of user authentication is a good idea. Basic authentication is very easy to setup and adds another layer of security to your proxy. <br />
<br />
You should decide what to do with the server logs. If you keep them, then you need to define log a retention policy and configure Linux/Apache accordingly. But, maybe the best idea is to give up on logs altogether. After all, we wanted an anonymous proxy, didn't we?<br />
<br />
Dealing with the road warriors is another matter. You will have to either limit their proxy access via the enterprise network, or change the entire security scheme by using a VPN or another encrypted channel between the proxy and the end users. <br />
<br />
The downside to all of the advanced setup ideas is that you will most likely have to configure a private AMI and add more software to support your needs. A future post will deal with some advanced scenarios. <br />
<br />
<br />
<h3>How to configure your users' browsers</h3><br />
This is beyond the scope of this article, but I added a few links nonetheless. <br />
Here are the instructions for Firefox<br />
<a href="http://support.mozilla.com/en-US/kb/Options%20window%20-%20Advanced%20panel?s=proxy&as=s#w_network-tab">http://support.mozilla.com/en-US/kb/Options%20window%20-%20Advanced%20panel?s=proxy&as=s#w_network-tab</a> , here for Chrome<br />
<a href="http://www.google.com/support/chrome/bin/answer.py?answer=96815">http://www.google.com/support/chrome/bin/answer.py?answer=96815</a><br />
and here for Explorer <a href="http://support.microsoft.com/kb/135982%20">http://support.microsoft.com/kb/135982 </a><br />
<br />
In a corporate environment, you should also take a look at Proxy Automatic Configuration scripts <a href="http://en.wikipedia.org/wiki/Proxy_auto-config%20">http://en.wikipedia.org/wiki/Proxy_auto-config </a><br />
You can set up a code based PAC server, that can supply different PAC files to different users based on various criteria.<br />
<br />
<h3> Caveats</h3><br />
Do remember that a proxy like the one we just discussed only hides the origin IP address. It does not hide any cookies the user accumulated in previous browser sessions, and it certainly does not hide you if you login to a subscription based service with your corporate email. I recommend using a browser dedicated only to sensitive operations. At least use a private session - all modern browsers support this feature. For example, if you use Chrome, pressing <b>CTRL+Shift+n</b> opens a new, incognito session. <br />
One more thing: it turns out that Flash does not respect the browser's proxy settings.Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-18260630781897706542010-11-21T18:38:00.001+02:002010-11-21T18:39:24.037+02:00Azure managementIn the last days of October, <a href="http://blog.toddysm.com/2010/10/remote-desktop-connection-to-windows-azure-instance.html">Microsoft announced that it is going to release remote desktop access to Azure instances</a>. <br />
That is very nice. However, despite the fact that you will be an admin on the instance and will be able to install and do anything, all changes will be ephemeral.<br />
There is no way to save the modified image, and the next time it is restarted all changes will be gone.<br />
<br />
This is worse than Google App Engine, who which gives your applications the illusion of a single hypercomputing instance. You don't have a VM for every role, at least not knowingly. By decoupling the application runtime from running VM instances, Google is ready to use whatever computing platform innovations that will happen. Azure, on the other hand, still gives you the responsibility to manage scalability by yourself. <br />
<br />
This is also worse than EC2. In EC2 you have the option to bundle a running instance to a private image, so any modifications you make can be saved for future sessions. <br />
<br />
The only good thing about this feature is the ability to debug your application, and I don't underestimate this new capability.Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-1302771971996053972010-10-18T10:56:00.002+02:002011-06-03T11:58:00.581+03:00Red vs BlueA couple of days ago, I had the privilege to listen to a webcast by Hasan Rizvi, Senior Vice President for Oracle Fusion Middleware and Java Product.<br />
Hasan spoke about the new Oracle Exalogic product, and about how this new product is going to revolutionize the datacenter and provide cloud computing capabilities.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div>Unfortunately Mr. Rizvi's talk was not about cloud computing, public or private. Neither it was about datacenters. <b>All I could see was a sniper rifle targeting IBM's mainframe business.</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.xbox360fanboy.com/media/2006/06/red_vs_blue_uppercut.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://www.xbox360fanboy.com/media/2006/06/red_vs_blue_uppercut.jpg" /></a></div>Oracle and IBM have a long lasting relationship, framed by both intense rivalry and business cooperation. Although they competed in the database market, an Oracle database running on AIX was a favorite enterprise configuration. This stable state of affairs was not heavily disturbed when Oracle started foraging into the enterprise integration business. It's SOA family of products, running off the Oracle Application Server, were mostly used by heavily committed Oracle shops, and were no match for the stability, configurability and performance offered by IBM's WebSphere brand and the other leaders of the pack.<br />
<br />
All of this changed since 2007, with the acquisitions of BEA and Sun.<br />
BEA's dowry included the WebLogic server, vastly superior to Oracle's OAS, and the AquaLogic suite, which was a better line than Oracle's own. Both lines offer real competition to the WebSphere brand. BEA's legacy also included Jrockit and Tuxedo. <br />
<br />
Tuxedo deserves a special consideration: It is used to offload applications from IBM's mainframe into distributed systems, and combined with Oracle's aggressive marketing organization it may now pose a real threat to IBM mainframe business.<br />
However, not all mainframe shops run CICS. Many modern mainframe workloads are today based on Java, specifically on the WebSphere Application Server for the z/os. Until now, there was no real substitute for the benefits of MIPS, integration and security of WebSphere based Java on z/os. <br />
<br />
And then came Sun... With the merging of Sun, Oracle now has full control over Java, and a private line of servers and storage. This doesn't spell like good news for IBM, that is heavily invested in Java based technologies, and is a major storage vendor. <br />
It all culminated in the aforementioned Exalogic webcast. All of the comparisons, benchmarks and jargon used, were aimed at either current IBM mainframe users or mainframe wannabes who want to have the power of a mainframe on distributed systems. Even the price/performance table on slide 21 compares Exalogic with IBM's Power 795 server. Who else is likely to dole out $1,075,000 for a piece of hardware?<br />
<br />
And what did IBM do? It ditched its support of the unassociated Java development in the Apache Harmony project,<a href="http://www.sutor.com/c/2010/10/ibm-joins-the-openjdk-community/"> and went into the Java bed with Oracle</a>. <br />
<br />
<br />
See the recorded webcast and the slides here:<br />
<a href="http://w.on24.com/r.htm?e=244865&s=1&k=7C946C00C82CA0F93EA4E95A5A6BA196" target="_blank">http://w.on24.com/r.htm?e=244865&s=1&k=7C946C00C82CA0F93EA4E95A5A6BA196</a><br />
<br />
To download only the slides use this link<br />
<a href="http://event.on24.com/event/24/48/65/rt/1/documents/slidepdf/webcastexalogic1012b.pdf">http://event.on24.com/event/24/48/65/rt/1/documents/slidepdf/webcastexalogic1012b.pdf</a><br />
<br />
IBM gives up on Apache backed Java and joins Oracle<br />
<a href="http://www.sutor.com/c/2010/10/ibm-joins-the-openjdk-community/">http://www.sutor.com/c/2010/10/ibm-joins-the-openjdk-community/</a><br />
<br />
Oracle sues Google over Java standards<br />
<a href="http://news.cnet.com/8301-30684_3-20013546-265.html">http://news.cnet.com/8301-30684_3-20013546-265.html</a>Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-91385981074155757342010-10-01T15:55:00.000+02:002010-10-01T15:55:39.448+02:00Cloud Security Architecture MattersNot all cloud services are created equal. Clouds are architected by flesh and blood men and women, and because the are no cloud standards yet, the architectural choices are invariably different. <br />
A couple of months ago I was looking for differences between various cloud vendors' API implementations, to show at a local <a href="http://www.owasp.org/">OWASP </a>meeting.<br />
<br />
And differences I found. The most important difference is in the way users present their credentials to the service, and how the service ensures that it receives a valid request. <br />
<br />
To make things comparable, let's look at the way 3 leading cloud vendors (<a href="http://aws.amazon.com/">Amazon Web Services</a>, <a href="http://www.gogrid.com/">GoGrid</a> and <a href="http://www.rackspace.com/">RackSpace</a>) authenticate and authorize usage of their cloud resources.<br />
<br />
We care about the way the service ensures that it receives a valid request, because we want to minimize the risk of account hijacking, and to minimize the risk of action replay.<br />
<br />
A hijacked account means that a third party does unintended actions using resources that are tagged as belonging to the account owner. The third party can use the account for unwanted activity, such as distribution of illegal materials or promotion of spam, and the bill is handed out to the account owner. <br />
<br />
Action replay means that someone may repeatedly send identical commands to the cloud infrastructure on behalf of the account owner. At best it is just a nuisance that may cost a little. <br />
At worst, it may enable a third party to replace valid content with invalid, or even malicious content. <br />
<br />
I am going to compare the basic action of listing the contents of the cloud objects container. <br />
To make long things short, here is a brief description of the results. <br />
<ul><li>All APIs make use of the HTTP protocol, with some custom extended headers. </li>
<li>All APIs require a public access identifier and a secret key shared between the account owner and the vendor. </li>
<li>AWS does not need the secret key in plaintext. It requires a unique signature for each request, making it extremely difficult to do any harm to the account, even if the request is somehow intercepted on the way. </li>
<li>GoGrid does not need the secret key in plaintext. It requires a signature as part of an authentication request, and returns a security token that is valid for a period of 10 minutes, and for all actions within 10 minutes. </li>
<li>RackSpace requires the secret key in plaintext as part of an authentication request, and returns a security token that is valid for a period of 24 hours, and for all actions within 24 hours. </li>
</ul><a name='more'></a><br />
<h3>Amazon S3 API security</h3>AWS S3 requires a unique signature for each request, making it extremely difficult to do any harm to the account, even if the request is somehow intercepted on the way. <br />
Here is how the HTTP request to list S3 buckets looks like: <br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">GET / HTTP/1.1<br />
Host: s3.amazonaws.com<br />
Date: Fri, 05 Feb 2010 12:00:00 GMT<br />
Authorization: AWS 15B4D3461F177624206A:xQE0diMbLRepdf3YB+FIEXAMPLE=</div><br />
The authorization header is the key to the request. It is comprised of a constant "AWS", a public access key that is managed on the account's Amazon control panel, and a base64 encoded hash that is unique to this specific request.<br />
In our case, the public access key is <span style="background-color: #cccccc;">15B4D3461F177624206A</span> and the hash is <span style="background-color: #cccccc;">xQE0diMbLRepdf3YB+FIEXAMPLE=</span><br />
<br />
To better understand the hash, here is a pseudocode representation of the way it is built. <br />
The SecretAccessKeyID is a 320 bit secret string that is shared between the account owner and AWS. <br />
The end result is a hash that contains the MD5 encoding of the request, and all the request headers including a timestamp. <br />
The timestamp must be within 15 minutes of the AWS internal time. <br />
<br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;<br />
<br />
Signature = Base64( HMAC-SHA1( UTF-8-Encoding-Of( SecretAccessKeyID, StringToSign ) ) );<br />
<br />
StringToSign = HTTP-Verb + "\n" +<br />
Content-MD5 + "\n" +<br />
Content-Type + "\n" +<br />
Date + "\n" +<br />
CanonicalizedAmzHeaders +<br />
CanonicalizedResource;<br />
<br />
CanonicalizedResource = [ "/" + Bucket ] +<br />
<HTTP-Request-URI, from the protocol name up to the query string> +<br />
[ sub-resource, if present. For example "?acl", "?location", "?logging", or "?torrent"];</div><br />
The end result is that each and every request has a unique signature, and there is no separate authentication and authorization step like with one of the following vendors. <br />
<br />
<h3>GoGrid API security</h3>GoGrid requires a signature as part of an authentication request, and the timestamp that must be encoded inside the signature makes the request valid for a period of 10 minutes. The same signature can be reused making it also valid for all actions within 10 minutes. This makes the account vulnerable for 10 minutes at a time, but the requirement to use https mitigates the risk.<br />
GoGrid uses a REST interface for its API. <br />
Here is how the HTTP request chain to list GoGrid objects looks like:<br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">https://api.gogrid.com/api/common/lookup/list<br />
?lookup=lookups<br />
&sort=name<br />
&asc=true<br />
&api_key=SK1mndkKSjsALKSExdsj30<br />
&sig=Lsmdfj3kw99sdklnasdkjhds<br />
&v=1.1</div><br />
The api_key is the public identifier, and the signature is a MD5 token created by a method like the php snippet listed below. <br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;"><?php<br />
$apikey = 'SK1mndkKSjsALKSExdsj30';<br />
$secret = ‘SharedSecretBetweenMeAndGoGrid';<br />
$timestamp = time();<br />
$sig = md5($apikey . $secret . $timestamp);<br />
?></div><br />
<h3>RackSpace API security</h3>RackSpace requires the secret key in plaintext as part of an authentication request, and returns a security token that is valid for a period of 24 hours, and for all actions within 24 hours. This method degrades the secret key to being just a simple password, and creates a double risk:<br />
<ol><li>The 24 hour validity of a security token is a very long period, during which the holders of the token can do whatever action they want to do with the account. </li>
<li>During login, the secret key is passed in plaintext in the request. Although the request must be over SSL, this still makes the RackSpace API the least secure one among its competitors. </li>
</ol>Here is how the login session looks like:<br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">GET /v1.0 HTTP/1.1<br />
Host: auth.api.rackspacecloud.com<br />
X-Auth-User: jdoe<br />
X-Auth-Key: a86850deb2742ec3cb41518e26aa2d89</div><br />
<br />
The response includes a number of tokens that must be used in subsequent requests. I color coded the relevant ones so it is easier to follow.<br />
<br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">HTTP/1.1 204 No Content<br />
Date: Fri, 05 Feb 2010 15:32:21 GMT<br />
Server: Apache<br />
X-Server-Management-Url: https://servers.api.rackspacecloud.com/v1.0/35428<br />
X-Storage-Url: https://<b><span style="color: #38761d;">storage.clouddrive.com</span><span style="color: blue;">/v1/CloudFS_9c83b-5ed4</span></b><br />
X-CDN-Management-Url: https://cdn.clouddrive.com/v1/CloudFS_9c83b-5ed4<br />
X-Auth-Token: <b style="color: red;">eaaafd18-0fed-4b3a-81b4-663c99ec1cbb</b><br />
Content-Length: 0<br />
Content-Type: text/plain; charset=UTF-8</div><br />
After we successfully authenticated, we must use the security token from the <span style="font-family: "Courier New",Courier,monospace;">X-Auth-Token</span> header, and parts of the URL from the <span style="font-family: "Courier New",Courier,monospace;">X-Storage-Url</span> header. <br />
<br />
The request to list cloud files looks now like this<br />
<br />
<div style="background-color: #cccccc; font-family: "Courier New",Courier,monospace;">GET <b style="color: blue;">/v1/CloudFS_9c83b-5ed4</b>?format=json HTTP/1.1<br />
Host: <b style="color: #134f5c;">storage.clouddrive.com</b><br />
X-Auth-Token: <b style="color: red;">eaaafd18-0fed-4b3a-81b4-663c99ec1cbb</b></div><br />
<br />
<h3>Summary</h3>I think that security matters, therefore Amazon comes up on top in this comparison. This gives me yet another reason to use AWS for me and my clients.Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-25087385659672402472009-09-13T17:08:00.005+03:002009-09-13T18:27:24.132+03:00How to get AWStats to show Intranet location statsI was asked by Teva's HR department to do something about the HR intranet "Portal". The requirement is to know how the HR intranet is used, based on the 3 W rules: What, When, and Where. Teva has multiple offices in Israel, and HR went through a major reorganization and restructuring of its services and methodology during the past years.<br /><br />It is good they came to me first, as some of my colleagues would have made it into a multi-million dollars data warehouse analysis project :)<br /><br />Having good experience with <a href="http://awstats.sourceforge.net/">Awstats</a>, I decided to use this tool to analyze the HR intranet logs.<br /><br />AWstats provides good enough stats about the What and the When, but I hit a blank wall regarding the Where. There are several plugins that enable geo-ip analysis, but according to the documentation, all of them are <a href="http://xkcd.com/55/">useless</a> for intranet only log files.<br /><br />The plugins from <a href="http://www.maxmind.com/">MaxMind</a> use a proprietary format, and do not include my 10.* network anyway.<br /><br />This left me with the IPFree database and plugin as the only viable option to add branch awareness into the intranet stats.<br /><br />Geo-IPfree can be found on <a href="http://search.cpan.org/search?query=ipfree&mode=all">CPAN</a>. First thing is to download it.<br />We need a couple of tools that may or may not be included in the IPfree package. <a href="http://cpansearch.perl.org/src/BRICAS/Geo-IPfree-0.6/misc/txt2ipct.pl">txt2ipct.pl</a> and <a href="http://cpansearch.perl.org/src/BRICAS/Geo-IPfree-0.6/misc/ipct2txt.pl">ipct2txt.pl</a> are required for us to make the modifications to the IPfree database.<br />First, extract the IP database from <span style="font-style: italic;">ipscountry.dat</span> into an editable text file.<br /><pre>perl ipct2txt.pl ./ipscountry.dat ./ips-ascii.txt<br /></pre>Let's suppose that you have 3 branches: London, Rome and Tel-Aviv. Your WAN network segments are respectively 10.1.*, 10.2.*, 10.3.*, and must be mapped into the IPfree database.<br /><br />Open the <span style="font-style: italic;">ips-ascii.txt</span> file with your favorite text editor and find your LAN/WAN IP range.<br />You have a line that looks like this<br /><pre>ZZ: 10.0.0.0 10.255.255.255</pre>We will map your locations into codes Z1, Z2 and Z3, because there are no such ISO country codes.<br /><br />So, replace your ZZ line with these five<br /><pre>ZZ: 10.0.0.0 10.0.255.255<br />Z1: 10.1.0.0 10.1.255.255<br />Z2: 10.2.0.0 10.2.255.255<br />Z3: 10.3.0.0 10.3.255.255<br />ZZ: 10.4.0.0 10.4.255.255</pre>Just for safety, rename your current <span style="font-style: italic;">ipscountry.dat</span> file, and execute<br /><pre>perl txt2ipct.pl ./ips-ascii.txt ./ipscountry.dat</pre>Now, create a new Geo folder under the plugins folder in the awstats installation, and copy the <span style="font-style: italic;">IPfree.pm</span> and <span style="font-style: italic;">ipscountry.dat</span> files into the new folder.<br /><br />We have to modify the <span style="font-style: italic;">lib/domains.pm</span> file to recognize the new Z1, Z2 and Z3 domains. Just add them to the end of the list, and keep the new domain names in lower case. The last line of the <span style="font-style: italic;">domains.pm</span> file will now look like this<br /><pre>'zm','Zambia','zr','Zaire','zw','Zimbabwe', 'z1', 'London', 'z2', 'Rome', 'z3', 'Tel Aviv'</pre>The last thing to do is to turn on the geoipfree plugin in the awstats configuration file.<br /><br />Have fun!!<br /><br />P.S.<br />Maybe I will initiate the multi-million dollar web data warehouse project after all ...Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com1tag:blogger.com,1999:blog-3119148538092600975.post-25711049942000284662009-09-02T23:19:00.007+03:002009-09-02T23:26:38.652+03:00Restoring my mobile phone backup to a new phone<div>My Sony Ericsson K610 mobile phone dropped dead. </div><div>The service guy removed my SIM and memory card from the fresh corpse, and stared me in the eye: "Do you have anything stored on the phone, sir? There are no phone numbers stored on the SIM"</div><div>I remembered storing everything on the phone instead of on the SIM. </div><div>I also remembered using Sony's PC Suite to backup the phone contents a couple of days ago.</div><div>"Don't worry, got backup" - I told the service guy. </div><div><br /></div><div><span class="Apple-style-span" style="font-size:large;"><b><span class="Apple-style-span" style="font-family:arial;">Problem</span></b></span></div><div>The Sony Ericsson PC Suite recognized the new phone easily. I selected Tools/Backup&Recovery and started to shiver. </div><div>No backup was listed, and my sync to Outlook was 4 months old.</div><div><br /></div><div><span class="Apple-style-span" style="font-size:large;"><span class="Apple-style-span" style="font-family:arial;"><b>Research</b></span></span></div><div>First, I determined that I must find the old backup. I created a new backup for my new phone, called it "backup2" and learned that </div><div>the phone backups are stored with extension .dbk inside "My Documents", in folder </div><div>"Sony Ericsson\Sony Ericsson PC Suite\Phone backup".</div><div>As if by magic, my old backup file also turned out in this folder. </div><div>I realized that the file contents must be tagged with the phone information.</div><div><br /></div><div>Opening the backup file with a text editor didn't help. It loooked almost completely binary. </div><div>However, there was a slight resemblance to something I saw before...</div><div><br /></div><div>I created a copy of the dbk file, and renamed it with a zip suffix. That did the trick. </div><div><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghDA57kfh5cXLBIzDvrKqBjMuQeQLXvm8A8ahx-eUP6MizO7ZGy3tH_k2uqNZDwPaJ2Zy6a9pTaWOEAckSNMRHRWJFZ2k1ah-wpYfwPxfwjWrMY87Ku_GvAM7pJlt7MHrUXxSZ6XfuBdPh/s320/old-shalom.zip.png" style="cursor:pointer; cursor:hand;width: 320px; height: 153px;" border="0" alt="" id="BLOGGER_PHOTO_ID_5376968760254095650" /></div><div><span class="Apple-style-span" style="font-size:large;"><b><span class="Apple-style-span" style="font-family:arial;">Solution</span></b></span></div><div>The key is in the phoneID.txt file. </div><div><br /></div><div><span class="Apple-style-span" style="font-family:'courier new';">Version=3.0</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">DeviceManufacturer=Sony Ericsson</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">DeviceModel=K610</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">HeartBeat=HB1-06</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">IMEI=3546xxxxxxxxxx</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">PhoneName=My K610</span></div><div><br /></div><div>I replaced the file with the one found in the dummy backup of the new phone</div><div><br /></div><div><span class="Apple-style-span" style="font-family:'courier new';">Version=3.0</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">DeviceManufacturer=Sony Ericsson</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">DeviceModel=W595</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">HeartBeat=HB1-07</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">IMEI=3529yyyyyyyyyyyy</span></div><div><span class="Apple-style-span" style="font-family:'courier new';">PhoneName=My W595</span></div><div><br /></div><div>Changed the zip file extension back to dbk. Now Sony Ericsson PC Suite showed the old backups and I was able to restore all of my important information to the new phone.</div><div><br /></div>Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-55559871912745058322009-01-26T12:40:00.007+02:002009-01-27T19:03:37.520+02:00Backing up to the cloudMy previous post was about using the <a href="http://jets3t.s3.amazonaws.com/index.html">Jets3t </a>synchronize tool on an iSeries server.<br />While synchronize works quite well, it is not flexible enough for what I had in mind. To be able to integrate Amazon S3 with a real life backup and DRP policy, I want a set of atomic functions that enable me to simply transfer particular data objects between S3 and the server.<br />Using the Jets3t rich set of S3 objects and the samples provided by the author, I found it quite easy to write additional proof of concept tools.<br /><br />I now have Create Bucket, Upload Single file, and DownLoad Single File programs. These are all barebones tools, use them at your own risk..<br /><br />All tools use the iUtils helper tool that provides for central management of parameters and credentials. In fact, iUtils uses exactly the same configuration file Synchronize does.<br /><br />I timed the Upload and Download tools with a 500MB savefile, and on my network it took less than 30 minutes to go either way.<br /><br />The usage is quite simple:<br /><ul><li>CreateBucket requires a single parameter, the new bucket name.</li><li>uploadSingleFile requires two parameters: the target bucket name, and the file name to upload</li><li>DownLoadSingleFile requires two parameters: the source bucket name, and the file name to download.</li></ul>Get the extended toolset files <a href="http://downloads.venera.com.s3.amazonaws.com/jets3t/jets3t-hacks.zip">here</a>. As I said before, these are barebones tools and there is a lot to improve. There is practically no graceful error handling, no compression and no encryption.<br /><br /><span style="font-weight: bold;font-size:130%;" >Using Amazon S3 in an iSeries backup/recovery scenario</span><br />S3 can be used as an offsite backup for some of your data, maybe even for all of your data. A library can easily be saved to a save file, and the save file uploaded to S3 storage until needed, at which time it can be downloaded and restored to your iSeries.<br /><br />For example, look at the following set of commands that saves changed objects to a save file, <a href="http://www.itjungle.com/mgo/mgo102302-story01.html">zips it</a> and sends it to S3. The extra zipping is required because the data compression built into the iSeries save commands is not very efficient, and because I have not implemented it yet as an integral part of the upload tool (although the functionality exists in the jets3t package).<br /><pre><br />/* create save file */<br />crtsavf s3wk200901<br /><br />/* backup objects changed since December 31 to save file */<br />savchgobj obj(*all) lib(mylib) dev(*savf) savf(s3wk200901)<br /> refdate('31/12/2008') clear(*all) dtacpr(*high)<br /><br />/* copy save file to IFS */<br />cpytostmf frommbr('/qsys.lib/mylib.lib/s3wk200901.file')<br /> tostmf('/backups/s3wk200901.savf') stmfopt(*replace)<br /><br />/* further compress save file */<br />qsh cmd('jar cMf /backups/s3wk200901.zip /backups/s3wk200901.savf')<br /><br />/* upload compressed save file to S3 */<br />qsh cmd('cpytos3.sh mybackup /backups/s3wk200901.zip')<br /></pre>Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com3tag:blogger.com,1999:blog-3119148538092600975.post-62060498069185925552009-01-25T18:35:00.010+02:002009-01-28T12:43:37.266+02:00Synchronize iSeries files with Amazon S3 virtual storageAmazon Web Services, or AWS, is one of the new and hot promises to revolutionize IT.<br />I subscribed to AWS almost two years ago, but started to really use it only on last October.<br /><br />AWS provides huge benefits to IT, developers and web site owners.<br />For IT, other than the ability to dynamically allocate servers and pay for the actual server utilization, AWS provides the ability to have a reliable, secure remote storage.<br /><br />I went out to see whether Amazon S3 lives up to my expectations and can play a significant part in a disaster recovery plan. Some smart folks already <a href="http://jeremy.zawodny.com/blog/archives/007624.html">did some math</a> a while ago, and the economic drivers for using S3 are now higher than ever.<br /><br />It turns out that there are quite a few S3 tools. For S3 management I ended up using the <a href="http://www.rjonna.com/ext/s3fox.php">S3 organizer</a>, a free FireFox extension, and the <a href="http://jets3t.s3.amazonaws.com/applications/applications.html#cockpit">S3 Cockpit tool</a>, which is actually part of a larger package, Jets3t.<br /><br />There is a major difference between personal tools and tools appropriate to use in a datacenter. The biggest challenge for me was to find those tools that support multiple platforms, and that once configured, work in unattended mode with no hassle.<br /><br />I settled for <a href="http://jets3t.s3.amazonaws.com/index.html">Jets3t</a> - a set of libraries and tools written in Java.<br />On a windows server, installation is straightforward: just unzip the distribution archive, set up some environment variables and you are ready to start. Of course, Java is a prerequisite.<br /><br />Jets3t consists of a set of libraries that wrap the S3 API with Java objects, and 4 working tools:<br /><ul><li>Cockpit, a GUI application/applet for viewing and managing the contents of a S3 account.</li></ul><ul><li>Synchronize, a console application for synchronizing directories on a computer with an Amazon S3 account.</li></ul><ul><li>CockpitLite, a GUI application/applet for viewing and managing the contents of an S3 account, where the S3 account is not owned by the application's user directly but is made available via the Gatekeeper servlet.</li></ul><ul><li>Uploader, a wizard-based GUI application/applet that S3 account holders (Service Providers) may provide to clients to allow them to upload files to S3 without requiring access to the Service Provider's S3 credentials.</li></ul> The most promising tool, as far as I am concerned, is Synchronize.<br />Working in a manner similar to rsynch, this tool can keep an S3 bucket and a local folder in synch with each other, while providing both compression and encryption in the cloud.<br /><br />After making it work on my PC, I set out to make it work on an iSeries server.<br /><br /><span style="font-weight: bold;font-size:130%;" >Setting up on iSeries</span><br />The first thing to do is to install Jets3t on your server, in any IFS directory you like.<br />I placed it in the /QIBM/UserData/aws/jets3t folder.<br /><br />Jets3t relies on some environment variables to be defined, specifically JETS3T_HOME and JAVA_HOME.<br />Make sure both are defined before executing the synchronize.sh script.<br />I set up both as global variables by running these iSeries commands:<br /><pre><br />addenvvar JETS3T_HOME '/QIBM/UserData/aws/jets3t' LEVEL(*JOB) REPLACE(*YES)<br />addenvvar JAVA_HOME '/QIBM/ProdData/Java400/jdk15' LEVEL(*JOB) REPLACE(*YES)<br /></pre><br />Additionally, the unix shell scripts that came with the package assume that the shell is in /bin/sh<br />This probably is not true on your iSeries server, so either modify the shebang lines or run this command to solve the problem<br /><br /><code>ADDLNK OBJ('/usr/bin') NEWLNK('/bin') LNKTYPE(*SYMBOLIC)</code><br /><br />Synchronize supports file compression and encryption, as well as various modes of operation that make synching a breeze. For example, to synch the /home/shalom/personal folder with a S3 bucket called shalom-personal, you will execute<br /><br />synchronize.sh UP shalom-personal /home/shalom/personal/<br /><br />You do not have to create the bucket prior to using synchronize, it will create the bucket if it does not exist. <span style="font-style: italic;">( tip: use the --noaction switch to verify your actions before commiting to any changes. )</span><br /><br />To learn how to actually use synchronize, <a href="http://jets3t.s3.amazonaws.com/applications/synchronize.html">read the manual</a>.<br /><br />To make it easier on the iSeries folks, <a href="http://downloads.venera.com.s3.amazonaws.com/jets3t/s3synch400.zip">here is</a> a CL program and CMD that wrap the synchronize tool in a familiar interface.<br /><br />In the next post I will explain how to extend Jets3t to a real on-demand storage device with some Java programming.Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com2tag:blogger.com,1999:blog-3119148538092600975.post-20421647509760606332009-01-22T17:24:00.007+02:002009-01-26T12:39:40.427+02:00Processing XML on iSeries, Part 2<span style="font-size:85%;">Read <a href="http://shalomcarmel.blogspot.com/2009/01/processing-xml-on-iseries-aka-system-i.html">here </a>the first part of this article, about XML and database integration using Saxon. </span><br /><br />In last October I looked at <a href="http://www.datadirect.com/products/xquery/index.ssp">DataDirect's XQuery tools</a>. These guys are serious. Their web site contains one of the best Xquery references on the web today.<br /><br />If you are an iSeries professional, you may ask yourself WTH is <a href="http://www.w3.org/TR/xquery/">xquery</a> and why you should care.<br /><br />Xquery is a bit more than yet another XML processing language.<br /><br />It attempts to deal with XML files in a manner similar to relational databases, and this is good. Many developers find it cumbersome to program an event driven process to deal with basically recurring events, like records in a XML message. It seems easier to write a couple of FOR loops (XQuery) rather than trapping template occurences (xslt). For some tasks, a plain procedural program flow rules. I had to develop some pretty complex XML transformation processes lately, and I found using xquery to be both more productive and more readable than XSLT. It is also easier to understand for non-xml experts.<br /><br />DataDirect support the standard xquery model exceptionally well. Their XML studio shines, and they also extended the standard and added database support, the result being that now it is really easy to define interactions between XML and databases.<br /><br />Like almost everything Java, I like to see if it is feasible to run it on iSeries, especially as I immediately realized the benefits of native execution: finally a tool to enable XML integration with DB2 tables, <span style="font-weight: bold;">NOT WRITTEN IN RPG</span>, maintainable by non-iseries buffs, and able to be inserted into native iseries business processes.<br /><br />The bulk of my experience is already detailed <a href="http://forums.datadirect.com/ddforums/thread.jspa?threadID=2581&tstart=0">here </a> and I will not repeat it in this blog. I can just say that it works.<br /><br />However, not everything is bright and shiny with DataDirect XQuery.<br /><br />First, there is performance: on my 520 it sucks. A program that took a couple of seconds on a PC runs for more than 30 seconds. There are simply too many dependencies and jars being loaded to execute a single XQuery program. I believe that this issue can be addressed by not spawning a distinct xquery instance for each xml file, but I haven't got to do anything about it.<br /><br />Then, there is the price... I will be fair to DataDirect and not disclose the price they wanted for an execution license on a System i model 520, but it was a bit on the high side. Had they asked for a price similar to their X86 license plan, we would have almost certainly bought it....Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0tag:blogger.com,1999:blog-3119148538092600975.post-79104689946103934752009-01-04T14:36:00.021+02:002009-01-29T16:32:09.647+02:00Processing XML on iSeries, Part 1I've always felt a bit sorry for the poor folks who insist on using <a href="http://en.wikipedia.org/wiki/IBM_RPG">RPG</a> to write new code on the <a href="http://en.wikipedia.org/wiki/AS/400">System i</a> platform. I mean no offense, but IMHO clinging to outdated technology and claiming it superior is not a smart move. RPG is simply a very bad fit for many modern information processing tasks. Like processing XML.<br /><br />Creating XML files is easy. You can think of it as a complex report, and as such a XML file can be created by a plain RPG II program with some O record wizardry.<br /><br />On the other hand, reading an XML file file is complex. There's white space to ignore, completion checks to do, schemas to validate, DTD to test against, naming spaces, private tags, encodings to parse, and more.<br /><br />On the <a href="http://en.wikipedia.org/wiki/Gripping_hand">gripping hand</a>, we can limit our problem to getting the data out of XML and into data entities that are more familiar to System i developers, like database tables.<br /><br />Fortunately, there are tools that let you map the relationship between data in XML and data in a table. Some are custom made for the System i and geared for RPG developers. I am deliberately going to ignore them in this post. If I have a choice, I prefer to use ubiquitous technology, that is easy to understand by hordes of programmers. Other solutions require deployment of ESB / ETL / EAI or similar infrastructures. While I am a big fan of ESB, sometimes you need a local solution rather than depend on external processes and processing, not to mention the costs of integration, licenses, and servers.<br /><br />That leaves us little choice but to focus on Java based tools.<br /><br />The first one is Saxon, to be found here <a href="http://www.saxonica.com/">http://www.saxonica.com</a> .<br />Saxon is among the leading XML transformation tools. In fact, it is embedded into a very large number of commercial XML processing tools and libraries.<br /><br />Saxon has an extension, called Saxon-SQL, that lets you embed SQL data modification statements into a XSLT transformation stylesheet.<br />You need Java at version of at least 1.4 , and the Saxon distribution. The free Saxon-B is good enough for our needs, but I recommend that you get <a href="http://www.saxonica.com/erol/erol.html">Saxon-SA</a> - at <b>£300.00</b> per server it is a real bargain.<br />The JT400 package is also required in order to use JDBC connections to the System i database.<br /><br />I unzipped the entire distribution into a new directory: <span style="font-style: italic;">/QIBM/UserData/saxon</span><br /><br />Now let's peek at the XML and database integration task.<br />Let's suppose that we are to insert into the Work Order application table data from XML.<br /><br />The XML looks like this:<br /><blockquote><?xml version="1.0"?><br /><WO_OUT><br /><ROW> <br /><WO>U10002</WO> <br /><ITEM>80012470</ITEM><br /><FACILITY>PL</FACILITY><br /><START_DATE>20081126</START_DATE><br /><DUE_DATE>20081129</DUE_DATE><br /><QUANTITY>654321</QUANTITY><br /><TV_TRANSACTION_ID>85246786</TV_TRANSACTION_ID><br /><INTERFACE_TRANSFER_DATE>20081015</INTERFACE_TRANSFER_DATE><br /><INTERFACE_TRANSFER_TIME>110903</INTERFACE_TRANSFER_TIME><br /></ROW><br /></WO_OUT><br /></blockquote><br />The mapping to the data table columns is:<br /><br />ITEM --> TCPROD<br />FACILITY --> TCFAC<br />START_DATE --> TCRDTE<br />DUE_DATE --> TCDDTE<br />QUANTITY --> TCQORD<br />TV_TRANSACTION_ID --> TCTRID<br />INTERFACE_TRANSFER_DATE --> TCDATE<br />INTERFACE_TRANSFER_TIME --> TCTIME<br /><br />The following XSLT maps the XML values into the corresponding database columns. It is based on the sample that comes with Saxon.<br /><br />Notice the following special interest items:<br /><ul><li>The stylesheet refers the DB2 JDBC driver. If you provide the correct library at runtime, this code can be executed on the System i server as well as on an external server. We will use the JT400Native.jar library for internal processing</li><li>The system name, user, password and library are resolved at runtime thru external parameters. The user and password are provided as part of the connection string.<br /></li><li><span style="font-style: italic;">normalize-space</span> is used to take care of extra white space like spaces, tabs, carriage returns</li><li>The <span style="font-style: italic;">INTERFACE_TRANSFER_DATE</span> and <span style="font-style: italic;">INTERFACE_TRANSFER_TIME</span> values are optional and possibly will not exist. Therefore <span style="font-style: italic;">TCDATE </span>and <span style="font-style: italic;">TCTIME </span>must be manipulated to set numeric defaults. </li><li>I recommend to set all database fields to allow null values in case of more missing data.</li><li>Do not use 1 char fields. <a href="http://sourceforge.net/forum/forum.php?thread_id=2642643&forum_id=94027">All char fields must have length of at least 2</a>.<br /></li><li>The database table should be journalled - there is an implicit commit.<br /></li></ul><br /><span style="color: rgb(51, 51, 255);"><pre><br /><?xml version="1.0" encoding="UTF-8"?><br /><xsl:stylesheet version="2.0"<br />xmlns:xsl="http://www.w3.org/1999/XSL/Transform"<br />xmlns:xs="http://www.w3.org/2001/XMLSchema"<br />xmlns:fn="http://www.w3.org/2005/xpath-functions"<br />xmlns:sql="java:/net.sf.saxon.sql.SQLElementFactory"<br />xmlns:saxon="http://saxon.sf.net/"<br />extension-element-prefixes="saxon sql"><br /><br /><!-- insert your database details here, or supply them in parameters --><br /><xsl:param name="driver" select="'com.ibm.as400.access.AS400JDBCDriver'"/><br /><xsl:param name="LIBRARY"/><br /><xsl:param name="SYSNAME"/><br /><xsl:param name="SYSUSER"/><br /><xsl:param name="SYSPASSWORD"/><br /><br /><xsl:param name="database"<br />select="concat('jdbc:as400://', $SYSNAME,<br /> ';user=' , $SYSUSER, ';password=',<br /> $SYSPASSWORD, ';naming=system;libraries=', $LIBRARY)"/><br /><xsl:param name="user"/><br /><xsl:param name="password" /><br /><br /><!-- This stylesheet writes the book list to a SQL database --><br /><br /><xsl:variable name="count" select="0" saxon:assignable="yes"/><br /><br /><xsl:output method="xml" indent="yes"/><br /><br /><xsl:template match="WO_OUT"><br /><xsl:if test="not(element-available('sql:connect'))"><br /><xsl:message>sql:connect is not available</xsl:message><br /></xsl:if><br /><br /><xsl:message>Connecting to<br /> <xsl:value-of select="$database"/>...</xsl:message><br /><br /><xsl:variable name="connection"<br /> as="java:java.sql.Connection" xmlns:java="http://saxon.sf.net/java-type"><br /><sql:connect driver="{$driver}" database="{$database}" <br /> user="{$user}" password="{$password}"><br /><xsl:fallback><br /> <xsl:message terminate="yes">SQL extensions are not installed</xsl:message><br /></xsl:fallback><br /></sql:connect><br /></xsl:variable><br /><br /><xsl:message>Connected...</xsl:message><br /><br /><xsl:apply-templates select="WO_OUT"><br /><xsl:with-param name="connection" select="$connection"/><br /></xsl:apply-templates><br /><br /><xsl:for-each select="ROW"><br /><sql:insert connection="$connection" table="KFXT"><br /> <sql:column name="TCCMPN" select="normalize-space(WO)"/><br /> <sql:column name="TCPROD" select="normalize-space(ITEM)"/><br /> <sql:column name="TCFAC" select="normalize-space(FACILITY)"/><br /> <sql:column name="TCRDTE" select="normalize-space(START_DATE)"/><br /> <sql:column name="TCDDTE" select="normalize-space(DUE_DATE)"/><br /> <sql:column name="TCQORD" select="normalize-space(QUANTITY)"/><br /> <sql:column name="TCTRID" select="normalize-space(TV_TRANSACTION_ID)"/><br /><br /> <sql:column name="TCDATE"<br /> select="if (string-length(normalize-space(INTERFACE_TRANSFER_DATE)) != 0)<br /> then normalize-space(INTERFACE_TRANSFER_DATE) else '0.0'"/><br /> <sql:column name="TCTIME"<br /> select="if (string-length(normalize-space(INTERFACE_TRANSFER_TIME)) != 0)<br /> then normalize-space(INTERFACE_TRANSFER_TIME) else '0.0'"/><br /><br /></sql:insert><br /><saxon:assign name="count" select="$count+1"/><br /></xsl:for-each><br /><br /><xsl:message>Inserted <xsl:value-of select="$count"/> records.</xsl:message><br /><sql:close connection="$connection"/><br /><br /></xsl:template><br /><br /></xsl:stylesheet><br /></pre></span><br /><br />To execute this, you will have to run the QSH java command:<br /><ul><li>Set the classpath to include saxon9.jar, saxon9-sql.jar and jt400Native.jar.</li><li>We will use localhost and the current user credentials. If you connect from a remote server, you will need the actual user and password.</li></ul>here it is embedded in a shell script:<br /><b><pre><br />#!/usr/bin/qsh<br />CP=/QIBM/UserData/Saxon/saxon9.jar<br />CP=$CP:/QIBM/UserData/Saxon/saxon9-sql.jar<br />CP=$CP:/QIBM/ProdData/OS400/jt400/lib//jt400Native.jar<br />java -Djava.version=1.5 -cp $CP net.sf.saxon.Transform <br /> input.xml insert-table.xslt LIBRARY=MYLIB SYSNAME=localhost<br /> SYSUSER=*current SYSPASSWORD=*current<br /><br /></pre></b><br /><br />and here it is in a CL program<br /><br /><b><pre><br />PGM<br />DCL &CP *CHAR 1024<br />dcl &XQ *char 1024<br />dcl &PATH2LIBS *char 1024 '/QIBM/UserData/Saxon'<br />dcl &PATH2JDBC *char 1024 '/QIBM/ProdData/OS400/jt400/lib'<br /><br />CD 'myDirectory'<br /><br />chgvar &XQ ('net.sf.saxon.Transform')<br />CHGVAR VAR(&CP ) VALUE( +<br /> &PATH2LIBS *tcat '/saxon9.jar' *tcat ':' *tcat +<br /> &PATH2LIBS *tcat '/saxon9-sql.jar' *tcat ':' *tcat +<br /> &PATH2JDBC *tcat '/jt400Native.jar' *tcat ':' *tcat +<br /> '.' +<br /> )<br /><br />JAVA CLASS(&XQ) +<br /> CLASSPATH(&CP) +<br /> parm( +<br /> 'input.xml' +<br /> 'insert-table.xslt' +<br /> 'LIBRARY=MYLIB' +<br /> 'SYSNAME=localhost' +<br /> 'SYSUSER=*current' +<br /> 'SYSPASSWORD=*current' +<br /> ) +<br /> PROP((java.version 1.5)) +<br /> OUTPUT(* )<br /><br />RETURN<br />ENDPGM<br /></pre></b>Shalom Carmelhttp://www.blogger.com/profile/15201874373375844470noreply@blogger.com0