Gregor's Ramblings
Home Patterns Ramblings Articles Talks Download Links Books Contact

Clouds and Integration Patterns at JavaOne

OCT 1, 2009

ABOUT ME
Gregor Hohpe
Hi, I am Gregor Hohpe, co-author of the book Enterprise Integration Patterns. I like to work on and write about asynchronous messaging systems, service-oriented architectures, and all sorts of enterprise computing and architecture topics. I am also the Chief Architect at Allianz SE, one of the largest insurance companies in the world.
RAMBLINGS
Here I share my thoughts and ideas on a semi-regular basis. I call these notes "ramblings" because they are typically based on my personal opinions and observations as opposed to official "articles".

I joined JavaOne this year as a panelist on Cloud Computing. As always, you’ll have to run s/Java/Java (TM) Technology Based over this article to satisfy Sun’s legal requirements.

Lary Ellison at JavaoneWe all expected the mood at JavaOne to be a bit more somber than during the fat years, but I’ll have to say it was out right depressing. I walked into the keynote fashionably late, just as James Gosling was introducing the Java application store. I have a lot of respect for James, but Jonathan Schwarz and James were simply stumbling over their own words. I would assume these guys are used to being on stage, so I am not sure what was going on. Apparently, other snafoos occurred earlier with one of the Sunsters referring to app store half an hour before it was actually announced. To add to the drama, Scott McNealy came back on stage, all sentimental, followed by Larry E. himself engaged in a somewhat staged (well, can we blame them?) dialog. While Scott was able to strike a chord with some in the melancholy department, the overall performance rally lacked in energy and coherence. It felt like a bunch of grumpy old men reminiscing about better times. Even the T-shirt cannon felt a bit like playing silly games at the kid’s birthday party after you’ve grown too old just to make grandma happy.

The distinct lack of entertaining (non-Sun-sponsored) parties and the regular party crowd only added to the somber mood. No Zebulon party, no Hani, James Strachan, or Cameron Purdy. Patrick Linsky was apparently sighted somewhere, but I did not manage to grab a pint with him.

The most striking feature on the expo floor (“pavilion”) was the complete absence of Oracle, and hence BEA. Rumor has it that this was due to legal restrictions during the planned acquisition, but it still left a noticeable hole. One vendor managed to drive a Lincoln Continental out of the show floor before the booth breakdown even started. The most allegorical moment must have been when I described Sebastian Meyen, who runs the JAX conferences in Germany, that JavaOne seems to be falling apart, just as the vendors started tearing off the decoration from the pavilion entrance.

Cloud Computing: Show me the money (or demo)

I had the honor to be teamed up on a cloud computing panel with representatives from Amazon, SalesForce, Intuit, Microsoft, and Sun. I love panels because they are interactive and require little preparation, but I was a bit nervous about having a panel with so many participants, each of which would give a demo. However, it actually went quite well at least from my perspective (the audience will be the ultimate judge via the feedback forms). Jeff Bar from Amazon showed an application that scrapes Seattle traffic data (like the weather, usually bad) and stores it on S3. Simon Guest from Microsoft deserves the bonus point for inviting everyone to hit his live Azure app, which queues up requests to validate prime numbers. Unfortunately, the app bogged down under the traffic, but came back up towards the end. I guess a perfect live demo without hiccups runs the risk of looking fake.

Naturally, I represented Google App Engine for Java (GAE/J). Since I expected most of my (friendly) competitors to come with ready-made demos, I joked that I forgot to make a demo and fired up Eclipse to throw together world’s simplest “Hello World” servlet. A simple button push deployed it out into the cloud, and there was really nothing more to tell. I was only half joking when I declared that it’s difficult to demo GAE because it’s so seamless that there’s really not much to show. The strategy seemed to have worked out as one attendee came to me afterwards to let me know that he appreciated the simplicity of my demo.

After my 2 minutes of demo chest chumping (and a round of applause for two lines of Java code), the audience posed a number of interesting questions to the panel:

  • One of the most common concerns relates to storage location and ownership of the data. This question touches both on privacy as well as regulatory concerns. Especially in the EU, legal requirements can restrict on which continent data can be stored. One of the recent Google products, the Secure Data Connector, brings some relief in this aspect, as cloud-based applications, such as App Engine programs can selectively access data residing inside the corporate firewall. I am sure there are other products in the market to address this need as well. Data access latency will likely be the biggest hurdle as fetching data from a cloud application across a secure channel will be an order of magnitude slower than running a direct query against Bigtable or SimpleDB.
  • Another question was the direct reaction to me hitting the “App Engine” button in Eclipse to deploy my application. The gentleman spoke from the operations perspective, to who “someone hitting a button in Eclipse to deploy an app is our worst nightmare.” He was curious to find out about staging procedures and check points. I was able to answer from the Google App Engine perspective, where we ask the user to authenticate to a Google account in order to deploy an application. For staging procedures, users typically setup separate instances of their application in the cloud, similar to what you would do internally.
  • Another listener was curious as to when using a cloud provider such as Amazon or Google is no longer economical compared to running your own datacenter, measured in storage and traffic size. It’s difficult to give general advice here. Usually there are additional aspects beyond size that motivate companies to run proprietary datacenter, such as the aforementioned privacy or regulatory concerns. The answer will also depend on the usage patterns. If usage fluctuates significantly, for example in seasonal patterns, the elasticity of the cloud solutions can become a major economical benefit (assuming not everyone customer is subject to the same seasonal pattern). An oft cited case study is that of the New York Times, which used Amazon’s EC2 to convert a mass of scanned newspaper images. It deployed a batch-style data processing job to EC2 and paid a few hundred Dollars for the whole event. For such one-time computing jobs, the cloud model is certainly economically appealing.

The last point relates to a discussion I had with David Chappell (not the Oracle one) over lunch at the Four Seasons. In a recent keynote presentation, he compared popular cloud computing platforms. Rather than ticking off feature checklists for each platform, he chose a scenario-based approach, comparing how suitable today’s cloud vendor platforms are to a set of five specific scenarios:

  1. Running an on-premises app unchanged in the cloud
  2. Creating a moderately scalable Web app
  3. Creating a very scalable Web application
  4. Creating a parallel processing application (as in the New York Times example above)
  5. Creating a very scalable Web application with background processing

He applied these scenarios against Amazon EC2, Mosso Cloud Servers, Azure, Google App Engine, and Salesforce.com. Check out his presentation to view his findings.

International AttendanceThe good news is that JavaOne nevertheless drew a number of good friends, who I rarely get to see. I had a good conversation with Martin Fowler, Erik Meijer, and Yuji over dinner at Ame, the fancy Japanese fusion restaurant at the St Regis hotel. The highlight of the dinner was that OGIS (Osaka Gas Information Systems division) covered the bill (arigatō!) The strong Yen may be helping. The fact that we just walked into the restaurant without any reservation during a major conference event is a certain indicator of the poor state of the US expense accounts. Martin shared his view that live demos do not add much value to a talk. I generally like (good) demos because they keep the audience engaged and demonstrate cause and effect of actions. If I write a certain piece of code, the demo shows what comes out the other hand. Unfortunately, this is where many (not useful) demos fail. Often too much “magic” is already prepared, making it difficult for the audience to build a suitable mental model of the system. Or pieces of code are added in various places without am obvious connection between them. Martin agreed that linking cause and effect is an important aspect for the audience, but he feels that one can accomplish the same effect with diagrams or explanation, which is certainly true. So doing a live demo is mostly a matter of style, with a tiny bit of geek chest chumping added into the mix.

Relating to Non-Relational Data

Another topic that came up in a few discussions is the recent rise of non-relational datastores, amongst them SimpleDB, Bigtable, and Azure Data Services. At Google IO, Martin voiced his concerns that developers are likely to get tangled up in the impedance mismatch between persistence frameworks and the underlying non-relational storage. David Chappell voiced a similar concern. Microsoft must have faced pressure a lot of pressure as they quickly put the SQL back into the Azure SQL Data Services. A recent discussion on the cloud computing mailing list offered additional insights into this topic.

I believe that a non-relational model can work reasonably well for “typical” Web applications, which are usually not based on a complex domain model. Data storage such as Bigtable makes up for the lack of relationships by providing a richer storage model inside a single table (I always like to highlight the fact that it’s called Bigtable and not Bigdatabase): Bigtable can store an arbitrary number of columns for a single row and is not tied to a fixed schema across rows – it’s essentially a persistent map. Because column names are not fixed, applications use them as data items as well, in addition to the date cells. Furthermore, each Bigtable cell can store multiple versions of data, providing a third dimension of data access beyond key and column name. Combined with an utter disregard for normal forms (compression is very effective for duplicated data), a single Bigtable can often provide the same storage model as a handful of tables in a relational database. The Bigtable storage model for AppEngine gives a nice example how clever use of row keys and prefix scans allows a variety of queries to be performed against a single Bigtable.

The biggest challenge I see is that it’s difficult for developers to understand the performance consequences of their application design when the details of the underlying storage model are not well publicized. For Google App Engine storage, I highly recommend Ryan Barrett's talk Under the Covers of the Google App Engine Datastore.

Integration Patterns all over JavaOne

Braking Down With all that somber mood at JavaOne, I was most excited to see a surprisingly strong presence of Enterprise Integration Patterns -- and even more excitingly neither Bobby nor I had anything to do with it.

  • First off, we were most pleased to see that our book made it onto the Top 10 Bestseller List from the conference bookstore!
  • This may have been largely a byproduct of Project Fuji Open Source ESB being part of the Tuesday morning general session / keynote. Their demo and tutorials make heavy use of our patterns. Sadly, I just walked in after the demo finished. This could have easily ranked as my second proudest conference moment after Grady Booch mentioning our book as one of the most influential pattern books.
  • Andreas Egloff and Bruce Snyder gave a Enterprise Integration Patterns in Practice talk, which demonstrates how our patterns are implemented in Apache Camel and Fuji / OpenESB.
  • The show floor aka Pavillion provided another nice EIP surprise: LogicBlaze-turned-Iona-turned-Progress Software created very nice Dzone RefCardz for Fuse aka Apache Camel, which implements many of our patterns. They adopted the icon language and diagrams from our book (which are available under the creative commons license) and mapped them to the corresponding implementation in Fuse.

Till Next Year?

What's going to be hot at JavaOne next year? I think many of us don't know what to expect. I feel that the transition of Java from the cool, exciting to bread-and-butter is sombering, but ultimately a good thing. In many ways, the transition had started quite a few years ago, and the transition to Oracle really just puts the icing on the cake. On the upshot, Oracle knows how to throw a good party. See you all there next year!

MORE RAMBLINGS    Subscribe  SUBSCRIBE TO GREGOR'S RAMBLINGS


Gregor is the Chief IT Architect of Allianz SE. He is a frequent speaker on asynchronous messaging and service-oriented architectures and co-authored Enterprise Integration Patterns (Addison-Wesley). His mission is to make integration and distributed system development easier by harvesting common patterns and best practices from many different technologies.
www.eaipatterns.com