Cloud Angels

watchit.by - having fun with videos in the cloud

We at CloudAngels like to use new technologies. Since Amazon is constantly launching new services we decided to make use of some of them and started a little public project a while ago that we had quite some fun with and put it up on Amazon's cloud.

The result is watchit.by - a website that constantly listens to the Twitter realtime stream, ranking the most interesting videos that are currently being discussed so you never miss out on the greatest videos. It's a leisure site really, but be warned, you can easily spend hours on it watching videos, that's how addictive it is (that's kind of the reason our little mascot fly over there has these big red eyes).

Feel free to give us feedback and follow the project on Twitter - the user is @watchitby

At its current stage the setup for the website is quite minimal, but we think once the site grows, we will have the opportunity to showcase some good use of Amazon's APIs.

 

Announcing the CloudAngels Youtube Channel

Starting today, CloudAngels is offering a new Youtube channel. Here we will continue to discuss with you the ins and outs of Cloud Computing and associated technologies. Go to http://youtube.com/user/cloudangelsdotcom

In the first available episode CloudAngels Co-Founder Antonio Agudo talks about the pros and cons of public and private clouds. The video is only available in German though.

Amazon storage and transfer key features roundup

Amazon is constantly improving their web services offerings, so it's easy to miss some great innovations if you don't pay close attention. To give you a quick overview of what recently happened in the storage arena, lets take a look back and see what kinds of new tools are at our disposal.

Google Atmosphere cloud computing CIO event

A great video about cloud computing

Just came across a great video about cloud computing in plain English - really like the analogies to the automative industry - see

 http://www.youtube.com/watch?v=XdBd14rjcs0#watch-main-area

Thought you might find it interesting.

AWS Start-Up Challenge 2009

Amazon Web Services just announced an extended deadline for their third annual start-up challenge that has been extended from the USA to UK, Israel and Germany this year! The five finalists (who will all be flown to San Jose, California this November) get chosen based on the following criteria: originality and creativity of the idea, likelihood of long-term success and scalability, how well it addresses a need in the marketplace, implementation of payments functionality, quality of presentation, and implementation of AWS infrastructure services and/or other paid services.

We think this is a great chance for European start-ups to prove that innovative business models do not compulsory come from the States. If you have a start-up idea or an existing company that is / will be running on AWS just apply here: http://aws.amazon.com/startupchallenge/. The deadline for applying online is September 25th. The Finalists will be notified on October 30th. Good luck =) !

What's new on S3? Data Import/Export and file organizers review

Amazon S3 is the source of many a success story. The enourmously popular storage service has a wonderful API and it comes at a price that is hard to beat for all the features it offers.

Data import/export

Today Amazon announced a feature that is sure to draw some more attention. The data import/export feature gives you essentially the possibility to snailmail Amazon your Harddisks for ingestion or extraction to or from S3.

So where you previously had to copy all your Terbytes over the intertubes, you can now just slap a stamp on your harddisks (better make sure they are well packaged and backed up) and send them on their way. It is no secret that despite all technical progress, physical transport of media often still offers a lot more bandwidth at a cheaper price than internet transfer. What's more, you can even save the transfer cost per gigabyte that AWS would charge you otherwise. Instead they charge with a more appropriate cost per hour/per device model at a pretty reasonable price. AWS even invented a packaging API to make sure there are no misunderstandings how you want to have your data stored.

Overall this makes S3 hugely more attractive to businesses with huge backup needs or media storage requirements, but also for a transition to or from traditional storage media. Handling of your S3 data just got way easier - which brings me to the next point.

S3 Organizers

One thing that always struck me as odd is the lack of good GUI tools written for S3. This situation has been improving dramatically lately.

Sure there is always jets3t's cockpit, but while great from a code perspective, it's is not exactly a crowd pleaser - just look at the GUI . And S3Fox may have mass appeal, but do I really want to risk the process instability of firefox when uploading large files and browsing at the same time? Is the colourful CloudBerry S3 Explorer mature enough for me to handle my data on a daily basis? Or should I pay 66$ for Jollat and have additional Ec2 capabilities at my hands? The only way to tell, obviously, is a tryout. So that's what we did.

JetS3t cockpit
Only few words can describe the beauty of the API that is the jets3t API. It's a masterfully crafted, stable piece of software you can rely on and it also offers basic CloudFront integration in the newest version. Unfortunately the GUI is not at all polished. Just what a unix hacker would like.

S3Fox
S3Fox was one of the first clients to support CloudFront. It is fully featured and leaves not much to be desired. Except of course an overview of the buckets size. We realize this is hard to do on large buckets since it would take a lot of LIST requests to estimate the whole size (amazon should really provide an API call to gather stats about the overall bucket size), but it would have been nice if the listed objects. Unfortunately the Client runs in-process with Firefox, so if the main process dies, so does your data transfer.

Jollat
Jollat looks nice on the surface. It integrates EC2 and S3 functionality and promises to be your command center to the cloud. But actually those 66$ are wasted money. The functionality of Jollat does not justify that price.

Bucket Explorer
This one is a fully featured, commercial client. It costs 50$ and offers an overwhealming set of features. The "versioning" and "Trash bin" functionality alone are worth the 50$ - no more accidentally deleted files. Incredibly this client even tracks your bucket usage via logfiles and reports on their usage as well as your SimpleDB usage. If you are a heavy user of S3 you might be interested in this client.

CloudBerry Explorer
CloudBerry is home to the hugely popular free version of CloudBerry Explorer, which is a feature complete S3 client, with all bells and whistles. It is a very mature piece of Software, that offers the most beautyful and polished UI of all Clients. Unfortunately it's a Windows only solution. Lately CloudBerry Labs released a pro version that offers advanced features like encryption, compression and folder synchronization. With these features it is set to become a worthy opponent of the Bucket Explorer. There is even a public beta of the Pro version that expires on first of July, so get it while it's hot.

Conclusion

If you are on Windows you are looking for a stable, free client with a hot UI and great features there is no way around the CloudBerry Explorer. If you are willing to pay and need more advanced functionality you should consider having a look at Bucket Explorer, while CloudBerry seems to be gaining ground fast.

CloudCamp Berlin spring 09 - the review

As some of you may have noticed, we recently organized a CloudCamp event in Berlin. For those of you unfortunate enough to not having been able to attend, here's a short summary of what happened there.

On 30th of April about 130 European early adopters and Cloud Computing industry experts met up at CloudCamp Berlin. Organized by Tom Leyden at Sun Microsystems cloud division and Antonio Agudo at CloudAngels which specialize in system integration services and solutions for clouds, this event was a real eye opener for the unsuspecting crowd.

The keynote was presented by Sun's cloud platform strategy director Niko Nelissen who is the co-founder of the company Qlayer, that SUN acquired last year. Niko highlighted the path ahead of the cloud industry as a whole, but also gave some interesting specifics about Sun's roll out of their cloud services and open questions that are waiting to be solved.

During the Lightning talks Simone Brunozzi, the European evangelist for Amazon Web Services talked about the way businesses can avoid hardware as a potential pitfall of fraction. Mark Masterson of CSC reminded us, with a lot of chuckling, that mathematically speaking IT security is not a binary variable. Further we had an advanced load balancing talk by Nick Bond of Zeus and a lessons learned talk from Grid computing researcher Morris Riedel from the Jülich Supercomputing Center regarding standardization processes. Max Ahrens from Zimory talked about recent cloud efforts, the company launched their public cloud earlier this year. Unbelievable Machine Company CTO Thorleif Wiik had inside information about how peering is managed on a cloud scale network for media distribution. Scott Wheeler of Directed Edge talked about future search of semantic data.  Max Robbins from AiCache presented his views on licensing and other issues within cloud computing, Dr. Frank Moser from Google concluded the lightning talks with his presentation about what benefits the cloud can bring to your enterprise IT.

Furthering the discussions in an unpanel and two productive unconference sessions, the foal of CloudCamp was achieved: Raise awareness about cloud solutions and provoke new ideas and thoughts. We were happy that many people came and we are pretty sure there will be more CloudCamps in the future. Here are a couple of links from around the web about the event.

Blog posts:
http://www.gruenderszene.de/allgemein/anders-aber-interessant-cloud-camp-in-berlin/
http://www.mashedup.de/blog/?p=102
http://www.stottmeister.com/blog/2009/05/04/review-of-first-cloudcamp-berlin/

Twitter:
http://search.twitter.com/search?q=ccberlin

Google AppEngine supports Java, heads for enterprise

Finally it's official. Google supports java web applications on their AppEngine architecture. Of course the usual suspects like GWT are supported out of the box. I think the most powerful aspect though is that GAE ships with both JDO and JPA support for the App Engine datastore. That means you could use jpox or Hibernate within your AppEngine webapp to access BigTable.

Groovy is already running fine, though Grails doesn't seem to work on GAE just yet. But support seems to be on the way according to the related tweets.

The java support really took a long time to implement. Apparently this was due to security restrictions that had to be put in place to prevent people to pry their way out of the jvm's sandbox. Google solved the problem by only allowing whitelisted classes from the JDK. Unsurprisingly, large portions of the JDK got scrapped. No swing, awt, printing, reflection, javax.management.*, java.rmi.* for the masses. If you are curious which classes are missing, I've got the full list here.

Currently you will have to contend yourself with the GAE for java and Eclipse development tools on the local machine though, since Google restricts online usage to a few alpha testers at the moment. Hopefully that will change soon.

What's interesting is that Google is clearly focusing on Enterprise customers with their new release. They even developed a kind of tunneling technique called Secure Data Connector or SDC, that lets privacy concerned customers define policies under which data within their firewall can be accessed from the GAE to solve common issues when moving sensitive material to the cloud.

Very interesting developments, that beg the question how platforms will compete and coexist in the future.

imageloop.com goes 100% cloud, saves 2/3 of operational costs

When Europe’s biggest slideshow contender imageloop.com, home of the web’s most awesome slideshow widgets, decided it had seen enough of its exploding operational costs late last year, it turned to Cloud Angels to look for ways to increase performance processing the steadily growing load of images that was piling on its servers while at the same time cutting costs for their server farm.

What kind of scale is imageloop.com dealing with?

Imageloop boasts no less than 20 Million slideshow views per month and each day people add about 30.000 Images to the stack. Of course, Imageloop needs a lot of bandwidth and storage for all those terabytes of big multi megapixel images. Additionally, resizing the images for slideshow consumption typically needs a lot of CPU power, and increases server load.

Imageloop before the switch

Multiple quad core Blade Servers were used for the front end, while incoming HTTP requests would be distributed by a load balancer and several huge disk arrays with redundant storage were accessed for mass storage. Of course as time goes by, disk arrays would have to be extended, servers had to be added to accommodate the increasing load, spare parts would have to be installed if something failed, etc. Let’s just say rack space and hardware isn’t exactly a bargain these days – so it was rather expensive to keep growing that way.

How did we solve the problem, you ask?

With a small team of just three expert professionals, Cloud Angels was able to migrate all of imageloop.com’s operations to Amazon AWS in just four weeks time. During the hot phase of the switch migration imageloop.com’s front end site had to be taken offline for about some hours overnight, while content delivery for all slideshow still kept working.

Imageloop after the switch

And since then, imageloop.com is running 100% on Amazon AWS, using S3 for Storage and EC2 for their front end servers of course. The dynamic nature of AWS enables them to add servers if the load spikes and shut servers down if they are no longer needed. Also, they certainly don’t worry about buying expensive disk arrays anymore, because Amazon S3 takes care of the redundant saving of data as well as the delivery of images. As a very pleasant side effect the front end servers are not involved in the delivery of images anymore. S3 does all the heavy lifting of big binary files, while the front end servers keep spewing out the interface markup. That way there is very little load and latency that stems from transfers of image files.

Happy End

The result is a site that is faster, more flexible and more scalable than ever. It also comes with a better backup strategy and more redundancy, while costing only a fraction of the money it took before to run it. With this single switch imageloop.com saved a whooping 2/3 of its monthly fixed costs. And by cleverly leveraging the benefits of the AWS interfaces, probably untold thousands of Euros will be saved by not having to deal with broken hardware and wasted phone calls to the hosting company.