Data sets are getting larger and more ubiquitous by the day. Scientists are able to get petabytes of data from large experiments and run complex data analytics using vast amounts of resources. These datasets are being generated on an ongoing basis. Recently Amazon announced their HPC offering with beefy machines and the promise of low latency and high bandwidth between VMs. The problem is that moving and storing data in the cloud is very expensive. If data was to be stored in S3 you're looking at 10 cents per gigabyte (starting November 1st) to move the data in, along with the storage cost of 5.5 cents per gigabyte (at the cheapest) a month. If you're dealing with just a terabyte of data, we're talking over 150 dollars a month as a starting point. Want to start talking about petabyte datasets? Multiple that by a thousand. As I have already mentioned, these data sets will grow larger and larger, and with just the data starting to cost that much it might be a wise decision to roll your own cloud with something like Eucalyptus, or go cloud-less altogether.

While I was interning at Lawrence Livermore National Labs this past summer I was thinking of what could they possibly do with a private cloud infrastructure. They aim to squeeze every iota of computation out of those machines and putting a level of virtualization better have some great benefits. The application area seems small enough that there really was not much of a need to be dynamic in the images one needs to run. A possible benefit was to allow users to specify whichever OS and application set they wanted to boot, but even this can be done simply with diskless booting over NFS.

Our diskless boot hadoop cluster had a very weird problem during startup. After running the start-all script, all the logs would stop on 

“Unable to create a new native thread”
“Could not obtain block”
“Task process exit with non-zero status”
 

This is because I've run out of memory (each machine has 24 gigs of ram) and my system does not have swap. TaskTrackers slowly start getting black listed and the job completely fails. What sucks is that the best configuration is very job dependent. Some jobs are CPU bound while others are IO bound. In my case all jobs are memory bound.

How many times have you run into a problem that other people have already run into, solved, and then blogged about it? Now how many times have you not found the solution online, solved it, but did not share your trials and tribulations with the community? Blogging about your technical feats, no matter how small may help someone later on. Furthermore, you may run into the same problem later on but lack documentation on how you previously solved it. Even if others have blogged about the solution you've come up with, blog about it anyway. Giving multiple view points helps others comprehend the material. I know it helps me and it gives me a feeling of joy knowing that I've contributed back to the community.