What I learned about Infrastructure

I learned a lot from Andrew Shaft's talk on building a private cloud. My biggest takeaway, perhaps not Andrew's main point, perhaps not even a point he was making, regardless... Clouds are not about virtualization. Virtualization is an implementation of a cloud infrastructure, one could just as easily build a cloud infrastructure with physical boxes.

A cloud infrastructure is more about automation and the ability to grow and manage one's infrastructure without physically logging into machines to manage them. Clouds are about tooling, not about a specifc tool for managing services. Machines are basically worthless, services provide all the value, focus on the service not the hardware or software. Scott McNealy brought up a simaar point years ago when I heard his keynote at JavaOne, back in 1999 (or 2000?), the JavaOne where we all bought real cheap Palm Vs! He pointed out that hardware by iteself and software by itself has no value. However, when you take a Sun box + Sendmail you get a service that has real value to people. Ok, so clouds ==  deployment and management of services via tools.

Everyone that talked about "the cloud" stated categorically that unless you are treating your infrastructure like you treat your code there is no way you are ready for the move. Obviously, this assumes that your code is managed in source control, you have plenty of autoamted tests, and that you have a repeatable build and release process. What we struggle with, here at Edmunds, is the configuration and deployment aspects. We have gotten a lot better at configuring our software, however, we have a long way to go with the deployment of our applications, the configuration of our systems, and the configuration of our environments in a repeatable manner.  Too often we find ourselves SSH'ing into a box to check or change a configuration. Once in the cloud you won't have enough time in the day to check all the boxes. For that matter, once we have additional data centers we won't be able to do this either.

Puppet was the tool of the day, everyone talked about Puppet, however, regardless of the tool, the goal is to have your infrastructure be self describing. With Puppet this means the Puppet DSL. The interesting point for me was that by using a tool that is close to the tooling your developers are using the dev and ops teams start using the same language and working through similar processes. Just like the code, the configuration is checked into source control, versioned, labeled, built, tested, and deployed. There is even a testing harness for Puppet using Cucumber!

Who writes the configuration? While the ops team traditionally manages all configuration it would make sense to have developers write configuration for their applications. After all, they wrote it they should know how to configure and deploy it. The real answer is both, just like code the configuiration, now in a language common to both teams, is a shared responsibility. There is no owner of our code base and there would be no owner of our configuration.

So we get it, configuration as code. We understand we can and should test our configurations and use all the standard change control systems we use for code. However, just like without a test there is no code, with infrastructure without a monitor there is no deployment. Monitoing is often a bunch of data exposed to the operations team that then has to figure out how to chart and alert off the data. The whole time the developer basically knows what a workng system looks like and what a broken system looks like. Rather than exposing a ton of data, why not start with a simple hey I'm working piece of data? The alternative is just let ops ping your service...really?!??

The final two lessons I learned (the first is really a recap but it is important):

1. If you don't have automated configuration management you can't build a cloud, once you have that you will probably realize it is cheaper to use a provider!
2. People and culture are usually the biggest barriers to appraoching infrastructure as code! (no suprise there same as with tdd!)

Next up...Continuous Deployments