Obviously, that’s an average, so some days there are more, and some days there are fewer.

How many times during that span has improperly tested code broken things in a way that impacted users? Just once, and it was fixed in just a few minutes by adding back the code that I over-zealously removed.

Also in that time, we went from 0% automated test coverage to 18% automated test coverage, refactored a lot of duplicate code, and added new features.

I’ve recently transitioned to working on a fairly ambitious overhaul of an existing project that has been around for a few years. The software has already reached a (mostly) working steady state and is used every day by real customers.

The very first thing I did, ahead of making any functional changes, was automate the deployment system using essentially the same kind of test gauntlet and approach for zero-downtime deployments I was using at Victors United. I added some very basic test automation, starting from the top of the test automation food pyramid with a simple “is the web server able to execute code, connect to the database server, and return an ‘OK’ rsponse?” test. From there, I’ve working my way down, just getting around to writing my first “pure” unit test just today.

Only after I was confident that I could do zero-effort and zero-downtime deployments that

1. Wouldn’t completely destroy the system (I had a test for that) and
2. Could be rolled back very easily if something went screwy

did I make my first functional change to the software.

And then, I made the smallest functional changes that could work. I tested them locally, added some test automation around them, and then let the automatic deployment system do its magic.

A few years ago, I was interested in doing continuous deployment, but I felt that the level of test automation wasn’t good enough, I wanted to be at least at around 90%.  Now I think that the less test automation you have, the more important it is to start doing continuous deployment of tiny incremental changes right away.

If you are working on software that’s actually in use and are interested in doing continuous deployment someday, there’s no better someday than today. Seriously. Do it now. It will start making your life better almost immediately.

I’ve also worked with development organizations that take the opposite extreme. Even the most trivial server updates needed to be scheduled weeks in advance. In order to push the updated code, you had to notify everyone, take all servers down in the middle of the night, run through a long series of manual steps, and then do a lot of manual testing. When things in production aren’t quite how they are in the test environment, this is followed by a hasty rollback or panic-induced middle-of-the-night attempts at bug fixing.

Both of these extreme approaches are (maybe) appropriate in some environments. But neither are appropriate for a new web startup that’s trying to move quickly and still provide a reliable trustworthy experience to their customers.

At the previous company I worked for, we often talked about IMVU-Style Continuous Deployment as our ideal process, but it was always something “in the future”. We were hesitant (some of us more than others) to do automatic deployment without at least a little manual intervention. We always wanted to have more test automation, or a smoother deployment system, or whatever.

Since it seemed to be hard (for me anyway) to move to an existing development organization to a continuous deployment system, I started to wonder what would happen if you do it that way from day one? I got a chance to answer that question when I co-founded a startup last year. One of the very first things I did, before we had anyone using the site, was to create an solid automated test & deployment system that was as fast and easy as possible without being dangerous and stupid.

Here’s the basic workflow that happens in our office multiple times every day.

Step 0. We make changes on our local dev envirnments, with a bias toward making the smallest possible change that adds value. That could be a bug fix, correcting a typo, a stylistic tweak, a stubbed-out new feature, whatever. Once I’m confident in my local (manual and automated) testing that the change is good (not perfect, not feature-complete, but just better), I push that to my github repository.

From there, the continuous integration server pulls down the new code and does the following:

Step 1. Does the code still compile. If not, the build fails and everything stops.

Step 2. The build agent runs the unit tests (where “unit tests” are defined as tests that run with no external dependencies, these take just a few seconds). For anything that does require external (generally slow) dependencies (network API, databases, filesystem, whatever) we use test doubles (fakes, mocks, stubs, whatever).

This first feedback loop is about catching and preventing errors in core business logic and is generally measured in seconds, not minutes.

Step 3. The build agent runs a set of tests that rebuild the database from a reference schema and exercises all of the repository layer code.

Step 4. The build agent runs another set of tests that test our dependencies on external APIs (twitter, geolocation services, etc.)

These two sets of tests run in a few minutes, so the feedback loop isn’t quite as tight, but it’s still pretty darn fast. Basically, they make sure that the assumptions that we make with our test doubles in our unit tests aren’t totally wrong.

I’ve written about these sorts of automated test distinction a couple of years ago, in a post about the Automated Testing Food Pyramid.

Step 5. Provided that the entire gauntlet of tests has passed so far, the code gets automatically deployed to a staging server.

Step 6. There’s an additional set of tests that run against the staging web server. These tests can find configuration problems and code that just does the wrong thing in a web context. These tests are pretty shallow. They hit all of the user-facing pages/JSON endpoints and fail if anything is totally broken.

Step 7. The build artifacts are copied from TeamCity to a new folder on our productionvserver, and then the web server is reconfigured to serve from that folder instead of the folder it had been serving from.

At this point, we’ve verified that the core business (game, in this case) is OK, verified that the persistence stack works as expected, that our integration with external APIs works as expected, and that the code doesn’t completely break in a web context. We’ve done a zero-downtime deploy to the production web server.

That’s cool, but we’re not quite done yet. There’s two more steps.

Step 8. Run a set of tests against the production web site to make sure that all of the pages that worked a few moments ago still work.

Step 9.  Have external monitoring systems in place, so if your changes make things slow or unresponsive. You’ll know.  We use pingdom.

Yikes! There’s a bunch of distinct steps here, and it seems really complicated (because it is). But it’s all totally automated. All I need to do ?

git push origin master

Because there’s zero deployment effort on my part, I do this all the time.  I find it very energizing to know that I can just do stuff in minutes instead of hours or days or (heaven forbid) months.

If (when) something goes wrong, I’ll know immediately. If a bad bit of code manages to roll through the test gauntlet, I can roll back easily (just reconfiguring the web server to use the last known good set of code). I’ve only had to roll back a couple of times over the course of several months and 326 deployments.

Just like when the folks at IMVU wrote about this process, I’m sure that some people in the audience convinced that I’m a crazy person advocating a crazy way of working. Here are the objections I’ve heard before.

Yeah, but this doesn’t give your QA team any time to test the code before it goes out!
We’re a small startup. We don’t have a QA team. Problem solved.

Yeah, but isn’t that incredibly dangerous?
No. The safest change you can make to a stable production system is the smallest change possible. Also, we design the individual parts of the system to be as encapsulated as possible, so we don’t tend to have crazy side-effects that ripple through and create unintended bugs.

When we make a change or add a new feature, we can manually test the hell out of that one thing in isolation (before checking in) instead of feeling like we need to spend a lot of time and effort manually testing everything in order to ship anything.

Yeah, but what about schema changes?
For schema changes that are backwards compatible with the code that’s out there (e.g. new tables, whatever). We have a simple system that executes the appropriate DML on application startup.

For non-compatible schema changes and things like server migrations, we have to take down the site and do everything manually. Fortunately, we’ve only had to do that twice now.

Yeah, but you have to spend all of that time writing tests. What a waste!
The time we spend writing tests is like the time a surgeon spends washing their hands before they cut you open. It’s a proven way to prevent bugs that can kill you.

Also, we get that time back, and then some, by not having to spend nearly as much time with manual testing and manual deployments.

Yeah, but what about big new features you can’t implement in just one sitting?
Traditional software development models (both waterfall and agile methods like Scrum) are organized around the idea of “multiple features per release (or iteration)“. Continuous deployment is organized around the idea of “multiple releases (or iterations) per feature“. As a result, we end up pushing a lot of code for features that aren’t done yet. For the most part, these simply unavailable through the UI or only exposed to users who have particular half-built features associated with their accounts. I credit Flickr with this general approach.

Yeah, but that might work for a solo developer, but it can’t work for a team.
There are actually three developers on the team.

Yeah, but I’m sure this only works with very experienced developers
One of the guys on the team has only been programming for the last year or so and hasn’t ever worked on a web project before. Tight feedback loops help everyone.

Yeah, but what about code you want to share with other that you don’t want released?
We use github, so creating additional branches and sharing with them is trivial. We also have a dedicated “preview” branch that triggers a parallel test/deploy gauntlet that sends code to a staging server instead of the production servers.

Yeah, but this will never work at my organization because…
OK. That’s cool. Don’t try it if you feel that it won’t work for you. You’re probably right. You’re not going to hurt my feelings either way. I found something that’s working really well for me, and I want share my experience to show other people that it’s possible.

What this really means

Half of what makes this process work is that we’re honest with ourselves that we’re human and will make mistakes. If we have multiple tight feedback loops between when we’ve broken something and when we know we’ve broken it, it’s faster and easier and cheaper to fix those mistakes and prevent similar mistakes from happening again.

The other half is the idea that if you design, implement, test, and release exactly one thing at a time, you know with certainty which change introduced a problem instead of having to ask the question “which of the dozen or so changes that we rolled out this month/sprint/whatever are causing this problem”.

About the site

Victors United is an online turn-based strategic conquest game. You can play asynchronously or in real time. You can play against robots or humans. If you’re playing against humans, you can play against your friends or against strangers. Unlike some other popular web based social games that I don’t like to mention, this is a real competitive game where strategy and gameplay matter.

About the tech 

The tech here is kind of beside the point. This general approach would work just as well with different technology stacks.

The front end is HTML5 + JavaScript + jQuery. The backend is IIS/ASP.NET MVC2/SQL Server/Entity Framework 4. Our servers are hosted in the SoftLayer cloud. External monitoring is provided by pingdom.

The test gauntlet is a series of distinct nUnit assemblies, executed by TeamCity when we push new code to GitHub. There’s a single custom PowerShell script that pulls down the build artifacts and tells IIS to change what directory it serves code from.

Yawn

At the risk of adding yet more noise, I did have a two nuanced thoughts that are at least new to me and I thought I would share them, along with a recent personal anecdote.  If this is obvious or old-hat to you, then I’m sorry.  If you think I’m  too stupid for words and that I’m drinking and/or selling snake-oil enriched kool-aid, I look forward to what will undoubtedly be informed and insightful feedback.

1. s/driven/aware/

The hardcore position of “never write a line of production code without a failing test for it” probably does more harm than good. Different kinds of code require varying degrees of effort (cost) to write and maintaintests for. Different kinds of code give varying degrees of benefit from test automation. Without always realizing it, developers make cost-benefit decisions all the time, and good development organizations empower their developers to act on those decisions.

That said, the cost-benefit decisions developers make must at least be informed decisions. A professional developer who hasn’t taken the time to learn how to use the appropriate test frameworks for their language/environment (jUnit, nUnit, whatever) is just plain negligent in 2009.  Test automation is just one tool in a competent developer’s toolbox, but a critical one.  I wouldn’t trust a carpenter who didn’t know what a hammer was, or a cardiologist who hadn’t bothered to learn about this newfangled angioplasty business.

Test-driven may not be appropriate for every context, but everyone needs to be at least test-aware.

2. s/first/concurrently/

Test-first is a really helpful approach, but it doesn’t work with the way everyone thinks, and mandating that everyone must always think in exactly the same way is the worst sort of micro-management.  The other extreme, writing test automation for a large system after it’s complete, is often prohibitively difficult and (frankly) boring as hell.

My advice is to always at least think about how you would write tests for your code before writing it. That will help keep you from painting yourself into untestable corners. Also, interlacing test writing immediately after you get a small subset of your system done is going to be much easier than testing the whole thing after the fact.  Personally, I move back and forth between writing the tests first and writing the code first. The key for me is that I’m working in short code-test-code-test cycles, using persistent (that is, I don’t throw it away when I’m done) test code as the primary mechanism for executing the code I’m writing as I’m writing it.  I don’t think of the process as being test-first, I think about testing concurrently with coding.

Recent Anecdote

Sure, anecdotes aren’t data, and they can’t prove anything, so take from it what you will

I just finished a pretty big refactoring project (that is a “pure” refactoring, the external interfaces and behavior of the existing stay the same but the underlying implementation was improved) of a system that had some decent test automation. Every time I got an edge case behavior wrong, introduced a side-effect, or removed a necessary side-effect (yuck), a test would go from green to red. This saved me at least a few days of development and testing time, and reduced the chances that I would release bugs to our QA guy (bad) or our production system (even worse).

To liven up the bus commute on non-bicycling days, I’ve put the audiobook edition of Stephen King’s On Writing on my iPod. I’m aware that Stephen King is somewhat out of fashion in some circles right now (I mentioned that I was making a novelist the main character of a story I was writing, and a good friend said “That sounds like something Stephen King would do” while wrinkling his nose) but until the best of my stuff surpasses the worst of his stuff, I’ll reserve snobbery. 

One thing that has come up in his advice to writers more than once is that you need to be a serious reader if you want to have any success as a writer. He’s clear that he doesn’t read fiction to directly improve his plotting, dialogue, or prose. He reads fiction because he enjoys reading fiction. His prose and dialogue just get better naturally with more exposure.

I’ve noticed a parallel in the software realm, where people who wouldn’t ever use a particular kind of software tool, are in charge of designing and building one. The results are predictably lackluster. The end product is stilted, like the work of a non-reading writer with a clumsy vocabulary or a flat ear for dialogue. You won’t get to greatness this way, you consider yourself lucky if it works at all.

I’ve made a conscious effort to make sure that I actually use software products similar to the ones that I’m building. I’ve been doing some media related work, and whenever I get stuck on an interaction, I ask myself, “What would iTunes do?” Not that I slavishly copy everything Apple does, but I know that they at least put a lot of time and energy into their interaction design. If humans didn’t solve problems in the context of how other people have solved similar problems, we never would have progressed beyond cave painting.

Now consider the iPod Amnesty Bin at the Microsoft Zune HQ.  I know it’s mostly a joke, but there’s still a message behind it: You had better not be seen using our competitor’s product, even though it’s a huge commercial success and generally considered one of the most well-designed machines in modern history.

And while the Zune is only in its second generation and has been steadily improving, it hasn’t exactly set the world on fire. I’ve only seen a small handful of them in the wild, while I see iPods everywhere. I sometimes joke that the Microsoft Connector commuter shuttles in the Seattle area should instead be called The Zune Express. It’s a magical, insular place where Microsoft employees don’t have to deal with the disturbing reality that nearly everyone on regular public transportation is using an iPod.

Can you imagine the scandal that would break out if you saw Stephen King reading a book by Dean Koontz? Or if you saw Michael Crichton reading a book by John Grisham? What? There’s no scandal at all. Just a recognition of the community of experts learning from each other’s successes and failures.

What’s next? I’m going to use as many software programs as I can get my hands on, and I’ve deviced to re-read The Shining with a critical eye, even if it is yet another story with a writer as the main character.

I saw a great example of this in some ASP.NET code I was working with recently. A tester had reported that some of the items in a display were being truncated inappropriately, while others weren’t being truncated at all. I looked at the screen and could immediately tell what was wrong. Every other row in the table was truncated. It corresponded perfectly with the ledger-paper-style alternating background colors.

Looking at the code, I saw the following things (simplified):


<asp:Repeater id="repeater" runat="server">
<ItemTemplate>
<tr bgcolor="#FFFFFF"><td><%# DataBinder.Eval(Container.DataItem, "Title")%></td></tr> </ItemTemplate>
<AlternatingItemTemplate>
<tr bgcolor="#CCCCCC"><td><%# DataBinder.Eval(Container.DataItem, "TruncatedTitle")%></td></tr>
</AlternatingItemTemplate>
</asp:Repeater>


 
The vast majority of the code in AlternatingItemTemplate has the same intention as the code in ItemTemplate. Whenever you have duplicated code (or near-duplicated code) you will eventually forget to update one of the two things that need to be updated in parallel. Hence the bug.

An alternative, would be to not have the duplicated/duplication-heavy AlternatingItemTemplate, but instead, call a method to get the row color, like this:


<ItemTemplate>
<tr bgcolor="<% = GetCurrentRowColor()%>">
<td><%# DataBinder.Eval(Container.DataItem, "TruncatedTitle")%></td></tr></ItemTemplate>


In the code behind (or even someplace more generalized, if you are going to use this layout more than once, of course), you could have this simple logic:


private const string initialRowColor = "#FFFFFF";
private const string alternatingRowColor = "#CCCCCC";
private string lastRowColor = initialRowColor;
public string GetCurrentRowColor()
{
 if(lastRowColor == initialRowColor)
{
lastRowColor = alternatingRowColor;
return alternatingRowColor;
}
else
{
lastRowColor = initialRowColor;
return initialRowColor;
}
}

 
One Courtesy Note: in the off-chance that my comrade who introduced this problem is reading this, I don’t mean to single you out or pick on you. It’s the sort of mistake that I’ve made a billion times myself, and wanted to capture what I felt was a reasonable solution to the AlternatingItemTemplate redundancy problem.

Also, in case any ASP.NET gurus are out there, I generally don’t like to use the DataBinder.Eval syntax, this is a simplified example, remember?

[Product X] companies were 25% more productive.  That’s like adding 25 people to a 100-person department, free!

Let’s just set aside the notion that you can accurately measure productivity of software teams in a meaningful way (you can’t). My immediate thought take on that was, “No, it’s like removing 25 people from a 100-person department. Hasn’t anyone read the Mythical Man-Month?”

I had just come off a project which had excellent pure unit test coverage, but not very good overall test coverage with actual collaborators. The integration points were much more difficult and painful than they needed to be. On the other extreme, I inherited a bunch of existing non-unit tests as part of the MVNO project. These were brittle, took a long-time to run, and failed for a bunch of different reasons. As a result, they weren’t run very often, became neglected, and eventually worthless.  There was no continuous integration in place, so the feedback loop between breaking something and finding what you broke was very loose.

To make matters just a little more dicey, I was working with another test-focused developer who had a strong preference for writing epic end-to-end tests, while I was trying to make the code more able to be tested in isolation. I resented the fact that “his” tests were keeping me from running the tests as part of the cruise control build.

I came up with what I called the “Food Pyramid” as a way of balancing these different forces and points of view. I broke the one massive test project into three distinct assemblies with their own goals.  The differences between the projects are best explained in a series of pictures:

So, did it work? Yes and no. We eventually had much smoother and more confident deployments thanks to the end-to-end tests. I also managed to refactor and add new code more quickly with fewer bugs thanks to my unit-test safety net. But, just as I was starting to get a handle on the project, the client filed for bankruptcy and stopped paying their vendors (which unfortunately included me). It seems that having a viable business model trumps all.

I have, however, used these and similar distinctions successfully on subsequent projects, and I know that at least one development organization I’ve explained the idea to has adopted the pyramid as part of their overall testing strategy.

Now, in a shameless appeal to authority, I find that Martin Fowler sees things the same way. From a recent post:

You can’t really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don’t do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing.

Right on!

One thought on drawing ideas directly from lean manufacturing: The ideas that work in both manufacturing and software (such as empowerment, incrementalism, tight feedback loops) don’t necessarily work in software just because they work in manufacturing. That’s a naïve mindset that can lead you to all sorts of bad conclusions.  Some ideas work in both due to the nature of the people involved in both. 

It’s easy to get ideas from manufacturing as it’s a human endeavor that’s more mature and has been very well studied and documented.  An observant and mindful person could surely draw lessons from all sorts of places.  It’s possible that someday we’ll all understand software development well enough that we won’t need to borrow concepts from other disciplines. It’s clear to me that we aren’t there yet, though.

I was taking wedding reception photos as a favor for an old friend. For the most part, I hate the entire idea of doing wedding photography. The risk/stress/reward/effort ratios don’t work out right.  This one actually worked out really well, as the couple and wedding party were all great. They also had a “real” photographer working (who had more invested in one lens than I have in my entire setup, alas) which freed me up to take some more experimental pictures, such as HDR still lifes of the venue, controlled motion blur shots of people dancing, and grainy black and white candids (I love tmax 3200 because of the grain, not in spite of it).

I took special care to make sure that all of my camera and flash batteries were charged before the event, but I didn’t spend a lot of time packing the bag before I left. I just threw everything into the camera bag and ran out the door. When I got to the venue and started putting all of the things together, I found I was missing a small but vital piece, the caddy that holds two batteries and slides into my battery grip. Without it, my DSLR wouldn’t work. Panic!

After calling home 15 times, I finally got in touch with my wife, who graciously drove across town with a small piece of plastic so I could actually turn the camera on. Fortunately, I brought a film SLR as a backup, and had just finished my last roll when she pulled up.

If I had made a simple packing checklist, much pain and fear would have been avoided.

And while I’m not a fan of oppressive standards, heavyweight processes, or detailed artifacts in software development (my thinking is that if you’re constrained by your conventions, the best you’ll ever be is conventional) simple “have I forgotten this” checklists are insanely valuable.  Based on what I’ve seen, they’re also underused. 

My first job in software was split between testing and support for a small company making technical graphics software. The testing department was pretty unstructured. We had reasonable automated test coverage (horrible by today’s standards, but OK for the time) but all manual testing was, “Hey Martin, I just fixed this bug, go poke at it!” and “We’re realeasing a beta next week, test everything!” and “We’ve got a release candidate, get the people from sales and marketing to play with it!”

So, for no other benefit than my own confidence, I made a some checklists, just to keep from forgetting to test specific features/permutations.  Eventually the company adopted my checklists and handed them around when we did our “all-hands” pre-release testing.  Just asking people (including me) to be mindful of the feature list when they were doing their exploratory testing helped us find many problems as well as places where we could improve the user experience.

I saw this again a few days ago when I was looking at a web application that made a lot of pretty common security mistakes (no SSL for login, emailing passwords in cleartext, etc.). “These are all essentially solved problems,”  I thought, “Shouldn’t there be a checklist for this sort of thing?” 

Many experienced folks have a sort of mental checklist. Things they know instinctively to look for. Like my old mentor who would always enter “O’Brien” into name fields to catch inappropriatley escaped SQL input. This is one of the reasons why domain knowledge is so valuable. How do we get people to capture and share this domain knowledge? Couldn’t a development group be able to use a set of mature “don’t forget to think about thing X” checklists as a competitive advantage for design/development/testing?

While writing this, I’m reminded of my favorite marketing professor at the UW. After spending a whole quarter discussing different theories, reading case studies, and pulling examples from real-life companies, the last day of class she said (and I’m paraphrasing here, because that was some time ago).

In the real world, when you start in marketing professionally, it’s just checklists: Have I identified the market for my product or service? Have I explained my offering to someone who doesn’t understand it? Have I underpriced? Have I overpriced? Am I saying something stupid or offensive in this ad? How will my competitors react to this change? How will my customers react to this change?  The answers are all easy, but bad marketers forget to ask the questions.

Sure, that doesn’t get you to greatness (or even guarantee goodness), but it at least keeps you from forgetting about the obvious.