Do Short Iterations Always Work?

One of the tenets of Agile is that quality software can be written via short iterations and refactoring. However, the opinion of most developers I know who are not "Agile developers" is that there are times when you need a relatively long architect, design, and initial implementation phase that can't be broken into two week iterations. If they are right, then there are a class of problems that Agile either can't solve, or can't solve as effectively as others methods.

I realize that the Agile community tends to refute this claim. I have to admit that I am skeptical though. I am pretty confident that there are times where you need a design/architect phase that doesn't fit into a short iteration cycle. However, it very well may be that the class of problems where this is true, doesn't actually occur often in practice, so it doesn't matter.

This feel like a study into this could be the basis for PhD thesis. Does anyone know if this has been rigorously studied?

Agile - Is It Worth the Hype?

I talk a lot about Agile methodologies here on this blog.  I do this, not because I am an expert, and not because I am a devout follower.  Rather I do it because I find it fascinating.

The promise of Agile is that the traditional cost of change vs time curve can be drastically flattened.  This flattening is what makes Agile agile.  Does it work?  I don't know, but if so, the techniques that help flatten this curve are incredibly valuable, whether used in an Agile framework or not.

Components of Agile

Short Iterations

Short iterations means that you get feedback from the customer early and often.  I don't know if this flattens the cost/time curve.  However, it does highlight where changes need to be made, early in the process.  i.e. Really this is just another way to do requirements gathering that is more likely to be effective in practice.

Refactoring

Accepted wisdom is that it is easier to maintain and extend clean software than hacked together systems.  Refactoring is a way of keeping the software clean when you don't have an exhaustive design stage upfront.  It also acknowledges the reality that you rarely know everything up front, so your design has to change.

While refactoring seems like a good idea, it doesn't seem to fundamentally change the shape of the cost/time curve.

Automated Unit Tests / Continuous Integration

Quality tests that are run automatically upon new changes can really decrease the odds of introducing new bugs upon changes.  This seems like something that really could flatten the cost/time curve.  Unfortunately, this seems to be the step which is not often followed through on.

Why not?

Continuous Integration isn't fundamentally that hard.  However, it does require being able to make builds in a single step.  (Question 2 on The Joel Test)  For some reason putting the work into making this happen often takes a lower priority than fixing bugs or implementing new features.

writing good tests is hardOn the other hand, writing good tests is hard.  Very hard!  I think most of the time it is easier to write the new code than it is to write the tests for the code.  Most developers aren't necessarily good at writing and designing tests, even if they are good at designing and writing the system to be tested.  And even if you can write good tests it takes time, often more time than it took to write the code being tested.

Do automated unit tests and continuous integration flatten the cost/time curve?   Well, it does raise the cost in the present, which has the effect of flattening the curve, but not in a helpful way.  It seems like it should make things cheaper in the future as it should catch new bugs as soon as they are written.  However, refactoring becomes much more expensive when you have a bunch of unit tests that have to be modified along with the code.

So to answer the question of whether this flattens the cost/time curve, I am going to give it a definite maybe.

Conclusion

There are good ideas in the Agile movement but like all fads, it probably isn't fully worth the hype.  Based on examples, it is clear that Agile can work well in the field, at least for certain problems.  However, most organizations that say they are adopting Agile only adopt parts of it.  Even though many organizations that claim to be Agile are really just trying to buzzword compliant and aren't implementing the right things to gain any benefit, I still think the ideas can be taken piecemeal.  They just can't be adopted blindly and expected to work.   As someone who doesn't like to just jump into the deep end of the pool, but rather get my feet wet first, I will continue to explore Agile techniques and talk about them here in a hope to understand what works well and why.

Software Process and Motivation

There are two main factors (besides the obvious lack of free time) that keep me from posting more frequently to my blogs.  The first is a fear of mistakes.  I fear saying something dumb, whether it is just a simple typo or if it is not thinking through my ideas enough before committing them to the Internet.  The second is that it is all for naught because no one reads what I write anyway.  However, if I want readers I have to post regularly, and if I want to get better at writing, I have to keep doing it.  So, for me, a large part of writing posts is a psychological battle with myself.

When writing software, there are also motivation issues that prevent or slow down development.  I think one of the key reasons Agile has such vocal followers is that it addresses the psychology of programming, keeping developers happy and productive.

Fear of Mistakes

While Agile may not explicitly say "Embrace Your Mistakes", the idea of constantly refactoring along with a large suite of automated tests and continuous integration attempts to address this fear.

Useful Software

The other day I was reading a friend's blog (May Contain Blueberries) and he mentioned that he read and enjoyed my blog.  This gave me a huge emotional boost, which encourages me to keep writing.  Similarly, besides all of the tangible benefits to quality that a release early and release often philosophy that Agile has, there is the huge psychological benefit to the development team having people use their software.  Nobody wants to work on something that never sees the light of day.

Staying Productive

Other ideas that Agile methodologies tend to push are Pair Programming and No Overtime.    The No Overtime philosophy is explicitly about programmer morale and mentality and keeping them sharp.   Pair Programming also has this affect.  When pair programming, you are less likely to be distracted by emails about lolcats.  You also have a compatriot who is in the same boat as you, sharing your miseries and successes.

Developers are People

Basically it comes down to the fact that developers are people.  Keeping us happy and motivated is a key component to continually getting quality software over time.   And, while free snacks are a tangible way of accomplishing this (and let me tell you, the Google snack rooms are pretty awesome), process and business practices are more important over the long haul.

As a postscript - it seems that I must feel strongly about this.  I just noticed that I had already posted on Psychology of Extreme Programming.  Oh well.  If its worth saying once, its worth saying twice.  :)  Hopefully I made enough new points to make it worthwhile.

Embrace Your Mistakes

All endeavors have their goof ups, and writing software is no exception.  There are two ways of handling with mistakes - try to prevent them, or try to fix them.  "Traditional" software engineering approaches focus on the former while "Agile" approaches focus on the latter.  Formal requirements and big-up-front-design attempt to prevent mistakes from getting into code.  Short iteration cycles with lots of customer interactions attempt to expose mistakes early and often so they can be fixed.

While the benefits of preventing mistakes is obvious, there are many less obvious benefits to embracing the mistakes and fixing them later.

Tangible

When you accept that mistakes are part of the business, you plan for them.  This leads to automated tests (to catch future mistakes), logging and other introspection techniques (like JMX) to catch and diagnose problems in the field, and hot patching or automated upgrades which have become par for the course in modern software - to fix bugs without a whole new release/deploy cycle.

These are obvious advantages and they can be (and are) used no matter what approach you take.  However, there are some psychological advantages to embracing mistakes which can really pay off in terms of getting software out the door and getting it right.

Psychological

Not being afraid of a mistake means you don't get paralyzed and can keep going.  When developing software there are countless decisions that have to be made, from small things like what to name a variable, method, or class to larger things like what data storage mechanisms to use.  With traditional approaches, your analysis step tells you everything you need to know to make the right decision.  However, in practice, often you don't have enough information, or things change.  This leads to extra long analysis and requirement gathering steps and the production of vaporware.

Accepting the idea that "it is ok to make a mistake now, because I can (and will) fix it later" is a huge step.  Once you do that, you can keep from getting bogged down with the unknowable.  Often it is by making a decision and seeing what the resulting software is, that you best understand the right decision.  For me, one of the huge advantages of subversion over CVS is the ability to change file names while keeping revision history.  Knowing that I can change a file name in the future keeps me from blocking while trying to determine exactly how a class will be used/morphed over time so I can give it the perfect name.  Instead I give it the name that makes sense today, and if the class changes over time, the name can be changed to match its new use/behavior.

However, allowing yourself to make mistakes is only useful if you also a willing to admit mistakes.  Admitting that a decision made in the past turned out to not be best is the only way that you can improve your software.  It can be hard to admit mistakes, which is why I titled this post the way I did.  If you embrace your mistakes, that means you are constantly willing to learn and improve, resulting in much higher quality software over time.

Decouple Ideas, Not Code

Here's a situation that I have witnessed (ok, been involved in) multiple times in my career. You are writing software and you make an effort to "future proof it" so that it will be easy to modify when new requirements come. However, the future comes and it is still a big effort to change. This phenomena has led to the YAGNI philosophy and to my post about not writing reusable software.

All is not lost, though.  There are still things you can do to future proof your software, and decoupling components may be one of the most powerful.  But decoupling code is often not enough, you need to decouple ideas.

Example

Imagine you are developing a .NET website in C# with dynamic data coming from multiple different sources.  One of your sources is a Content Management System (CMS) where authors and editors can update some of the web pages in production without any developer interaction.  The CMS system works by providing the authors an editors with an interface friendly to them.  It stores their work in database tables, which  your .NET application can access.

In this particular application the CMS system has two types of pages "Generic Pages" and "Articles".  The Articles database table has columns like Content and Title which contain HTML data and NumberOfWords which is an integer.  The GenericPages database table has columns:  Body, DisplayTitle, and SidebarContent, which also contain HTML data.

An obvious approach to solving this problem is to create classes GenericPage and Article which know how to read from the database and have accessors for the various fields.  You probably also want a CMSHandler class which has methods for taking a unique identifier (e.g. relative path) and returning the appropriate GenericPage or Article.  Your website code access these GenericPages and Articles via the CMSHandler.

Decoupling

These CMS classes that you wrote seem like they should be decoupled from the rest of the application.  It doesn't seem farfetched that the CMS system will change.  Maybe the company providing it will be bought by Microsoft.  Maybe you'll decide to write an in-house system.  Who knows?

So with this thought in mind you create interfaces IGenericPage, IArticle, and ICMSHandler.  Now the rest of your code base is coupled to the interface, rather than the implementation.  i.e. your website code access an IGenericPage or an IArticle via an ICMSHandler.

Success! ... Right? ... Maybe not.

Future

Fast forward two years.  Your CMS provider is bought by Microsoft, and you decide to go with a cheaper competitor.  However, the new system is a little bit different.  Besides GenericPages and Articles, it also has FAQPages.  And the Articles table in the database doesn't have a NumberOfWord columns.

While you did decouple your code base from the CMS classes, you didn't actually decouple your code from the CMS system and all its assumptions.  Now that you have a new system, you realize it doesn't work the same way as the old system.  Your code base has to change anyway to handle the new system.

Decouple the Idea

So what's the solution?  Decouple your code from the assumption in the system you are using, not just the classes.  Do you really need to handle GenericPages and Articles differently?  Or are you just going to show the pages the same, but not show sidebars for the Articles?  Do you really need a word count?  Rather than looking at the features that are provided, consider the features you really need.  Write your interface classes to provide those features.   This may mean the interface hides some of the functionality that could be provided by the underlying mechanism.  It also means that you may have to add some functionality that wasn't there.  (for example, you decide you really do need a word count, even if it isn't provided by the CMS system).

In this example it means that your website accesses an ICMSPage via an ICMSHandler.  The fact that there are different types of pages like GenericPages or Articles is hidden from your website, since the website doesn't need to know this.

If you have successfully decoupled the assumptions of your code base from the assumptions of the CMS system, then you shouldn't have trouble replacing the CMS system.  Your interface should represent the minimal set of assumptions that your application makes.  Any CMS system that satisfies these assumptions should be easy to drop in.  Or at least much easier than if your decoupling mechanism still allowed all the assumptions of the CMS system to leak through to the website itself.

Comparing Two Decoupling Approaches

At work this week I suggested an alternative to how we currently decouple code.  I got asked the eminently reasonable question, "How is that way any better?"  This post discusses the two approaches.  Note that I think that the approach I favor can be improved upon, but this is a starting point.

The Problem

Imagine you have a Frobulator class.  You use this object in many places in your code.  However, you can foresee a situation where you might replace the Frobulator class because of a change in the underlying data source.  If and when this occurs you don't want to have to go through your entire code base and change every occurrence of Frobulator to FrobulatorPrime.  In short, you want to decouple your code base from the Frobulator object.

Solution 1

Create a GenericFrobulator class.  The GenericFrobulator class has the exact same methods as the Frobulator class.  Its implementation is to have an instance of a Frobulator and for every method to just redirect the call to the Frobulator class.  The rest of your code base uses the GenericFrobulator object.  If and when you replace the Frobulator class, all you have to do is change the one class.

Cons

This requires writing a bunch of "dummy" code.  I.e. one method for every method in Frobulator each of which is a single line of pass through code.

There is also a performance hit of the extra call - though realistically this is almost definitely negligible.  Also, on errors or debugging stack traces will have an additional level in the stack, but this is only a minor level annoyance.

Pro

Successfully decouples the code base from the Frobulator object.  Only one class has to change (in addition to the creation of the new FrobulatorPrime class) when a new implementation is used.

Solution 2

Create an IFrobulator interface (we're working in C#, hence the 'I' prefix on interfaces) that has the same interfaces as the Frobulator object, and make the Frobulator class implement this interface.  Create a FrobulatorFactory class which has a single method which returns an IFrobulator object.  The implementation creates a Frobulator object.  The rest of the code base uses this FrobulatorFactory class to create the IFrobulator that it uses.

Cons

You have to create two additional classes (IFrobulator and FrobulatorFactory), one of which doesn't actually do anything.

Pros

It successfully decouples the code base from the Frobulator implementation.

Comparison

On the surface, these two approaches are pretty similar.  They both require similar amounts of code.  With the Interface approach you don't have to create "pass through" methods for every single method, but on the other hand you do have to create an extra class - the Factory class.  Both approaches involve a similar amount of work to change out the Frobulator class -- create the FrobulatorPrime class, and either modify the FrobulatorFactory or the GenericFrobulator class.

Real Difference

The real benefit to the interface approach is the primary benefit to OO in general.  Its not that it saves code, but rather it models your intent.  By having an Interface and a Factory, it is explicit what the purpose of each class is.  It is obvious that the Factory is for creating objects, and the Interface is for decoupling.

It also guards against new developers breaking the design.  With the GenericFrobulator approach, it is always possible that down the road some developer will decide to insert logic (besides a simple call-through) into this class.  Now all the decoupling power is lost.  Interfaces (at least in languages like C# and Java) can't contain any code, so this can't happen.

The Interface method also supports a hybrid model.  Imagine that you have a scenario where part of your code wants to use the Frobulator class, and part of it wants to use the FrobulatorPrime class.  How would you do this with the GenericFrobulator approach?  With the Interface and Factory approach, you can either create a new Factory, which some classes use, or you can have an additional Factory method/parameter to determine which object gets created.  (or you could pass in the IFrobulator object and make it the calling code's responsibility).

I also feel that the Interface and Factory approach is closer to the real panacea of decoupling, though this is a fuzzier argument.

Failed Decoupling Goals

There are other goals of decoupling, that both of these approaches fail at.  Both can be modified to potentially achieve them, though I feel that the Interface/Factory approach more naturally fits these modifications.

The first goal is to make it possible to change the Frobulator class out without having to make any changes to the code base.  I.e. without having to change the Factory class or the GenericFrobulator class.  An example of this is the java.db packages which you get via putting the right jar file in your classpath and using the right configuration string to get your db classes.  (you are using a config file for this, right?)

The second goal is to be able to test the rest of your code base without having a Frobulator instance at all.  Sometimes it is nice to be able to have a StubFrobulator class which creates test data, and use that to test the rest of the object.  If your code is properly decoupled, this should not be difficult.

Summary

In short, I think the Interface and Factory approach is the better approach because it more directly describes the intent, more succinctly solves the problem, and is easier to modify to handle more advanced decoupling issues.

Is there yet a different way to decouple software?  How do you do it?

Don't Try To Write Reusable Software

First a story

Early in my first job, another fresh out of school co-worker and I decided that we were going to rewrite this one library we used to make it more reusable for all of the upcoming projects.  When we asked for advice from a more experienced developer, he basically told us not to bother, because we were going to fail.  We scoffed at his cynicism, knowing that our design skills and knowledge of the requirements were going to result in this awesome library.

Fast forward a year and the "reusable" library that we spent so much time and effort on still had to go through considerable customizations to handle cases that we hadn't even envisioned when we started.

Did we fail because of incompetence?  While our skills probably weren't quite as good as we thought at the time, no.  We failed because it required the unknowable - knowing what the future would bring.  The Agile premise of YAGNI summarizes this.

The Moral

I summarize this experience with the title to this post, don't try to write reusable software.

I can hear your CS hearts shudder at this. 

How can this be?  Isn't writing reusable software what is taught in our programming classes?  You can't really be saying that we should be cutting and pasting code all over the place.  Besides there are plenty of successful reusable libraries out there, like printf or java.util.*.  This must be a false premise!

Again, I say, "Don't try to write reusable software!"

But...but...that goes against everything I learned.

The Real Moral - Discover Reusable Software

Don't try to write reusable software, discover it instead.

Huh?

Obviously you shouldn't be cutting and pasting code.  Obviously you should still be using methods and classes.  The DRY principle is a good one and it implies reusable code.  However, reusable code is the means, not the goal.  

Let me repeat that, because it is important.  Writing code that is reusable (methods, classes, etc.) is the means to achieving the goal of a clean design.  But your goal isn't the reusability, your goal is to only specify everything once.  Only write your algorithm once.  Only specify a value once.  Only put your business logic on one place.  If you strive to achieve those goals, you will end up with reusable components.  

Even better, you will end up with code that is maintainable and written in a timely manner.

Madking is back

It has been almost 2 years since I last posted here.  During that time I changed jobs twice.  Once to take a job that was too good to be true.  Then again when it turned out that it was too good to be true and they cancelled the project, leaving me looking for a job.

Not only have I not blogged in 2 years, but I also haven't competed in programming contests in that time frame.  I am finally getting back on that hobby horse.  I am participating in the TopCoder Open 2009, and as of right now I am one of 120 still standing (one of 9 Americans, if I counted right) in the Algorithm contest.  Tomorrow they narrow down to 45, so we'll see how that goes.

I figured if I could get back to the programming contests, I could also get back to the blog.  I am not going to promise any particular schedule, so I'd suggest subscribing to the RSS feed if you want to actually read what I write.  I will try to write more often than every 2 years.  :)

One other development since I last posted, is that we have gotten our own domain, and I picked a hosting site that specializes in hosting Ruby on Rails applications.  I have a couple applications that I am playing with on the site.  I am envisioning that as I work with them, they will inspire me for things to blog about.

I never know how to end posts, so I  am just going to end this one with a statement about ending posts.