Monday, August 9, 2010

JNDI, SQL, and JUnit

So I recently had a problem that I would've assumed many people in Java land have had. I have some code that will live in an app container (Tomcat, JBoss, or Weblogic probably), and I want to write unit tests for it.

Here is what the code does.
  • Get a database connection using JNDI (from the app container)
  • Does some database stuff
Here are my self imposed goals for the unit tests:
  • Runnable out of the box. (i.e. if you can checkout and build the project, you can run the unit tests)
  • Does not require an app container. I don't want to require a running instance of JBoss just to have access to JNDI so I can run unit tests.
  • Each developer has their own unit testing database sandbox to run unit tests in, so they won't/can't collide with each other.
  • Ideally the unit testing database is checked into the repository.
These seemed like reasonable goals to me, and I figured a few minutes of Google searching would give a couple of options of how this is typically done. Since it took more than that, I'll write up what I did.

HSQLDB

StackOverflow led me to HSQLDB. With HSQLDB I am able to get an in-memory database, which is perfect for unit testing. And I can make sure all other team members can use it, just by including the jar file in the check-in. Well, that seems easy enough.

JNDI

Now all I had to do was to make the HSQLDB DataSource available via JDBC. This should be easy enough, right? Right? Ugh.

I found this blog that seems to be the standard way to inject a JNDI datasource without an app container. Seems easy enough, except that when I tried to follow the directions, I got a ClassNotFoundException for org.apache.naming.java.javaURLContextFactory. It seems that this class is included with Tomcat. I suppose I could've gone and gotten the Tomcat jar to include in the project, but this didn't feel right with my goal to be app container agnostic.

So with a little more searching I found an in-memory JNDI context on SourceForge. Well, that seems to be what I want.

[edit - As I wrote in my follow up post, Don't Reuse. Rewrite!, I ended up writing my own in memory JNDI context, since all I needed were the bind and lookup methods.]

Huh - now that I write it up, it seems much less painful than it was when I was trying to track all this down myself.

Implementation Summary

Download two packages:
I added 3 jars to my CLASSPATH.
  • hsqldb.jar - comes with the HSQLDB download.
  • jndi.jar - comes with the in-memory JNDI context. [edit - don't use any more]
  • util.jar - also with the in-memory JNDI context. (and yes - they really did name the jar file unit.jar) [edit - don't use any more]
As for the actual unit testing code, I am using JUnit 4. This means that I can put the setup code once per class. As such my code looks something like:
  1 import java.sql.Connection;
2 import java.sql.Statement;
3
4 import javax.naming.Context;
5 import javax.naming.InitialContext;
6 import javax.sql.DataSource;
7
8 import jndi.naming.provider.MemoryContextFactory;
8 import mypackage.MockInitialContextFactory;
9
10 import org.hsqldb.jdbc.JDBCDataSource;
11 import org.junit.BeforeClass;
12
13 public class MyUnitTest {
14 private static final String JNDI_NAME="my_JNDI_name";
15
16 @BeforeClass
17 public static void setUpBeforeClass() throws Exception {
18 createJNDIContext();
19 createDBTable();
20 }
21
22 public static void createJNDIContext() throws Exception {
23 JDBCDataSource ds = new JDBCDataSource();
24 ds.setDatabase("jdbc:hsqldb:mem:mymemdb");
25 ds.setUser("SA");
26
27 System.setProperty(Context.INITIAL_CONTEXT_FACTORY,
28 MemoryContextFactory.class.getName());
28 MockInitialContextFactory.class.getName());
29 InitialContext ic = new InitialContext();
30 ic.bind(JNDI_NAME, ds);
31 }
32
33 private static void createDBTable() throws Exception {
34 DataSource ds = (DataSource)new InitialContext().lookup(JNDI_NAME);
35 Connection conn = ds.getConnection();
36 String create = "CREATE TABLE MY_TABLE ("
37 // Insert column description here.
38 + " )";
39 Statement stmt = conn.createStatement();
40 stmt.executeUpdate(create);
41 conn.close();
42 }
43
44 // Unit Tests go here
45 }
46
Note - this code was colorized by: http://puzzleware.net/codehtmler/default.aspx.


Saturday, March 13, 2010

ASP.NET UpdatePanel must be added before Event Handling step

This post is specific to AJAX programming in the ASP.NET environment. This is mostly so that if I forget this issue 6 months from now I'll have something to remind me, since I was unable to find anything on the web about this. So if you have no interest in AJAX in the .NET environment, feel free to stop reading now.

Background

Just in case anyone without .NET experience is still reading I will give a brief high level overview of what is going on. When you hit a .aspx page, the execution goes through multiple stages including PageInit, PageLoad, and EventHandling. I am not going to go into the intricacies here, just suffice it to say that they happen in order and you can define specific functionality for each step.

UpdatePanel is the ASP object that you use to wrap parts of your web page that you want to update asynchronously. Content in an UpdatePanel can be updated without a full page reload, and buttons clicked in an UpdatePanel operate in a AJAX fashion and don't cause a full page reload.

Oh, and content to your web page can be added either declaratively in the .aspx file (similar to php) or programmatically in the C# (or other language) code behind file.

Problem

The problem that I have run across is that UpdatePanels that are added programmatically during the event handling phase don't work in AJAX fashion. This means that if you add an UpdatePanel to your page as the result of a button click it doesn't work.

Demo Program
Here is a little demo program that has 3 update panels. The first is defined declaratively, the other two programmatically, one during the init phase, and one during the load phase. There are also two buttons. The "AJAX Update" button causes an AJAX event which causes all 3 panels to update themselves - so they will all show a new time.

The "Add Panel" button adds a fourth UpdatePanel to the page. This one is added during the event handling phase (button click handling). In theory, this panel should act like the others. However after adding this panel, if you click the "AJAX Update" button, you'll see that only the first 3 panels update their time, and not the last one.


Code

Here is the code for the above demo. In case you are curious, code formatting on this post was done by: http://www.manoli.net/csharpformat/ I hope the styles apply correctly when I post this.

ViewForm3.aspx


   1:  <%@ Page Language="C#" AutoEventWireup="true" CodeBehind="ViewForm3.aspx.cs" Inherits="ViewForm3" %>
   2:  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
   3:  <html xmlns="http://www.w3.org/1999/xhtml" >
   4:  <head runat="server">
   5:      <title>Test Update Panel</title>
   6:  </head>
   7:  <body>
   8:    <form runat="server">
   9:      <asp:ScriptManager runat="server" />
  10:      <asp:UpdatePanel runat="server">
  11:        <ContentTemplate>
  12:          Last Updated at: <%= DateTime.Now %>
  13:          <asp:Button runat="server" Text="AJAX Update" />
  14:        </ContentTemplate>
  15:      </asp:UpdatePanel>
  16:      <asp:PlaceHolder runat="server" ID="PlaceHolder" />
  17:      <asp:Button runat="server" Text="Add Panel" OnClick="Add_Click" />
  18:    </form>
  19:  </body>
  20:  </html>

ViewForm3.aspx.cs


   1:  using System;
   2:  using System.Web.UI;
   3:   
   4:  public partial class ViewForm3 : Page
   5:  {
   6:    protected void Page_Init(object sender, EventArgs e)
   7:    {
   8:      AddUpdatePanel("Init updated at: " + DateTime.Now);
   9:    }
  10:   
  11:    protected void Page_Load(object sender, EventArgs e)
  12:    {
  13:      AddUpdatePanel("Load updated at: " + DateTime.Now);
  14:    }
  15:   
  16:    protected void Add_Click(object sender, EventArgs e)
  17:    {
  18:      AddUpdatePanel("Click updated at: " + DateTime.Now);
  19:    }
  20:   
  21:    private void AddUpdatePanel(string message)
  22:    {
  23:      UpdatePanel up = new UpdatePanel();
  24:      LiteralControl lit = new LiteralControl(message);
  25:      up.ContentTemplateContainer.Controls.Add(lit);
  26:      PlaceHolder.Controls.Add(up);
  27:    }
  28:  }


Why this is?

I have no idea why this is, and that bugs me. I was unable to find any examples on the web of people doing this, so didn't learn anything there. If anyone who reads this understands why this behavior is as it is, please email me, or add a comment to this post. (Note - this was done using Visual Studio 2008 with ASP.NET 3.5, if that makes a difference). If I learn more - I will post an update.

Friday, July 3, 2009

Do Short Iterations Always Work?


One of the tenets of Agile is that quality software can be written via short iterations and refactoring. However, the opinion of most developers I know who are not "Agile developers" is that there are times when you need a relatively long architect, design, and initial implementation phase that can't be broken into two week iterations. If they are right, then there are a class of problems that Agile either can't solve, or can't solve as effectively as others methods.

I realize that the Agile community tends to refute this claim. I have to admit that I am skeptical though. I am pretty confident that there are times where you need a design/architect phase that doesn't fit into a short iteration cycle. However, it very well may be that the class of problems where this is true, doesn't actually occur often in practice, so it doesn't matter.

This feel like a study into this could be the basis for PhD thesis. Does anyone know if this has been rigorously studied?

Sunday, June 7, 2009

Agile - Is It Worth the Hype?


I talk a lot about Agile methodologies here on this blog.  I do this, not because I am an expert, and not because I am a devout follower.  Rather I do it because I find it fascinating.

The promise of Agile is that the traditional cost of change vs time curve can be drastically flattened.  This flattening is what makes Agile agile.  Does it work?  I don't know, but if so, the techniques that help flatten this curve are incredibly valuable, whether used in an Agile framework or not.

Components of Agile

Short Iterations

Short iterations means that you get feedback from the customer early and often.  I don't know if this flattens the cost/time curve.  However, it does highlight where changes need to be made, early in the process.  i.e. Really this is just another way to do requirements gathering that is more likely to be effective in practice.

Refactoring

Accepted wisdom is that it is easier to maintain and extend clean software than hacked together systems.  Refactoring is a way of keeping the software clean when you don't have an exhaustive design stage upfront.  It also acknowledges the reality that you rarely know everything up front, so your design has to change.

While refactoring seems like a good idea, it doesn't seem to fundamentally change the shape of the cost/time curve.

Automated Unit Tests / Continuous Integration

Quality tests that are run automatically upon new changes can really decrease the odds of introducing new bugs upon changes.  This seems like something that really could flatten the cost/time curve.  Unfortunately, this seems to be the step which is not often followed through on.

Why not?

Continuous Integration isn't fundamentally that hard.  However, it does require being able to make builds in a single step.  (Question 2 on The Joel Test)  For some reason putting the work into making this happen often takes a lower priority than fixing bugs or implementing new features.

writing good tests is hardOn the other hand, writing good tests is hard.  Very hard!  I think most of the time it is easier to write the new code than it is to write the tests for the code.  Most developers aren't necessarily good at writing and designing tests, even if they are good at designing and writing the system to be tested.  And even if you can write good tests it takes time, often more time than it took to write the code being tested.

Do automated unit tests and continuous integration flatten the cost/time curve?   Well, it does raise the cost in the present, which has the effect of flattening the curve, but not in a helpful way.  It seems like it should make things cheaper in the future as it should catch new bugs as soon as they are written.  However, refactoring becomes much more expensive when you have a bunch of unit tests that have to be modified along with the code.

So to answer the question of whether this flattens the cost/time curve, I am going to give it a definite maybe.

Conclusion

There are good ideas in the Agile movement but like all fads, it probably isn't fully worth the hype.  Based on examples, it is clear that Agile can work well in the field, at least for certain problems.  However, most organizations that say they are adopting Agile only adopt parts of it.  Even though many organizations that claim to be Agile are really just trying to buzzword compliant and aren't implementing the right things to gain any benefit, I still think the ideas can be taken piecemeal.  They just can't be adopted blindly and expected to work.   As someone who doesn't like to just jump into the deep end of the pool, but rather get my feet wet first, I will continue to explore Agile techniques and talk about them here in a hope to understand what works well and why.

Tuesday, May 26, 2009

Software Process and Motivation

There are two main factors (besides the obvious lack of free time) that keep me from posting more frequently to my blogs.  The first is a fear of mistakes.  I fear saying something dumb, whether it is just a simple typo or if it is not thinking through my ideas enough before committing them to the Internet.  The second is that it is all for naught because no one reads what I write anyway.  However, if I want readers I have to post regularly, and if I want to get better at writing, I have to keep doing it.  So, for me, a large part of writing posts is a psychological battle with myself.

When writing software, there are also motivation issues that prevent or slow down development.  I think one of the key reasons Agile has such vocal followers is that it addresses the psychology of programming, keeping developers happy and productive.

Fear of Mistakes

While Agile may not explicitly say "Embrace Your Mistakes", the idea of constantly refactoring along with a large suite of automated tests and continuous integration attempts to address this fear.

Useful Software

The other day I was reading a friend's blog (May Contain Blueberries) and he mentioned that he read and enjoyed my blog.  This gave me a huge emotional boost, which encourages me to keep writing.  Similarly, besides all of the tangible benefits to quality that a release early and release often philosophy that Agile has, there is the huge psychological benefit to the development team having people use their software.  Nobody wants to work on something that never sees the light of day.

Staying Productive

Other ideas that Agile methodologies tend to push are Pair Programming and No Overtime.    The No Overtime philosophy is explicitly about programmer morale and mentality and keeping them sharp.   Pair Programming also has this affect.  When pair programming, you are less likely to be distracted by emails about lolcats.  You also have a compatriot who is in the same boat as you, sharing your miseries and successes.

Developers are People

Basically it comes down to the fact that developers are people.  Keeping us happy and motivated is a key component to continually getting quality software over time.   And, while free snacks are a tangible way of accomplishing this (and let me tell you, the Google snack rooms are pretty awesome), process and business practices are more important over the long haul.


As a postscript - it seems that I must feel strongly about this.  I just noticed that I had already posted on Psychology of Extreme Programming.  Oh well.  If its worth saying once, its worth saying twice.  :)  Hopefully I made enough new points to make it worthwhile.

Saturday, May 16, 2009

Embrace Your Mistakes


All endeavors have their goof ups, and writing software is no exception.  There are two ways of handling with mistakes - try to prevent them, or try to fix them.  "Traditional" software engineering approaches focus on the former while "Agile" approaches focus on the latter.  Formal requirements and big-up-front-design attempt to prevent mistakes from getting into code.  Short iteration cycles with lots of customer interactions attempt to expose mistakes early and often so they can be fixed.

While the benefits of preventing mistakes is obvious, there are many less obvious benefits to embracing the mistakes and fixing them later.

Tangible

When you accept that mistakes are part of the business, you plan for them.  This leads to automated tests (to catch future mistakes), logging and other introspection techniques (like JMX) to catch and diagnose problems in the field, and hot patching or automated upgrades which have become par for the course in modern software - to fix bugs without a whole new release/deploy cycle.

These are obvious advantages and they can be (and are) used no matter what approach you take.  However, there are some psychological advantages to embracing mistakes which can really pay off in terms of getting software out the door and getting it right.

Psychological

Not being afraid of a mistake means you don't get paralyzed and can keep going.  When developing software there are countless decisions that have to be made, from small things like what to name a variable, method, or class to larger things like what data storage mechanisms to use.  With traditional approaches, your analysis step tells you everything you need to know to make the right decision.  However, in practice, often you don't have enough information, or things change.  This leads to extra long analysis and requirement gathering steps and the production of vaporware.

Accepting the idea that "it is ok to make a mistake now, because I can (and will) fix it later" is a huge step.  Once you do that, you can keep from getting bogged down with the unknowable.  Often it is by making a decision and seeing what the resulting software is, that you best understand the right decision.  For me, one of the huge advantages of subversion over CVS is the ability to change file names while keeping revision history.  Knowing that I can change a file name in the future keeps me from blocking while trying to determine exactly how a class will be used/morphed over time so I can give it the perfect name.  Instead I give it the name that makes sense today, and if the class changes over time, the name can be changed to match its new use/behavior.

However, allowing yourself to make mistakes is only useful if you also a willing to admit mistakes.  Admitting that a decision made in the past turned out to not be best is the only way that you can improve your software.  It can be hard to admit mistakes, which is why I titled this post the way I did.  If you embrace your mistakes, that means you are constantly willing to learn and improve, resulting in much higher quality software over time.


Saturday, May 2, 2009

Decouple Ideas, Not Code

Here's a situation that I have witnessed (ok, been involved in) multiple times in my career. You are writing software and you make an effort to "future proof it" so that it will be easy to modify when new requirements come. However, the future comes and it is still a big effort to change. This phenomena has led to the YAGNI philosophy and to my post about not writing reusable software.

All is not lost, though.  There are still things you can do to future proof your software, and decoupling components may be one of the most powerful.  But decoupling code is often not enough, you need to decouple ideas.

Example



Imagine you are developing a .NET website in C# with dynamic data coming from multiple different sources.  One of your sources is a Content Management System (CMS) where authors and editors can update some of the web pages in production without any developer interaction.  The CMS system works by providing the authors an editors with an interface friendly to them.  It stores their work in database tables, which  your .NET application can access.

In this particular application the CMS system has two types of pages "Generic Pages" and "Articles".  The Articles database table has columns like Content and Title which contain HTML data and NumberOfWords which is an integer.  The GenericPages database table has columns:  Body, DisplayTitle, and SidebarContent, which also contain HTML data.

An obvious approach to solving this problem is to create classes GenericPage and Article which know how to read from the database and have accessors for the various fields.  You probably also want a CMSHandler class which has methods for taking a unique identifier (e.g. relative path) and returning the appropriate GenericPage or Article.  Your website code access these GenericPages and Articles via the CMSHandler.

Decoupling



These CMS classes that you wrote seem like they should be decoupled from the rest of the application.  It doesn't seem farfetched that the CMS system will change.  Maybe the company providing it will be bought by Microsoft.  Maybe you'll decide to write an in-house system.  Who knows?

So with this thought in mind you create interfaces IGenericPage, IArticle, and ICMSHandler.  Now the rest of your code base is coupled to the interface, rather than the implementation.  i.e. your website code access an IGenericPage or an IArticle via an ICMSHandler.

Success! ... Right? ... Maybe not.

Future

Fast forward two years.  Your CMS provider is bought by Microsoft, and you decide to go with a cheaper competitor.  However, the new system is a little bit different.  Besides GenericPages and Articles, it also has FAQPages.  And the Articles table in the database doesn't have a NumberOfWord columns.

While you did decouple your code base from the CMS classes, you didn't actually decouple your code from the CMS system and all its assumptions.  Now that you have a new system, you realize it doesn't work the same way as the old system.  Your code base has to change anyway to handle the new system.

Decouple the Idea



So what's the solution?  Decouple your code from the assumption in the system you are using, not just the classes.  Do you really need to handle GenericPages and Articles differently?  Or are you just going to show the pages the same, but not show sidebars for the Articles?  Do you really need a word count?  Rather than looking at the features that are provided, consider the features you really need.  Write your interface classes to provide those features.   This may mean the interface hides some of the functionality that could be provided by the underlying mechanism.  It also means that you may have to add some functionality that wasn't there.  (for example, you decide you really do need a word count, even if it isn't provided by the CMS system).

In this example it means that your website accesses an ICMSPage via an ICMSHandler.  The fact that there are different types of pages like GenericPages or Articles is hidden from your website, since the website doesn't need to know this.

If you have successfully decoupled the assumptions of your code base from the assumptions of the CMS system, then you shouldn't have trouble replacing the CMS system.  Your interface should represent the minimal set of assumptions that your application makes.  Any CMS system that satisfies these assumptions should be easy to drop in.  Or at least much easier than if your decoupling mechanism still allowed all the assumptions of the CMS system to leak through to the website itself.