Madking's Musings

Chasing Technology

2011-10-31T09:00:00.000-04:00

Maybe I'm just getting old, but keeping up with technology seems to be getting harder and harder. As I've mentioned before, I am a big fan of learning your libraries, languages, and tools. But how are you supposed to do that, when the libraries keep upgrading and there are constantly new tools to use? Just this past week there was a post on slashdot suggesting that tech skills have a 2 year half life. The comments on that thread seem to mostly disagree with the hypothesis, but there is definitely some truth to it.

The two languages that I use the most at work did not exist when I started college. However, languages are easy to keep up with. They tend to move slowly and are very well documented. Tools, libraries, and frameworks, on the other hand, are a nightmare to chase. They are constantly being upgraded, and often times the best documentation are the question/answers out there on the web at places like stackoverflow. The problem with these sites is that they are only useful after a tool has been out long enough for a critical mass of people to use.

So what's a person supposed to do?

Well, first, as many commenters from the slashdot article stated, the most valuable skills a software engineer has is knowing how to design, develop, and debug software. These skills don't go away just because you are using a new language or library. So play to your strengths. If you design and write clean code, it'll be easier to fix/update as you learn the language/library/tool better.

Second, choose your upgrades carefully and intentionally. Just because there is a newer version of something out there, doesn't mean you need to use it. Are the benefits worth it to upgrade now? Does it make more sense to wait until it has been popularly adopted? Or maybe it makes sense to skip a version or two.

Third, don't become an expert in everything. Most of us have finite amounts of energy and time. So, while I'm a big believer in being an expert in the tools you are using, that isn't always practical. If a particular tool/library/framework is bound to be upgraded/replaced in the near future, it's probably ok if you use the existing library in a less than optimal way. Focus your energies on the timeless (design and coding skills) or the long lasting (languages).

constantly learnFourth, and most importantly, constantly learn. If your employer doesn't provide opportunities for professional growth, take it anyway. Just as you don't ask for permission for every single test that you write, or every whiteboard design that you draw, you don't need permission to explore new technologies and tools. Whether they know it or not, your employer pays you to be a good and competent software developer. This requires you to be constantly learning new and better ways of doing things. If you are not taking ten to twenty percent of your time exploring and learning, you are shortchanging yourself and making yourself obsolete. I realize that it seems there is never time to do this, but like other aspects of good software engineering (appropriate design, testing, documentation), in the long run you will be worse off if you don't make the investment now.

Vexing Bugs

2011-10-10T09:00:00.000-04:00

While bugs are a part of development, there are a few types of bugs that I find particularly vexing: intermittent bugs, library bugs, and bugs that only happen in production.

Intermittent Bugs

It's annoying to perform the exact same set of steps 4 times, have it work 3 of the times, and fail once. When Murphy has his way, it works when you, perform the steps or are watching, but fails when the user does it on their own. How do you diagnose a bug that you can not reliably reproduce?

Library Bugs

The majority of library bugs aren't actually bugs in the library, but rather a bug in how you are using the library. Either way, having cryptic error messages coming out of the bowels of a library, when there is no apparent connection between the error and your code, is not fun. I find the process of googling error messages or library usages to be much more frustrating than tracking down errors that are entirely in my own source code.

Production Bugs

While we strive to have our development environment similar to the production environment, there are certain discrepancies that always creep in. Things like debug information, optimization level, database source, etc. When in development, you don't want to be messing with production data, and your willing to give up some performance to better track the code. But since it isn't the same, code that works fine in development doesn't always work in production. Ugh.

This Week

So what prompted this post? Well, this week I had a bug that was at the intersection of these bugs. We had deployed a new version of our application, and suddenly I started getting server emails about errors. It was some cryptic error happening within the library we use for making Web Service calls. And, of course, the errors only happened sometimes. Even more frustrating, when trying to reproduce the error on my local box using WEBrick in development mode, the error never happened.

After setting up a development server on my local box that was configured just like the production box, I was able to intermittently reproduce the error. Using my standard debugging technique of adding printfs, I eventually noticed output that looked like:

Starting Web Service Call 1
Starting Web Service Call 2
Exception Caught from Web Service Call 2
Ending Web Service Call 1

Unfortunately, it didn't catch my eye right away, but I eventually noticed that every time there was an error, the web service calls were getting interleaved. i.e. Web Service Call 2 was starting before Web Service Call 1 started. And now it suddenly made sense.

The two web service calls were being made as the result of two separate AJAX callbacks from a web page. Since the the two web service calls were being done in two different web requests, they were being handled in parallel resulting in a race condition. As it turns out, the web service library we were using is not thread safe. However, apparently WEBrick in development mode was serializing these requests (i.e. not processing 2 until 1 was completely handled), and so in development there were no threading issues.

As with most bugs, once the problem had been diagnosed, it wasn't that hard to fix. In this case, we put the web service calls in a critical section, forcing them to be serialized. For our particular use case, the higher latency of serializing the web service calls was acceptable, allowing for a fairly simple solution.

Moral

This post wasn't really intended to have a moral. It was mostly just me talking about my week, but I suppose there are a couple good ideas.

If you think its a library bug, you are probably misusing the library. (i.e. Learn your libraries)
Intermittent errors are indicative of threading issues.
Know the differences between your development and production environments.
oh, and maybe most importantly of all
When frustrated, take a break. I didn't actually track down the bug until I walked away from it for a while and then came back to it with fresh eyes.

Learn Your Libraries

2011-09-26T09:00:00.000-04:00

Story 1
I still remember the month long final project from my first ever CS class. Actually, I remember almost nothing from the project except for a bug that caused me to have to throw away my first three weeks of work. I had just learned about enumerations in Pascal and thought they were the neatest thing since sliced bread. I designed my entire project around enumerations. Users would enter their menu choices via enumerations. I would display the enumerations as options. All of this depended very heavily on my understanding of enumerations which, as it turned out, was in no way related to what enumerations actually are.

Unfortunately, I didn't find this out until I had spent two weeks writing the entire program and tried running it. It didn't work at all. Like any good developer I randomly changed various lines of code until it worked. A week later, with the project deadline looming, my project was not any closer to working. It was time for drastic action. Since I couldn't seem to get the enumerations to do what they were "supposed" to do, I actually read up on them. Imagine my dismay when I realized that all the assumptions that I had made were wrong. And not just a little bit wrong. The reality (enums are a typesafe way to have named constants) and my vision (they'll do everything I want, including cooking dinner and solving the halting problem) were completely unrelated. I spent the rest of the day griping and complaining about why would anybody create such a useless design feature. When I realized that that wasn't getting me any closer to a program that I could actually turn in, I finally sat down and rewrote the entire program using only Pascal features that I actually understood, and eschewed enumerations entirely.

Story 2
The other day a friend asked me what I knew about Java Serialization. After asking for details, it turns out that he was working on a project where they were doing bit-diddling on the Serialized output of an object. One of the assumptions that they had was that if you had an objects instance1 and instance2, and you serialize both of them, that the serialized byte data would be identical if instance1 and instance2's fields are all identical. They had run a bunch of experiments and it seemed like their assumption wasn't always holding. I think the question to me was hoping that I would point out something they had missed that would give them an easy fix.

Upon check the Serialization Spec, it turns out their assumption was not valid. My suggested solution was to not use Java's Serialization, but rather write their own conversion to bytes that would fit their specific needs. This may seem like reinventing the wheel, but as I've mentioned before, I believe that rolling your own is often the way to go.

Story 3
On a project I once worked on, the decision was made that all data was going to be handled as XML. All of the data was stored as an XML document in a single column in a database. To process individual data items, XPath queries would be made. HTML reports would be created by running giant XSLT transforms on the data. The assumption was that because XML has such great support, it would be easy to add new data (just turn it into XML and merge it with the other data), and create new reports (just write a new XSLT).

The reality is that just because you can create a general tool to handle syntactically correct XML, you still need to customize the logic for the semantics. Which meant every new data source added still required code and logic changes. As for generating the HTML via XSLT transformations, its very difficult to use modern HTML features, like Ajax, this way. And by making such a large portion of the code base XSLT and XPath, we couldn't take advantage of the IDE's ability to do easy code completion, navigation, and refactoring, or leverage the development team's OO skill to create small, easily testable, and reusable components. And then there's the performance issues, both in time and space, that were encountered with trying to manipulate and store large quantities of large XML documents.

Moral
The moral of these stories is that you need to know your tools. Enumerations, Serialization, and XML are all great, when used appropriately. When not... well, invalid assumptions lead to bugs. Invalid assumptions about crucial components of your software lead to large scale rewrites. To avoid the large scale rewrite, we often have a period of denial where we try to make the unworkable work. The worst case is that with enough bubble gum, chicken wire, and ingenuity we do make it work, at least some of the time. This results in a maintenance nightmare keeping it working and large scale rewrites are even less likely to happen once you've got "working" code.

While we all know that assumptions can be bad, the danger is when we don't realize we are making assumptions. In the first story I thought I know what enumerations did. The developers in the second story had used Java Serialization on many projects and thought they fully understood it. In the third story the benefits of XML were considered without awareness of all of the limitations and restrictions that came along. i.e. We didn't know what we didn't know.

The conservative solution to this problem is to only use technologies you have already used successfully on all your projects. While this seems to be favored by many people, I am too much a fan of shiny objects and so like to try new things. The key to using new technologies (or existing ones in new ways) is to limit the scope until you have used it successfully. If it is going to be integral to your design, create a small throw-away prototype first to make sure you find the dark corners and sharp edges first.

The most important solution, though, is to be willing to admit you were wrong. Its impossible to get very far in life without making assumptions, and despite your best precautions, sooner or later, you will be wrong about an assumption. Rather than try to patch on hack after hack to try to make a bad assumption viable, be willing to break from the past and rewrite/redesign code given your new knowledge. Even if it means throwing away a lot of work.

RSpec Lessons Learned

2011-09-19T09:00:00.000-04:00

Previously I showed some unit tests written in RSpec. Those test were fairly ugly, partially because I didn't really understand RSpec. In an effort to have better unit tests, I have learned a little more about RSpec.

Read the Latest Documentation
You'd think this would be obvious, but given the lack of links to the documentation on the various "here's advice on using RSpec" blogs, it must not be done a lot. Worse, somehow I had been only looking at the old documentation for version 1 of RSpec. So, if you are going to use RSpec, I suggest you read latest documentation, which as of when I am writing this is RSpec 2.6. This will definitely be helpful.

Make Tests Self Documenting
Each test (it or specify block) can take a description. Sometimes this is necessary, but any time you have documentation (and that's what this description is), you risk having the documentation be out of sync with the code. While not everyone agrees, I am a fan of letting the tests document themselves, whenever possible.

Make Use of Context
Use the "context" keyword to describe what you are doing in a "before" block to set up the state necessary for testing.

describe Thing do
  let(:thing) {Thing.new}
  subject {thing}
  context "empty object" do
    #insert tests
  end
  context "with inherited object" do
    let(:base) {Thing.new}
    before(:each) do
      base["foo"] = 5
      thing.prototype = base
      thing["baz"] = 15
    end
    #insert tests
  end
end

Make Use of Describe
Use the "describe" keyword to describe either the noun that is being tested, or the actions that are under test. i.e. if you have actions in a "before" block that are the actions being tested, you should you use "describe" rather than "context"

describe Thing do
  let(:thing) {Thing.new}
  subject {thing}
  context "empty object" do
    describe "when assigning via properties" do
      before(:each) do
        thing["foo"] = 5
        thing["bar"] = "hello"
      end
      # insert checks
    end
  end
end

Make Use of Subject
If you have multiple tests on a single object, make it the RSpec Subject and put it in a describes block. This way, all of your "should" comparisons will be implicitly on this object.

describe Thing do
  let(:thing) {Thing.new}
  subject {thing}
  context "empty object" do
    describe "keys" do
      subject {thing.keys}
      it {should_not be_nil}
      it {should be_empty}
    end
  end
end

Use Its
Often times you want to test the properties of an object. You can use the "its" method to have the implicit subject of "should" comparisons be the result of the method or array dereference specified.

describe Thing do
  let(:thing) {Thing.new}
  subject {thing}
  context "empty object" do
    its(['newProperty']) {should be_nil}
    its('newMethod') {should be_nil}
  end
end

Let and Subject are Lazy Loaded
The "variables" defined in "let" calls and the "subject" aren't actually evaluated until they are used. So if you never reference a variable specified in a "let", then that code is never executed. This also means that order isn't important. i.e. the following will work:

let (:a) {b + 1}
let (:b) {5}
specify("show using let") {a.should == (b+1)}

Shared_examples and shared_contexts are Global
I actually haven't found this documented, so I may be doing something wrong. But I have found that if I have two different rspec files that each have a "shared_examples_for 'test this object'", this causes problems. I can test each rspec file in isolation and it is fine. But if I try to test both at the same time, I get an error saying that a shared example already exists with the name "test this object".

There are two different solutions to this problem that I have found. If the shared examples are the same, pull them into a common Module that is included. Note, this is better than having repeated code anyway. If the examples are different, then you have to be more unique with the names of the shared examples.

Final Thoughts
It seems that the approach of RSpec is to write tests such that they are self descriptive and so that each test tests exactly one thing. While in theory, this sounds good, I am finding that this results in very verbose test files. That's even with the cleaning up that I have done after learning RSpec better. i.e. I like a lot of what RSpec does, but I am not convinced that it is the best way to write unit tests.

Rewritten Tests
Using what I have learned, I have rewritten the unit tests from before. Here is what they look like now:

require 'spec_helper'

describe Thing do
  let(:thing) {Thing.new}
  subject {thing}

  shared_examples_for "simple object" do |map, self_keys|
    describe "keys" do
      subject {thing.keys}
      it {should have(map.size).items}
      it {should include(*(map.keys))}
      it {should_not include("noSuchProperty")}
    end
    describe "self_keys" do
      subject {thing.self_keys}
      before(:each) { self_keys ||= map.keys}
      it {should have(self_keys.size).items}
      it {should include(*self_keys)}
      it {should_not include("noSuchProperty")}
    end
    describe "fields" do
      map.each do |k, v|
        its([k]) {should == v}
      end
      its(['noSuchProperty']) {should be_nil}
    end
    describe "methods" do
      map.each do |k, v|
        its(k) {should == v}
      end
      its('noSuchMethod') {should be_nil}
    end
    describe "to_hash" do
      subject {thing.to_hash}
      it {should_not be_nil}
      it {should have(map.size).items}
      it {should include(*map.keys)}
      it {should include(map)}
    end
  end
  
  context "empty object" do
    describe "keys" do
      subject {thing.keys}
      it {should_not be_nil}
      it {should be_empty}
    end
    its(['newProperty']) {should be_nil}
    its('newMethod') {should be_nil}
    describe "to_hash" do
      subject {thing.to_hash}
      it {should_not be_nil}
      it {should have(0).items}
    end
    
    describe "when assigning via properties" do
      before(:each) do
        thing["foo"] = 5
        thing["bar"] = "hello"
      end
      it_should_behave_like "simple object", 'foo'=>5, 'bar'=>"hello"
    end
    
    describe "when assigning via methods" do
      before(:each) do
        thing.foo = 5
        thing.bar = "hello"
      end
      it_should_behave_like "simple object", 'foo'=>5, 'bar'=>"hello"
    end
  end
  
  context "with inherited object" do
    let(:base) {Thing.new}
    before(:each) do
      base["foo"] = 5
      base["bar"] = "hello"
      thing.prototype = base
      thing["baz"] = 15
      thing["bye"] = "bye"
    end
    it_should_behave_like "simple object", 
      {'foo'=>5, 'bar'=>"hello", 'baz'=>15, 'bye'=>"bye"}, ['baz', 'bye']
    describe "when overriding values" do
      before(:each) do
        thing["foo"] = 25
        thing["bar"] = "hola"
      end
      it_should_behave_like "simple object", 'foo'=>25, 'bar'=>"hola", 'baz'=>15, 'bye'=>"bye"
    end
  end
end

Below is what the output looks like. As you can see, if you read it, it describes the tests that are being run more clearly than the old version of the tests.

$ rspec spec/models/thing_spec.rb 

Thing
  empty object
    keys
      should not be nil
      should be empty
    ["newProperty"]
      should be nil
    newMethod
      should be nil
    to_hash
      should not be nil
      should have 0 items
    when assigning via properties
      it should behave like simple object
        keys
          should have 2 items
          should include "foo" and "bar"
          should not include "noSuchProperty"
        self_keys
          should have 2 items
          should include "foo" and "bar"
          should not include "noSuchProperty"
        fields
          ["foo"]
            should == 5
          ["bar"]
            should == "hello"
          ["noSuchProperty"]
            should be nil
        methods
          foo
            should == 5
          bar
            should == "hello"
          noSuchMethod
            should be nil
        to_hash
          should not be nil
          should have 2 items
          should include "foo" and "bar"
          should include {"foo"=>5, "bar"=>"hello"}
    when assigning via methods
      it should behave like simple object
        keys
          should have 2 items
          should include "foo" and "bar"
          should not include "noSuchProperty"
        self_keys
          should have 2 items
          should include "foo" and "bar"
          should not include "noSuchProperty"
        fields
          ["foo"]
            should == 5
          ["bar"]
            should == "hello"
          ["noSuchProperty"]
            should be nil
        methods
          foo
            should == 5
          bar
            should == "hello"
          noSuchMethod
            should be nil
        to_hash
          should not be nil
          should have 2 items
          should include "foo" and "bar"
          should include {"foo"=>5, "bar"=>"hello"}
  with inherited object
    it should behave like simple object
      keys
        should have 4 items
        should include "foo", "bar", "baz", and "bye"
        should not include "noSuchProperty"
      self_keys
        should have 2 items
        should include "baz" and "bye"
        should not include "noSuchProperty"
      fields
        ["foo"]
          should == 5
        ["bar"]
          should == "hello"
        ["baz"]
          should == 15
        ["bye"]
          should == "bye"
        ["noSuchProperty"]
          should be nil
      methods
        foo
          should == 5
        bar
          should == "hello"
        baz
          should == 15
        bye
          should == "bye"
        noSuchMethod
          should be nil
      to_hash
        should not be nil
        should have 4 items
        should include "foo", "bar", "baz", and "bye"
        should include {"foo"=>5, "bar"=>"hello", "baz"=>15, "bye"=>"bye"}
    when overriding values
      it should behave like simple object
        keys
          should have 4 items
          should include "foo", "bar", "baz", and "bye"
          should not include "noSuchProperty"
        self_keys
          should have 4 items
          should include "foo", "bar", "baz", and "bye"
          should not include "noSuchProperty"
        fields
          ["foo"]
            should == 25
          ["bar"]
            should == "hola"
          ["baz"]
            should == 15
          ["bye"]
            should == "bye"
          ["noSuchProperty"]
            should be nil
        methods
          foo
            should == 25
          bar
            should == "hola"
          baz
            should == 15
          bye
            should == "bye"
          noSuchMethod
            should be nil
        to_hash
          should not be nil
          should have 4 items
          should include "foo", "bar", "baz", and "bye"
          should include {"foo"=>25, "bar"=>"hola", "baz"=>15, "bye"=>"bye"}

Finished in 0.0831 seconds
78 examples, 0 failures

Birth of a Bug

2011-09-12T09:00:00.001-04:00

This is a a story of a bug. It was a simple bug. Like many simple problems, on the surface it had a simple cause. Also like many simple problems, in reality it had many contributing factors. Before I tell you about the bug, let me tell you about the application.

let people configure which emails they getAt my place of employment we have various electronic forms that need to be signed by people. For example, if you want to purchase something, there are people who need to approve that purchase - probably your supervisor plus whoever has signature authority for the project you are billing the purchase to. To keep the process flowing people get emails letting them know when they have a form to sign. As it turns out, not everyone wants these emails, and sometimes people only want the emails under certain circumstances. So I wrote an app to let people configure which emails they get.

I deployed this app the other day after work, and promptly the next morning people started using it to turn off email notifications. Shortly thereafter they started sending in complaints that they were still receiving email.

Why didn't it work? How did such a fundamental bug get into the application?

Before answering that question, I'll tell you what the bug was. One of the features of the configuration is that people could turn off email notification, but add a financial exception. As an example, some people only want to get email notifications for requests that are for more than $10,000. Somewhere in the user's NotificationPreferences object is a line that looks similar to:

boolean checkFinancialOverride(Double amount) {

 return (amount >= mMoney);

}

where amount is the amount the form is for ($53.24 stapler), and mMoney is the threshold the user set ($10,000). There's a catch though - what is mMoney if the user didn't specify a threshold? Turns out that there are two possible values, depending on how the value got set. If they never have set any preferences at all, this value is NULL, but if they have set a preference for configuration, but never set a threshold, this value gets set to 0. This means that we do NOT want to send a financial override if the mMoney value is either NULL or 0. The actual code looked like:

boolean checkFinancialOverride(Double amount) {

  return mMoney != null && amount != null && mMoney > 0 && amount >= mMoney;

}

Don't bother looking for the bug in that code, as the bug isn't there. At least not yet. It didn't show up until deployment day. While the bug was simple, there were a number of "best practices" that were violated that led to the birth of this bug.

Don't Make Last Minute Changes
The plan was to deploy the code at 5PM. At 3PM, I had a coworker who suggested that "amount" was a better name for the database column for the threshold than "money" because "amount" is consistent with other projects. (and, of course, "mAmount" would be the Java variable containing the threshold value.)

Anytime you make a change, the risk of breaking something is non-zero. Make the change at the last minute and you won't have time to exhaustively test it. As you have figured out, we broke the code. Then in my "smoke testing" I only chose examples that didn't trigger the new bug. Which is how I unknowingly release code with this bug.

This wasn't a critical user bug. Heck, this wasn't anything that was going to affect the user in any way, shape, or form. I never should've agreed to make this change.

Beware of Semantic Code Changes
When making this last minute name change from "money" to "amount" we split the work up between me and the developer who recommended the change, even though all the code involving this had been written by me. This means that he was changing code that he wasn't familiar with. Here is what the updated code looked like:

boolean checkFinancialOverride(Double amount) {

  return mAmount != null && amount != null && amount >= mAmount;

}

As you can see, the condition for checking if "mAmount > 0" got dropped. Why was it dropped? I haven't asked, so I don't know. I don't know if it was accidental or intentional. Either way, there are related best practices that should have been followed.

Understand Your Changes
Ever run across code in a working system that just can't be right? Before just assuming that the bug is being hidden by other functionality and going ahead and fixing the code, make sure you understand exactly what is happening. If you have access to the original developer (in this case I was just across the hall, a slightly raised voice away), ask why the code is the way it is. It might be a bug, but you might also not be understanding it.

Leverage Refactoring Tools
If you are just renaming a variable, let your IDE rename it for you. This is the type of change that can almost always be done automatically by tools. Use those tools.

Check Your Changes Before Committing
This is something I am a big enough believer in that I may turn it into its own blog post at some point. After making code changes, no matter how large or how trivial, run diff on the code before checking it in. Verify that every change that you are committing is one that you mean. This serves as a simple and quick code review, as well as ensuring that you don't accidentally commit left in debugging statements. Not to mention catching those places where you accidentally inserted or deleted code because of careless clicks.

Run Your Unit Tests
After making any changes, no matter how trivial, make sure you rerun your unit tests. Especially before deploying. You do have unit tests, don't you? Ok, I'll admit it, we had no unit tests on this project, and this is exactly the type of bug that would've gotten caught by unit tests.

Write Understandable Code
While you can follow all the best practices under the sun, you can't force your coworkers to do the same. The best thing you can do to defend against that is to write code that is not only correct but understandable and obvious. Let's look at that query one more time:

boolean checkFinancialOverride(Double amount) {

  return mMoney != null && amount != null && mMoney > 0 && amount >= mMoney;

}

Checking for null is fairly obvious as it prevents NullPointerExceptions when the autounboxing happens. The comparison "amount >= mMoney" is the comparison that is needed. But what's with the "mMoney > 0"? If I hadn't explained it above, would you have guessed why it is here? Maybe, maybe not, either way I would argue that this is a piece of non-obvious code. I will offer up two possible suggestions for how this code could be made more obvious. The first is to add a comment:

boolean checkFinancialOverride(Double amount) {

  // if mMoney <= 0, it is not a valid value and should be ignored

  return mMoney != null && amount != null && mMoney > 0 && amount >= mMoney;

}

But since I think comments are often a sign of confusing code, my other suggestion involves changing the code:

boolean checkFinancialOverride(Double amount) {

  return isThresholdValid() && amount != null && amount >= mMoney;

}



boolean isThresholdValid() {

  return mMoney != null && mMoney > 0

}

Conclusion
Write clean code. Don't make last minute changes. Understand your changes. Write unit tests and use them. These are the steps to avoid unexpected and avoidable bugs.

Unit Testing a Simple Property Model

2011-09-05T09:00:00.009-04:00

Previously I showed the beginning of a Ruby implementation of a simple property model. Well, I decided to create some unit tests to help verify that my property model correctly handles properties, especially with the prototype delegation.

RSpec seems to be popular in the rails world, so I used rspec to write my tests. This exercise provided me with examples of the good, the bad, and the ugly with unit tests. The good was straightforward. I don't have an application yet, just this simple library and unit tests provided an easy way to test it. Even better, at one point I changed an implementation detail that required some rewriting of the property model, and by rerunning the tests, I had confidence that I didn't break anything.

The ugly? This is definitely a good example of ugly. I have the unit testing code at the bottom of this post. I found I had a lot of duplication, so I created "shared_examples_for" to reduce redundancy. However, the resulting test code is still really long and while the shared examples helps with sharing tests, I am not sure they help with readability. Now, as Trevor commented, its possible (ok, likely) that the unit tests are ugly because they weren't written well. I will admit to being a newbie at RSpec. I am sure that they could be written better, but I am not convinced they wouldn't still be somewhat ugly.

The bad? Well, there isn't too much bad, yet. It did take time to create the tests, but I had to validate my code somehow. And it is possibly (likely?) that my tests aren't as thorough as they should be, so maybe I have some false confidence in the correctness of my code. But all in all, despite what I said last paragraph, this wasn't that good of an example of the bad. At least not yet. The bad will come when I spend time trying to clean up these tests, rather than using that time to write new code.

And without further ado, here are these unit tests.

require 'spec_helper'



describe Thing do

  let(:foo) {5}

  let(:bar) {"hello"}



  shared_examples_for "simple properties" do

    describe "keys" do

      it {thing.keys.size.should == 2}

    end

    describe "foo" do

      it {thing['foo'].should == foo}

      it {thing.foo.should == foo}

    end

    describe "bar" do

      it {thing['bar'].should == bar}

      it {thing.bar.should == bar}

    end

    describe "baz" do

      it {thing['baz'].should be_nil}

      it {thing.baz.should be_nil}

    end

    describe "to_hash" do

      it {thing.to_hash.should_not be_nil}

      it {thing.to_hash.size.should == 2}

      it {thing.to_hash['foo'].should == foo}

      it {thing.to_hash['bar'].should == bar}

    end

  end



  shared_examples_for "nested properties" do

    describe "keys" do

      it {thing.keys.size.should == 4}

      it {thing.self_keys.size.should == 2}

      it {thing.self_keys.include?('b2').should be_true}

      it {thing.self_keys.include?('bye').should be_true}

      it {thing.self_keys.include?('bar').should be_false}

    end

    describe "foo" do

      it {thing['foo'].should == 5}

      it {thing.foo.should == 5}

    end

    describe "bar" do

      it {thing['bar'].should == "hello"}

      it {thing.bar.should == "hello"}

    end

    describe "b2" do

      it {thing['b2'].should == 15}

      it {thing.b2.should == 15}

    end

    describe "bye" do

      it {thing['bye'].should == "bye"}

      it {thing.bye.should == "bye"}

    end

    describe "baz" do

      it {thing['baz'].should be_nil}

      it {thing.baz.should be_nil}

    end

    describe "to_hash" do

      it {thing.to_hash.should_not be_nil}

      it {thing.to_hash.size.should == 4}

      it {thing.to_hash['foo'].should == 5}

      it {thing.to_hash['bar'].should == "hello"}

      it {thing.to_hash['b2'].should == 15}

      it {thing.to_hash['bye'].should == "bye"}

    end

  end

 

  describe "no inheritance" do

    let(:thing) {Thing.new}

    subject {thing}



    describe "new object" do

      describe "keys" do

        it {thing.keys.should_not be_nil}

        it {thing.keys.should be_empty}

      end

      describe "properties" do

        it {thing['foo'].should be_nil}

        it {thing.foo.should be_nil}

      end

      describe "to_hash" do

        it {thing.to_hash.should_not be_nil}

        it {thing.to_hash.size.should == 0}

      end

    end

   

    describe "adding properties to object" do

      before(:each) do

        thing['foo'] = 5

        thing['bar'] = "hello"

      end

      it_should_behave_like "simple properties"

      it {thing.self_keys.size.should == 2}

      it {thing.self_keys.include?('foo').should be_true}

      it {thing.self_keys.include?('bar').should be_true}

      it {thing.self_keys.include?('baz').should be_false}

    end

    describe "adding fields to object" do

      before(:each) do

        thing.foo = 5

        thing.bar = "hello"

      end

      it_should_behave_like "simple properties"

      it {thing.self_keys.size.should == 2}

      it {thing.self_keys.include?('foo').should be_true}

      it {thing.self_keys.include?('bar').should be_true}

      it {thing.self_keys.include?('baz').should be_false}

    end

  end



  describe "inheritance" do

    let(:thing) do

      base = Thing.new

      base.foo = 5

      base['bar'] = 'hello'

      foo = Thing.new

      foo.prototype = base

      foo

    end

    subject {thing}

 

    describe "new object" do

      it_should_behave_like "simple properties"

      it {thing.self_keys.include?('bar').should be_false}

      it {thing.self_keys.include?('baz').should be_false}

    end

   

    describe "adding properties to object" do

      before(:each) do

        thing['b2'] = 15

        thing['bye'] = "bye"

      end

      it_should_behave_like "nested properties"

    end

    describe "adding fields to object" do

      before(:each) do

        thing.b2 = 15

        thing.bye = "bye"

      end

      it_should_behave_like "nested properties"

    end

    describe "adding redundant fields to object" do

      let(:foo) {15}

      let(:bar) {"bar2"}

      before(:each) do

        thing.foo = foo

        thing.bar = bar

      end

      it_should_behave_like "simple properties"

      it {thing.self_keys.size.should == 2}

      it {thing.self_keys.include?('foo').should be_true}

      it {thing.self_keys.include?('bar').should be_true}

      it {thing.self_keys.include?('baz').should be_false}

    end

  end

end

And here is the output of running the above tests:

$ rspec spec/models/thing_spec.rb



Thing

  no inheritance

    new object

      keys

        should not be nil

        should be empty

      properties

        should be nil

        should be nil

      to_hash

        should not be nil

        should == 0

    adding properties to object

      should == 2

      should be true

      should be true

      should be false

      it should behave like simple properties

        keys

          should == 2

        foo

          should == 5

          should == 5

        bar

          should == "hello"

          should == "hello"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 2

          should == 5

          should == "hello"

    adding fields to object

      should == 2

      should be true

      should be true

      should be false

      it should behave like simple properties

        keys

          should == 2

        foo

          should == 5

          should == 5

        bar

          should == "hello"

          should == "hello"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 2

          should == 5

          should == "hello"

  inheritance

    new object

      should be false

      should be false

      it should behave like simple properties

        keys

          should == 2

        foo

          should == 5

          should == 5

        bar

          should == "hello"

          should == "hello"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 2

          should == 5

          should == "hello"

    adding properties to object

      it should behave like nested properties

        keys

          should == 4

          should == 2

          should be true

          should be true

          should be false

        foo

          should == 5

          should == 5

        bar

          should == "hello"

          should == "hello"

        b2

          should == 15

          should == 15

        bye

          should == "bye"

          should == "bye"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 4

          should == 5

          should == "hello"

          should == 15

          should == "bye"

    adding fields to object

      it should behave like nested properties

        keys

          should == 4

          should == 2

          should be true

          should be true

          should be false

        foo

          should == 5

          should == 5

        bar

          should == "hello"

          should == "hello"

        b2

          should == 15

          should == 15

        bye

          should == "bye"

          should == "bye"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 4

          should == 5

          should == "hello"

          should == 15

          should == "bye"

    adding redundant fields to object

      should == 2

      should be true

      should be true

      should be false

      it should behave like simple properties

        keys

          should == 2

        foo

          should == 15

          should == 15

        bar

          should == "bar2"

          should == "bar2"

        baz

          should be nil

          should be nil

        to_hash

          should not be nil

          should == 2

          should == 15

          should == "bar2"



Finished in 0.06217 seconds

106 examples, 0 failures

Simple Property Model in Ruby

2011-08-29T09:00:00.003-04:00

I've talked before about how JavaScript follows what Steve Yegge calls the Universal Design Pattern. I can see the value of associating an arbitrary number of properties with an object, and to be able to say that one object is "like" another object the way JavaScript's prototype property works. I even have ideas for using this. However, I don't do my server side programming in JavasScript. Here is my first cut at a very bare bones property model using Ruby.

Code

First I'll show you the class that stores a property. It is a just a simple key/value pair.

class Property

  attr_accessor :name, :value

  

  def initialize(name, value)

    @name = name

    @value = value

  end

end

Next, is my code for the Object class (which I call Thing):

class Thing

 

  attr_accessor :prototype, :type, :properties



  def initialize

    @properties = {}

  end

 

  def self_keys

    @properties.keys

  end

 

  def keys

    self_keys | (@prototype ? @prototype.keys : [])

  end

 

  def [](key)

    return @properties[key].value if @properties.has_key? key

    @prototype ? @prototype[key] : nil

  end

 

  def []=(key, value)

    if @properties.has_key? key

      @properties[key].value = value

    else

      @properties[key] = Property.new(key, value)

    end

  end

 

  def to_hash

    hash = @prototype ? @prototype.to_hash : {}

    @properties.each_value {|p| hash[p.name] = p.value}

    hash

  end

 

  def method_missing(method, *args)

    name = method.to_s

    if (name.end_with?('='))

      self[name[0..-2]] = args[0]

    else

      return self[name]

    end

  end

end

As you can see a "Thing" consists of a type, a prototype object, and a map of key/property pairs. Property lookups (which can be done using either array type syntax or the dot syntax of methods and fields) first look at a thing's properties, and then at its prototype's properties. All in all, it is a fairly simple pair of classes.

Thoughts

There are a couple of decisions here that seem odd, so I'll explain my thinking.

Thing

Why did I call the Object type Thing? Well there is already an Object class in Ruby and I didn't want to worry about naming collisions. Like object, "thing" is a generic enough word that if I want to create a class with that name, I should probably rethink my design. So it satisfies the qualifications of describing "something" without causing any likely naming collisions.

Property
Why do I have a Property class rather than doing the more straightforward thing of having Thing just containing a Hash of key/values? Well, this is an example of me violating the YAGNI principle. I am going to want to persist this information to a database so I am envisioning a Thing table and a Property table. My assumption is that when I do that, I'll want the key/value pairs broken out into their own Property class. So I am (perhaps foolishly) breaking that out ahead of time.

method_missing
Like JavaScript, I want to be able access properties via either array syntax (thing['property']) or method syntax (thing.property). Ruby provides the functionality to handle unknown method names, via the method_missing method. As you can see, I wrote a bare bones implementation that does no error checking. If it gets passed a method that ends with '=', it assumes that it is a property assignment, otherwise it assumes it is a property read. I suppose in the future I could/should add error checking like that there are the right number of arguments to the method. I should probably also implement respond_to so it will return true for any property that exists.

Lack of Functionality

Do these classes actually buy me anything? I'm not sure. I am sure that I am going to need a bunch of additional functionality (like database persistance) before they are really useful to me. But I have to start somewhere. So rather than blog about a fully baked solution, I've decided to show you some of the raw ingredients as I go. I am hoping that this gives me a good starting point.

Wrong is Better Than Vague

2011-08-22T09:00:00.006-04:00

Vague
I've got this idea for the next killer app. It's this game, you see, that'll leverage the unique properties of tablets. Do you want to work on implementing it with me?

What's the game? Oh, something fun.
So this will be for the iPad? That and other tablets.
Who's going to play it? Everyone.

Wrong
I am going to create a 3D minesweeper for the iPad and I'll sell it for $473.26 per copy. Minesweeper is already popular on the Windows, and there is this huge market of iPad owners who have already shown they are willing to pay a lot for gadgets. And with the 3D angle we can get rich off of this. Do you want to work on it with me?

The Difference
The problem with vague is that most ideas sound either good or bad depending on the listener's optimism. Vague is how a small project turns into a years long boondoggle, or how the next billion dollar idea gets missed.

In my experience most new ideas from management types (and a disturbing number of research scientists) are of the vague variety. They have these great pie in the sky ideas, but if you try to actually evaluate them for engineering or business practicality, you can't. Without details there is no way to determine if the idea is good or completely infeasible.

Take the vague example above. Would you work on it? Is it going to be successful? Who knows. What about the wrong example? Of course you wouldn't work on it, as is. But because it is concrete, it could be improved.

Nobody pays $400 for apps! Oh, then I guess we can charge $4.95.

The iPad doesn't support 3D. Hmm.. then maybe we should target those cell phones that do.

Minesweeper really doesn't lend itself to 3D. Well, what about this other [fill in specific details here] game?

How much work will this be? Oh, given those details it'll take about 3 months.

So how many do have to sell to make it worthwhile? Oh, my time is cheap, probably just 10 copies.

Then you'll probably be successful, but I don't think I'll help you.

As you can see, with concreteness, even wrong concreteness, decisions can be made in an informed way.

Even more important than concrete plans, are concrete implementations. There's a reason people create research prototypes, whether it is for a physical product or software. You can learn so much more from what goes wrong than you can from just the high level ideas.

So the next time you have an idea, make it concrete. A wrong implementation is worth much more than just the vague idea itself.

Emailing Server Errors

2011-08-15T09:00:00.001-04:00

I've talked about checking the server's health, but its even nicer if the server lets you know if it has a problem. In particular, you don't want user's experiencing errors and for you to remain in blissful ignorance. A simple fix is to have your server email you whenever a user encounters an error. Here is how I did this in rails.

Catch the Error
The first thing to do is to capture any exceptions that get thrown. This can be done easily by adding the following line to your ApplicationController.

unless Rails.application.config.consider_all_requests_local # don't rescue on local testing

  rescue_from Exception, :with => :handle_error

end

As of this writing, apparently Rail 3 doesn't support 404 errors this way, so a fix for this is to add a catch-all route at the bottom of your config/routes.rb that looks something like:

# this must be the LAST rule in the file

# it allows our custom 404 error to execute

match '*path', :to =>'application#routing'

Handle the Error
Now that I have caught unhandled exceptions and 404 errors, you need to do something with them. These methods in the the ApplicationController are the ones called from the above code:

def routing(exception = nil)

  record_error "404 error", exception

  render :template => "/errors/404.html", :status => 404

end

 

def handle_error(exception)

  msg = exception ? exception.message : "unknown error"

  record_error(msg, exception)

  render :template => "/errors/500.html", :status => 500

end



def record_error(msg, exception)

  return nil if Rails.application.config.consider_all_requests_local # don't email during local testing



  referrer = request.env['HTTP_REFERER']

  url = request.url

  app_name = Rails.application.class.to_s

  user = current_user

  name = user ? user.username : "UNKNOWN"

   

  error_msg = msg

  if exception

    error_msg += " #{exception.message}\n#{exception.backtrace.join '\n'}"

  end

   

  ip = request.remote_ip

  remote_ip = request.env['HTTP_X_FORWARDED_FOR']



  logger.warn "Error #{error_msg} for url [#{url}] coming from url [#{referrer}] from user [#{user}] at ip [#{ip}] forwarded from ip [#{remote_ip}]"

   

  mail = ErrorMailer.log_error(msg, app_name, url, referrer, ip, remote_ip, name, exception)

  mail.deliver

end

You can make the 404 and 500 error pages that these methods render be whatever you like, though it is suggested that you make them useful. The record_error method that these methods calls, collect a bunch of information about the request, log them (in case mailing fails), and then emails them. The emailing is done using Rails mailing capability. Since the actual configuration for talking to the mail server will depend on your situation, you should read the documentation for that.

Emailer
To send the email, I created mailers/error_mailer.rb which is analogous to a controller. All it does is package all the variables into instance variables that the view will have:

class ErrorMailer < ActionMailer::Base

  default :from => "myApplication@myurl.com"

  helper :application



  def log_error(subject, app_name, url, referrer, ip, remote_ip, user_name, error)

    @app_name = app_name

    @url = url

    @referrer = referrer

    @ip = ip

    @remote_ip = remote_ip

    @user_error = user_error

    @user_name = user_name

    @error = error

    mail :to => "myAddress@somewhere.com", :subject=>"RAILS ERROR in #{@app_name}: #{subject}"

  end

end

The view (views/error_mailer/log_error.text.haml) turns this information into an email message:

There was an error in app #{@app_name} at time #{Time.now}.



URL:  #{@url}

Referrer:  #{@referrer}

IP:  #{@ip}

IP via a portal:  #{@remote_ip}

User: #{@user_name}

 

- if @error

  Error class: #{@error.class}

  Error message: #{@error.message}

  Error Stack Trace:

  - @error.backtrace.each do |line|

    = line

- else

  No exception to log

Warning
I do have one piece of advice: Make sure the subject you give your error emails is something very distinct that you can write mail filter rules. After I deployed a similar solution at work, the security teams scanner found my app and proceeded to run through its entire litany of tests against the server. I came in the next morning to more than 4,000 emails from 404 errors. The fact that I had put the entire development team as the recipients of these emails was just the icing on the cake.

Maybe emailing every error isn't great, since you can look in the log files for the same information. However, I find the immediacy and in your face nature of an email is good for making sure that errors get fixed and aren't just silently experienced by your users (and ex users). However, be aware that if something goes wrong, you have the potential to receive a lot of email. (not to mention choice comments from anyone else who is also getting those emails).

One year of regular posts

2011-08-08T09:00:00.003-04:00

Though I started this blog years ago, it was one year ago that I started posting regularly. This was partially inspired by Jeff Atwood's advice to post regularly. I have no idea how he regularly posted 6 times a week given that I have a hard time posting once a week. And I think that Syd (a commenter on that post) hits the nail on the head that there is more to being a popular blogger than just being reliable. However, being reliable is definitely a prerequisite.

So how have I done? Well, I missed two weeks in the past year. And I feel like I am struggling each week to keep up. However, Google Analytics shows that I've been at least a little bit successful. I've trended up pretty consistently since I started posting regularly.

100 page views a week is far from a popular blog. But it is a number that is high enough that I know that it isn't just me and my closest friends reading my blog. So that is some measure of success.

If you've been following my blog you've noticed that I have two different types of posts, my philosophical posts and my practical/tutorial posts. When I started this blog, my goal was to post primarily philosophical posts and to generate a discussion on each post. My secret goal was to learn more from my comments than I put into my posts. Unfortunately, so far, I don't get many comments, so that isn't working out. At some point I decided to also write tutorial type posts describing how I solved a specific problem. I've learned much from others who have posted what they know, so I feel I should give back. Based on Google Analytics, the majority of people who see my blog come to view these posts. Unfortunately, if the readers don't leave comments, I have no idea if what I posted was useful or just a waste of time. I can only hope that some people are finding it useful. Even more, I hope that some of these people become regular readers (and commenters) on my more philosophical posts. I am the eternal optimist and still hope to achieve my original goal.

Of course, to accomplish that, I need to continue Jeff Atwood's advice and post regularly. While I have plenty of ideas, it is somewhat daunting to me to actually turn those ideas into posts each week. Well, here's hoping that next August I get to bore you all with another post like this.

Documentation and Context

2011-08-01T09:00:00.001-04:00

A few weeks ago Scott Adams (of Dilbert fame) wrote a blog post about the context of a message. One of his points was that the context that a message is presented in affects how the message is received. Saying the same words at a standup comedy club and at a funeral won't have the same affect. This is related to a thought I've long had about software design documentation. We all have a context in our mind, but that of a reader is probably different.

A specific example of this difference came up at work recently. Asok, Alice, and I (not their real names) were discussing a new feature we wanted added. The feature was something Alice and I needed to support our project, but it required a change in Asok's code. Over the course of an hour long meeting Alice and I kept suggesting changes to Asok's code and he kept telling us that what we were suggesting didn't make any sense.

So why did Asok keep telling us our suggestions made no sense? It wasn't arrogance, he was right. Our suggestions didn't make sense in the context of his code. It was a lack of understanding on Alice and my's part. We were looking at database and classes, and based on their names we were making certain assumption that turned out to be wrong.

Why didn't Alice and I understand Asok's design well enough to make good suggestions? We both have plenty of experience in the field, so it wasn't that. It would be easy to blame poor naming of databases and classes on Asok's part, but now that I understand it, I'm not sure I can come up with anything better. Really it was just a matter of context. Asok had worked on this system for months, and everything fit into a model he had in his head. Alice and I didn't have that model, so we made up our own. As often happens, the model we assumed didn't match reality.

How should [we] document our designsThis, to me, is the big problem of software design documentation. I have been on both sides of this problem. I have seen people completely mangle my code because they didn't see the "correct" ways and places to extend it. I have also mangled others people's code for the same reason. This is how unmaintainable behemoths get created. The challenge is: how should Asok, Alice, or I document our designs such that the context we have in our head is conveyed to the next person?

Some people say that the code is the documentation. While the code is the reality of today, the context of the code shows how to extend, modify, and use that code. Sharing this context from the code writer's mind is crucial to code being able to be maintained beyond the original author.

So, I guess this is another one of those posts where I point out a problem and have no solution. Do you have a solution?

Checking Web Application Health

2011-07-25T09:00:00.001-04:00

There are certain things you want to know about your web application to make sure it has started correctly and to help diagnose problems.

Is it up?
Can it connect to the database(s)?
Can it connect to the web service(s)?
Can it do any other actions that may independently break?
What are the server settings?
What version of the code is running?

The first question is easy to answer? But what about the others? You can obviously determine if the other dependencies are up by going to the appropriate place in your application, but it'd be nice to have a single location to check all of these. And what about server settings or code version? Ever had a bug reported that you know you fixed? It's nice to also be able to make sure that you really are running the version you think you are.

Here's how I've accomplished this on a Ruby on Rails project.

Single Location
First, I created a controller with a single page:

prompt> rails generate controller Admin index

Here is the beginning of the controller class.

class AdminController < ApplicationController
  before_filter :require_admin

  def index
  end
end

As you can see, I've protected it, so only administrative users can view this page. Here is the views/admin/index.html.haml where I show server state.

This is the status page.
%br
%h2 Version
%pre= @version
%h2 Environment
%br
%b RAILS_ENV:
= ENV['RAILS_ENV']
%br
%hr
%h2 Site Tests
.test
  = button_to "Test DB", {:action=>"test_db"}, :remote=>true, :onClick=>"startTest()"
%hr
%h2 Test Output
#testOutput

:javascript
  function startTest() {
    $("#testOutput").replaceWith('<div id="testOutput">Loading...</div>');
  }

Dependency Tests
I've implemented each test as a separate Ajax call. In the view code above, this everything from "Site Tests" on down. This consists of the buttons (one for each test, just one here), a place to show the output, and the javascript to make this work. For each test, I just add a new button in the .test div. As you can see in the code above, the button makes an Ajax call (:remote=>true) and calls a javascript function to show that the result is loading. It is up to the controller to then update the testOutput div with output status.

In my admin_controller.rb I have a function for each button that looks like:

def test_db
  standard_test do
    # insert test code here
  end
end

The helper function standard_test looks like:

def standard_test
  begin
    yield
    @success = true
  rescue Exception => e
    @success = false
    @exception = e
  end
  respond_to do |format|
    format.html {
      flash[:error] = "This should've been ajax"
      redirect_to admin_index_path
    }
    format.js {
      render 'test_output', :layout=>false
    }
  end
end

All it does is executes the test code (via the yield statement) and catches any thrown exceptions. If there is no exception, it considers the test a success. Assuming it was called via an Ajax call, it then renders the javascript template test_output.js.haml which is simply:

!= "$('#testOutput').replaceWith('#{escape_javascript render(:partial=>'test_output')}')"

As you can see, this merely replaces the testOutput div with what is rendered by the _test_output.html.haml partial:

#testOutput
  - if @success
    %b SUCCESS
  - else
    %b ERROR
    %br
    %b Error Class:
    = @exception.class
    %br
    %b Error Message:
    %pre= @exception.message
    %b Error Stack Trace:
    %pre
      - @exception.backtrace.each do |line|
        = line
    %b Error Inspect
    %pre= @exception.inspect

Of course, don't forget to make sure your routes are in your config/routes.rb.

get 'admin/index'
post 'admin/my_test' # separate route for each test

Version
Including version information is going to be very specific to your environment. We use Mercurial for our version control, and since we have a number of Java project, we use ant for our deployment process. Here is how I connected these together.

In the ant deployment script, I added a target that looks like:

<target name="write.hg.version">
  <exec executable="hg" dir="${project}" output="${project}/config/version.txt">
    <arg value="summary" />
  </exec>
</target>

All that is left is to get the contents of this version.txt file into the @version variable that is displayed by index.html.haml. This is done by simply having the index method in admin_controller.rb read in the file.

def index
  name = File.join(RAILS_ROOT, 'config', 'version.txt')
  if File.exists? name
    f = File.new(name)
    @version = f.read
  else
    @version = "Unknown version"
  end
end

Conclusion
This really wasn't too much work, and now I have a single page that I can go to and at a glance see what version of the code was deployed and which rails environment is being used. Adding new tests is just a simple matter of adding a new button to the view, which calls a new test method in the controller. By leveraging the standard_test method, new tests can consist of just the code they are intended to test.

Windows Command Line Cut & Paste Rant

2011-07-18T09:00:00.000-04:00

Just a quick rant this week. When selecting text in the windows command console, why does it do a "block" mark rather than a line based mark?!?! In every other text based environment I've used, including Windows Notepad and MS Word, if you start highlighting text in the middle of one line and move down to the middle of the next line it select the end of the first line and the beginning of the next line. i.e. all the text that starts where you started the mark and ends where you ended the mark. For some reason the Windows Command shell only marks the text that is actually in the box, i.e. a couple characters in the middle of both lines.

Is there a time that this behavior is useful? If you know, please share it with me.

For me, I often want to copy either a long command line I ran, or the output of a command, to put in an email, IM, a wiki, or even to paste into another terminal to run. But if the line is wrapped, then to make sure I get everything I want, I have to also copy the command prompt. This means I have to add an intermediate step of editing my selection before I can send it.

What's even more bizzarre to me, is that in searching on the web, I can't even find any other comments about this behavior. Nor can I find a way to change this. Am I the only one who finds this annoying? More importantly, does anyone know how to change this behavior?

(I know, I know. I shouldn't be using Windows, and I shouldn't be cutting and pasting. Well, I have to admit that I sometimes do both, and I was doing a lot of both this past week.)

Attack of the Clones

2011-07-11T09:00:00.000-04:00

As I mentioned, I was recently at ICSE. One of the big topics there was Clones. How do you detect them? How do you remove them? The talk that I found most interesting on the topic was one where they tried to measure how bad clones really are.

First, the definition for those of you aren't familiar with the term clone in the context of software engineering. (I wasn't before this conference). A clone is just what it sounds like, duplicated code. Clones are most often created via cut and paste.

The idea of removing duplicated code through common modules, whether they are libraries, classes, functions, procedures, or macros is an old one. Probably about as old as programming. In recent years, removing duplicated code has even got a hip acronym: DRY. The idea that clones are bad was basically a given at this conference. I have pretty much internalized that idea, so this didn't seem strange to me at all.

So how bad are they? Well, as I mentioned above, there was research that tried to answer this question. Researchers analyzed the version control history of a few different projects and tracked all of the clones in these projects. They then tried to evaluate some of the problems. For example, one of the problems with clones is that if you change the code, you have to remember to do that change in multiple locations. What they found was that in the vast majority of the time, clones either never changed, or only changed once. Meaning that the majority of the time, the work of refactoring the code to remove the clones would be at least as large as the work of just maintaining them.

What about the situation where one clone is modified, but another copy doesn't get updated. Well, they found that this is also very rare, and a lot of the time when this does happen, its intentional. And even of the times when they are unintentionally out of sync, the resulting code is typically not a bug.

So what does all this mean? I don't know. Maybe their sample projects weren't representative of most software projects. Or maybe, just like most other engineering decisions, there are trade offs, and there isn't a 100% right answer, independent of context. I still find that clones offend my sense of software ascetics most of the time, but I am trying to be a little more open-minded and be critical about what and when I refactor.

GCJ 2011 T-Shirt

2011-07-04T09:00:00.014-04:00

Due to the holiday weekend, I am just making a short post this week.

Last week I mentioned that I got knocked out of the TCO 2011 competition, and so will get no shirt for it. Well, a couple days later FedEx delivered my Google Code Jam 2011 t-shirt. It's not as cool as getting the trip to Japan for the finals, but I guess I'll take what I can get. Since a picture was requested, here it is. And, no Alan, programming contests don't completely eliminate the need to go clothes shopping, though I do have around two dozen t-shirts plus a nice winter coat and hat from various programming contests.

No T-Shirt For Me

2011-06-27T09:00:00.002-04:00

This past Saturday my brain turned off and I failed to get even a single problem right in the 2^nd round of the TCO 2011. This was the round you had to get past to get a t-shirt.

So what happened? One thing I like to do with problems is work out a couple simple inputs by hand to make sure I understand the problem. As one of my former co-workers liked to say, if you don't know how to do something, you don't know how to do it with a computer. On the first problem from Saturday's contest, the input was a positive integer, and we had to return an integer based on the problem description. I worked out the answers for the inputs 1 - 6. Unfortunately, I had come up with the wrong answer for an input of 6. Since 6 wasn't one of the sample inputs, I didn't know I was wrong.

Not surprisingly, trying to generalize from my wrong solution to a general solution didn't work so well. I eventually gave up, but not in time to code up a correct solution to the next problem.

walk away from the problemThis is a problem that I've had before, and I've seen it in others. Once you make a mistake, it can be really hard to see that mistake. Your mind tends to stay in that rut and you repeat the mistake when you redo the problem. In the real world (i.e. not the artificially constrained world of programming contests), there are a couple of ways to solvie this type of mistake. The first is to talk through the problem with an external party. If you can, though, don't explain what you did, but rather let them solve it themselves. That way you don't lead their thought process into the same rut as your own. The second approach is to walk away from the problem. Often times, just walking away from a problem is enough to let you see the solution when you pick it up the next day.

Unfortunately, neither of those were options for me with the programming contest. So, instead, I get to kick myself for a year. Oh well. Maybe next year will go better.

Testing in Production

2011-06-20T09:00:00.001-04:00

A co-worker of mine has the picture to the right posted on her door. Last week we put that into practice. We deployed a new internal app which depended on two other components which were also new. On Thursday at 4:30 PM an email went out to tell people they could use it. By 4:45 our server had crashed. Thursday was a late night, but we eventually got the system where it could run without crashing the server. On Friday morning (when it really got used), we had to scramble to fix some less obvious (at least to the end user) bugs, but just as critical.

So what happened?
The short answer is, we didn't do enough testing. Every bug that we've found (so far), arguably should've been found in testing. All in all, it was a pretty embarrassing experience. Especially since part of it was our first internal Rails app which I had pushed hard to get us to use. However, in hind-sight if I had it to do all over again, I think I'd have done the same thing. Why?

Obviously, it's not good to have such a big failure. Plus, I never like to *have* to work late. However, nothing is in a vacuum. Every decision has to be compared to the alternative. We probably could've found most or all of the bugs with another week of testing. So which was better to the Lab? That we test for another week, with the opportunity cost that two developers can't be working on other projects? Or that we deploy a poorly tested app, give a few users a poor experience, and have a couple people work one long day on a Thursday?

The fact of the matter is, 15 minutes of real user testing found bugs that would've taken at least a week of developer testing. In terms of people hours, it was much more efficient to get some real use than to continue testing. As Jeff Atwood says, release your buggy software. So what was the negative? In this particular situation it was an internal app so there are no customers to lose or investors to turn away. The only negative is that a few people probably think less of me and my division. On the plus side, getting the app in front of people helped define requirements better than any amount of up front analysis. Also, the majority of this code will be reused on future apps, so this testing should allow for future high quality releases, allowing us to regain some of that lost reputation.

Releasing finds missing requirementsSo what's the take home message? Well, I am still embarrassed by how many problems we had with this release. However, given that there were no real negative consequences, it was probably in everyone's best interest for us to release (potentially) buggy software, rather than delay it for more exhaustive testing. Releasing finds missing requirements and shows how the code is really used. No amount of testing can do that.

In general, I try to have this blog be a voice for higher quality software, and to not subject users to a bad experience. However, at times it might be better to give a temporary bad experience so that the majority can have a good experience sooner.

Too Slow

2011-06-14T09:00:00.029-04:00

I have been remiss in getting my posts out. A few weeks ago I was at ICSE. I had intended on making a post based on that conference, but as soon as it was over, I went into vacation mode. While on vacation I found a number of things to do that kept me away from computers. About the only computer thing I did do was compete in the 2^nd round of the Google Code Jam.

This was more challenging than normal for me because Google tries to schedule their contests to be at a somewhat reasonable hour for most of the world. They've made the decision that there is this great big ocean without a lot of people living in it, and so they schedule it so the contest is in the middle of the night for the Pacific. Of course, I happened to be in Hawai'i at this time and so got up at O'dark-thirty for the 4AM contest start. Surprisingly, I not only did well enough to get a t-shirt (top 1000), but did well enough to advance to the next round (top 500).

So this is why I missed the last two posts. What about this post? Well, after two weeks away, we had a lot of catching up to do at home and at work. The little bit of free computer time I was able to find went to practicing for the 3^rd round of the Google Code Jam where they would select the top 25. Unfortunately, that was for naught, as I did not advance.

Too Slow
24 out of 25 of the advancers solved the first 3 problems, as well as the small input set on the 4^th problem. I, also, know how to solve the first 3 problems as well as the small input set on the 4^th problem. So what's the difference between the advancers and me? On this problem set, speed. At the same time as I was submitting my solution to the second problem, 25^th place was submitting his final solution.

The top competitors were probably a little faster at coming up with algorithms and at coding the algorithms. However, what killed me was debugging. In the first problem, I had numerous bugs in my code that I had to spend time tracking down. Careless mistakes on this problem singlehandedly knocked me out of the running to advance. On the third problem, bugs were again my downfall. I didn't take my own advice and I didn't spend a couple minutes thinking about how to best structure my data structures. I just used the first thing that popped into my head which ended up being very error prone. After the contest, when I had a couple minutes to think without the pressure, I came up with a much simpler way to code the same algorithm.

So what's the moral of this story? Don't make mistakes! :) And trying to do things in a rush just makes it more likely that you will make mistakes. Oh, and going on vacation during a contest is probably not conducive to preparing.

As for why this post is a day late... well, I am just a little bit slow.

Moore's Law Doesn't Catch Imagination

2011-05-23T09:00:00.002-04:00

Back in 1992 I had an opportunity to use an iPSC super computer. If I recall correctly, the particular computer I used had a total of 192 MB of RAM. This was just an outrageously huge amount. It was enough that I didn't really pay too much attention to the memory I was using on the program I was writing. When it was suggested that maybe my program was failing due to using too much memory, I was incredulous. But upon further inspection of my program, sure enough, I was trying to allocate arrays requiring a few hundred megabytes of memory.

Using a generalization of Moore's Law that says that processing power should double every 18 months (I know, that's not the real law), and given that approximately 12 18month periods have passed since then, computers should be 4,000 times as powerful. And sure enough, you can find supercomputers with terabytes of RAM, and the laptop I just bought has 8GB of RAM, which is 4,000 times what was common then. I am actually kind of amazed at how accurate this rule of thumb is.

However, my programming goals apparently grow just as fast. Just the other day, as I was practicing for programming contests, I tried to allocate a 4 terabyte array. Needless to say, my program failed. And for some reason, instead of just crashing, it ground my computer to a halt, eventually requiring a reboot. I suspect in another two decades I'll be crashing programs by trying to allocate exabyte arrays.

Conclusion
For many business or web apps, network or database latency is going to be your biggest bottleneck. Because of this we often treat memory as infinite and processing time as zero. The emphasis on writing readable, maintainable code as opposed to the most efficient is typically the right trade-off. However, memory isn't infinite, and processing time isn't zero. Before coding a module that requires pulling your entire database into memory, think about whether this is a couple megabytes of memory or a hundreds of terabytes. Modern computers are powerful, but they still aren't as powerful as we'd all like.

Rails form_for_hash

2011-05-16T09:00:00.010-04:00

Ruby on Rails has some nice form building helpers if you are building an HTML form for populating an ActiveRecord model. But what if you want your front end forms to not exactly map to your back end models? Maybe you are presenting a logical view that corresponds to parts of different models. Well, Rails still provides helpers that let you roll your own form. However, now you have to manually handle reading and writing values to the form. It would be nice if you could get the form_for functionality with just a simple Hash.

Well, it turns out that with a little bit of work, you can. Based on an idea I found at http://pullmonkey.com/2008/1/6/convert-a-ruby-hash-into-a-class-object/, I created a simple Object which wraps a Hash and behaves like an ActiveRecord. This class looks like:

class HashObject
  def initialize(hash={})
    @hash = hash
  end

  def method_missing(sym, *args, &block)
    @hash[sym]
  end
end

Then to make it easy to use, I put a helper function in helper/application_helper.rb

def form_for_hash(hash, name, url, html_options={}, &proc)
  object = HashObject.new(hash)
  form_for object, :as=>name, :url=>url, :html=>html_options, &proc
end

Now I can use it in the view as such: (e.g. new.html.haml)

= form_for_hash @person, :person, special_people_path do

  = f.label :first_name, 'First Name'
  = f.text_field :first_name

  = f.label :last_name, 'Last Name'
  = f.text_field :last_name
  %br
  = f.label :phone_number, 'Phone Number'
  = f.text_field :phone_number
  -# etc etc etc

Then in your SpecialPeopleController you would have methods like

def new
  @person = {:first_name=>'Jenny', :phone_number=>'867-5309'}
end

def create
  @person = params[:person]
  # logic for populating/updating models based on hash goes here
end

And that is all you need to do to be able to use a Hash to populate your forms.

Programming Contests and Languages

2011-05-09T09:00:00.054-04:00

This past weekend was the qualifying round for the Google Code Jam. Google allows you to use any language you would like in this contest. This year I have decided to use Ruby as my language of choice. In the past I have always used Java, as that is the language that I currently know the best. However, I think Ruby will allow me to write cleaner, more compact code which will let me code faster and hopefully write fewer bugs.

Simplicity

Why do I think this? Let me give you a simple example. A common task in these contests is to be able to read in a line of space separated integers and convert it into an array for later processing. Here is how I accomplish this task in Java:

  1 BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
  2 String[] parts = reader.readLine().split(" "); 
  3 int[] array = new int[parts.length]; 
  4 for (int i = 0; i < array.length; i++) {
  5   array[i] = Integer.parseInt(parts[i]); 
  6 }

Here is how I do the same thing in Ruby:

array = gets.split.map { |i| i.to_i}

Bugs
Imagine I now want to loop through the array and determine how many array elements are the same as their position. I.e. if I have the array [0, 2, 1, 3, 5, 4], elements 0 and 3 are the same as their index and so I'd want to get a value of 2. Here is Java code that accomplishes this task:

  1 int count = 0;
  2 for (int i = 0; i < array.length; i++) {
  3   if (array[i] == i) count++; 
  4 }

Here's Ruby code that does the same thing:

count = 0
array.each_index {|i| count += 1 if array[i] == i }

Now these two code samples are almost identical. Some people would argue that its just a matter of style which you prefer. However, as someone who has written countless loops like this under the time pressure of programming contests I can tell you there is one crucial difference. In the Java code I am explicitly incrementing the index variable i in the for loop and checking it against the length of the array. In Ruby, the method each_index does this for me. Why does this matter? Well, imagine that you have nested loops. Well, it can be really easy to accidentally write your inner loop like:

for (int j = 0; j < array2.length; i++)

and now you suddenly have an infinite loop. More insidious, you could write:

int[][] map = new int[ROWS][COLS]; 
for (int r = 0; r < map.length; r++) 
   for (int c = 0; c < map.length; c++)
     array[r][c] = r + c;

when you meant to write the column loop:

for (int c = 0; c < map[r].length; c++)

and now you have code that works whenever ROWS == COLS, but not otherwise.

Having had many conversations with other competitors at contests, I can tell you that I am not the only one who has wasted large amounts of time tracking down dumb bugs just like these. Nor am I the only one who has unknowingly submitted a buggy solution because I only tested on square maps, and so failed a problem because of such a silly bug. My hope is that by using Ruby I will spend less time dealing with bugs like this, allowing me more time to actually solve the problem at hand.

Performance
Here's the catch. The dynamic aspect of Ruby comes at a cost. Many problems in programming contests like the Google Code Jam are very computationally intensive and there is always a time limit for how long your code can run. In fact GCJ's FAQ warns that Python's slowness may cause problems for you on some problems. My assumption is that Ruby has similar performance characteristics to Python, so I decided to do some simple comparison's between Ruby and Java performance on some programming contest type tasks.

Test 1
As my first test, I came up with a solution for GCJ's 2008 Round 1B's Problem C. Mousetrap. My Ruby solution takes about 40 seconds to correctly process Google's large input set. The same algorithm in Java takes about 4 seconds to process the same data set.

Test 2
My second test was based on a suggestion of a friend who was inspired by Problem D. GoroSort on this year's qualification round. What I did was write a program which creates every permutation of N integers, and then for each permutation counts up how many numbers are the same as their index. In addition to Java and Ruby, I dusted off my C and wrote a C implementation, as well, for comparison. Here are the results I get on my Macbook Pro. (all values are in seconds)

N	Ruby	Java	C
6	0.015	0.20	0.003
7	0.028	0.21	0.004
8	0.15	0.22	0.007
9	1.3	0.24	0.021
10	14	0.55	0.16
11	160	3.9	2.4
12	1,980	43	36

Conclusion
There is a definite and meaningful difference in performance between Ruby and the other languages for larger data sets. Still, for the majority of problems Ruby will be fast enough. I will just need to make sure I do a big-O analysis of my algorithms before I implement them, and if there is any doubt as to performance, write it in Java rather than Ruby. For any problems where speed doesn't look to be an issue, I think I will try to use Ruby. Hopefully this will work for me.

Session Timeouts on Rails

2011-05-02T09:00:00.021-04:00

Session Timeouts are a perfect example of the conflict between usability and security. From a security standpoint you don't want to leave a user logged in for an extended time period, because someone else may get access to their browser and then act as that person. As an example I have a a Facebook friend who frequently has posts made in her name by friends who get a hold of her phone. On the other hand, as a user, it is really frustrating when I have to keep logging back into a site.

In general, be only as secure as you need to be. I am ok with my bank logging me out after a short period of inactivity. However, I am willing to risk the occasional prank post on Facebook to not have to keep logging back in. As long as the social sites require a re-entering of a password before allowing a password change or any other personal information changes, they are secure enough from my point of view.

Session Expiration Tasks
If you want to expire sessions, you need to do a couple of things:

Keep track of the time of the user's last action
Show the timeout on the client side, by having a javascript function running on the browser which will load the login page after the session has timed out
Enforce the timeout on the server side by having the server redirect to a login page if a different page is requested after a timeout
Make sure the redirect also happens on Ajax updates

Rail Implementation
For purposes of this code, I am assuming that you have a login system like shown in Michael Hartl's "Ruby on Rails Tutorial".

Tracking Timeout
Following the example I found at http://snippets.dzone.com/posts/show/7400 I made my ApplicationController look like:

class ApplicationController < ActionController::Base
  protect_from_forgery
  before_filter :session_expiry
  before_filter :update_activity_time

  include SessionsHelper

  def session_expiry
    get_session_time_left
    unless @session_time_left > 0
      sign_out
      deny_access 'Your session has timed out. Please log back in.'
    end
  end

  def update_activity_time
    session[:expires_at] = 30.minutes.from_now
  end

  private

  def get_session_time_left
    expire_time = session[:expires_at] || Time.now
    @session_time_left = (expire_time - Time.now).to_i
  end

end

In this code we store an expiration time in the session scope, and we make sure the session hasn't expired before performing any actions. update_activity_time accomplishes the goal of tracking the user's last action and session_expiry enforces the timeout.

Support Ajax Calls
A complication comes up if your web application has Ajax calls. If an Ajax request comes in after the session times out you want to do a full page redirect to the login page, not just a partial update. To accomplish this I modified the deny_access method in the SessionsHelper module to look like:

def deny_access(msg = nil)
  msg ||= "Please sign in to access this page."
  flash[:notice] ||= msg
  respond_to do |format|
    format.html {
      store_location
      redirect_to signin_url
    }
    format.js {
      store_location request.referer
      render 'sessions/redirect_to_login', :layout=>false
    }
  end
end

As you can see, on an Ajax call we store the referrer, i.e. the page that the user was on when they made the Ajax call, and then render a javascript fragment. The file 'views/sessions/redirect_to_login.js.haml' just consists of the one line of javascript:

window.location.replace( '#{escape_javascript signin_url}' );

which tells the browser to redirect to the login page.

Client Side Timeout
It can be a frustrating user experience to perform an action on a page, only to find out that you have timed out. Since we've already decided that we will timeout the user, we can't prevent that behavior, but we can at least let the user know they have timed out by automatically redirecting them to the login page when they time out. To accomplish this we add the following javascript lines to our public/javascripts/application.js file

function checkSessionTimeout(url) {
 $.getScript(url);
}
function setSessionTimeout(url, seconds) {
 setTimeout("checkSessionTimeout(\'" + url + "\')", seconds*1000 + 15);
}

checkSessionTimeout is just a method which makes an Ajax call (using jQuery) to a URL and executes the returned javascript. setSessionTimeout sets a javascript timeout to call checkSessionTimeout after the specified delay.

To call these functions we add the following to our views/layout/application.html.haml

-if @session_time_left
  :javascript
    $(function() {
      setSessionTimeout('#{check_session_alive_sessions_path}', #{@session_time_left})
    });

Basically, if the @session_time_left variable is set, we use the jquery functionality to call the setSessionTimeout method when the page has been loaded. We pass the url that corresponds to the check_session_alive action on the sessions controller which looks like:

def check_session_alive

  get_session_time_left
  if @session_time_left > 0
    render 'sessions/check_session_alive', :layout=>false
  else
    sign_out
    deny_access 'Your session has timed out. Please log back in.'
  end
end

If the session has timed out, we just call deny_access which we showed above and already handles the redirect. If the session hasn't timed out, we render sessions/check_session_alive.js.haml which just calls setSessionTimeout again, with the new timeout value:

setSessionTimeout('#{check_session_alive_sessions_path}', #{@session_time_left})

One Catch
We don't want to enforce that the session is valid on the session actions like login, logout, and check_session_alive. To fix this we add the following to the top of our SessionsController:

class SessionsController < ApplicationController

  skip_before_filter :session_expiry
  skip_before_filter :update_activity_time

Conclusion
With that you have a bare bones session expiration system on your Rails app. I will leave it as an exercise to the reader to add extra features, like giving a user a 5 minute warning that they are about to be logged out.

Programming Contest Season

2011-04-25T09:00:00.016-04:00

It is the time of year for programing contests. Google Code Jam has its qualifying round on May 6^th and the TopCoder Open starts the qualifiers for their algorithm contest on May 14^th. Assuming nothing unexpected comes up, I intend to participate in both of these contests again. If I am to have any hope of doing well in either of the contests I will have to prepare.

There are 3 main skills you need to have to able to do well in these type of contests.

Problem Solving
These contests are not software development contests. You don't need to be able to build an application better than other people, or even at all. What you need to do is be able to solve puzzles. Sometimes this is just a simple matter of following the instructions in the problem statement. More often it is a matter of realizing that if you look at the problem the right way, you can map it to another problem (e.g. graph traversal, binary search, etc.) that you already know how to solve. And occasionally you'll have to come up with a completely novel solution on the spot.

There are a few things you can do to get better at this. The first is to learn the known solutions to common computer science problems. It doesn't do you any good to recognize that a problem requires finding the shortest path between two nodes in a graph, if you don't know an algorithm that accomplishes that. The second is to practice recognizing places where these algorithms apply. Practice solving problems helps with this. Seeing how other people solved the same problem can also help.

The more knowledge you have of existing algorithms and the more experience you have at recognizing the use of these algorithms, the less often you'll come across a truly novel problem. And even when you do, you'll be better prepared to find a solution.

Programming
The problems posed in these contests don't require a lot of software engineering. It is rare for a solution to run as long as 100 lines, even in a relatively verbose language like Java. However, it is still a key skill. If you can't convert an algorithm into code accurately and quickly, you won't last long in a contest. When learning algorithms, don't just read about them in a book or online, code them up. And then code them again a different way. That experience will be helpful when coding under the gun.

Besides being able to code algorithms, you need to know your language. Do you know how to format strings, parse strings, and round numbers? Do you know the collection libraries provided and can you use them without having to look up API documentation? Do you know the performance characteristics of various features like autoboxing or string manipulation? How long will it take you to execute an n⁴ algorithm when n = 50? Does it matter what calculations you perform in the inner loop? Can you create an array of length one million? Ten million? One hundred million? How big of a value can you store in an integer? a long? What happens on overflow? Are you familiar with the precision issues that can crop up when using floating point numbers?

If you don't know the answers to the above questions, learn them. Not just from a book, but by writing programs. Knowing what is feasible and what isn't can help you find a workable algorithm for solving a problem.

Speed
These contests are timed. It doesn't do you any good if you can solve a problem, but it takes 6 hours. The only way that I know to improve speed is to practice, practice, practice. There are a couple of hints I can give you, though.

The first is that bugs are the enemy. Nothing kills your time more than having to debug a program. There are things you can do to reduce bugs. One is to be consistant. Be consistant with how you name variables and types, where and when you use curly braces and how you write algorithms. Another is to use as much structure as you need. Even though it is quicker to type one letter variable names and to create one long function, if meaningful variable names and separate functions and/or classes helps structure your code in a way that you can understand it, the time lost in typing will be more than made up in debugging.

Take your time.The last, but most important piece of advice in being quick is to take your time. Plan your solution. If it is complicated, sketch out your solution on paper and pencil before coding. Spend a couple minutes analyzing the performance characteristics of your solution. A couple minutes of thinking can save 20 wasted minutes of programming a flawed solution. Taking a couple extra minutes coding up a solution carefully and cleanly can save hours of debugging time. (and the contests don't last hours, so that is critical).

Out Coding
Well, I guess I should stop procrastinating and get to practicing so that I have a hope of advancing in these contests.

attr_accessible and Security in Rails

2011-04-18T09:00:00.001-04:00

This is a story of how easy it can be to inadvertently put security flaws in your programs.

Given that I had implemented authentication and authorization, I wanted a place where users could edit their profile information. However, I wanted to always keep the user's original name and email address that was provided by OpenID authentication. So my thought was that I would provide the user with a "preferred email address" and a "display name" which I would pre-populate based on what OpenID provided, but which they could change to anything they wanted. I'd still keep the original information, but just not ever show it to them.

Step One - New Database Columns
This was just a matter of creating a new migration:

class AddDisplayNameToUser < ActiveRecord::Migration
  def self.up
    add_column :users, :display_name, :string
    add_column :users, :preferred_email, :string
  end

  def self.down
    remove_column :users, :display_name
    remove_column :users, :preferred_email
  end
end

and running

rake db:migrate

Step 2 - Create Default Values in Model

Pre-populating these new columns was easy, all I needed was to use the "after_initialize" hook in ActiveRecord. To do this, I added the following lines to my User model:

after_initialize :init

def init
  self.display_name ||= "#{self.last_name}, #{self.first_name}"
  self.preferred_email ||= self.email
end

Step 3 - Create Web Pages

I created a Profile resource with the following line in routes.rb

resource :profile, :only=>[:show, :edit, :update]

In the "show" action, I show the user their display name and preferred email. The edit page allows them to edit these two pieces of information. The HAML for this is:

%h1 Edit Your Profile
= form_for @user, :url=>profile_path do |f|
  = f.label :display_name
  = f.text_field :display_name, :size=>50
  %br
  = f.label :preferred_email, 'Preferred email address'
  = f.text_field :preferred_email, :size=>50
  %br
  = f.submit 'Update Profile'

The update method in the controller, which gets called when the form is submitted looks like:

def update
  current_user.update_attributes params[:user]
  redirect_to profile_path
end

And with that, I have a page where a user can update their own information. I'm done, right?

Security Flaw

Do you see the security flaw?

The problem is one that frequently comes up in client/server settings, particularly on the web. The server is putting too much trust in the client. It's all well and good to restrict the user's edit page to just the Display Name and Preferred Email fields, but a wily user will just bypass the web browser and directly make an HTTP request with parameters set to change fields that we don't want changed, like the :identifier_url.

So how do we fix this? Well the obvious way is to replace the line

current_user.update_attributes params[:user]

in the update method with code that explicitly just sets the fields that we care about. But then we lose the convenience of letting this Rails library method do the work for us. Luckily there's a solution, and I am sure the title of this post has not given you any hint to that solution. If you add the line:

attr_accessible :display_name, :preferred_email

to your User model class, you are telling Rails to allow those fields to be updatable via a mass assign command like update_attributes. More importantly, as soon as you've declared at least one field as attr_accessible, then all of the non-declared fields can't be update this way.

Take Home Message

Never Trust the ClientI am sure you know this already, but it bears repeating: Never Trust the Client. There is nothing stopping a malicious user from writing their own client and sending your server any data they want. Unfortunately, this requires eternal vigilance and learning. Sometimes holes, like the one above, exist not because of code you wrote, but because of the code you didn't write. If I hadn't read about this exact situation in Michael Hartl's Ruby on Rails 3 Tutorial, I might never have even realized the security hole I had created.

New Languages Can Broaden Your Horizons

2011-04-11T09:00:00.110-04:00

"A language that doesn't affect the way you think about programming is not worth knowing." -Alan PerlisIn linguistics there is theory called the Sapir–Whorf hypothesis which asserts that our language limits and defines our cognitive processes. In the computer realm Alan Perlis once said that, "A language that doesn't affect the way you think about programming is not worth knowing." I don't know how true these statements are, but I do know that just about every programming language I have learned has taught me new ways to think about software development. In many cases I wasn't even aware of what I didn't know until the new language opened my eyes.

Here is an overview of languages that have changed my way of thinking about programming, in the order that I used them. I realize that some of the concepts that I learned in one language could've been learned in another, but this is the order I personally learned things. Sometimes just the act of doing something new can show you things that you could have done with the old tools, but hadn't considered in the past.

Languages

BASIC (in particular, Commodore 64 BASIC) - my first programming language. While much of what I learned at this time I later had to unlearn, learning that computers aren't magical but just do what the programmer tells them has had lifelong value.

Assembly/Machine Language - I learned what the computer actually did - i.e. what does high level code get turned into? What instructions can the CPU actually perform?

Pascal - with Pascal (well, technically Karel the Robot which we did for a month, first) I was introduced to the concepts of functions and procedures, as well as how to use looping constructs effectively. I also learned about pointers and data structures in Pascal. (i.e. the basics of structured programming as we know it). I was also first exposed to separating interfaces from implementations with Pascal (Turbo Pascal's Units), but it didn't really take.

C - C was a combination of Pascal and Machine Language for me. I still could use all of the structured programming concepts that I knew, but with C you could see how it tied to machine code. e.g. pointers are just memory addresses. They say that C is "A language which combines the flexibility of assembly language with the power of assembly language." For me this was its beauty. While not being quite at the Assembly level, I could understand exactly what my programs were doing. I also learned about the difference between compile time and link time, which is critical to understanding how to build large systems.

C++ - This was my first real exposure to OO. However, it didn't really click. I liked the encapsulation aspect of grouping methods with data in classes, but at this point in my programming life I still didn't "get" OO. What I mainly got from C++ was a way to structure my C code.

LISP - besides having Lots of Irritating Stupid Parenthesis, it introduced me to functional programming. I learned it is possible to program with variables that don't vary and without iterative constructs (Lisp has looping constructs, we just didn't use them till later on in the class). Very powerful ideas, though I have to admit that I didn't use LISP for long (even though I thought it was really cool), so I probably didn't absorb as much from it as I could've from the language.

Java - with Java I finally got OO. (or at least I think I get it now. :) ). I think the key feature that helped me was the concept of an interface. Why would you want to create a "class" that doesn't do anything? It was the struggle to answer that question that turned on the light bulb in my head for getting OO. The other cool idea Java introduced me to was the idea of namespaces. While C++ was starting to get them at that time, I had never used them, and had been burned by class name collisions. Namespaces are a good idea for large programs, and a necessary one for large scale code reuse.

When Java 5 came out and annotations came into the language, this was another feature that didn't seem to have any value at first. I have since seen that there can be pretty large value in declarative programming though I don't think I have fully internalized how to use the power of it yet. (i.e. I can see the power of this idea, but it hasn't actually transformed me yet).

C# - when I first learned C#, it didn't change my way of thinking. It was just Java with slightly different syntax. But then I got a chance to play around with LINQ. There are two powerful ideas that I got out of LINQ. The first is directly using query language notation inside a "normal" programming language. The second is that of late evaluation. i.e. if you had something like

DataContext context = new DataContext(<db stuff>); 
Table<Person> people = context.getTable<Person>(); 
var query = people.Where(p => (p.age >= 20 && p.age <= 25))
                  .OrderBy(p => p.name).Select(p => p); 
foreach (var person in query) {
  Console.WriteLine("{0} is {1} years old", person.name, person.age) 
}

I would've naively implemented the getTable method to return the entire datatable, Where to return the reduced datatable, OrderBy would sort, and then Select would return an array with the values. This is obviously much less efficient then just letting the database do the query for you. One of the clever thing that the LINQ folks did is they accomplished this. Where, OrderBy and Select all return objects describing their actions, not the results of their actions. Its not until you try enumerating the elements in the query in the foreach loop that any data is actually retrieved or manipulated. At that point the code is able to take advantage of everything you specified to act efficiently. Compare that to the Unix command line:
find . -name "*" -print | grep "taxes" | sort -r
I am used to this type of environment where find does a calculation and passes data to grep which then passes data to sort. The idea of just passing metadata around and then doing the calculation at the end requires greater coordination between components, but when you have that the result is improved efficiency.

Ruby - with Ruby I am learning that the static typing I learned to love in Java might not be as important as I believed. Duck typing is a great idea. Even cooler are ruby blocks, i.e. closures. Being able to pass blocks of code around is HUGE and definitely a transformative idea. Prior to learning Ruby, I would occasionally run into situations where I was frustrated by Java's lack of function pointers. But now! Now I have a hard time writing code in Java, because I keep wanting to solve my problems using blocks.

Rails - while Rails is just a framework, and not truly a language, I feel it deserves its own entry. Rails leverages Ruby in such a way that it feels like you are using a domain specific language. Cool ideas I learned from Rails are convention over configuration and how to actually use the MVC pattern.

Javascript - from Javascript I learned that OO can be done in multiples ways. OO in Javascript is nothing like OO in C++/Java/C#. In fact, everything is just an associative array. The power of what you can do with such a simple idea, in conjunction with closures, is eye opening.

Conclusion
I am not a language expert, and as such there are many important features of the above languages that I omitted. The point of this post isn't what features are in what language. Rather it is a personal history showing how new languages expanded my horizons and abilities as a software developer.

Have you learned a new language lately?

Role Based Authorization in Rails

2011-04-04T09:00:00.004-04:00

Previously, I created an OpenID based login system. Now I want to create a role based authorization system. What do I mean by role based authorization? Well, in many systems you have actions that you want to protect so that only admins can perform them. Obviously, we could implement this by adding an "admin" flag to our User model. However, sometimes, we have more than just two classes of users. For example, if you have a website with comments, you might want to have a class of users who aren't full admins, but do have authority to filter comments for spam.

A simple way to accomplish all of this is to have Role model and then let each user have 0 or more roles associated with them. Here is how you can do this, assuming that you've already got an authentication system as described in the previous post.

Data Model
First, create database tables, which will just consist of a Role table with a name column, and then a mapping table to map roles to users.

rails generate model Role name:string
rails generate model RolesUser role_id:integer user_id:integer
rake db:migrate

Then tell the User model about the roles, by adding the following line to the User class (app/models/user.rb):

has_and_belongs_to_many :roles, uniq=>true

Role Views
Now that we have a Role model, we need a way to add and delete roles. Since this is functionality that we'll want to reserve only to top level admins, let's put this functionality in an admin directory.

rails generate controller Admin::Roles index

This generated the controller and view, but not the right routes. So delete the

get "roles/index"

line that was added to routes.rb file and replace it with:

namespace :admin do
  resources :roles, :only=>[:index, :create, :destroy]
end

This allows a showing of roles and the RESTful creation and destruction of roles. Since roles are just a name, we don't really need the separate show, edit, and update paths.

The controller (at app/controllers/admin/roles_controller.rb) looks like:

class Admin::RolesController < ApplicationController

  def index
    @roles = Role.all
    @new_role = Role.new
  end

  def create
    Role.create params[:role]
    redisplay_roles
  end

  def destroy
    Role.find(params[:id]).destroy
    redisplay_roles
  end

  private

  def redisplay_roles
    redirect_to admin_roles_path
  end
end

The method redisplay_roles is a placeholder for now. A little bit further down we'll modify this to support adding of roles via Ajax calls.

So now we have to create the HTML, app/views/admin/roles/index.html.haml

%h1 Roles
= render 'role_list'
%br
= form_for [:admin, @new_role] do |f|
  = f.label :name, 'Name of new role:'
  = f.text_field :name
  = f.submit

There are two things of note here. On line 4, the array [:admin, @new_role] results in the URL for creating a Role object in the admin namespace, which is what we want. On line 2, we render the partial 'role_list' (app/views/admin/roles/_role_list.html.haml), which is shown below. The reason it is a partial is to make it easier to Ajaxify everything.

#RoleList
  %h2 There #{is_pluralize(@roles.length, 'role')}
  %ul
    - @roles.each do |role|
      %li
        %b= role.name
        [#{link_to 'x', [:admin, role], :method=>:delete, :confirm=>'You sure?'}]

is_pluralize on line 2 is a helper function I wrote (in application_helper.rb) that uses the appropriate form of to be along with the pluralized noun. It looks like:

def is_pluralize(count, noun)
  verb = (count == 1) ? "is" : "are"
  "#{verb} #{pluralize(count, noun)}"
end

The rest of the partial is just looping through the roles and showing them. On line 7, a link to the destroy action is created, by specifying the method as 'delete'.

AJAX

The above is all you need to be able to add and delete roles. Since we are redirecting back to the index page after each add or delete, I thought it would be nice to make those Ajax actions, and just stay on the page. Turns out this is pretty easy.

As a first step, if you haven't already, install jQuery. (yes you could use prototype or many other javascript libraries, but jQuery is what I am using, so it's what I'll describe.) To do this, put the line

gem 'jquery-rails'

in your Gemfile and then run the commands:

bundle install
rails g jquery:install

Now that jQuery is installed, modify the add/remove calls to be AJAX calls by adding the parameter ":remote=>true" to the link_to call on line 7 of _role_list.html.haml and to the form_for call on line 4 of index.html.haml.

Now we modify redisplay_roles to look like

def redisplay_roles
  respond_to do |format|
    format.html { redirect_to admin_roles_path }
    format.js {
      @roles = Role.all
      render :redisplay_roles
    }
  end
end

Line 3 handles the case if the method is called in a non-Ajax way. Line 5 ensures that @roles variable is set. Line 6 is where the magic happens. Rather than returning html like normal, we will return javascript, so we create a view file app/views/admin/roles/redisplay_roles.js.haml. This javascript HAML file just has one line:

$('#RoleList').replaceWith("#{escape_javascript(render :partial=>'role_list')}")

$('#RoleList') is the jQuery selector to choose the div we are modifying and replaceWith is a jQuery function which will replace the contents of that div with the argument that is passed in. To create that argument, we just render the partial 'role_list' again.

With just those simple changes, now the adding and removing of roles is done via Ajax calls.

User Views
Now that we can create roles, we need to be able to add roles to users. This will be very similar to what we did with Roles above. The command:

rails generate controller Admin::Users index show

generates our controllers and views. Again, we will delete what got added to routes.rb and replace it with:

namespace :admin do
  resources :users do
    member do
      post :add_role
      delete :delete_role
    end
  end
  resources :roles, :only=>[:index, :create, :destroy]
end

This includes the route for Roles from above. While we aren't using the edit and update routes, we will leave them in for future enhancements. We are also adding the routes add_role and delete_role which will do what you'd expect. The users_controller looks like:

class Admin::UsersController < ApplicationController
  before_filter :load_user, :except=>[:index]

  def index
    @users = User.all
  end

  def show
  end

  def add_role
    role = Role.find_by_name params[:role]
    @user.roles.push role if role
    redisplay_roles
  end

  def delete_role
    @user.roles.delete(Role.find params[:role])
    redisplay_roles
  end

  private

  def load_user
    @user = User.find params[:id]
  end

  def redisplay_roles
    respond_to do |format|
      format.html { redirect_to [:admin, @user] }
      format.js { render :redisplay_roles }
    end
  end
end

The index.html.haml just shows a list of all the users with links to show them.

%h1 Users
%h2 There #{is_pluralize(@users.length, 'user')}
%ul
  - @users.each do |user|
    %li= link_to user.email, [:admin, user]

show.html.haml looks like:

%h1 User
%b Id:
= @user.identifier_url
%br
%b Name:
#{@user.last_name}, #{@user.first_name}
%br
%b Email:
= @user.email
%hr
%h2 Roles
= render :partial=>'role_list'
#addRole
  = form_tag [:add_role, :admin, @user], :remote=>true do
    = label_tag :role, 'Add Role:'
    = text_field_tag :role
    = submit_tag 'Add'
%hr
= link_to 'Return to User List', [:admin, User]

Just like above, we put the roles listing in a partial. _role_list.html.haml looks very similar to the one in the roles view:

#RoleList
  This user has #{pluralize @user.roles.length, 'role'}
  %ul
  - @user.roles.each do |role|
    %li
      = role.name
      [#{link_to 'x', delete_role_admin_user_path(@user, :role=>role.id), :method=>:delete, :confirm=>'Are you sure?', :remote=>true}]

which means that redisplay_roles.js.haml is identical to the Roles version of this. I suppose I could pull this into the shared directory and let it be shared, but I am not convinced that it will always remain identical, so I'll leave the duplication for now.

Authorization
Now that we can create roles, and add users to roles we need a way to use this information. First we'll add a method to User to determine if a user has any of a given role or roles. I would like this to work if the passed in roles are Role objects or strings or symbols defining the role name. To do this I overrode to_s in Role

def to_s
  self.name
end

and added the following methods to User

def has_role?(*role_names)
    self.roles.index {|role| includes_role role_names, role}
  end

  def includes_role(role_list, role)
    role_list.index {|r| r.to_s == role.name}
  end

As you can see, has_role? takes a variable number of roles and returns true if the user has any of those roles. Now that we can make this determination, we will create a method called ensure_role. We could place it in the controller, but since we will be using it from multiple controllers we will be placing it in the authentication_helper.rb that was created in the previous post. It looks like:

def ensure_role(*role_names)
    if signed_in?
      role_names.push 'admin'
      unless current_user.has_role? *role_names
        flash[:error] = 'You do not have permission to view this page'
        redirect_to actions_index_path
      end
    else
      ensure_signed_in
    end
  end

On line 3, we push the role 'admin' onto the list of roles. We do this, because we want the 'admin' role to have all permissions and this way we don't have to explicitly list it every time we are ensuring a role. We add the line

before_filter { ensure_role 'admin' }

to our controllers in the admin directory to ensure that only users with the admin role can access these pages. Technically we could leave off the 'admin' argument, since it is implicitly added, but I think it makes the controller much more readable to have it explicitly there in the case where 'admin' is the only role allowed.

Rake Task
Here's a brief quiz. Do you see the problem that we've created?

We now have a chicken and egg problem. You can't create an admin role or assign it to yourself unless you already have it. The way we'll solve this is by creating rake tasks that you can run at the command line to create roles and assign them to users. All the details of rake are beyond the scope of this blog post which has gone on for way to long already, so I'll just tell you what I did. I created a file lib/tasks/roles.rake which looks like:

namespace :roles do
  desc 'Creates a role'
  task :create, [:role_name] => :environment do |cmd, args|
    Role.create :name=>args[:role_name]
    puts "Created role #{args[:role_name]}"
  end

  desc 'Add User to Role'
  task :add_user, [:email, :role_name] => :environment do |cmd, args|
    user = User.find_by_email args[:email]
    role = Role.find_by_name args[:role_name]
    unless user
      puts "No such user #{args[:email]}"
      return
    end
    unless role
      puts "No such role #{args[:role_name]}"
      return
    end
    user.roles.push role
    puts "added #{role.name} to #{user.last_name}, #{user.first_name}"
  end
end

Now you can run the commands:

rake roles:create[admin]

rake roles:create[<your email address>,admin]

to create an admin role and assign your user to it.

Conclusion

Wow, this post went on a lot longer than I meant it to. I guess there was a lot more ground to cover than I realized. I'll try to keep my tutorial based posts shorter in the future...

Visiting a Foreign Land

2011-03-28T09:00:00.001-04:00

This past week I took a trip to a foreign country, MacLand. I have many friends who either live there or are frequent visitors and they all love it, despite the very high cost of living there. I decided it was time to see what all the fuss was about.

When I first got off the plane and opened up my new 13" MacBook Pro, it was welcoming. I was through security and on the internet in almost no time at all. However, it didn't take long for me to become discombobulated. The customs here are strange, like the fact that they drive on the left side of the road here with their window controls and they don't leave the menus at the tables as I am used to. I'm obviously going to have to learn the local dialect. What is a "Finder" or "Preview" anyway, and what are the funny symbols (arrows and cloverleafs?) I am seeing on many of the menus? And my instincts as to what to do in any given situation tend to be wrong. I am worried that I'll commit a serious faux pas and end up in trouble. Well, I guess I'll just have to trust that my friends that live here will bail me out if that happens.

To achieve some level of comfort, I installed Chrome and Emacs. It's always nice to see a familiar face when abroad. In the lobby of my MacBook there are a number of brochures of local activities, like going on Safari, that I suppose I should try out. When I get up some courage, I guess I will try to branch away from the strictly familiar.

So why did I decide to take this journey? Well some years ago, there was a revolution in MacLand, and the new government, OS X, was based on Unix. I have long preferred Unix to other approaches, but historically, Unix run places were not places for individuals to dwell, unless they had a strong Do It Yourself mentality. My own hobby interests lie elsewhere, I just want my infrastructures to work. My couple personal trips to Unix resulted in more hassles than it was worth to me. With OS X, MacLand promised the power of Unix while also providing a nice friendly infrastructure that just works.

Between this promise, the numerous friends I have who love it, and the fact that Ruby on Rails is supposed to be easier in MacLand, I decided it was time to take the trip. We'll see how I like it, but so far I am cautiously optimistic that this will be a good land. (as long as there are no betrayals)

Google OpenID via Rails

2011-03-21T09:00:00.007-04:00

So you've decided to write a web application. You are probably going to want to allow users to log in. Maybe its because you don't want to mess up password security, or maybe its because you feel the web doesn't need yet one more username/password combination, but you've decided to use OpenID. Here's how I've done it in rails.

For this, I am very much leveraging http://blog.sethladd.com/2010/09/ruby-rails-openid-and-google.html. Rather than repeat everything that Seth Ladd said, I will just include the differences from him.

First, I am starting completely from scratch, so the first step is to create a rails project.

rails new login

Then I need a user model:

rails generate model User identifier_url:string email:string
first_name:string last_name:string
rake db:migrate

The above two steps replace step 0 from Seth Ladd's blog post.

For step 1, here is what my Gemfile looks like:

  1 source 'http://rubygems.org' 
  2  
  3 gem 'rails', '3.0.4'
  4 gem 'sqlite3' 
  5 gem 'haml' 
  6 gem 'haml-rails' 
  7 gem 'ruby-openid' 
  8 gem 'rack-openid' 
  9 gem 'mongrel', '>= 1.2.0.pre2'
 10  
 11 group :development do
 12   gem 'rspec-rails' 
 13 end 
 14  
 15 group :test do
 16   gem 'rspec' 
 17   gem 'webrat' 
 18 end

There are only a couple things of note in this Gemfile. As I've mentioned before, I like haml, hence lines 5-6. Lines 7-8 are the lines from Seth's post. Line 10 is because WEBrick causes errors when I try to use the OpenID authentication and the current version of mongrel isn't working for me (Windows platform). Oh, and lines 11-18 are lines I pulled from Ruby On Rails 3 Tutorial by Michael Hartl. Even though I am not doing any testing here, it seems like a good idea to enable it.

Then, of course, I ran:

bundle install

Steps 2-6 I exactly followed Seth's post. For step 7, I basically followed his post, but used Haml rather than Erb. For step 8, I created an action controller:

rails generate controller Actions view

My action controller looks like:

  1 class ActionsController < ApplicationController
  2   before_filter :ensure_signed_in  
  3  
  4   def view 
  5   end
  6 end

and my app/views/actions/view.html.haml looks like:

  1 %p  
  2   = current_user.first_name  
  3   = current_user.last_name  
  4   (  
  5   = current_user.email  
  6   )  
  7 %p= current_user.identifier_url

And with that I have a simple app that uses OpenID for authentication. I test it by running:

rails s

and then going to http://127.0.0.1:3000/actions/view in my browser. As expected, I get redirected to google, and then after logging in, I get sent back, and I can see my information. Doing something useful with this will be left as an exercise for the reader.

When Does X Not Equal X and Other Oddities

2011-03-14T09:00:00.001-04:00

In many programming languages:

For what value of x is the condition (x == x) false?
For what integer value of y is the condition (y < 0) && (-y < 0) true?

i.e. define x on line 3 and y on line 10 in the following Java code sample so that lines 7 and 12 execute:

  1 public class Odd {
  2   public static final void main(String[] arg) {
  3     // define x here 
  4     if (x == x) { 
  5       System.out.printf("Reflexive%n"); 
  6     } else { 
  7       System.out.printf("Not Reflexive%n"); 
  8     } 
  9  
 10     int y = 0; // change 0 to something else
 11     if (y < 0 && -y < 0) { 
 12       System.out.printf("How can they both be negative?%n"); 
 13     } 
 14   } 
 15 }

In SQL:

What database value x is the condition (x = x) false?
What about values y and z such that ((y = z) OR (y != z)) is false?

These next few lines are intentionally left blank, in case you wan to think about these before viewing possible answers.

Answers
In mathematics, one of the properties of the equality operator is that it is reflexive. It turns out that the Java (and almost every other language) equality operator does not have this property. If you put the following on line 3:

double x = 0.0/0.0;

and run the program, it'll print out "Not Reflexive". Why is this? Well, according to the IEEE standard 754, 0/0 = NaN. What is NaN? Well, it is not a number, corresponding to the mathematical concept of an indeterminate number. Because it could be any number, the powers that be decided that NaN != NaN.

So what about the second example? Well, in a 2's complement system of encoding numbers, there is room for one more negative number than positive numbers. (e.g. a 16 bit signed value encodes the values -32,768 to 32,767). What this means is that if you try to negate the most negative number (e.g. -32,768), the corresponding positive number won't fit in the memory location, and so you actually get the same number back. This means an answer to line 10 is:

int y = Integer.MIN_VALUE;

What about the SQL questions? Well, it turns out that SQL treats NULL like NaN, i.e. it is an indeterminate value. SQL goes even further than Java. Not only is the expression (NULL = NULL) result in false, but (NULL != NULL) also results in false. Well, actually that's not quite true. SQL uses three-valued logic, and so both the expressions actually evaluate to undefined, but undefined is treated as false when it comes to matching rows.

Why does this matter?
Have you ever written code where it is assumed that if two numbers have the same value they will be equal? How about that Math.abs() will always return a negative value? Eric Lippert had a blog post showing how this assumption can cause the standard sort functions to fail.

I personally have generated two reports by running queries like:
SELECT * FROM Table WHERE columnA = columnB -- report 1
SELECT * FROM Table WHERE columnA != columnB -- report 2
and then was mightily confused when, at some point during the analysis, I realized that there were rows in my database table that weren't in either report.

These may feel like nitpicky details that only show up on those gotcha type quizzes. However, not understanding these details can lead to bugs in your code. If you are going to be an expert programmer, you have to know and understand the rough edges of your language.

Crop and Upload Image - Server Side

2011-03-07T09:00:00.043-05:00

Previously I showed some HTML and JavaScript for cropping and uploading an image. Here is what I did on the server side to support this.

First I make sure I had rails 3 installed, configured for HAML and jquery. I also download the jqueryui and jcrop javascript and css files and add them to my project. Then, I use the rails scaffolding functionality to get started:

ruby script\rails generate scaffold image content:binary width:integer height:integer name:string

After running:

rake db:migrate

I have a database table for storing an uploaded image, including the binary content, the height and width of the image and a name. Now that I have a place to store the image, I need to modify the generated web pages, so I can upload it.

Since I prefer HAML, the first thing that I do is rename the two erb files that I need to change: views/layouts/applications.html.erb to views/layouts/applications.html.haml and app/views/images/_form.html.erb to app/views/images/_form.html.haml. The applications.html file is the template for all the pages, and I need to modify that to allow pages to add their own javascript into the header. _form.html is the partial that is used for both uploading new images and editing images.

Converting the applications.html file from erb to haml is just a straightforward transformation. Here is what it looks like when I am done:

  1 !!! 
  2 %html 
  3   %head 
  4     %title Image Upload 
  5     = stylesheet_link_tag :all 
  6     = javascript_include_tag :defaults 
  7     = csrf_meta_tag 
  8     = yield :head 
  9   %body 
 10     = yield

The only real line of note is the named yield on line 8. This is what will allow me to insert custom stylings and javascript into specific pages.

The only thing left is to rewrite _form.html.haml so that it generates HTML and Javascript like discussed in the client side post. Here is what this looks like.

  1 - content_for :head do 
  2   = stylesheet_link_tag 'jquery.Jcrop' 
  3   = javascript_include_tag  'jquery.Jcrop.min'
  4   %style 
  5     -# insert styling described in client post here 
  6   :javascript 
  7     // insert javascript described in client post here
  8 = form_for(@image) do |f| 
  9   - if @image.errors.any? 
 10     #error_explanation 
 11       %h2 
 12         = pluralize(@image.errors.count, "error") 
 13         prohibited this image from being saved:
 14       %ul 
 15         - @image.errors.full_messages.each do |msg|
 16           %li= msg 
 17   .field 
 18     = f.label :name 
 19     %br 
 20     = f.text_field :name 
 21  
 22   Drag and drop image here: 
 23   #cropWidget{:width=>400, :height=>400} 
 24     #surface 
 25     %canvas#canv1{:width=>400, :height=>400} 
 26  
 27   = f.hidden_field :content64 
 28   Uploaded image is here: 
 29   %canvas#canv2{:width=>200, :height=>200} 
 30   .field 
 31     = f.label :width 
 32     %br 
 33     = f.text_field :width, :readOnly=>true 
 34   .field 
 35     = f.label :height 
 36     %br 
 37     = f.text_field :height, :readOnly=>true 
 38   .actions 
 39     = f.submit :id=>"imageSubmit", :onClick=>"uploadSelection();", :disabled=>true;

I left out the specific CSS styles and JavaScript as they are described in the previous post. Notice that lines 1-7 are inserted into the head of the template as described by yield :head on line 8 of the application.html.haml file above. Lines 9-16 are the boilerplate error handling that was generated by the scaffold command. The remainder just generates the HTML for displaying the image and for entering the name of the image.

We are now almost, but not quite, done. If you notice the field that we are storing the image content in is content64, while the database column for the binary data is just called content. We need to take the base64 encoding of the image that is sent over the wire as part of the HTML POST and decode it to binary before saving in the database. To do that, we just need to add new methods in our app/models/image.rb Image class.

  1 require "base64" 
  2 class Image < ActiveRecord::Base 
  3   def content64 
  4     return nil unless content 
  5     return "data:image/png;base64," + Base64.encode64(content); 
  6   end 
  7  
  8   def content64=(c64) 
  9     index = c64.index(','); 
 10     self.content = Base64.decode64(c64[index..-1]); 
 11   end 
 12 end

This way when the content64 attribute is assigned in our controller class, it'll actually write the binary data to the content field so it can be saved to the database. The only clever part of this is that the content64 data that is passed from the HTML form has the initial string "data:image/png;base64,". Lines 9-10 calculate where this prefix ends and just decodes the remainder of the string. Line 5 adds this prefix back on to the encoded binary content.

Other Work
So the above will allow you to upload a new image that has been cropped and save it to the database. However, to be able to view that image or edit it, more work is needed. Probably the most important thing to do is to add a new action which will return the image's binary content. The url for this action can be used in the src attribute of <img> tags. The other scaffold generated pages for viewing and listing the uploaded images are then modified to display the image rather than showing the binary content. The last step is to modify the javascript described previously to check if an image already exists (i.e. we are editing an existing uploaded image rather than uploading a new image), and preload that. Since this post has gone on long enough, I will leave all of those details as an exercise for the reader.

The Problem with Component Based Web Architectures

2011-02-28T09:00:00.001-05:00

Imagine the following scenario:

You browse to a web site and log in.
After doing whatever you needed to do, you log out.
You keep your browser tab (which is now sitting at a login page) open.
The next day you return to this browser tab and enter your username and password.

Assuming that you enter your username and password correctly, I think it is a safe assumption that the behavior you would expect is for the site to log you in. Anything else could be considered a bug.

Why do I bring up such a simple scenario? Because if the website was written using JSF and they just used out of the box settings (for example what you would get if you use seam-gen with a Seam application), the above scenario will result in an error!

Component Web Architecture
Both JSF and ASP.NET use a component based web architecture. What this means is that the developer's markup language, whether it is aspx, jsp, or facelets, doesn't directly define HTML. Rather it describes a tree of components. The HTML is generated after the component tree has been built and populated, either by asking the components to render themselves, or by having a separate renderer that traverses the component tree.

Advantages
Component based architecture attempts to leverage the fact that almost all modern programming languages are object oriented. Components are just objects. By using a paradigm that most developers are familiar with the hope is that it'll be easy to use, maintain, extend, etc.

Problem
As you know, the web is stateless. The browser and web server interact by sending messages back and forth. However, the client (browser) and server are two completely disconnected entities. The conversation between them might not actually even be between two entities. A user might save a hyperlink from one page, and then follow the link from a different browser on an entirely different computer. Because of load balancing, or server restarts, the server that handles one message in a conversation isn't necessarily the same as the one that handled the previous message. And even if its the same server, the server might be in a very different state due to handling requests from other clients or other external changes like database state changes.

This presents a myriad of challenges to web developers. Web frameworks provide tools to overcome the above difficulties and still write useful applications. JSF and ASP.NET protect the developers from all of the ugliness through their use of components. Through clever use of cookies, hidden form variables, and session information, these frameworks provide the illusion that you have objects which persist from one page request to the next and that links, form fields, and buttons change the state and or perform actions on these objects.

The problem is that the long lived object model is a leaky abstraction over the reality of stateless messages being sent back and forth. My previous complaints about ASP.NET are caused by trying to paper over this leak.

Component Web Mismatch
To correctly process any form submission (for example login), the first thing the webserver has to do is recreate the exact same component tree structure that was used to generate the page. The appropriate components then get loaded with whatever form data data was submitted, and the component corresponding to the button that was clicked then can have its action called. The challenge is ensuring that the component tree structure that was used when the page was created is the EXACT SAME as the component structure on the form submission. For the simple cases, this is easy. But imagine that the component tree is based on a database query (e.g. listing of items on sale). It's possible that the database has changed between page load and form submission (e.g. items sold to another user). If you naively build the component tree the same way you did originally (e.g. by iterating over results from database), you will get a different component tree which can cause all types of wonky behavior. For example if the user tried to buy the 5^th item on the list, they may end up purchasing the wrong item, since the 5^th item they saw on the screen isn't the same as the 5^th item currently returned by the database.

To solve this problem, when a form is created JSF saves the component tree so that it can be recreated when the form is submitted. This can be stored in a hidden field in the form sent to the client, but since this can be quite large it typically isn't done. By default, JSF saves this in a server session variable. However, if the session times out, this data is lost, and you can no longer recreate the state. This is the cause of the javax.faces.application.ViewExpiredException that happens in the scenario at the top of this post.

Solution
We need web frameworks that don't try to recreate the exact same state on two different requests. There are some very smart people who have (and continue to) work on JSF and ASP.NET. The fact that their solutions are so complex and still tend to drop people through the cracks (based on the questions out there on message boards) is indicative that the approach is flawed, rather than just the implementations. Frameworks like Ruby on Rails which ease the process of writing web apps without hiding the fact that each web request is its own independent action are the long term future.

Crop and Upload Image - Client Side

2011-02-21T09:00:00.047-05:00

I want to be able to extend my moving icon program to allow a user to upload their own icon. However, I want to give the user the ability to crop the image so they just upload the portion that they want. Here is how I do that.

Technologies Used
I use the HTML 5 file capabilities to upload the image to the browser. I use the Jcrop plugin to jQuery to actually specify the crop region. I also leaned heavily on another demo I found on the web (http://www.pandamatak.com/people/anand/blog/2010/11/clientside_image_crop_and_then.html) to figure out how to do it, though I have tweaked it some.

HTML
First you need a place to drag the image too.

<div id='cropWidget' height='400' width='400'>  
  <div id='surface'></div>  
  <canvas id='canv1' height='400' width='400'></canvas>  
</div>

The surface is where an image can be dragged from your file system and dropped to. canv1 is the canvas object that will handle image manipulation.

Next we need a place to show the cropped portion of the image.

<canvas id='canv2' height='200' width='200'></canvas>

The only other portion is a control to put the content, two text controls which will show the size of the cropped image, and a button to upload the image.

<input id="content64" name="content64" type="hidden" />
<div class='field'>  
  <label for="width">Width</label>  
  <br /> 
  <input id="width" name="width" readOnly="true" size="5" type="text" />  
</div>  
<div class='field'>  
  <label for="height">Height</label>  
  <br /> 
  <input id="height" name="height" readOnly="true" size="5" type="text" />  
</div>  
<div class='actions'>  
  <input disabled="disabled" id="imageSubmit" name="commit" 
         onClick="uploadSelection();" type="submit" value="Create Image" />  
</div>

There are a couple of things to note here. First the "readOnly" attribute on the text inputs. The user won't be entering the width and height, they will be calculated by JavaScript. Also, the imageSubmit button is disabled. It does not become enabled until an image has been uploaded. And then on click, the JavaScript function uploadSelection will be called.

CSS
The surface element and the canv1 element need to be superimposed on top of each other for this to look right. This CSS accomplishes this, plus adds a little bit of styling.

<style>  
  #cropWidget {position:relative; width 400px; height:400px; padding:5px;} 
  #surface {position:absolute; top:0px; left: 0px; background:blue;  
            width:400px; height:400px; z-index:100; opacity: 0.50;} 
  #canv1 {position:absolute; top:0px; left: 0px; border:1px solid blue; 
          width:400px; height:400px;} 
  #canv2 {position:relative; border:1px solid yellow; width:200px; height:200px;} 
</style>

JavaScript
Here is all of the javascript. I will explain the parts below.

  1 var cropWidget = document.getElementById('cropWidget');
  2 var surface = document.getElementById('surface');
  3 var canv1 = document.getElementById('canv1');
  4 var canv2 = document.getElementById('canv2');
  5 var imageSubmit = document.getElementById('imageSubmit');
  6 var ctx1 = canv1.getContext('2d');
  7 var ctx2 = canv2.getContext('2d');
  8  
  9 function resize(comp, width, height) { 
 10   comp.width = width; 
 11   comp.height = height; 
 12   comp.style.width = img.width + 'px'; 
 13   comp.style.height = img.height + 'px';
 14 } 
 15  
 16 function displayImage(img) { 
 17   resize(canv1, img.width, img.height); 
 18   resize(surface, img.width, img.height); 
 19   resize(cropWidget, img); 
 20   while(surface.childNodes[0]) surface.removeChild(surface.childNodes[0]);
 21   surface.appendChild(img); 
 22   ctx1.drawImage(img, 0, 0, img.width, img.height);
 23   jQuery("#" + img.id).Jcrop({ aspectRatio: 1, onSelect: cropImage });
 24 } 
 25  
 26 function loadImg(imgFile) { 
 27   if (!imgFile.type.match(/image.*/)) return;
 28   var img = document.createElement("img"); 
 29   img.id = "pic"; 
 30   img.file = imgFile; 
 31  
 32   var reader = new FileReader();
 33   reader.onload = function(e) { 
 34     img.onload = function() { displayImage(img); }; 
 35     img.src = e.target.result; 
 36     resize(canv2, 200, 200);
 37   }; 
 38   reader.readAsDataURL(imgFile); 
 39 } 
 40  
 41 function cropImage(c) { 
 42   var w = c.x2-c.x; 
 43   var h = c.y2-c.y; 
 44   resize(canv2, w, h); 
 45   ctx2.drawImage(canv1, c.x, c.y, w, h, 0, 0, w, h);
 46   $("#width").val(w); 
 47   $("#height").val(h); 
 48   imageSubmit.disabled = false; 
 49 } 
 50  
 51 function uploadSelection() { 
 52   var imgData = document.getElementById('canv2').toDataURL("image/png"); 
 53   $("#content64").val(imgData); 
 54   alert("This would've upload your image to a server, if there was one."); 
 55   return false;
 56 } 
 57  
 58 surface.addEventListener("dragover", function(e) {e.preventDefault();}, true); 
 59 surface.addEventListener("drop", function(e) {e.preventDefault();  
 60                                            loadImg(e.dataTransfer.files[0]);},
 61                          true); 
 62 ctx1.fillText("Drop image here", 60, 100); 
 63 ctx2.fillText("Preview", 60, 100);

Lines 1-5 just create variables for the various HTML objects, and 6-7 get the drawing contexts for the canvases. resize (lines 9-14) is a helper method for changing the size of an HTML component. Lines 12-13 resize the display of the component, while lines 10-11 resize its internal structures, if the component is a canvas. displayImage (lines 16-24) first resizes the HTML components to fit the image. Then any previous image is removed from the surface. Then the image is added to the surface and drawn to the canvas. Finally jcrop is called to enable cropping of this image, calling the function cropImage when a region is selected. cropImage (lines 41-49) resizes canv2 and then draws the portion of the image onto this canvas. Then it stores the width and height into the HTML width and height components and enables the submit button.

Jumping to the end, lines 62-63 put some text onto the canvases. Lines 58-61 enable the drag and drop functionality, setting loadImg as the function that is called when a file is dropped. loadImg (26-39) first ensures that it is actually an image being dropped. Then it creates an HTML <img> tag to contain the image. Line 32 creates an HTML5 FileReader and line 38 reads the file that was dropped. The function defined on lines 33-37 is called when the file has been read. Line 35 sets the source of the <img> tag that was created a few lines above. Line 34 calls the displayImage function once the image has been loaded fully by the browser, and line 38 resizes the crop canvas to a fixed size of 200x200, which also has the effect of erasing any previous image drawn on it.

That is basically the extent of the drag and drop functionality. The only remaining method is uploadSelection (lines 51-56) which is called when the "Create Image" button is clicked. This function gets the base64 encoding of the cropped image and copies it to the HTML element content64 where, if this were actually connected to a server, it would be posted as part of the form back to the server.

Demo
Here is what all of this looks like.

Image Cropper

Drag and drop image here:

Cropped image is here:

Width

Height

Unit Testing - the Ugly

2011-02-14T09:00:00.046-05:00

So you've weighed the benefits and drawbacks to unit testing and decided to do some unit testing. Now you will learn a rarely talked about fact, unit testing is ugly.

A unit test is composed of 3 steps:

Set up state
Execute component
Validate state

For any type of complex components, the code for setting up the state and validating the state can be quite large and tedious. If you look at a set of tests for a single component you'll also find that they tend to be very redundant.

To a certain extent, there is nothing you can do. Comprehensive tests, by necessity, will contain a lot of data and a lot of validation. If you are a believer in the DRY principle, the redundancy will drive you batty. Through standard refactoring ideas you can pull out the common setup and validation code into their own methods and helper classes. However, there are two catches.

Almost the Same

Each of your unit tests are presumably testing different aspects of your component. As such the state will have to be different for each test. And the validation steps will be different for each test. Trying to parameterize your extracted setup and validation methods to support your scenarios can be as challenging a design problem as any portion of your application proper.

Self Contained

One goal of unit testing is that each test is self descriptive. That way when the testFrombulatorWithNull fails you can quickly narrow down the problem. Unfortunately, if you have refactored your unit tests according to the DRY principle, the setup/validation information won't be solely in the specific test method. You'll have to understand how the helper methods and classes interact so you can understand the test so you can understand what went wrong.

Example
Imagine we modify the PhoneFormatter example from before to now take a formatting string in its constructor. Now, we have a lot more cases to test, which might look something like this:

  1 import static org.junit.Assert.*;
  2 import org.junit.Test; 
  3 import org.junit.Before; 
  4  
  5 public class PhoneFormatterTest {
  6  
  7   private PhoneFormatter mFormatter; 
  8  
  9   @Test public void testWithSpace() {
 10     mFormatter = new PhoneFormatter("(###) ###-####"); 
 11     check("(757) 555-1234", "7575551234"); 
 12     check("(757) 555-1234", "17575551234"); 
 13     check("555-1234", "5551234"); 
 14     check("911", "911"); 
 15   } 
 16  
 17   @Test public void testNoSpace() {
 18     mFormatter = new PhoneFormatter("(###)###-####"); 
 19     check("(757)555-1234", "7575551234"); 
 20     check("(757)555-1234", "17575551234"); 
 21     check("555-1234", "5551234"); 
 22     check("911", "911"); 
 23   } 
 24  
 25   @Test public void testNoParen() {
 26     mFormatter = new PhoneFormatter("###-###-####"); 
 27     check("757-555-1234", "7575551234"); 
 28     check("757-555-1234", "17575551234"); 
 29     check("555-1234", "5551234"); 
 30     check("911", "911"); 
 31   } 
 32  
 33   @Test public void testDots() {
 34     mFormatter = new PhoneFormatter("###.###.####"); 
 35     check("757.555.1234", "7575551234"); 
 36     check("757.555.1234", "17575551234"); 
 37     check("555.1234", "5551234"); 
 38     check("911", "911"); 
 39   } 
 40  
 41   /* Test more formats here */ 
 42  
 43   private void check(String expected, String number) {
 44     checkImpl(expected, number); 
 45     checkImpl(expected, "9" + number); // check leading 9 
 46   } 
 47  
 48   private void checkImpl(String expected, String number) {
 49     PhoneNumber pn = new PhoneNumber(number);
 50     assertEquals(expected, mFormatter.format(pn)); 
 51   } 
 52 }

As you can see the unit test is now a lot longer. And this is for testing a very simple class that takes a very simple input and has a very simple output. Imagine how much uglier this would be if PhoneFormatter took more complex objects as parameters to its methods and returned objects?

Even with such a simple class under test there is a ton of duplication, each unit test method is almost identical to each other. This has the usual problems of cut-and-paste. For example, if you want to add international numbers to your test, you need to make sure you add this to each and every method, and it could be easy to miss one.

There's also the possibility for confusion. Notice the method check (lines 43-46) checks numbers both with and without a leading 9. A future developer, if they don't carefully follow the code, may not expect this, making it unclear why a particular test fails. In this case, it is probably worth this extra indirection to cut the size of the test class almost in half, but you are bound to come up with examples that aren't so clear.

Conclusion

Obviously, the goal to find the happy medium where your test code is structured to minimize redundancy while still maximizing readability, just like writing any other code. However, the goals of unit testing (test component, find bugs, show example of how component is used) are not the same as writing other code. This means that you may make different decisions as part of your design trade-offs than you would elsewhere in code. When combined with the fact that comprehensive unit tests can't be concise (otherwise they wouldn't be comprehensive), it is likely that at least some of your unit tests will be ugly.

Java Web Services Without Code Generation

2011-02-07T09:00:00.001-05:00

With Java 6 it is possible to use annotations to create web services and to create clients to consume those web services without using 3rd party libraries like Axis. If you have control of both the server and the client code you can easily write these pieces without any code generation tools. Here is an example of that.

Server
To create your web service, perform the following steps.

Create a directory for the project. (mkdir server; cd server)
Create a package for your code. (mkdir hello)

Create the interface for the web service. (hello/HelloPort.java)

  1 package hello; 
  2  
  3 import javax.jws.WebService; 
  4  
  5 @WebService 
  6 public interface HelloPort {
  7   String talk(String msg); 
  8 }

Create the web service. (hello/Hello.java)

  1 package hello; 
  2  
  3 import javax.jws.WebService; 
  4 import javax.xml.ws.Endpoint; 
  5  
  6 @WebService 
  7 public class Hello implements HelloPort {
  8   public String talk(String msg) { 
  9     return "Hello, " + msg; 
 10   } 
 11  
 12   public static void main(String[] argv) {
 13     Endpoint.publish("http://localhost:8080/Hello", new Hello()); 
 14   } 
 15 }

Compile your code. (javac hello/Hello.java)
Run the server. (java hello.Hello)

Now that your server is running, you can view the wsdl by going to http://localhost:8080/Hello?wsdl. If you wanted to, you could've skipped the step where you created the HelloPort interface (and had Hello not implement any interface), but we will use HelloPort in the client.

Client
To create a client that consumes this web service:

Create a directory for the project. (mkdir client; cd client)
Create a directory for the Hello interface. (mkdir hello)
Copy the interface. (cp ../server/hello/HelloPort.class hello)

Create the client code (Client.java)

  1 import java.net.URL; 
  2 import javax.xml.namespace.QName; 
  3 import javax.xml.ws.Service; 
  4 import hello.HelloPort; 
  5  
  6 public class Client {
  7  
  8   static class HelloService extends Service {
  9     public HelloService() throws Exception {
 10       super(new URL("http://localhost:8080/Hello?wsdl"), 
 11             new QName("http://hello/", "HelloService")); 
 12     } 
 13     public HelloPort getHelloPort() { 
 14       return super.getPort(new QName("http://hello/", "HelloPort"), 
 15                            HelloPort.class); 
 16     } 
 17   } 
 18    
 19   public static void main(String[] args) throws Exception {    
 20     HelloService hs = new HelloService(); 
 21     HelloPort port = hs.getHelloPort(); 
 22     System.out.printf("ws returns [%s]%n", port.talk(args[0])); 
 23   } 
 24 }

Compile the client (javac Client.java)
Run the program (java Client <message>) or
java -Djava.util.logging.config.file=log.prop Client <message>

And just like that you have written both a client and a server which talk to each other via SOAP. The secondary run line (with the -Djava.util... option) is there in case you don't want to see the logging messages that get printed by default when you run the client.

The URL specified on line 10 is the same URL that you specified on line 13 of the server up above, except with a "?wsdl" parameter. The QName defined on line 11 comes directly from the WSDL. If you look at the wsdl, http://localhost:8080/Hello?wsdl, it'll look something like:

  1 <?xml version="1.0" encoding="UTF-8"?> 
  2 <!--  
  3   Published by JAX-WS RI at http://jax-ws.dev.java.net. RI's version is JAX-WS RI 2.1.6 in JDK 6. 
  4 --> 
  5 <!--  
  6   Generated by JAX-WS RI at http://jax-ws.dev.java.net. RI's version is JAX-WS RI 2.1.6 in JDK 6. 
  7 --> 
  8 <definitions  
  9   xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" 
 10   xmlns:tns="http://hello/" 
 11   xmlns:xsd="http://www.w3.org/2001/XMLSchema" 
 12   xmlns="http://schemas.xmlsoap.org/wsdl/" 
 13   targetNamespace="http://hello/" 
 14   name="HelloService"> 
 15   <!-- service definition here --> 
 16 </definitions

The targetNamespace and the name (lines 13 and 14 above) correspond to the QName used in the client.

Final Thoughts

This example was for a very specific case where you control both the client and server code and want it to be standalone. Of course, your web service needs may not match this model. If you already have a web container (Tomcat, etc), you can always deploy the Hello web service in that, rather than running it as a standalone application. And, of course, you could write your client in any language, as long as it conforms to the WSDL. Or you can just give customers the WSDL and let them write their own client.

If you have to write a client and don't have handy interface classes (hello.HelloPort in the above example), you can do that as well. You'll just have to generate classes using the wsimport command which comes with Java, but that is beyond the scope of this particular blog post.