Semprebon on Software

Ruby Nation Roundup

2011-04-05T18:51:00.000-04:00

It was a cold and rainy Friday morning when I headed out to Reston, Virginia for RubyNation, a local ruby conference. Attending were hundreds of the Washington area’s finest ruby developers, as well as programmers from as far away as India. What follows are what I consider the highlights of the conference.

The opening keynote by Ruby godfather Dave Thomas epitomized one trend I noticed, by not being about Ruby at all. In fact, out of 26 main talks at RubyNation (I didn’t count lightning talk sessions), only 11 were actually about Ruby. Has the rabid enthusiasm for ruby on rails finally abated? At least among the elite developers who actually take the time to attend conferences, this seems to be the case. There was speculation in the hallways as to which big-name ruby guru would be the first to break ranks. Talks by Aaron Bedra on Clojure, Jerry Cheung on Node,js, and Blake Mizerany on, well, just about everything else, highlighted the advantages of avoiding ruby for tasks it isn’t particularly suited for: low-latency services, concurrency, analytical analysis, and more. Even Joe O’Brien’s talk on Ruby on Android was more about ruby’s failure as a viable Android programming language in comparison to the ruby-like Mirah programming language.

Of course, if you’re going to go all polyglot on your project, you need a way for those languages to intercommunicate. The preferred lingua franca, at least for lower-level integration, seemed to be the Java Virtual Machine, with jRuby getting several mentions and one full talk by Russ Olson. Ben Scofield’s lecture on “Thinking Small”, on the other hand, dealt with integrating at a larger scale - in particular, with the value of creating a large system from many smaller, more focused services, communicating via databases and HTTP/REST. You know, we should really be doing more of that here at GeoIQ.

Second only to Ruby in conference air time, Javascript was well represented. In a pirate-themed talk complete with hats, patches and the shouting of “Arrrg!”, Chris Williams covered the history of JavaScript, from its humble and rushed beginnings to its current effort at world dominance, ending with his call for google-bombing to promote good JS documentation. The previously mentioned talk on Node.js covered the growing use of JS on the server, while Trotter Cachion’s talk on doing graphics in Raphaël covered the client side. My favorite javascript alternative, coffeescript, seems to be getting some traction as well, with several mentions and one Lightning Talk.

Avdi Grimm gave an excellent talk on writing “Confident Code” which clearly spelled out how to make ruby more readable by structuring methods in 4 parts: Input,Work, Results, Errors. He also mention a number of neat techniques, like using Null Objects to eliminate nil checks, that I was not familiar with.

As usual, there were a sprinkling of talks on non-programing topics - David Keener’s on business models (money for code!), Keavy McMinn’s talk on training for an ironman triathlon (You think you have it bad?), and Dr. Nic Williams closing keynote on ruby user groups (We need to talk) were all more or less successfully pitched as relevant to the ruby developer.

One reason I like going to conferences like RubyNation is to learn about new tools and even old tools that somehow escaped my notice. Some of the those that got my particular attention were Mirah (static ruby for jvm), Protocol Buffers (cross-platform binary serialization), Sass/Compass (programmable css), Rails Installer (easy Rails for Windows), and Raphaël (JS alternative to Flash or Canvas)

Of course, the main reason for a conference is to hang out with like-minded folks for stimulating conversations, and RubyNation didn’t disappoint, whether it was discussing the pros and cons of cross platform applications (Mac/iPhone users are picky, Linux users are happy to have any decent UI) or the best places to live outside the United States (Chile?). One of the great things about these smaller local programming conferences is that you end up knowing a lot more of the people. In all, a successful conference which can be summarized in one word: Arggg!.

Javascript, RIAs and the Future of Software Development

2011-02-24T18:28:00.002-05:00

In 5 years, 90% of new code will be written in or compiled into javascript

A bold statement, no doubt, but the trend is clear. In a few years, "web apps" will be developed as simple static HTML/CSS with business logic written in javascript, supported by a database server serving up JSON. Mobile apps will be javascript compiled to native code or run in-browser with native add-on code only as needed. This can already be done, and greatly simplifies the development of cloud-based applications that cab be accessed by web, desktop app, or mobile app. And in a year or two, if your mobile app can't be accessed on the web, or your web app can't be accessed on a mobile device, your users are going to go elsewhere.

The Past

To understand where we are going, it helps to look at where we came from. Back in the 80's, client server was all the rage. Rich client applications contained all your business logic, and accessed a remote back-end database for persistence and interconnectivity. This worked well, except that distributing the client was a pain in the butt. Within an enterprise, this could be semi-controlled with the appropriate software, but for the general public, it only worked for well-established and fairly static protocols like POP, HTTP, etc.

In the late 90s, business found they could eliminate the distribution problem by moving to web applications. Unfortunately, these had limited functionality. Web forms were functionally equivalent to gussied up 3270 screens. In addition to the poor UI, web sites also required entire pages to be built repeatedly on the server, resulting in higher bandwidth and more load, while increasingly more powerful PC clients sat mostly idle.

In the mid 2000s, we got Dynamic HTML, which once it was rebranded as AJAX, quickly became a popular way to add more functionality to traditional web applications. As AJAX became more prominent, we had to essentially implement two apps in one - one feeding html to the user's browser, and another feeding JSON to the javascript app.

Later in the decade, web APIs became popular, allowing web applications to be integrated with other applications (web or otherwise). If we were extremely lucky and forward-looking, these APIs just used our existing AJAX interface, but more typically, we had to create yet another http interface in our increasingly overburdened web application.

So in web development as it stands today, we end up writing server code to generate HTML pages; and more server code to generate AJAX JSON; and maybe more server code to support our API, And if we want to support mobile, its another app for each mobile OS.

This is absurd.

The Future

The solution is clear. Write the entire app - all the business logic, in javascript or its equivalent. This code should access a RESTful database server that accepts and serves up JSON, and can handle security and validation. You will also need a web server to do what web servers do best - serve up static pages (.js, .html, .css, .png, etc). The javascript will then fetch dynamically fetch non-static data from the database server and update the static page. This is basically the old rich client from the 80s, but implemented in javascript, and using the Internet for distribution. Our servers now become remarkably simpler. And since we are now using our own API, there is no extra work implementing an API for external consumption. In fact, our app is basically on equal footing to those external apps that use our APIs.

There are already frameworks like SproutCore for writing these types of applications, and I expect to see more. What we really need is better support for a JSON-serving database that allows custom validation (triggers implemented in javascript would be ideal) and security. Until then, the back-end will need to be coded using conventional methods.

There are, of course, some criticisms of this approach:

Isn't implementing client side javascript insecure?

If you are going to implement a full and rich external API anyway, the security has got to be in the JSON server anyway. Because the JSON server is much simpler than the old-school dynamically generated html server, security should be easier.

Yuck, I have to write all my code in Javascript?

Javescript is a semantically powerful object oriented language, saddled with a poor syntax. Fortunately, there are languages like coffeescript that can be compiled to javascript and offer better syntax. I expect this to be a growing trend, with better languages in the future, and maybe even precompiling into an abstract syntax tree that is sent to the browser.

What about mobile development?

Frameworks like Appcelerator and Phonegap allow you to create mobile applications for a variety of mobile OSs from javascript code, so much less work is needed compared to developing a fully native app for each mobile OS. While these environments don't give you the same level of control and access to features as native apps, I expect that this will be less a problem as these platforms mature. And you always have the option of enhancing your javascript-based app with native code where its really needed

Where does flash fit in?

Flash is basically the browser equivalent to native app code - used only as necessary, and increasingly replaced by javascript functionality.

So that's where I think web development is going. Now all we need is a catchy name...

Playing With CoffeeScript and HTML5

2011-01-04T20:51:00.003-05:00

I spent today playing around with CoffeeScript, a nice little pythonesque language that is compiled into javascript. Very nice. This was for a little mini wiki app I'm working on as a learning exercise. The idea originally was to have a Scala-based server side running on Google App Engine that acted as a simple RESTful JSON database. All the processing is done in the browser with javascript (i.e., CoffeeScript), using HTML5 storage for off-line availability with eventual synchronization when the server becomes available.

Unfortunately, I ended up abandoning the idea of using scala for the backend part - doing Java (i.e., Scala) development on Google App Engine turned out to be rather slow and tedious, partially due to broken (or at least not well documented) APIs, but mostly due to tools that didn't quite work together. I did get it to work, barely, but after a day and a half of frustration, I switched to Python and got the whole server side part done in a couple of hours. Much easier and simpler.

Javascript Idioms - Namespaces

2010-02-28T02:53:00.003-05:00

This is brain-dead simple. In order to avoid sticking everything in the public namespace, we just create an object to hold it, and add our name-spaced objects to it:


var mystuff;
if (mystuff === undefined) { mystuff = {}; }

mystuff.foo = function() { return "Hello!"; };

mystuff.foo(); # => "Hello!"

I define it this way so that I can add things to the namespace in multiple files without having to worry about what's included or which is first. For more complex module-like behavior with private variables and such, see my previous post.

Javascript Idioms - Private Attributes and Modules

2010-02-28T02:30:00.002-05:00

Ok, I admit my Javascript skills are a bit lacking, and I've recently been doing more of it, so after watching Douglas Crockford talk about the good parts, and reading up a bit, I think I now understand some of the stuff going on those javascript libraries. So there are more or less notes to myself so I don't forget.

Closures

So to really understand what's going on, you have to know a little bit about closures. Now some people seem to be under the impression that a closure is just an anonymous function, but its a little more complex - its an an anonymous function that has access to the context in which it was defined. The reason that last bit is important is that it lets you maintain state outside the function itself. For example:


var x=5
function foo() { ++x; return x; }
foo(); // returns 6
foo(); // returns 7

You can even share state between funtions:


function bar() {--x; return x; }
bar();  // returns 6
bar(); // returns 5

Dynamic Contexts for Closures

In this case, the context is just a global variable, so this probably doesn't look too impressive, but we can actually create some context dynamically using a "self-invoking function' (which strikes me as a bit of a misnomer, as the function isn't invoking itself, its returning a function that is then invoked. Anyway, in simplest (and most useless) form, it looks like this:


(function() { /* stuff */ }) ()

So why would we do that? Well, in that little region I've put the "stuff" comment is actually a context where we can define variables and functions. Now, ordinarily, all that context would disappear once the function completes running, but if we define another function in there and make it available outside our bit of code, it actually has access to the "stuff" (since its really a closure) even after the function returns. This allows us to create private data and functions that are only accessible by the "public" functions we make available. For example, we can define a function that can access some private data:


var normalize = function {
  var DELIM = /\W+/
  return function(s) { s.split(DELIM).join('_'); };
} ();

In this case, we can access the internally defined function because its returned by our context-creating anonymous function, but we could also assign it to some other object in the global scope, like window. The internally defined DELIM is private to the function.

The Revealing Module

So now we can use the closure to create a module that has private and public functions and variables. There are a couple of ways of structuring the code, but I think the Revealing Module Pattern is one of the better ones I've seen:


var foo = (function() {
  var DELIM = /\W+/
  function normalize(s) { s.split(DELIM).join('_'); };
  function capitalize(s) { toUpperCase(s.charAt(0)) + s.slice(1); }
  function mixed_case(s) {
    var arr = s.split(DELIM);
    for (var i = 0; i < s.length; ++i) { arr[i] = capitalize(arr[i]); }
    return arr.join();
  }

  // constructor should return object with public parts
  return { normalize: normalize, mixed_case: mixed_case }; }
}();

// how to use it:
foo.mixedCase('hello world!'); # => 'Hello World!'
foo.normalize('Hello world!); # => 'Hello_world'

So what this is doing is creating a private context, defining some variables and functions in that private context, then returning an object with just the parts (in this case, the normalize and mixed_case functions) that we want to have as the public api. These functions are then assigned to the public variable foo.

DroidDice open sourced and now on github

2009-12-18T05:53:00.002-05:00

I've open-sourced my DroidDice android application under the GPL. It is available at GutHub

Seting Custom Headers in ActiveResource Requests

2009-06-04T09:53:00.003-04:00

Recently had to figure out how to send a custom http header when accessing a resource through ActiveResource. Basically, you need to add the header to the connections default header hash. In your active resource class:


class ActiveResource::Connection
  alias :static_default_header :default_header

  def set_header(key, value)
    default_header.update(key => value)
  end
end

class FunkyResource < ActiveResource::Base
  ...
  connection.set_header('SHOE_SIZE', '12')

end

Now all HTTP requests for that resource will include the HTTP header SHOE_SIZE.

Taming Shoulda

2009-02-22T07:31:00.003-05:00

Ok, so I've been using shoulda for the last couple of weeks, and while its certainly reduced the amount of duplicated code in my test, and given me more meaningful test descriptions, I've also noticed that its a little hard to find things with all the indented levels of contexts, setups, and shouldas. And I find that I still end up with duplicated setup and testing code. Fortunately, this is fairly easy to address by writing your own shoulds and context methods.

Custom shoulda's

Lets look first at writing your own shoulda method. Lets say you are testing a method that parses some file and returns the extracted data, along with a lit of any errors it found:


result, errors = MyClass.parse(file)

We want to test this with a number of different test files, so we end up with tests looking like this:


context "when parsing abc.tst" do
 setup do
   @result, @errors = MyClass.parse("abc.tst")
 end
 should "have no errors" do
   assert @errors.empty?
 end
 should "have some expected result" do
   ....
 end
end

So a number of our tests is going to be checking that @errors is empty. While its only a few lines, it does add some visual clutter, and is easy to clean up with our own shoulda method:


def self.should_have_no_errors
 should "have no errors" do
   assert @errors.empty?
  end
end

This needs to be before your tests are defined, since it is a class method which will be executed during the loading of the class. Now, we can replace three lines with one:


context "when parsing abc.tst" do
 setup do
   @result, @errors = MyClass.parse("abc.tst")
 end
 should_have_no_errors
 should "have some expected result" do
   ....
 end
end

Our shoulda methods can also take parameters, so if we wanted to test for the presence of errors, we could add a should_have_error method:


def self.should_have_error(regex)
 should "have error matching #{regex.to_s}" do
   assert_match regex, @errors.first
  end
end

which we would use with


should_have_error /Invalid Attribute/

Custom Contexts

The next thing we notice is that a lot of our tests are going to have similar setup sections, but we can't use a common context because the file names are different. To address this, we can create a custom context:


def self.when_parsing(filename, &block)
 context "when parsing #{filename}" do
   setup do
     @result, @errors = MyClass.parse(filename)
   end
   block.bind(self).call
end

Now, our tests look like this:


when_parsing("abc.tst") do
 should_have_no_errors
 should "have some expected result" do
   ...
 end
end

I find this particularly convenient for handling logins when testing rails functional tests:


def self.logged_in_as(sym, &block)
 context "logged in as #{sym.to_s}" do
   login_as(sym)
 end
 block.bind(self).call
end

logged_in_as(:quentin) do
 [test normal user stuff]
end

logged_in_as(:admin) do
 [test admin access]
end

Shoulda

2009-02-05T18:27:00.004-05:00

Recently, I've been using shoulda for improving my test code. There are a number of things I like about shoulda:

Unlike rspec, shoulda works with your existing tests
It lets you separate out common set-up and tear-down code that isn't needed for every test
Adds some organization to your test files, making it easier to find things
Allows you to describe test cases in real English, insteadOfMixedCaseIdentifiers

When I first started using shoulda, I organized my tests the same way I had in my pre-shoulda days - be method tested, so I might have:


class StackTest
 context "Stack" do
   setup { @stack = Stack.new }
 context ".pop" do
   should "Return nil if stack empty" do ...
   should "Return top of stack" do...
   ...
 end
end

However, I have come to the conclusion that a better approach is to organize by the state of the object under test, since this will allow you consolidate your setup/teardone code more effectively, and also force you to think in terms of the object rather than the individual methods:


class StackTest
 context "An empty stack" do
   setup { @stack = Stack.new }
   should "Return nil on pop" do
   should "Have one item after a push" do ...
   should "return top of stack" do...
   ...
 end
end

There are a number of other things that make shoulda tests more maintainable:

Don't nest contexts too deeply, as it makes it harder to find things
Don't just throw tests together in the same context because they happen to share some setup code - think in terms of the state of the object or class being tested
You can use contexts without setup to organize - like having a context for grouping together class methods
Write your own shoulda clauses for commonly tested conditions
You can make it easier to parse output into contexts/shouldas by capitalizing each description

Android Applications

2008-11-28T13:13:00.004-05:00

So I've spent the last month writing a few of applications for the Android phone. I got a TMobile G1 a couple of months ago, and I have to say, its very neat to program for.

This means that, with a few exceptions, there will probably be very little money in writing Android applications. As we have seen with the iPhone, expect a flood of free/cheap apps drive user expectations as to what an Android app should cost. The lower barrier to entry for Android development as compared with the iPhone means that developers will flock to the platform, so there will be plenty of competition driving prices down, as well as open source efforts.

There may still be a few ways to make some decent money as an Android developer, but none are particularly easy:

Make up for low price by being very popular - yeah, much easier said than done.
Provide a combined web/android service that people are willing to pay a subscription for - of course, you'll still be competing with a lot of low price/free offerings, but at least you'll be getting a steady revenue stream from your users.
Develop custom enterprise applications - not particularly fun, but it pays the rent
Get someone else other than the user to foot the bill, advertising being the most obvious way to do this.

Still, I plan to write some more apps. For one thing, its a lot of fun. For another, I'm not impressed by very many of the Android programs I've tried out so far. More about that later.

2008-11-09T22:16:00.002-05:00

So I just got myself a T-Mobile G1 Android phone. I don't know how it will fare against the iPhone, which in many ways is better designed (but alas, not open source), but I'm guessing it will do well against its other competitors.

Android

2008-10-23T11:03:00.002-04:00

I started playing with Google's Android SDK today - long enough to implement a simple "Hello World" app. Unfortunately, between the android emulator, firefox, and NetBeans, my four-year-old laptop is struggling to keep up. While android in not as popular as the iPhone (and may never be), writing in Java has got to be better than working in Objective-C - an ugly languange from what I've seen, at least syntactically. And with android, there is at least some hope that other languages would be available - although the only success I've heard of is with scala.

Another posibility, is, of course, javascript, especially once Google ports Google Gears to Android, although the only problem there is that it won't neccessarily have acess to all the phone's features.

2008-10-13T08:33:00.000-04:00

The consensus at Ruby DCamp this weekend was that we should be using gems rather than plugins for sharing rails code (of course, not everyone agrees. This was primarily because gems are versioned and handle dependencies automatically. This being the case, why are so many rails enhancements packaged as only plugins? Couldn't we do everything as gems?

I think a lot of people view plugins as specific to a specific rails application while gems apply to ruby as a whole. I guess a lot of people don't know how easy it is to set up a separate gem repository for your application. With plugins, you can also easily identify which plugins an application uses. A new change to rails, gem dependencies, makes this even easier, and allows you to specify what gems your application depends on, and even gives you a task to install any required gems.

Writing Good Rails Tests

2008-09-16T07:07:00.009-04:00

Some automated testing strategies I've seen implemented on various projects:

The Nil Strategy

I see this all too often. It is exemplified by the Rails test class below:


require File.dirname(__FILE__) + '/../test_helper'

  class RecommendationTest

  # Replace this with your real tests.
  def test_truth
    assert true
  end

end

Enough said. You know who you are.

One Test Per Method

I used to see this done all the time on java projects. For each method in the class you wan to test, you have a single test method. This simplistic approach may work for fairly simple methods, but for a method of any complexity, they are nearly always inadequate. They just test the most obvious test case for the method in question.


def test_add
  set = OrderedSet.new
  set.add("X")
  set.add("Y")
  set.add("X")
  assert_equal ["X", "Y"], set.to_a
end

Now obviously, you COULD put all sorts of tests in that single test method:


def test_add
  set = OrderedSet.new
  // adding to an empty set
  set.add("X")
  assert_true set.include? "X"
  assert_equal 1, set.size

  // adding item to set that doesn't contain it
  set.add("Y")
  assert_true set.include? "Y"
  assert_equal 2, set.size

  // adding item to set that does contain it
  set.add("X")
  assert_equal 2, set.size

  // ensure that added items preserve order
  100.times do { |i| set.add(i.to_s) }
  assert_equal set.to_a.sort, set.to_a
end

But now its harder to determine what exactly we are testing for. Its not too bad for this method, but its still a fairly simple method. What we really need to do is refactor the test into several tests, but we can't do that if we cling to OTPM. Which leads to

One Test Per Assertion

The way I usually test is to have a method for each test condition I'm trying to test for. By convention, I name this method in a form like


def test_add_should_not_add_item_if_its_in_set
  set = OrderedSet.new
  set.add("X")
  set.add("X")
  assert_true set.include? "X"
  assert_equal 1, set.size
end

The method name includes both the name of the method I'm testing and what condition is being tested. The test itself sets up, calls the method, and checks a single fact about what the method under test should do.

Note that this may not literally be a single assertion. Sometimes I use several assertions to verify a single fact. I also sometimes use assertions to verify my assumptions about the setup conditions - this is particularly useful in Rails to verify some assumption about your test fixtures before you use them:


def test_member_in_good_standing_gets_discount
  assert_true @user.in_good_standing?
  ...
end

The main reason to do this is not to let you know if you inadvertently screwed up your fixture (although it will do that), but to serve as documentation of the assumptions the test is making if it isn't obvious elsewhere in the code.

Rails Unit Testing Without Fixtures

2007-08-16T19:57:00.001-04:00

When testing models with complex relationships, fixtures can be a real pain. They require the details of a single test to be spread around several files; they aren't very dry, since you have to repeat a lot of details in each instance that may not be relevant to the situation you are trying to set up, and you are forced into putting a bunch of for unrelated tests into a single file.

The Test With Fixtures

Lets take the classic example of an Order item, that has_one product. If you are testing to ensure that the correct total is computed. With fixtures, it might look like:


def test_total_should_return_price_times_quantity
assert_equal 33.00, order_items(:first).total
end

To see that this is valid, you would have to look in order_items.yml to find out the quantity for item :first is 3, then look in products.yml to find that the price is 11.00.

There have been a number of alternatives to fixtures proposed, including Jay Fields' idea of using mocks to avoid database access, which seems like going a bit too far for my tastes; and fixture scenarios, which only solve part of the problem. Here's what I do:

The Test Without Fixtures

First, I create a build method for each of my model objects. This method will create a valid object with reasonable default values, except as modified by a hash parameter. For simple objects with no associations, it looks like this:


def build_product(attrs = {})
Product.new({
  :name => "Test Product",
  :price => 10.00,
}.merge(attrs))
end

For more complex objects, the builder should create any reasonable associated objects or use those past in:


def build_order_item(attrs = {})
OrderItem.new({
  :quantity => 1,
  :product => build_product,
}.merge(attrs))
end

These builder methods I usually put in test_helper.rb, since they ussualy need to be called from multiple test files. With these in place, out total test becomes:


def test_total_should_return_price_times_quantity
order_item = build_order_item(
  :quantity => 3,
  :product => build_product(:price => 11))
assert_equal 33.00, order_item.total
end

This test doesn't touch the database, and all the details relevant to what is begin tested are in one place.

Unproductive Haiku

2007-04-17T11:45:00.000-04:00

No work done today.
The parts in which it's been cut
Are less than the whole.

2007-04-16T15:21:00.000-04:00

My contract at a large government agency recently came to an end, and while I've been working on a project of my own which will hopefully be in beta soon, I've also been looking for more regular work. In the process, I've noticed an interesting dichotomy.

Most of the companies I've talked to have been unimpressive. Although their websites claim that they are passionate about quality, enhancing the customer experience, and hiring only the best and brightest, that's not the impression you get from talking to them. Their ads on one of the big job sites have a long list of technologies you are expected to have mastered. They do little to sell you on why you should be working for them and not one of the other body shops. The recruiter gives the impression they are just trying to find the right peg for the staffing hole in some large bureaucracy.

Two companies so far have stood out. Interestingly enough, I didn't hear about any of them from a job board. One, a group at Booz Allen, I heard about through a friend. Sure, they do typical government work, but the developers I talked to were all really pretty sharp. The other was at BrowserMedia, and although the location is not quite ideal (but doable), they sound like they do a lot of interesting work and really made me want to work there. I had heard about them from a post on the local ruby users group, but I notice they also have an ad on Joel on Software's job site, so I'm guessing their's going to be stiff competition.

I think I'm going to have to start paying more attention to the more focused job lists like Joel's.

Managing User Attention

2007-04-03T14:31:00.000-04:00

One feature of the web application I helped developed for a client was a list of messages that was displayed when you first logged in. This feature had been in the system since day one, and included both system-wide messages entered by management, as well as messages auto-generated by the system. I wasn't originally responsible for the message feature, but when I first saw it, I estimated that, since most of the messages were not actionable, it would only be a matter of weeks before employees became habituated to the messages and ignored them. Sure enough, that is exactly what happened.

Although most of the messages were fairly irrelevant, roughly twice a year, management would put out an important messages that everyone was supposed to read. Once the managers realized that users were ignoring the message, they requested the feature be changed to make it more prominent. First, my coworker moved it to a separate page you had to click through before you got to the main page of the application. And of course, after a while, users became habituated to that as well. Then we started adding extra code to highlight the most important messages on the login page, which required the source that generated that page to be edited and redeployed each time the message was changed. Then we had to put them in bright colors. Almost a year ago when we got another request to change the login page message, and I decided to put an end to this attention arms race.

I modified the original message scheme to show messages in large, red letters or the first few weeks, then revert to the normal font. And I got rid of the other places we were displaying "special" messages. This actually went well beyond the user's original request, but in the change request form, I carefully documented what I was doing, and why. The system-wide message would be displayed prominently, and we wouldn't need to redeploy the application each time the message was changed.

Some time after putting that change in, I get a system change request asking us to modify the message. "We don't need a change request for this" I replied, "it isn't a software change - they can do it themselves now". I was told it had to go through the CR process anyway. After a bit of questioning, it comes out - apparently, management doesn't actually read the change requests that they routinely approve, and had no idea of what I had done. They too had become habituated.

Cleaning Up Code

2007-03-10T09:21:00.001-05:00

It was a single, 6000 line module, and it was a mess. The original developer had been gone for 6 months, and it had gone through several hands since then. It was constantly having problems, with new bugs discovered weekly, but nobody wanted to touch it because every time someone did, they would break something else. It landed on my desk with several as yet unfixed bug reports attached to it.

After looking through it I came to the conclusion that there was a relatively simple program buried inside, and went to work. There were no unit tests, but the main function of the program was to generate a number of output files. I could compare the correctness of my revised version by running both it and the original on the same data, and then comparing the output files.

I deleted redundant and unused code, refactored, and generally cleaned up the program. After untangling a particularly convoluted bit of logic, my comparison test failed. After half a day of research, I came to the conclusion that in this particular case, the original program had been in error. Two weeks later, I checked in a program that was 30% smaller, better documented, and much easier to follow.

The supervisor was a bit suspicious of this. His philosophy was that we should be changing just the bare minimum needed to implement a fix, since previous changes had a high probability of introducing new bugs. After the program passed the first round of verification using a single run (about 8000 records) with flying colors, he insisted on testing a whole two week cycle. Since I had been testing with several months worth of data, I wasn't particularly worried. When it passed that test, he finally authorized it to be moved into production.

Unfortunately, I never got a chance to finish cleaning up the code, since there were no additional bug reports in the six months that followed.

Feedback

2007-03-09T11:08:00.000-05:00

I suppose we've all been in this situation. We have done something (revised our resume, changed hairstyles, whatever) and we ask a friend how it looks. "looks fine" we hear, or maybe they make some minor criticism - "you have a mispelling in the third sentance". The fact is, the people we deal with on a daily basis want to be nice to us, and are therefor reluctant to criticise or suggest improvements, even when that criticism is what we really need to do better.

Getting good feedback is hard, and yet it is extremely important to any effort to improve, whether improving your image or your software development process. And yet, so many organizations not only don't encourage feedback, they actively discourage it. This can be done in both subtle, and unsubtle ways.

One manager I know would dominate the conversation during the status meetings he held. He would prompt his underlings with specific questions about their work, then quickly move on. While this did help to keep the amount of wasted time down, it meant that issues that were not on the manager's agenda never got aired at all.

Now it is certainly true that developers with some problem to be addressed could have sent an email, or otherwise brought it to the managers attention, but "process" is one of those things the project manager is supposed to be in charge of this requires the developers to both recognize that there is a problem that can be solved or an improvement that can be made, and take the initiative to raise it. This is one of the reasons I think the sadly underused "retrospective" meeting can be so effective. It creates a forum specifically for addressing project wide and process issues that are often ignored in the heat of development.

Sometimes, even when issues are raised, they are repeatedly ignored by management, either by being put off "for later" or by not responding at all. Management in this case is sending a clear message that feedback is not wanted.

While not encouraging feedback can be a problem, even worse is discouraging it. At one organization I know of, a friend of mine once suggested some improvements to the current development process, and was told in no uncertain terms that his role was to accept the process as handed down by management, not to suggest improvements, and that if he continued with this "insubordination", he would be terminated. Needless to say, this organization was inefficient, had a lot of communications problems, and morale was poor to boot. Managers would complain that they had a hard time keeping track of the status of things. Hardly surprising if developers were fearful of giving any more information than what they had been specifically asked for.

Remember, everyone can have good ideas, and so its important to capture those ideas. Feedback is vital to any effort to improve. Encourage it.

rspec

2006-12-15T15:04:00.000-05:00

After hearing Aslak Hellesoy talk about rspec on the rails podcast I figured I would give it a try. Its a neat way of seeing test driven development from a different perspective.

First, I tried creating a new spec for an existing model using the rspec_model generator, but this hung up (maybe because of some configuration issues on my XP box).

Then I got the following error when running a test:

c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/errors.rb:94:in `check': SQL logic error or missing database (SQLite3::SQLException)
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/resultset.rb:76:in `check'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/resultset.rb:68:in `commence'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/resultset.rb:61:in `initialize'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/statement.rb:158:in `execute'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/database.rb:211:in `execute'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/database.rb:186:in `prepare'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/database.rb:210:in `execute'
from c:/ruby/lib/ruby/gems/1.8/gems/sqlite3-ruby-1.1.0-mswin32/lib/sqlite3/database.rb:620:in `rollback'
... 11 levels...
from c:/ruby/lib/ruby/gems/1.8/gems/rspec-0.7.4/lib/spec/runner/context_runner.rb:22:in `run'
from c:/ruby/lib/ruby/gems/1.8/gems/rspec-0.7.4/lib/spec/runner/command_line.rb:26:in `run'
from c:/ruby/lib/ruby/gems/1.8/gems/rspec-0.7.4/bin/spec:4
from c:/ruby/bin/spec:18

The problem was apparently related to using embedded ruby expressions which used the model I had previously tried to create a spec for. The generator had overwritten my old model with a new blank model (I though generators were supposed to prompt before overwriting files), so the embedded ruby expression was not valid. Anyway, thinks seem to be working now.

Rails Woes

2006-09-19T19:54:00.000-04:00

I read somewhere (yeah, I know, I should have a link, but I forget) that to have a really effective, widely read blog, you need to blog every day. So here I am. The problem I have is that, during the day, I'm always coming up with great ideas for something to blog about, but by the time I actually have to sit down and write something, I forget what I was so worked up about.

Anyway, my friend George, recently forcibly retired, was asking me some questions about PHP, and so I pointed out to him that all the cool geeks now use Ruby on Rails (and not just because I don't know diddly about PHP), and he ought to give that a try. So he's been trying to get it installed, and he's even forsaken his corporate Windows security blanket and is actually trying to do this on Linux, and he's been trying for two weeks, and still hasn't gotten past the install yet. Despite the fact that he's been a manager for the last few years, George is a real smart guy, and if he can't get this thing working, then its TOO DAMNED HARD!

Unfortunately, I've been of limited help here. One of his main problems has been with MySQL, and for quick toy databases I use SqlLite (zero configuration and I can check the entire database file into subversion) and if I need a real database than, as a long time Oracle expert, I use PostgreSQL. He's also running on Fedora, and I tried to explain that all the cool geeks are running Ubuntu these days (except for those who have gone to the dark side and embraced OS X, but that just wasn't an option for a guy who has two dozen old Windows XP boxes of various vintages in his garage). Well, it turns out that a former employee is also using Fedora and swears by it, and ol' George wants to stay compatible, so that's fine. But between Yum and RPM and Gem something is just not right (and thank god he didn't ask me why Linux needed three different package managers to install a simple application - I kept my mouth shut about apt and aptitude...)

To be quite honest, a lot of this is just growing pains. In a year, all the top Linux distributions will have their own Rails package and he'll be able to type install rails and be there, but in the mean time, ITS A PAIN IN THE BUTT! And its not like my choice of SqlLite is that much better - I had to go through the dance of getting just the right packages installed so the gem could compile too. But I guess I knew a little bit better what to do, and as a long time Linux user, I knew how to use Google for my tech support. George, having recently come from a windows background, didn't quite have the hang of that.

If he calls tomorrow, I'll suggest he just compile from scratch. Or buy a Mac. Or use Windows.

Striving Forward, Dragged Back

2006-08-16T10:28:00.000-04:00

A Tragedy in Three Acts

Act I - Java

The existing system I helped develop and currently maintain (as I write this in the Summer of 2002) is a Java/J2EE based web application that supports around 16,000 users. The Organization I work for hasn't gotten the Agile message yet, so its been up to individual developers to use agile techniques or not. The proprietary development tool they chose precludes writing automated unit tests, but I've got a good collection of automated functional tests now. The tool-generated code is highly coupled, but I've been refactoring where I can. We've been using CVS for version control since we first started developing the system five years ago. The Java culture values things like automated testing, refactoring, and version control. Not all our tools were perfect, but the Java developers on our project at least understood the reasoning behind agile development.

The system has very few reported bugs. I don't get many enhancement suggestions from users, but not because its perfect, or even particularly good. The Organization had gotten very good at saying "No" to users. If the response is always "No", you give up asking.

So for a while, I found myself with not much to do. Every month or so, The Organization would mandate some change. I'd spend a few hours or days implementing it. The help desk still got plenty of calls from confused users about how to use the system. To me, these were usability bugs; the valuable feedback I needed to keep improving the system. But The Organization discouraged dangerous things like user input. The help desk calls were summarily dismissed as "enhancements" or "user training issues". The system met its requirement. No software change was required.

Instead, I worked on honing my software development skills and improving what I could that wasn't part of the system proper. I automated (to the extent possible) the migration and deployment process. I wrote additional functional tests. I taught myself ruby and started doing some data conversion, analysis, and ad-hoc reporting with it. I found the Ruby culture much like the Java culture, but with more focus on simplicity and elegance. And Ruby tools were certainly better at agile development than Java tools. Work life wasn't perfect, but it wasn't so bad.

Act II - PeopleSoft

The Organization has another big system - PeopleSoft. It doesn't support as many users, but it seems to have a lot more change requests and bugs. The Organization completed a big system change over a year ago, and a lot of the changes these days are fallout from that. It was supposed to simply things by out-sourcing the payroll functionality, but the PeopleSoft team is busier than ever. And with all the bug fixes and contractually or legally mandated changes, there just isn't time to make any enhancements to that system. This is why The Organization have gotten very good at saying "No" to users. Its an effective way to manage the schedule when you are swamped with changes.

Besides, The Organization plans to throw the system out in a few years anyway, and out-source the rest of the PeopleSoft functionality. They don't want to invest any more than they have to on the PeopleSoft system.

Many years ago, I too had been a PeopleSoft developer. It was back in the days when the web was in its infancy. Companies were tossing out their old main-frame systems and buying newfangled client-server applications like PeopleSoft. There was plenty of contractor money to be made for those with "PeopleSoft" on their resume, and for several years, I rode that gravy train. But it wore me down -- the proprietary, non-object-oriented languages; the existing code base of poorly structured code; the culture of ad-hoc testing; the problems of deploying client software on dozens, if not hundreds of desktops. It was obvious to me that the future was the web. I had to leave my job and take a significant pay cut in order to get into web development (with Pearl at first, then later Java), but I was a lot happier, and I never regretted taking that step.

Act III - The Tradgedy

Despite my efforts to keep it secret, The Organization eventually found out about my previous PeopleSoft experience. It was little things at first - helping someone with a bit of SQL; or making a minor change to an SQR. I can hardly blame them - a resource sitting idle, and them having so much PeopleSoft work to do.

Now, most of what I do here at The Organization is PeopleSoft. Eight years, and I'm back to where I started. The PeopleSoft culture hasn't changed much in that time. They don't use automated tests or use version control effectively. The culture is "change as few lines as possible", rather than refactoring, and "cut-and-paste" rather than "Don't Repeat Yourself". Massive change take months to implement. The tools have changed little, or not at all. The Organization is running an obsolete version of PeopleSoft, because they don't see the need to invest in an upgrade.

Instead of pushing my skills to the next level, I have to struggle just to make them relevent.

Epilogue

Its clearly time for me to move on. I have lived in the Java culture too long to tolerate the culture and tools of the PeopleSoft team. Unfortunately, that culture is way too common. I struggled with it when we first started our Java development. Deciding to leave is easy. Finding a new job where they really believe in the culture of agile development is less so. I had a phone interview with one place where a friend worked. They claimed to be doing XP, but were working 80 hour weeks, lied to the client about what they were actually working on, and had a massive list of requirements that they were trying to shoe-horn into their 6 month scheduled delivery date. Didn't sound too agile to me.

In the mean time, at least The Organization is paying me well.

Oracle 10g Express up on Debian

2006-05-17T16:10:00.000-04:00

After a bit of work, I've got Oracle's new 10g Express database up on my debian server! Now I can start playing around with some rails applications that use Oracle on the back end. The main difficulty is that Oracle (as with their previous versions) always seems to assume that you are running X Windows on your server box. In the case of 10g Express, they provide a web application for administration, but it requires a graphical browser and can only be accessed from the local machine. Fortunately, there was a way to open it up so I could access it from my workstation:

$ su oracle
$ cd /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin
$ ./oracle_enc.sh
$ ./sqlplus / as sysdba
sql> exec dbms_xdb.setlistenerlocalaccess(false);
sql> exit
$

When I tried Rails with Oracle several months ago, I found Rails' oracle support to be a bit lacking. The SQL it generated was slow, and not very flexable. Apparently, that situation has improved somewhat, so I'd like to try it out.

In the Belly of the Beast

2006-03-10T11:04:00.000-05:00

I can almost feel the enthusiasm being crushed from me. Some people think I have a great job - I don't have much work to do, and the pay is really great. But its so friggin boring. Developers are stuffed two at a time in grey featureless cubes. The organization is so risk-averse any new changes are difficult to get approved. The development process in place requires that even a minor 2o minute fix requires a two week turn-around time when it does get approved. All the software we work with is years out of date, and they end up paying license fees for software the vendor no longer even supports.

Wow

2006-03-03T11:09:00.000-05:00

I thought Steve Yegge's article on Ruby was a pretty amazing piece of writing.

The Annual Blog Post

2006-03-03T11:05:00.000-05:00

Hmm, only one post in 2005. Well, that's because I was hosting a blog on my corporate website, www.eqsystems.com, but I've decided maintaining something there is more trouble than its worth.

2004-07-21T15:59:00.000-04:00

Figures. A day after putting out that last post about Java alternatives, I come across this article by Eric Raymond, who comes to the same conclusion, but is much more convincing.

Beyond Java

2004-07-20T09:32:00.000-04:00

In my previous post, I described some of the reasons I felt that Java was lacking when using agile development methods. In this article, I'll go over some of the alternatives. Each of these languages has more expressive power than Java, but each has its own particular advantages and disadvantages.

Lisp and Scheme

Putting Lisp on this list might be surprising, but it has long been held in esteem by digital cognoscenti. It was, of course, conceived in the fabled early days of computers - one of only handful of programming languages that are older than I am. Despite this, Lisp adherents today assert that it is more powerful than any other language out there. I had spent a few weeks trying to learn Lisp in high school 25 years ago, but never really saw the point of it. My recent interest was sparked after reading several articles in its favor by Paul Graham. Lisp, and its more modern cousin, Scheme, are blessed with a trivially simple syntax, along with very powerful features such as macros and continuations that take advantage of that simple syntax. There are plenty of packages that extend the language.

So, if Lisp is so great, why hasn't it swept the world by storm. First of all, with dozens of different implementations of Common Lisp, and dozens more of Scheme, the Lisp world appears fragmented. While there appears to be plenty of modules out there for doing things such as HTTP servers, graphics, GUIs, and such, many appear to be specific to some implementation or another. Also, while its support for programming in the small are quite impressive, its not clear that it offers the same advantages for programming in the large. Finally, Lisp's culture of terse and often obscure naming conventions and reliance on tricky, if concise programming techniques probably works against it. Paul Graham argues that, because of the simple syntax, Lisp is easy to learn. The problem is not the syntax, its the concepts. Functional programming, continuations, macro-programming, and recursion as a replacement for loops are all going to be hurdles for the average corporate programmer. Not that these are difficult to learn per-se, just that they are far from the coding mainstream. I may come back to Lisp/Scheme some day, but for now the effort required to assembly the pieces I need, and learn the techniques, is too great.

Python

I've been working with Python quite a bit recently, and like it a lot. The syntax is elegant and consistent. It provides metaprogramming facilities. Python also comes with a lot of useful libraries provides everything from HTML and XML parsing, to graphics, to an interface to Tk (Tkinter) for doing GUI applications.

Perl

I've never been much of a fan of Perl, although I used it extensively in one major project I worked on. It works, but the syntax rules are confusing. In the version I was using (5.6), object-oriented programming was more effort than it should have been. The best thing Perl has going for it is CPAN, which serves as a clearinghouse for Perl modules. Its cool because Perl comes with a program for automatically installing modules from CPAN, which makings taking advantage of this resource a snap. One of the reasons I've liked jEdit is the automated installation of plugins. More developers need to do this kind of thing.

Ruby

The Pragmatic Programmers recommend Ruby as a great programming language. While it seems a bit less wordy than Python, I don't really see that it has much advantage over that language.

The Rise and Fall of Java

2004-07-02T18:22:00.000-04:00

The Early Promise

Nearly a decade ago, I slung code in C, C++, and Object Pascal. C was a decent language, if somewhat hard to debug, and it put food on the table. But as I adopted the OOP mind-set, C became less and less appealing. I moved on to C++. It was James Coplien's Advanced C++ Programming Styles and Idioms that really clued me in to both the real advantages of object oriented programming, as well as how best to code C++ classes. But I wondered, why did I have to build all that code for every class? Shouldn't the system be doing that kind of rote coding for me? It seemed to me that C++ always involved a lot more work than it should.

I played a bit with Object Pascal, to be specific, Borland's Delphi product. Object Pascal had automatic memory management, and a much simpler object model than C++. You could whip up a nice, GUI app in no time compared to C++. It was easy. It was fun. Alas, it was also Windows only, and proprietary.

Then, around 1996, came Java. It did objects and managed memory for you. It did cross-platform GUI applications. It came with a great class library. Java seemed perfect. I switched over to java for all my own development Java quickly started accumulating an impressive collection of library classes as Java mind-share grew. Life was good. And it looked like you good actually make a living doing this java stuff to boot! When it came to getting things done, Java was a sailboat to C++'s rowboat. Sure, there were some places you couldn't get to with Java, but for those you could, you could do it with much less effort and a lot more speed and style.

The Decline and Fall

But in the last few years, the sail has started to tatter. I've done some coding in Perl, and marveled at the ease of moving data into and out of methods. I played around a bit with Python, and was impressed with the depth and flexibility of its object model and the cleanness of its syntax. I've struggled with Java APIs that were much more complex than they should have been. I've seen how poor Java has evolved to keep up with changing development techniques. Coding in Java today, I'm beginning to feel the same way I felt 10 years ago coding in C++ - wondering why I need all this extra crap to express what I wanted to. Java was becoming more an anchor than a sail.

Java just doesn't fit my more agile style of development. I do a lot more refactoring and automated unit testing now, and as a result, I've changed sides on the debate between static and dynamic typing. I've seen the work required to get a fairly inflexible language like Java to do things like database persistence well. The increase in computer power has reduced the risk of using interpreted pure object-oriented languages. Short turn-around times has put a premium on developing quick, functional code that doesn't require a lot of extra scaffolding.

The developer community seems to be coming around to this view as well. Those that came to Java from Smalltalk have always complained, of course. At a recent Java conference, there were a lot of complaints about the complexity of enterprise Java, and EJBs in particular. The latest 1.5 version of Java does address some of these concerns. Autoboxing of primitives, for example, is a nice addition, as is variable-length parameter lists. But ultimately, I feel this is too little, too late. I'll still be doing most of my coding in Java, since its paying the rent, but my heart is just not in it anymore. I've come to the conclusion that Java is not part of the best way to develop software. It still has a great class library and unparalleled third-party support, so it will remain a tool in my toolbox. Just not my favorite.

Next... Beyond Java - A look at some alternatives