August 2007 - Posts - Jeremy D. Miller -- The Shade Tree Developer

Trying to answer hard questions about Agile development

My immediate organization (formerly Finetix) largely uses XP/Scrum practices, but much of our larger parent organization is still new to Agile development. Since more and more clients are asking for Agile project delivery, several of my coworkers and I were asked to participate in an Agile roundtable event. The roundtable was asked a series of five questions one at a time and then went around the table. To the best of my recollection, here is a distillation of my responses.

Yes, my answers are very biased, but I've come by these bias honestly. I've had mostly positive experiences with XP/Scrum, and nothing but irritation from working with waterfall mentalities. It's probably true that many shops don't really do a strict waterfall in practice, but in a way that's even worse to give lip service to the waterfall while working adaptively on the side. Those shops are living a lie. They're trying to work adaptively since that's the only real way to succeed at complex projects, but their project lifecycle simply doesn't support that adaptation. Any decent usage of an Agile process is built around gathering feedback and making adaptations. We ought to just start telling ourselves and management the truth and bring this adaptation out into the open sunshine where it's easier to control.

What about Fixed Bids?

I am largely passing on a question about doing Agile on a fixed bid, fixed deliverable project. I don't have any first hand experience, and my cheeky response is that it works the exact same way as any other process. You do your damndest to estimate the work by breaking it down into as fine grained tasks as possible then work stupidly long hours when the estimate turns out to be wrong. Yeah, the extra upfront work to do the estimate isn't necessarily Agile, and probably makes the whole of the work more expensive than it would be otherwise, but the situation is what it is.

I don't really think the Agile answer to a fixed bid is any different from any other process. I do think that Agile practices and project management can give you far more control and feedback on the "Iron Triangle" of resources, time, and features. Agile/RUP/CMMI/waterfall whatever, the iron triangle constraints still apply. If you try to lock all three constraints you're in for either pain, unhappiness, or protective sandbagging in your estimates. I would still choose to use Agile delivery for fixed bid projects because I think that is the most efficient way to execute and allows for the ability to fail "softly" with some fraction of the features instead of total abject failure to deliver any features on time like a waterfall project.

How do you create agile requirements? Some teams define stories poorly and then discover later that implementing these stories takes much longer than estimated. Does a note card’s worth of information provide adequate detail for a good estimate? Is it a problem if estimates are highly inaccurate?

I'm going to be brief here because I have a "Jeremy length" essay on Agile requirements coming soon that hits this topic in detail. The short answer is that user stories on note card's are definitely not enough information for good estimates. That's okay though, because that isn't all the information. Story cards are primarily a project management tracking device plus a way of creating a common language for the team. The note cards are simply an icon representing some finite amount of work. The actual estimates are produced by the team with the developers tasking out the larger feature with plenty of help from the anaylysts, customers, and project manager, but that conversation absolutely has to happen. In an Agile project you are very consciously trading in intermediate deliverables for far more face to face communication. If I'm allowed to have my way, the detailed requirements will be captured in the form of acceptance tests and largely automated - but only close to the time that a given feature is actually put into development. There's no use in doing detailed analysis for some feature that ends up getting scrapped. That's just waste.

Check out this link from Jim Shore: Beyond Story Cards: Agile Requirements Collaboration

How is Agile different from RUP? Each has development and release in an iterative process. Unlike waterfall, both can have late inclusion of requirements.

I had fun with this one. I should probably say that while I have plenty of "book" knowledge about RUP, I've never used it in anger. I've spoken to many people that very happily transitioned from RUP to XP or Scrum and refused to ever go back. I did try to champion a conversion to RUP at a former employer before running away to join the XP circus. This is a partial repeat of an earlier post, but who cares?

Subject	RUP	XP/Scrum
Are they Iterative?	RUP is supposed to be iterative and the founders of RUP will turn their faces blue saying this. The problem for me is that RUP still includes a lot of waterfall mentality baggage in the form of project phases. The iterations are much longer and seem to really amount to mini-waterfall projects in their own right. The typical RUP iteration lengths I've heard ranged from 6 weeks to a couple months	Extreme Programming uses 1 to 2 week iterations, Scrum teams originally worked in 30 day sprints, but the cross pollination with XP has led to shorter iterations. Testing is engaged much earler in an Agile team.
Ceremony	RUP is commonly disparaged for its dizzying array of intermediate deliverables. Most are optional and teams are meant to pick and choose which deliverables are appropriate for their project. Some of the RUP deliverables may simply be an excuse to justify the purchase of the Rational lifecycle products.	XP and Scrum used to be described as low ceremony, but that might be a bald faced lie. The project management "ceremonies" are simpler, but have to be followed closely. XP in particular will have more impact on the minute by minute activity and behavior of the team than any other process.
Tools	Rational is a software company that makes their living from selling their lifecycle tools.	There are tools from vendors that support or aid with Agile processes and practices, but by and large, Agile teams use far more open source tooling than non-Agile teams.
Origin	I think that RUP is largely created by people with C++ experience. Coding is nasty, brutish, and hard. The only way to succeed is regimented discipline. Quality gates and the creation of documents like the Software Architecture Document.	Most of the early Agile leadership had a background in Smalltalk. Coding can be productive and pleasant, but the extreme flexibility of the language requires an internalized discipline. There might not be many intermediate deliverables in XP/Scrum, but XP/Scrum requires a very disciplined approach from the developers in the form of Test or Behavior Driven Developement, Continuous Integration, and Simple Design (much harder than that sounds).
Design	All UML all the time. The Three Amigos were all architects and RUP has a strong emphasis on architecture.	The Simplest Thing that could Possibly Work, the Last Responsible Moment, You Aren't Gonna Need It (basically, don't ever do anything more complicated than what you need right now). Most Agile proponents eschew UML, but I'd write this off more to personal preference than a real prohibition. I do think that CRC cards and Responsibility Driven Design is more effective inside rapid iterations than even informal UML.

You're an Agile team, but inside of a Waterfall environment. Is it possible to remain Agile? What compromises must you make?

You will absolutely have to compromise. Your team may be flexible in regards to your feature ordering and iteration planning, but the waterfall team isn't. If a waterfall team needs something from the Agile team, that feature simply needs to be made a priority and played in the next iteration. The other way around is more tricky. Waterfall teams generally work off of longer plans and don't have that type of flexibility. If you're going to be dependent upon work from a waterfall team you have to treat that dependency as a constraint.

Here's where I think the bitter irony is, the waterfall teams purport to be more predictable because they have a linear project plan, but those plans are rarely accurate unless the plans are constantly adjusted in the face of feedback. Because those plans are never truly accurate, we need to be able to adapt if it turns out the waterfall team is late with their work. I think that the flexible delivery schedule of rapid iterations should give us more ability to simply switch to working on other features

As an Agilist I try to make all design decisions at the Last Responsible Moment. In the case of a dependency on a waterfall team, my Last Responsible Moment comes much, much earlier than it would if that same feature was completely controlled by the Agile team. Much of software design is being cognizant of what design decisions have to be made and the appropriate time to make that decision. In other words, you need to decide when to decide. In this particular case, I have to make decisions earlier than normal just to determine the concrete needs from the waterfall team early enough to get those needs into the waterfall teams project plan.

There is a very serious mismatch in terminology and vocabulary between teams using XP or Scrum and other teams doing more traditional waterfall work. They think we're out of control and we think they're largely nuts. You will have to invest some time with the other management to make them understand, or agree on a compromise for, how the Agile team is going to communicate progress. There are no real intermediate deliverables on a typical Agile project. I take the fairly common viewpoint that the only real measure of progress is features completed that are potentially able to be shipped.

Project staffing can be the killer in my experience. To really do an iterative lifecycle we really need to include the analysts and testers as part of the holistic team -- and leave them there! Part of the enduring allure of nice, linear waterfall lifecycles is the belief that analysts can be rolled off of a project early and testers only need to be on the project at the end. It's a nice little myth, but I've seen nothing but severe pain from that type of project staffing in practice.

Posted Aug 31 2007, 08:03 AM by Jeremy D. Miller with 32 comment(s)

Filed under: Featured

StoryTeller Beta 0.7 Is Here! Come and get it!

I owe some folks a couple blog posts, but I had to get this puppy out of the way. This white whale is swimming free, but I've got StoryTeller in my harpoon sights.

I announced StoryTeller nearly a year ago and its finally getting close to a real release. The tagline is simply this: "StoryTeller is the FitNesse killer for .Net." My express goal is to fulfil the promise of FitNesse with less frustration and friction. This isn't really about criticism of FitNesse so much as just trying to make the next step. This tool would not exist without the inspiration or FitNesse.

I have absolutely zero intention at this moment at making StoryTeller run Fit tests in anything other than .Net. StoryTeller is intended to provide the best possible experience for .Net development without compromising for cross language support. Besides, there are other tools for Java and other languages.

From the original announcement:

StoryTeller is a new tool for efficient creation and management of automated testing of .Net code with the NFit/FitNesse engine. StoryTeller is specifically created to support an Acceptance Test Driven Development strategy. All existing .Net FitNesse tests will run under StoryTeller. Features will include editing, tagging, and integration with source control, CruiseControl.Net, NAnt and/or MSBuild, and support for application versioning.

It's taken a long time, and it's not really done yet, but I finally feel like I've got enough here to warrant a real release. The UI client is still rough, but it does work. I'm thrilled with the response to a call for beta testers, so here it is:

StoryTeller Beta 0.7

Fixed.

And I did the amateurish thing by creating the package by copying all the files in bin/debug which inevitably misses GAC'ed assemblies. So also go download the telerik assemblies at http://storyteller.tigris.org/svn/storyteller/trunk/tools/telerik.

~~I'll get the zip repackaged tonight.~~

Download the zip file, unzip, and double click on the executable and you should be good to go.

Things to Know

StoryTeller is aimed very squarely as a replacement for FitNesse on .Net projects. It still uses the same test engine.
StoryTeller stores the test files as either HTML or Wiki text. Some options aren't supported with the Wiki text. One of my goals was to make the tests easy to edit by hand or your html tool of choice.
You edit the tests inside the StoryTeller GUI with a subset of the FitNesse Wiki format. See the FitNesse website for more details.
SetUp & TearDown works a little differently than FitNesse. You can optionally define a SetUp and/or TearDown for any Suite. When a test executes, StoryTeller will execute the SetUp, if one is found, for each anscestor Suite. For example, a test with the path "Suite1.Suite2.Suite3.Test1" will first run "Suite1.SetUp" then "Suite1.Suite2.SetUp" then "Suite1.Suite2.Suite3.SetUp" before executing the actual test. TearDown's work similarly, but in reverse order. The TearDown's are executed by the inner most parent in this order: "GrandParent.Parent.TearDown" then "GrandParent.TearDown" and so on.

Starting a New Project

The first thing to do is to configure a new StoryTeller project. Inevitably, it's an Xml file. There is a form in the client to configure the Xml file, but for the sake of documentation I'm going to describe the Xml file (it's just a serialized object by the way). The sample project file looks like this:

<?xml version="1.0"?>

<Project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <!-- List of all possible tags for the project-->

  <Tags>

    <string>jeremy</string>

    <string>bob</string>

    <string>derrick</string>

  </Tags>

  <!--

       The only valid Project type is "Local" for the moment.  I had a "FitNesse" project at one time and plans for a "Remote" project.

       All it means is that the tests are run locally

-->

  <ProjectType>Local</ProjectType>

  <!--

    The directory containing the assemblies and configuration files for the SystemUnderTest.  The StoryTeller.dll must also be in this folder

    The path can be either absolute or relative to the project file

-->

  <BinaryFolder>..\TestApp</BinaryFolder>

  <!--

    Optional, but recommended.  As assembly qualified name of a class that implements the StoryTeller.IFixtureLibrary interface.

    The role of the IFixtureLibrary interface is simply to register Fixture type's with the Fit engine.

-->

  <FixtureLibraryTypeName>StoryTeller.IntegratedTesting.TestingFixtureLibrary,StoryTeller.IntegratedTesting</FixtureLibraryTypeName>

  <!--

    The folder that contains the StoryTeller tests.  The root Suite's will each be a directory under this folder.

    The path can be either absolute or relative to the project file

-->

  <TestFolder>SimpleTestApplication</TestFolder>

  <!-- The displayed name of the SystemUnderTest -->

  <Name>Test Application</Name>

  <!-- Controls the format that the StoryTeller engine uses to persist tests.  At this time I *strongly* suggest the HTML option-->

  <StorageFormat>HTML</StorageFormat>

  <!-- Set a timeout on test execution.  Any number <= 0 will be treated as infinity.  The TimeOut is somewhat problematic at the moment-->

  <TimeoutInSeconds>0</TimeoutInSeconds>

  <!--

    Purely desperation.  If you need a completely clean AppDomain for each and every test run, choose this option.

    Needless to say, this will slow down the execution time to a painful crawl.  And yes, I had to have this in a previous

    project.

-->

  <UseCleanAppDomainForEachTest>false</UseCleanAppDomainForEachTest>

</Project>

Running from the Command Line

There is a command line runner called StoryTellerRunner.exe in the download for running StoryTeller tests from a command line. This is the tool for integration with build scripts. I originally intended to do this with NAnt tasks, but decided on the command line approach for maximum reach. The command line is:

StoryTellerRunner.exe ProjectFile OutputFile \n [-description:AAA]\n [-suite:AAA]\n [-tag:value] (can be repeated)

ProjectFile is the path to the StoryTeller project file
OutputFile is where you want the TestReport to be saved
-description:Text is simply a way to add a description to the TestReport
-suite:Suite Path is the path to the Suite you want to run. Use this to limit the test run to only this Suite
-tag:value - run tests with this tag
-status:Acceptance or -status:Regression - Only run tests in this state. Conceptually, you could make one test run that executes all the Acceptance tests but doesn't fail the build, and a second regression test run that will fail the build on any test failures.

What's Left

It's not complete, and I'm hoping to do one more beta before 1.0. I'm releasing this now to get feedback. From the response I've gotten, there's an obvious demand for something like StoryTeller. The stuff that I've definitely got left is:

Syntax checking - Simply highlight test rows that don't match up with the Fixture classes
A Fixture explorer in the UI tool
Exporting reports and results from the client
Reloading a project
Canceling tests in the execution queue
Creating playlists on the flow
Adding the component versions to the test reports
Making the UI not be butt ugly
Docs, lots of docs

Posted Aug 29 2007, 12:22 PM by Jeremy D. Miller with 12 comment(s)

Filed under: StoryTeller

I've heard this story before

"It started as a prototype, but got pushed into production"

All of the very worst systems I've worked with have an origination story that can be summed up by the 10 words above. If you hear these words, either run or go into legacy code mode.

Prototypes are dangerous. Dangerous in that they can easily go on way past the point of diminishing returns, and dangerous in the sense that there's too much temptation to put it into production. When you prototype, I think you need to decide right off the bat what the limited goal of the prototype is and set a timebox for the effort. There has to be an acknowledgement that we're temporarily coding without the safety regulator of good practices and plan on throwing away the prototype code later.

I said much more earlier at Of Spikes and Prototypes

Posted Aug 27 2007, 03:16 PM by Jeremy D. Miller with 13 comment(s)

Filed under: Ranting

Don't ever, ever build domain objects that can't...

...run without the database -- Active Record implementations like Trade trade = new Trade("tradeId") where the constructor immediately reaches into the database to grab the rest of its properties. One of the reasons that I won't use the Entity Framework by choice is the lazy fetch implementation: invoice.Items.Load(). I understand their reasoning for doing this, but it totally screws up unit testing.

...run without a configuration file -- Extra stuff that has to be copied around and valid for the code to work

...run without needing half of the rest of the application -- "Trade trade = new Trade("tradeId", tradeRepository)" is no bueno

...be at least partially set up in a few lines of code -- An ObjectMother can help tremendously. Yeah, an ObjectMother is often times deoderant, but still...

and just for any Java folks that might have stumbled into this accidentally.

...run without some sort of heavyweight container*

But Jeremy, I just needed to "Git 'R Done!" and I don't have time to worry about ivory tower things like separation of concerns! Careful design *is* pragmatic. Keeping service and infrastructure concerns out of your domain objects should help in the "Git 'R Done" category, because:

Your business logic lives in the domain objects. It's more efficient and safer to be able to work on domain logic without other services getting in the way. I want to write and read code that works with domain concepts. I want to work on one thing at a time, and in this case that one thing is domain logic. More signal and less noise makes it easier to verify the business logic code by inspection, which is good considering that that code is very likely to change.
Much of the rest of the system reacts to and interacts with your domain objects. When I'm working with a controller class or a repository class I want to develop and test those things without incurring a heavy tax from setting up connected domain objects with whatever container.

*Of course, the great thing about being in .Net is that the Java guys bump into most of the problems first since they're always a year or two ahead. Just pay attention to what causes the Java community pain and avoid it when some clown tries to introduce the pain into .Net.

Posted Aug 27 2007, 09:17 AM by Jeremy D. Miller with 19 comment(s)

Filed under: Featured, Ranting

I'm looking for some Beta testers for StoryTeller (OSS tool for Fit testing)

If anybody's game, I'm making the first beta release of StoryTeller next week. I'll use the latest version over the next week at work to work out the kinks, but other sets of eyes would be appreciated. If you're interested, could you drop me a line via the contact link on this page. Thanks.

Why do I care?

Bottom line: If you do .Net development and you want to use Fit/FitNesse for automated story or acceptance tools, StoryTeller is going to be a better way to use the Fit engine than FitNesse.

Posted Aug 25 2007, 05:20 PM by Jeremy D. Miller with 4 comment(s)

Filed under: StoryTeller

To all of you Functional Programming gurus out there

What's some examples of programming problems that would be good for learning functional programming? I'm looking for categories of problems that are better addressed with functional programming instead of OO.

I'm thinking about using F# as part of StoryTeller to write a syntax checker for Fit tests against the Fixture classes to provide something like compile time checks for Fit test syntax.

Thanks in advance.

Posted Aug 21 2007, 03:30 PM by Jeremy D. Miller with 9 comment(s)

I love JetBrains

"Develop with pleasure" -- Says it all. I'm dipping into a legacy .Net 1.1 codebase for the next month and lo and behold, discovered that my templates from .Net 2.0/ReSharper 2.5 work just fine in .Net 1.1 (except for the ones that used generics of course).

I've also learned today so far that things have really improved from VS 2003 to VS 2005 and even more so from newer ReSharper versions. Onward and somewherewards!

Posted Aug 21 2007, 08:47 AM by Jeremy D. Miller with 5 comment(s)

So much wrong in one little statement

I'm completely blocked from personal email at work, but I can see the headers come across. I just got one with this message:

Start earning the salary you deserve by obtaining the proper credentials!

A.) "Earning the salary you deserve" - Damn, I don't want to take a paycut

B.) What certification is really that important or useful? Once you're pass the entry level, what good does it really do? I know they've gotten better, but the last certification test I took from MS was pure memorization of API's.

C.) You don't want to work anywhere where they use certifications to make judgements about employees. I think that that implies that they don't look too carefully at each individual. I'll buy it as part of a developer's "personal growth" plan, but not otherwise.

Posted Aug 15 2007, 12:38 PM by Jeremy D. Miller with 8 comment(s)

Test Automation Week -- Fail Fast or Just Plain Fail

Let me start with a little story from a couple years ago. My team inherited a less than desirable codebase from another team* . There were some blackbox regression tests that we could use as a safety net, but we found that they were extremely fragile and unhelpful. The system had a multitude of external dependencies on COM objects, configuration files, external systems, remoting points, virtual directories, and web services. If there was anything wrong with any of that environment, the application wouldn't run properly. The main problem in testing was that some processing that relied on these external dependencies happened in a background thread. When there was an exception the application would log the exception then swallow it. The testing harness would just say that the test failed. That hurt. Debugging code when the problem could be almost anywhere isn't an efficient exercise. Not to worry though, once I started to sink my talons into the codebase I made a relatively small change to make the code...

Fail Fast

Fail Fast is a long standing piece of advice for software construction, but even more so for automated testing. From Jim Shore,

Failing fast is a non-intuitive technique: “failing immediately and visibly” sounds like it would make your software more fragile, but it actually makes it more robust. Bugs are easier to find and fix, so fewer go into production.

"Fail Fast" simply means this: when something goes wrong in the code, fail immediately and directly. Don't pass go, don't hide the real exception behind some sort of human readable message, don't log the exception and then pretend that nothing is wrong, and finally, don't put up a friendly "System Failure, please try again in a few minutes" message box. Some of these choices are proper in normal functioning. Even desirable in production mode, but hiding the gory details can make your life sheer hell when it comes time to debug a failing test. Just throw the raw exception in a way that you can immediately relate the exception to the test failure.

My little solution to the problem I described above was pretty simple. I typically make exception logging and sometimes auditing run through some sort of interface that's pulled out of StructureMap. Part of the reason to do this is to simply make unit testing via mock objects easier when auditing is a first class requirement. The second reason is to create a testing mode where the exceptions are just thrown right back up. Here's an example from my current project.

The interface is just this:

    public interface IMessageLogger

        void LogMessageFailure(string messageId, object request, object response);

        void LogMessageFailure(string messageId, object request, Exception ex);

...

        void LogApplicationFailure(string messageCaption, string MessageBody, Exception ex);

In the projection mode the real concrete class is just a simple wrapper around log4net. In testing mode I use this:

    public class FailFastMessageLogger : IMessageLogger

        public static string LastTradeId;

        public void LogMessageFailure(string messageId, object request, object response)

            throw new ApplicationException(response.ToString());

        public void LogMessageFailure(string messageId, object request, Exception ex)

            throw ex;

...

And in a "BootstrapFixture" class that I call in my StoryTeller SetUp, I do this to inject a FailFastMessageLogger:

        public void StubTheServer()

            ObjectFactory.ResetDefaults();

            ObjectFactory.Profile = UserInterfaceRegistry.STUBBED;

            ObjectFactory.InjectStub<IMessageLogger>(new FailFastMessageLogger());

            MessageBoxRecorder.Start();

The call to ObjectFactory.InjectStub<T>(T stub) "spring loads" StructureMap to return a FailFastMessageLogger anytime an IMessageLogger is requested.

I really wouldn't mind some feedback here. Like this solution, think it's ugly, or something else? Comments are open.

Defensive Assertions

I'm not a big fan of defensive coding in general, but I think it adds value at strategic spots in your code. Inside test automation code you can make test failures far easier to diagnose by strategic usage of assertions when the test harness can detect something strange. Instead of allowing the test harness code to stay quiet, make the automated test blow up right there.

Here's an example that comes to mind right off the bat. I can happily test my WinForms client through Fit tests by manipulating all the little UI widgets (yes, I'm going to do a post in Build you own CAB showing how I do this). There's a danger in doing the tests in this way. Think of a button. If you're doing things programmatically, you can quite happily make the button's Click event fire even when the button is either disabled or hidden. Your test could give a false positive. Instead, do this:

        public void ClickButton(string label)

            // Just don't worry about *how* we find the Button for the moment

            ButtonElement button = _binder.FindElementByLabelOrAlias<ButtonElement>(label);

            assertElementExists(button, label);

            // Throw an exception if the inner button is not enabled and visible

            button.AssertVisibleAndEnabled();

            button.Click();

That'll stop a false positive in its tracks. A more pernicious problem is a silent failure. In one of my main Presenter classes I perform a validation on the Model class before a creation or update message is passed to the server. If the Model object is not valid, the validation messages are shown on the UI and the call to the server is cancelled. Early on, I noticed some regression tests suddenly failing with no discernable reason. With some debugging, I quickly realized that the issue was invalid data being set up in the test case. Originally, the test data was perfectly fine, but as we added more and more validations the test data managed to trip off some of the new validation rules -- silently. I fixed that issue with this:

        public Fixture UpdateTradeWithProperties(string tradeId)

            Trade trade = findTrade(tradeId);

            ITradeFixture fixture = TradeFixtureFinder.FindTradeFixture(trade.TypeDescription);

            fixture.Trade = trade;

            fixture.RowsFinished += delegate

                                            assertTradeIsValid(trade);

                                            _tradeUpdateLog = _repository.UpdateTrade(trade);

};

            return (Fixture) fixture;

Note the line of code in bold. If the Trade object definined in the test doesn't pass it's validation rules, the call to assertTradeIsValid() blows up and puts all of the validation messages into the exception message. This little bitty thing has made the tests more resilient to the code changing by making it easier to fix up the test data. There's another lesson there about dealing with test data, but I'll let that one slide.

Summary Points

Debugging is hard. Debugging big tests that involve lots of classes and some external dependencies can be wicked hard. Keep that debugger turned off as much as possible.
Make the test failure messages reflect the real reason for the test failures by failing fast
Don't let hidden issues like clicking a hidden button or exception logging make test failures harder to diagnose
Think about creating an alternative "testing" mode where exceptions aren't trapped or logged

* Of course, the guys that wrote the miserable code are still there and going strong, while my team that wrestled with that crappy code up are all long gone.

Posted Aug 12 2007, 08:15 PM by Jeremy D. Miller with 6 comment(s)

Filed under: Featured, Test Automation Week

Test Automation Week

My wife and son are going away for two weeks soon to visit family in Texas. I'm seizing the opportunity to ~~fire up ye olde XBox~~ finally wrap up StoryTeller 1.0 (basically a tool for maximizing the usefulness of Fit testing and a .Net centric replacement for FitNesse). As a runup to the first real StoryTeller release, and to continue the thread on Fit/FitNesse testing and executable requirements that's been going around, I'm going to spit out some little posts (plus one big post) all week on tips and tricks for Fit testing and some lessons I've learned on automating acceptance tests. I haven't used every possible tool out there, but I've done a lot of test automation with a mixed bag of results. Check in occasionally and I might accidentally post something to help out some test automation newbies.

I'm thinking:

Fail Fast
Dependency Injection trickery
Extending FitnesseDotNet with custom type handlers
Don't Repeat Yourself
Jumping back into "Build a CAB" to do the Subcutaneous Testing chapter because Ian Cooper keeps mentioning it and a client developer and I are plotting a way to do just that in his code next week.
Maybe try one more stab at creating a DSL with FitnesseDotNet's DoFixture and comparing that to either a Fluent Interface in C# and/or a Ruby equivalent. That might take a bit of effort, but I'm dying to see how the three choices with "feel"

Anyway, the first one will hit tonight.

Posted Aug 12 2007, 07:04 PM by Jeremy D. Miller with no comments

Filed under: Test Automation Week

Why I'm suddenly down on Fit/FitNesse

Quick Note: I love *what* Fit testing accomplishes, but the mechanics are an issue. There's a rumor floating around that a bunch of test automation gurus are going to work up some ideas for how to "save" Fit testing. Anybody feel like commenting? I don't know anymore details that that.

I've had doubts for months, but I think I finally soured on Fit testing within about 5 minutes last week. Here's why:

To fix an existing test I caught myself using 2-3 different low level tricks within Fit. I built a couple different custom type handler and injected them, overrode a DoTables() method, and some other bit of hackery that involved low level knowledge of Fit internals. I found myself using 4-5 different Fixture classes in a single class. It suddenly occurred to me that there was no way in hell that I was going to be able to completely transition this stuff to the client's testers and developers before I leave (especially since I work beside them instead of working with them, but that's a different battle). It just requires too much internal knowledge of Fit mechanics to do anything outside of basic set-based testing.
Nat Pryce aptly described the Fit engine mechanics as "reflectopornographic", and I don't want to use something described as "reflectopornographic." If you've worked with Fit testing before, you're probably laughing along with that right now.
Some things are easier in Fit tests than xUnit tests, but for larger flow based tests I'd rather just write code alone. If I had it to do over again, I'd keep Fit (StoryTeller in reality) for set-based things, but would have written a, wait for it, fluent interface test harness for the flow based testing I do against the user interface. I don't have any participation from the customer and little from the tester, so why not use the tools I'm most comfortable with? I'm spending too much time hunting down test grammars from the Fixture classes to remember what to type into the Fit editing window. Intellisense anyone?

When I say set-based here, I'm talking about tests that can very accurately be expressed completely in 2 dimensional tables. X number of inputs leading to Y number of outputs. Fit excels at that with the ColumnFixture and RowFixture tests.

What about StoryTeller?

I've just done a reset on the StoryTeller UI and I'm committed to finishing it. StoryTeller is aimed very squarely as a FitNesse killer for better management and editing of Fit style tests, not as a replacement of the actual FitnesseDotNet engine. I think in the longer term I'm keeping StoryTeller for the things it is good at, but start moving more towards the xBehave tools for the flow style testing that I use DoFixture's for today.

Regardless of whether or not you're using Fit happily, with reservations, or just want to think about starting, I'm putting up a lessons learned post in the next couple days on Fit testing in specific and automated acceptance tests in general.

Posted Aug 12 2007, 09:08 AM by Jeremy D. Miller with 21 comment(s)

Filed under: StoryTeller

That's what I meant

I've written a couple posts on teams and the desire for multi-skilled people and eliminating specialists. I hope I made my case, but it doesn't matter because Tate Stuntz did it for me in fantastic style in The demise of the Gantt Chart in Agile Software Projects. Please focus on why he says Gantt charts can be rendered unnecessary rather than flying off the handle by saying that Agile projects don't plan. You can, and have to, change the way you might be working in order to achieve any gains from an Agile process. The way you write code. How you divide responsibilities in the team. When you do certain activities. How you collaborate. It all needs to be put up for discussion and analyzed to make a successful shift to working in an Agile manner.

Posted Aug 07 2007, 01:29 PM by Jeremy D. Miller with 2 comment(s)

Fred George is blogging

Fred on The Secret Assumption of Agile. I didn't even know he had a blog. I promise you this guy will be worth reading.

Posted Aug 06 2007, 08:36 PM by Jeremy D. Miller with 2 comment(s)

On Software Teams

It's been about half a year since I threw out my My Programming Manifesto, and I've started to collect some things that I missed the first time around. As always, you should be reading this as "..while there is value in the items on the right, *I, Jeremy D. Miller, sole author of the Shade Tree Developer (who is not acting as an agent of my employer in this article)* value the items on the left more." It's also the case that you only have so much time and resources to complete any given project and we all want to play that time and resources against activities that create more value and minimize the time we spend on lower value added activities. Part of my formulation below is the belief that the stuff on the left provides more bang for the buck than the stuff on the right. In a few cases I also think the things on the left provide a more pleasant work environment than the things on the right, and that formulation simply cannot be disregarded. Happy developers are productive developers.

Also, this isn't even remotely complete. This just represents some areas that I'm seeing in clients and otherwise where I think the balance between the two opposing choices are being tipped the wrong way. Overall, let me just say that I'm an optimist in terms of people and their behavior. Yes, I'm as snobbish as any alpha geek wannabe and I'm prone to thinking that most developers out there are bad, but I also think those same people could be much better if given a better environment. I also see a lot of good people being held back by bad environments. Raw talent and effort just isn't everything.

I tried and tried to come up with pithy subtitles, but just couldn't do it.

Internalized Discipline over Externally Enforced Discipline

I got in a bit of an argument the other day with one of the project managers at my client (which ended with a mutually acceptable compromise). They use Jira from Atlassian for project management and tracking. Great right? Atlassian is one of the most visible Agile software companies in the world. I was objecting to their very complicated directed workflow that they enforced inside of Jira. My point was that if you wanted Jira to accurately reflect the current status, dump the directed workflow to make the tool easier to update without jumping through the process hoops. I also stated that I didn't think they derived any value out of the directed workflow. The response I got was that the developers wouldn't test their code at all without the "quality gates" enforced by Jira. Yet another trip down "the developers can't be trusted (but I can)." That last statement is what would concern me the most.

Quality gates are a very common tool in software development processes to enforce quality by making the people on the project demonstrate some sort of compliance to a standard or process before the project is allowed to proceed. I'm not sure that I'd say that quality gates are necessarily a bad thing at all, but by themselves, quality gates are absolutely worthless in promoting software quality. The quality gates happen too late in the project and are far too often conducted by people who are not qualified to judge the real quality. Instead, it's often an exercise in checking off the adherence to a defined process.

In my experience, software quality is something that happens as a result of a team's day to day behavior, not from externally enforced strictures. How well you unit test the code. How diligent the developers are about producing well structured code. How the team refines the design as they work. How conscientious the testers are about testing the behavior of the system. How well the analysts can capture and guide the evolution of the requirements. How the team strives to constantly fine tune their processes as the project proceeds. These are the things that make for software quality. The internal discipline and approach of the team performing the work every single day is more important that externally applied governance. You don't get quality with enforcement and reviews, you get it by making quality a daily exercise.

Back to the idea of what activities pay off more. Quality gates and external discipline doesn't happen at the time that code is crafted, requirements are gathered, and tests executed. It comes after the fact. That externally enforced discipline isn't around when things are actually getting done. Even if the quality gate finds problems, it's awfully late in the game. Those quality gates still take up time and resources to do, even if they go flawlessly. Why not redirect that time towards better testing or more code reviews?

Quality should be the responsibility of the team doing the work, because they're the only people who can make it happen. One of my most deeply held beliefs about software development is that people should have the power over the way that they work, and then adequately take responsibility for that work. The team doing the work is most likely the folks best qualified to know what they do and do not need to do.

But Jeremy, that might be fine for you, but I can't trust my developers to just do the right thing. I need governance. First, what are you doing to change their behavior? Secondly, why did you hire them in the first place if they can't be trusted, or can you tighten up the hiring process. Three, is that really the case? I've heard this line about not being able to trust the developers far too many times. Why, or how, is the manager responsible, but the developers aren't?

As far as tools go, I'm a hold out. I think these kinds of tools are only justified in distributed teams and even then, they better be easy to use and not cause any friction. I think that I can create far more project transparency while only using crude project management artifacts like story cards, but upping the communication and collaboration of the team. I think these tools are often perfume over smelly team mechanics.

Coaching over Enforcement

Too many times in my career I've seen situations where a team is created that doesn't contain the adequate skills and experience to execute the project that they're given, but there's a quality body of some kind ready to eviscerate them for not complying to some sort of standard. I've seen plenty of "Quality Assurance" teams that came into project retrospectives loaded for bear, but were sight unseen during the project's execution. It's far more useful in the long run to roll up your sleeves and help your teammates do things well than it is to be the big guy up top who ensures that the other people are doing things well.

My career goal for the near future onwards is to become the sort of person who can help everyone else around him to be better. To compare it to basketball, I want to be more like Magic Johnson circa 1988 rather than a younger Michael Jordan -- even though I started the other way around. Same thing with coaches. I played a lot of sports when I was younger, thinner, and had intact ACL's. The coaches that I remember fondly are the patient ones who were great teachers. The yellers and screamers I could do without.

Collaboration over Direction

In the following two discussions, which would you rather be a part of?

This is what we're going to do. Use the Supervising Controller pattern here
What I'm thinking is this. I think we should use the Supervising Controller pattern for this screen because...

Most people want some form of autonomy, and many people are perfectly capable of thinking for themselves. Giving a person exact direction is often tantamount to turning off their brain and removing free will from their work. I've led teams on and off for 7 years now. I've noticed a very severe trend in how developers react to the direction I give them. If I give them very detailed instructions, down to the class and method level with UML diagrams, they try to follow those marching orders verbatim, often to very poor results. On the other hand, when we go up to a whiteboard together and sketch out a design together, the results are far, far better. In some part I think it's because they understand the design better for having taken part in its formulation, but mostly because they don't stop thinking.

It's not just the touchy, feel-y aspect of it either. People work better when they understand the context, reasoning, and goals behind a course of action. I call this "understanding the why." When you understand the why behind your technical direction, you're much better able to make decisions and follow that direction.

Another thing that simply cannot be avoided. People can't just pick up a task cold and run with it. There's always some bootstrapping involved before someone understands a task well enough to proceed. Tomorrow I'm starting work on a financial report I know nothing about in a totally different architecture and codebase than what I've been working on (I will be positive about the opportunity to flex my legacy code slaying muscles instead of whining, I swear). I'm not going to develop at the same speed I do in my own Jeremy friendly codebase. I also wasn't a part of the formulation of this little side adventure, so I'm effectively starting from scratch tomorrow.

I think you can cut down on that ramp up time considerably by working more collaboratively between project/product management and the developers. Nobody should ever be surprised by their work assignments. Whether it be through iteration kickoffs, project planning, or simply discussions leading up to a task, a developer should never, ever see an unfamiliar task assignment pop up in their TODO list. Yet another reason why I don't like project management tools that try to take the place of human interaction.

Testers too for that matter. Dumping new features on unsuspecting testers is one of the worst things that we developers can do to the testers. It takes time to develop an effective test plan, and we need to accommodate this in our planning. Get the testers involved as early as possible to ease the turnover from development to testing. You can't ship until the testers are done with their part (assuming that you're on one of those advanced teams that actually includes testers), so it behooves you to make their job as easy as possible. Besides, a tester can't just test what you tell them to test. They need to exercise their tester superpowers and find those pesky edge cases that you didn't.

I don't want to work on either side of "direction" from above. To me, working in a collaborative environment is much more rewarding than having to be the guy at the top that makes every decision for the minions below. It's tiring and unrewarding to be "the guy." It used to be an ego boost to be the One Truck Guy, but I've had a chance since to work with strong developers in a collaborative setting. It's a much more enjoyable experience than being "the" guy. I'm old enough now to realize that not every idea I have in regards to a design is necessarily right or the best solution. I want feedback on my ideas. I enjoy interaction. "Ok, whatever you say" or "I trust you" is nice, but it's more rewarding to have a real discussion. Besides, I just don't quite know what to make of a developer that isn't opinionated. Wait Jeremy, they're just being pragmatic and open minded. No they're not, they're just being wishy washy.

Anybody that worked with me five years ago would be laughing their butts off at the idea that I'm for more collaborative team structures rather than a hierarchy with me at the top. People do change. Or, you know, at least try to.

Other Stuff

I've tackled subjects related to teamwork before and I think it's just as important if not more than the technology used or your individual habits.

There's more coming, someday, but I really wanted to click the "Publish" button at some point this evening.

Posted Aug 05 2007, 09:41 PM by Jeremy D. Miller with 15 comment(s)

Call me a Utopian, but I want my teams flat and my team members broad

One of many things I love about Ayende is the tagline in his blog "Specialization Is For Insects," which brings me to my point this morning: over the long run a flat team of generalists with healthy overlap between skillsets will blow the doors off a hierarchical team composed of specialists.

I hit the issue of a self organized team versus a strict hierarchical team before in Self Organizing Teams are Superior to Command n' Control Teams, but this morning I'm more irritated over the specialist thing. Two issues yesterday brought it out for me in stark contrast:

I've got about five weeks sunk into a new financial product in our system in the .Net UI client. Sometime this morning (knock on wood) I'm going to declare it ready to deploy, but it's not. We can't deploy it because the server side work to integrate with a 3rd party system hasn't even been started. This is obviously an extreme case with a wide variance between skills, but it renders my work on the .Net client effectively useless. Yes, it will be ready to deploy when the integration, and the issues that uncovers, are complete. In the meantime though, we could have played my time and effort into features that could be deployed and start providing value now.
Myself and my ersatz team were brought in specifically for WinForms work, but I insisted on being involved with the development of the Java web services as well. It's led to some pain on my part learning how to get Apache Axis to work for web service development but it's paid off. This week we had a defect related to a field not being saved properly between the UI and server. Because I have visibility and understanding of the entire system from front to back, it was relatively easy for me to diagnose the error in an iBatis map on the server side. A UI specialist would have just spun their wheels for a while. Since all my Java server side "resources" are temporary assignments, I would have been screwed if I hadn't worked on the Java end as well.

Case 1 is a bad break because of specialization. Depending on specialization makes planning and scheduling more difficult, especially when you're planning across multiple project streams. If your plan has a fixed dependency on a time bound utilization of a particular specialist, you're plan is now exceedingly brittle. A self sufficient team that can take care of all of its needs can be far more resilient in the face of scheduling changes and unforeseen events. And how often does an unforeseen event happen in a nontrivial software project? Moreover, it takes far more detailed planning when you have specialists and dependencies between the workflows of the specialists. A team of generalists can happily shift around depending on what needs done at that moment. Unsurprisingly, the type of adaptive planning used in XP/Scrum projects works vastly better with a flatter team of multi-skilled individuals.

Case 2 was a good outcome that was enabled by having at least one person (me) involved on both the client and server side. I can build most complete features from front to back without any assistance from anyone else, making the turn around for new features less than it would be waiting for the server side guys to come available. Troubleshooting and testing in a full integration mode is much easier for me than it would be for two people handling either side alone, and most of the problems are integration issues.

More things:

A team of generalists can be a smaller team of folks dedicated to the project instead of a strung out group of specialists who come on and off the project. This cuts down the communication overhead and enables Agile practices to be more successful. Communication overhead is a significant cost in any project. Cutting it down can only help.
You can raise your Truck/Lottery Number. You shouldn't be crippled because one team member is on vacation.
The team members have a much better understanding of the technical solution as a whole, and I think this contributes to better decision making.

Posted Aug 03 2007, 08:24 AM by Jeremy D. Miller with 19 comment(s)

Jeremy D. Miller -- The Shade Tree Developer